Algorithm Performance Dataset from NASA Open-Source Software

Abstract

NASA Langley Research Center has recently developed and released the open-source software Multi Model Monte Carlo with Python (MXMCPy- LAR-19756-1) as a general capability for computing the statistics of outputs from an expensive, high-fidelity model by leveraging faster, low-fidelity models for speedup. Given a fixed computational budget and a collection of models with varying cost/accuracy, multi model Monte Carlo (MC) seeks a sample allocation strategy across the models that results in an estimator with optimal variance reduction. MXMCPy is a versatile tool that enables convenient access to many existing multi-model MC approaches (over a dozen algorithms available) within one modular and extensible package [1]. With MXMCPy, users can easily compare existing methods to determine the best choice for their particular problem,while developers have a basis for implementing and sharing new variance reduction approaches. However,there is currently very little understanding about which algorithm will perform best for a given problem (defined by the correlation between and relative cost of the available models) without a brute force search.