Search NASA⌕ Search

SEARCH · Search NASA

Results for “Statistical emulation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Constructing a Simulation Surrogate with Partially Observed Output

Gaussian process surrogates are a popular alternative to directly using computationally expensive simulation models. When the simulation output consists of many responses, dimension-reduction techniques are often employed to construct these surrogates. However, surrogate methods with dimension reduction generally rely on complete output training data. This article proposes a new Gaussian process surrogate method that permits the use of partially observed output while remaining computationally efficient. The new method involves the imputation of missing values and the adjustment of the covariance matrix used for Gaussian process inference. The resulting surrogate represents the available responses, disregards the missing responses, and provides meaningful uncertainty quantification. In conclusion, the proposed approach is shown to offer sharper inference than alternatives in a simulation study and a case study where an energy density functional model that frequently returns incomplete output is calibrated.

42 ENGINEERING↗

Cosmological Evidence Modelling: a new simulation-based approach to constrain cosmology on non-linear scales

Extracting accurate cosmological information from galaxy–galaxy and galaxy–matter correlation functions on non-linear scales (${\lesssim } 10 \, h^{-1}{\rm {Mpc}}$) requires cosmological simulations. Additionally, one has to marginalize over several nuisance parameters of the galaxy–halo connection. However, the computational cost of such simulations prohibits naive implementations of stochastic posterior sampling methods like Markov chain Monte Carlo (MCMC) that would require of order $\mathcal {O}(10^6)$ samples in cosmological parameter space. Several groups have proposed surrogate models as a solution: a so-called emulator is trained to reproduce observables for a limited number of realizations in parameter space. Afterwards, this emulator is used as a surrogate model in an MCMC analysis. In this work, we demonstrate a different method called Cosmological Evidence Modelling (CEM). First, for each simulation, we calculate the Bayesian evidence marginalized over the galaxy–halo connection by repeatedly populating the simulation with galaxies. We show that this Bayesian evidence is directly related to the posterior probability of cosmological parameters. Finally, we build a physically motivated model for how the evidence depends on cosmological parameters as sampled by the simulations. We demonstrate the feasibility of CEM by using simulations from the Aemulus simulation suite and forecasting cosmological constraints from BOSS CMASS measurements of redshift-space distortions. Our analysis includes exploration of how galaxy assembly bias affects cosmological inference. Overall, CEM has several potential advantages over the more common approach of emulating summary statistics, including the ability to easily marginalize over highly complex models of the galaxy–halo connection and greater accuracy, thereby reducing the number of simulations required.

79 ASTRONOMY AND ASTROPHYSICS↗

Validating an Air Traffic Management Concept of Operation Using Statistical Modeling

Validating a concept of operation for a complex, safety-critical system (like the National Airspace System) is challenging because of the high dimensionality of the controllable parameters and the infinite number of states of the system. In this paper, we use statistical modeling techniques to explore the behavior of a conflict detection and resolution algorithm designed for the terminal airspace. These techniques predict the robustness of the system simulation to both nominal and off-nominal behaviors within the overall airspace. They also can be used to evaluate the output of the simulation against recorded airspace data. Additionally, the techniques carry with them a mathematical value of the worth of each prediction-a statistical uncertainty for any robustness estimate. Uncertainty Quantification (UQ) is the process of quantitative characterization and ultimately a reduction of uncertainties in complex systems. UQ is important for understanding the influence of uncertainties on the behavior of a system and therefore is valuable for design, analysis, and verification and validation. In this paper, we apply advanced statistical modeling methodologies and techniques on an advanced air traffic management system, namely the Terminal Tactical Separation Assured Flight Environment (T-TSAFE). We show initial results for a parameter analysis and safety boundary (envelope) detection in the high-dimensional parameter space. For our boundary analysis, we developed a new sequential approach based upon the design of computer experiments, allowing us to incorporate knowledge from domain experts into our modeling and to determine the most likely boundary shapes and its parameters. We carried out the analysis on system parameters and describe an initial approach that will allow us to include time-series inputs, such as the radar track data, into the analysis

Statistical emulation↗

Climate Change Impact Uncertainties for Maize in Panama: Farm Information, Climate Projections, and Yield Sensitivities

We present results from a pilot project to characterize and bound multi-disciplinary uncertainties around the assessment of maize (Zea mays) production impacts using the CERES-Maize crop model in a climate-sensitive region with a variety of farming systems (Panama). Segunda coa (autumn) maize yield in Panama currently suffers occasionally from high water stress at the end of the growing season, however under future climate conditions warmer temperatures accelerate crop maturation and elevated CO (sub 2) concentrations improve water retention. This combination reduces end-of-season water stresses and eventually leads to small mean yield gains according to median projections, although accelerated maturation reduces yields in seasons with low water stresses. Calibrations of cultivar traits, soil profile, and fertilizer amounts are most important for representing baseline yields, however sensitivity to all management factors is reduced in an assessment of future yield changes (most dramatically for fertilizers), suggesting that yield changes may be more generalizable than absolute yields. Uncertainty around General Circulation Model (GCM)s' projected changes in rainfall gain in importance throughout the century, with yield changes strongly correlated with growing season rainfall totals. Climate changes are expected to be obscured by the large inter-annual variations in Panamanian climate that will continue to be the dominant influence on seasonal maize yield into the coming decades. The relatively high (A2) and low (B1) emissions scenarios show little difference in their impact on future maize yields until the end of the century. Uncertainties related to the sensitivity of CERES-Maize to carbon dioxide concentrations have a substantial influence on projected changes, and remain a significant obstacle to climate change impacts assessment. Finally, an investigation into the potential of simple statistical yield emulators based upon key climate variables characterizes the important uncertainties behind the selection of climate change metrics and their performance against more complex process-based crop model simulations, revealing a danger in relying only on long-term mean quantities for crop impact assessment.

Maize↗

MARGInS: Model-Based Analysis of Realizable Goals in Systems

Under NASAs Constellation effort, the Exploration Technology Development Program funded research toward a system validation capability that applied machine learning and test-case generation techniques to the analysis of black-box system behavior. The behavior analysis capability scaled to spaces of hundreds of input parameters and tens of thousands of test cases. Aerospace systems at the vehicle level, especially those systems which contain some level of autonomy, are best described by hybrid and non-linear mathematics. Even simplified models of such systems need parameter dimensionalities in the hundreds or thousands of parameters in order to capture sufficient fidelity. The System Safety Assessments (such as those described in the SAE ARP 4761A Safety Assessment Process guidelines) for these systems are prone to errorinteractions between the vehicles subsystems are complex, and can display emergent behaviors. NASA captured this new analysis in the Model-based Analysis of Realizable Goals in Systems (MARGInS) tool and applied it to the Pad Abort 1 (PA-1) simulation as part of the independent validation and verification cycle before the PA-1 flight test in May of 2010. MARGInS evaluated the adherence of the high-fidelity simulation to its requirements, and deter- mined the margins to failure from the expected nominal input conditions. Following the PA-1 test, the capabilities within the MARGInS framework have been extended with sophisticated statistical and white-box test case generation techniques and applied to other NASA missions. The frame- work now includes a critical factors analysis that was applied to NASAs Orion simulation and design. NASAs Aeronautics Research Mission Directorate (ARMD) leveraged the existing MARGInS framework for work on aviation safety for civil transport vehicles and for research on autonomy issues. The NASA ARMD effort created a time series output prediction capability that has been used to characterize trajectories for a plane with an adaptive control system, and a safety boundary detection capability that has been applied to an air traffic control concept of operation for the Federal Aviation Administration. The statistical and machine- learning based techniques within MARGInS have been successfully combined with concolic execution to improve the coverage of a critical unit by driving system-level inputs. The use case driving the concolic execution and MARGInS integration was inspired by the Air France 447 disaster in which the loss of a critical functionality (the airspeed calculation from the pitot tubes) led to loss of the entire plane with the people aboard. To illustrate capabilities and limitations, we will highlight the analyses for the applications listed above. We will then discuss the future plans for MARGInS and its interfaces with other tools.

Validation↗

Linking big models to big data: efficient ecosystem model calibration through Bayesian model emulation

Data-model integration plays a critical role in assessing and improving our capacity to predict ecosystem dynamics. Similarly, the ability to attach quantitative statements of uncertainty around model forecasts is crucial for model assessment and interpretation and for setting field research priorities. Bayesian methods provide a rigorous data assimilation framework for these applications, especially for problems with multiple data constraints. However, the Markov chain Monte Carlo (MCMC) techniques underlying most Bayesian calibration can be prohibitive for computationally demanding models and large datasets. We employ an alternative method, Bayesian model emulation of sufficient statistics, that can approximate the full joint posterior density, is more amenable to parallelization, and provides an estimate of parameter sensitivity. Analysis involved informative priors constructed from a meta-analysis of the primary literature and specification of both model and data uncertainties, and it introduced novel approaches to autocorrelation corrections on multiple data streams and emulating the sufficient statistics surface. We report the integration of this method within an ecological workflow management software, Predictive Ecosystem Analyzer (PEcAn), and its application and validation with two process-based terrestrial ecosystem models: SIPNET and ED2. In a test against a synthetic dataset, the emulator was able to retrieve the true parameter values. A comparison of the emulator approach to standard brute-force MCMC involving multiple data constraints showed that the emulator method was able to constrain the faster and simpler SIPNET model's parameters with comparable performance to the brute-force approach but reduced computation time by more than 2 orders of magnitude. The emulator was then applied to calibration of the ED2 model, whose complexity precludes standard (brute-force) Bayesian data assimilation techniques. Both models are constrained after assimilation of the observational data with the emulator method, reducing the uncertainty around their predictions. Performance metrics showed increased agreement between model predictions and data. Our study furthers efforts toward reducing model uncertainties, showing that the emulator method makes it possible to efficiently calibrate complex models.

58 GEOSCIENCES↗

Computational budget optimization for Bayesian parameter estimation in heavy-ion collisions

Abstract Bayesian parameter estimation provides a systematic approach to compare heavy-ion collision models with measurements, leading to constraints on the properties of nuclear matter with proper accounting of experimental and theoretical uncertainties. Aside from statistical and systematic model uncertainties, interpolation uncertainties can also play a role in Bayesian inference, if the model’s predictions can only be calculated at a limited set of model parameters. This uncertainty originates from using an emulator to interpolate the model’s prediction across a continuous space of parameters. In this work, we study the trade-offs between the emulator (interpolation) and statistical uncertainties. We perform the analysis using spatial eccentricities from the T R ENTo model of initial conditions for nuclear collisions. Given a fixed computational budget, we study the optimal compromise between the number of parameter samples and the number of collisions simulated per parameter sample. For the observables and parameters used in the present study, we find that the best constraints are achieved when the number of parameter samples is slightly smaller than the number of collisions simulated per parameter sample.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Algorithms for Spectral Decomposition with Applications to Optical Plume Anomaly Detection

The analysis of spectral signals for features that represent physical phenomenon is ubiquitous in the science and engineering communities. There are two main approaches that can be taken to extract relevant features from these high-dimensional data streams. The first set of approaches relies on extracting features using a physics-based paradigm where the underlying physical mechanism that generates the spectra is used to infer the most important features in the data stream. We focus on a complementary methodology that uses a data-driven technique that is informed by the underlying physics but also has the ability to adapt to unmodeled system attributes and dynamics. We discuss the following four algorithms: Spectral Decomposition Algorithm (SDA), Non-Negative Matrix Factorization (NMF), Independent Component Analysis (ICA) and Principal Components Analysis (PCA) and compare their performance on a spectral emulator which we use to generate artificial data with known statistical properties. This spectral emulator mimics the real-world phenomena arising from the plume of the space shuttle main engine and can be used to validate the results that arise from various spectral decomposition algorithms and is very useful for situations where real-world systems have very low probabilities of fault or failure. Our results indicate that methods like SDA and NMF provide a straightforward way of incorporating prior physical knowledge while NMF with a tuning mechanism can give superior performance on some tests. We demonstrate these algorithms to detect potential system-health issues on data from a spectral emulator with tunable health parameters.

Srivastava, Askok N.↗

Community Project for Accelerator Science and Simulation (ComPASS)

After concluding our initial exercise (solving a simplified statistical inverse problem with unknown parameter laser intensity) of coupling Vorpal and our parallel statistical library QUESO, we shifted the application focus to DLA. Our efforts focused on developing a Gaussian process (GP) emulator within QUESO for efficient optimization of power couplers within woodpiles. The smaller simulation size (compared with LPA) allows for sufficient “training runs” to develop a reasonable GP statistical emulator for a parameter space of moderate dimension.

43 PARTICLE ACCELERATORS↗

Ab initio predictions link the neutron skin of 208 Pb to nuclear forces

Heavy atomic nuclei have an excess of neutrons over protons, which leads to the formation of a neutron skin whose thickness is sensitive to details of the nuclear force. This links atomic nuclei to properties of neutron stars, thereby relating objects that differ in size by orders of magnitude. The nucleus 208 Pb is of particular interest because it exhibits a simple structure and is experimentally accessible. However, computing such a heavy nucleus has been out of reach for ab initio theory. By combining advances in quantum many-body methods, statistical tools and emulator technology, we make quantitative predictions for the properties of 208 Pb starting from nuclear forces that are consistent with symmetries of low-energy quantum chromodynamics. We explore 10 9 different nuclear force parameterizations via history matching, confront them with data in select light nuclei and arrive at an importance-weighted ensemble of interactions. We accurately reproduce bulk properties of 208 Pb and determine the neutron skin thickness, which is smaller and more precise than a recent extraction from parity-violating electron scattering but in agreement with other experimental probes. This work demonstrates how realistic two- and three-nucleon forces act in a heavy nucleus and allows us to make quantitative predictions across the nuclear landscape.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Exploring uncertainties in global crop yield projections in a large ensemble of crop models and CMIP5 and CMIP6 climate scenarios

Concerns over climate change are motivated in large part because of their impact on human society. Assessing the effect of that uncertainty on specific potential impacts is demanding, since it requires a systematic survey over both climate and impacts models. We provide a comprehensive evaluation of uncertainty in projected crop yields for maize, spring and winter wheat, rice, and soybean, using a suite of nine crop models and up to 45 CMIP5 and 34 CMIP6 climate projections for three different forcing scenarios. To make this task computationally tractable, we use a new set of statistical crop model emulators. We find that climate and crop models contribute about equally to overall uncertainty. While the ranges of yield uncertainties under CMIP5 and CMIP6 projections are similar, median impact in aggregate total caloric production is typically more negative for the CMIP6 projections (+1% to −19%) than for CMIP5 (+5% to −13%). In the first half of the 21st century and for individual crops is the spread across crop models typically wider than that across climate models, but we find distinct differences between crops: globally, wheat and maize uncertainties are dominated by the crop models, but soybean and rice are more sensitive to the climate projections. Climate models with very similar global mean warming can lead to very different aggregate impacts so that climate model uncertainties remain a significant contributor to agricultural impacts uncertainty. These results show the utility of large-ensemble methods that allow comprehensively evaluating factors affecting crop yields or other impacts under climate change. The crop model ensemble used here is unbalanced and pulls the assumption that all projections are equally plausible into question. Better methods for consistent model testing, also at the level of individual processes, will have to be developed and applied by the crop modeling community.

crop yield projections↗

A Bayesian approach for parameter estimation and prediction using a computationally intensive model

Bayesian methods have been successful in quantifying uncertainty in physics-based problems in parameter estimation and prediction. In these cases, physical measurements y are modeled as the best fit of a physics-based model eta(theta), where theta denotes the uncertain, best input setting. Hence the statistical model is of the form y = eta(theta) + c, where epsilon accounts for measurement, and possibly other, error sources. When nonlinearity is present in eta(center dot), the resulting posterior distribution for the unknown parameters in the Bayesian formulation is typically complex and nonstandard, requiring computationally demanding computational approaches such as Markov chain Monte Carlo (MCMC) to produce multivariate draws from the posterior. Although generally applicable, MCMC requires thousands (or even millions) of evaluations of the physics model eta(center dot). This requirement is problematic if the model takes hours or days to evaluate. To overcome this computational bottleneck, we present an approach adapted from Bayesian model calibration. This approach combines output from an ensemble of computational model runs with physical measurements, within a statistical formulation, to carry out inference. A key component of this approach is a statistical response surface, or emulator, estimated from the ensemble of model runs. We demonstrate this approach with a case study in estimating parameters for a density functional theory model, using experimental mass/binding energy measurements from a collection of atomic nuclei. We also demonstrate how this approach produces uncertainties in predictions for recent mass measurements obtained at Argonne National Laboratory.

Higdon, Dave↗

Comparing emulation methods for a high‐resolution storm surge model

Abstract Realistic simulations of complex systems are fundamental for climate and environmental studies. Large computer systems are often not sufficient to run sophisticated computational models for large numbers of different input settings. Statistical surrogate models, or emulators, are key tools enabling fast exploration of the simulator input space. Gaussian processes have become standard for computer simulator emulation. However, they require careful implementation to scale appropriately, motivating alternative methods more recently introduced. We present a comparison study of surrogates of the Sea, Lake, and Overland Surges from Hurricanes (SLOSH) simulator—the simulator of choice for government agencies—using four emulation approaches: BASS; BART; SEPIA; and RobustGaSP. SEPIA and RobustGaSP use Gaussian processes, BASS implements adaptive splines, and BART is based on ensembles of regression trees. We describe the four models and compare them in terms of computation time and predictive metrics. These surrogates use proven and distinct methodologies, are available through accessible software, and quantify prediction uncertainty. Our data cover millions of response values. We find that SEPIA and RobustGaSP provide exceptional predictive power, but cannot scale to emulate experiments as large as the one considered in this paper as effectively as BASS and BART.

Hutchings, Grant↗

THE MIRA–TITAN UNIVERSE: PRECISION PREDICTIONS FOR DARK ENERGY SURVEYS

Large-scale simulations of cosmic structure formation play an important role in interpreting cosmological observations at high precision. The simulations must cover a parameter range beyond the standard six cosmological parameters and need to be run at high mass and force resolution. A key simulation-based task is the generation of accurate theoretical predictions for observables using a finite number of simulation runs, via the method of emulation. Using a new sampling technique, we explore an eight-dimensional parameter space including massive neutrinos and a variable equation of state of dark energy. We construct trial emulators using two surrogate models (the linear power spectrum and an approximate halo mass function). The new sampling method allows us to build precision emulators from just 26 cosmological models and to systematically increase the emulator accuracy by adding new sets of simulations in a prescribed way. Emulator fidelity can now be continuously improved as new observational data sets become available and higher accuracy is required. Finally, using one ΛCDM cosmology as an example, we study the demands imposed on a simulation campaign to achieve the required statistics and accuracy when building emulators for investigations of dark energy.

79 ASTRONOMY AND ASTROPHYSICS↗

Incorporating Snow Albedo Feedback into Downscaled Temperature and Snow Cover Projections for California’s Sierra Nevada

California’s Sierra Nevada is a high-elevation mountain range with significant seasonal snow cover. Under anthropogenic climate change, amplification of the warming is expected to occur at elevations near snow margins due to snow albedo feedback. However, climate change projections for the Sierra Nevada made by global climate models (GCMs) and statistical downscaling methods miss this key process. Dynamical downscaling simulates the additional warming due to snow albedo feedback. Ideally, dynamical downscaling would be applied to a large ensemble of 30 or more GCMs to project ensemble-mean outcomes and intermodel spread, but this is far too computationally expensive. To approximate the results that would occur if the entire GCM ensemble were dynamically downscaled, a hybrid dynamical–statistical downscaling approach is used. First, dynamical downscaling is used to reconstruct the historical climate of the 1981–2000 period and then to project the future climate of the 2081–2100 period based on climate changes from five GCMs. Next, a statistical model is built to emulate the dynamically downscaled warming and snow cover changes for any GCM. This statistical model is used to produce warming and snow cover loss projections for all available CMIP5 GCMs. These projections incorporate snow albedo feedback, so they capture the local warming enhancement (up to 3°C) from snow cover loss that other statistical methods miss. Capturing these details may be important for accurately projecting impacts on surface hydrology, water resources, and ecosystems

54 ENVIRONMENTAL SCIENCES↗

Emulating ab initio computations of infinite nucleonic matter

We construct efficient emulators for the computation of the infinite nuclear matter equation of state. These emulators are based on the subspace-projected coupled-cluster method for which we here develop a new algorithm called small-batch voting to eliminate spurious states that might appear when emulating quantum many-body methods based on a non-Hermitian Hamiltonian. The efficiency and accuracy of these emulators facilitate a rigorous statistical analysis within which we explore nuclear matter predictions for > 10 6 different parametrizations of a chiral interaction model with explicit Δ -isobars at next-to-next-to leading order. Constrained by nucleon-nucleon scattering phase shifts and bound-state observables of light nuclei up to He 4 , we use history matching to identify nonimplausible domains for the low-energy coupling constants of the chiral interaction. Within these domains we perform a Bayesian analysis using sampling and importance resampling with different likelihood calibrations and study correlations between interaction parameters, calibration observables in light nuclei, and nuclear matter saturation properties. Published by the American Physical Society 2024

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Modeling Redshift-space Clustering with Abundance Matching

Abstract We explore the degrees of freedom required to jointly fit projected and redshift-space clustering of galaxies selected in three bins of stellar mass from the Sloan Digital Sky Survey Main Galaxy Sample (SDSS MGS) using a subhalo abundance matching (SHAM) model. We employ emulators for relevant clustering statistics in order to facilitate our analysis, leading to large speed gains with minimal loss of accuracy. We are able to simultaneously fit the projected and redshift-space clustering of the two most massive galaxy samples that we consider with just two free parameters: scatter in stellar mass at fixed SHAM proxy, and the dependence of the SHAM proxy on dark matter halo concentration. We find some evidence for models that include velocity bias, but including orphan galaxies improves our fits to the lower-mass samples significantly. We also model the clustering signals of specific star formation rate (sSFR) selected samples using conditional abundance matching (CAM). We obtain acceptable fits to projected and redshift-space clustering as a function of sSFR and stellar mass using two CAM variants, although the fits are worse than for stellar-mass-selected samples alone. By incorporating nonunity correlations between the CAM proxy and sSFR, we are able to resolve previously identified discrepancies between CAM predictions and SDSS observations of the environmental dependence of quenching for isolated central galaxies.

79 ASTRONOMY AND ASTROPHYSICS↗

Design of Experiments, Model Calibration and Data Assimilation

This presentation provides an overview of emulation, calibration and experiment design for computer experiments. Emulation refers to building a statistical surrogate from a carefully selected and limited set of model runs to predict unsampled outputs. The standard kriging approach to emulation of complex computer models is presented. Calibration refers to the process of probabilistically constraining uncertain physics/engineering model inputs to be consistent with observed experimental data. An initial probability distribution for these parameters is updated using the experimental information. Markov chain Monte Carlo (MCMC) algorithms are often used to sample the calibrated parameter distribution. Several MCMC algorithms commonly employed in practice are presented, along with a popular diagnostic for evaluating chain behavior. Space-filling approaches to experiment design for selecting model runs to build effective emulators are discussed, including Latin Hypercube Design and extensions based on orthogonal array skeleton designs and imposed symmetry requirements. Optimization criteria that further enforce space-filling, possibly in projections of the input space, are mentioned. Designs to screen for important input variations are summarized and used for variable selection in a nuclear fuels performance application. This is followed by illustration of sequential experiment design strategies for optimization, global prediction, and rare event inference.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗