mystic: tools for AI, UQ, ML, model validation, and robust design under uncertainty
Explore the source record for details and available documents.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Abstract not provided.
Abstract not provided.
Abstract not provided.
MOCkingbird LDRD final poster.
Explore the source record for details and available documents.
Final performance report for our DOE grant
Validating models of airspace operations is a particular challenge. These models are often aimed at finding and exploring safety violations, and aim to be accurate representations of real-world behavior. However, the rules governing the behavior are quite complex: nonlinear physics, operational modes, human behavior, and stochastic environmental concerns all determine the responses of the system. In this paper, we present a study on aircraft runway approaches as modeled in Georgia Tech's Work Models that Compute (WMC) simulation. We use a new learner, Genetic-Active Learning for Search-Based Software Engineering (GALE) to discover the Pareto frontiers defined by cognitive structures. These cognitive structures organize the prioritization and assignment of tasks of each pilot during approaches. We discuss the benefits of our approach, and also discuss future work necessary to enable uncertainty quantification.
Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.
Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.
Two issues in the treatment of tropical convection in general circulation models are examined. First, several studies have found significant gradients in clear sky longwave fluxes near large convective systems. Increased upper tropospheric moisture associated with deep convection may explain the reduction in the longwave emission. Similar local gradients are not apparent in measurements from the Earth Radiation Budget Experiment (ERBE), an important data set for model validation. Thus the average cloud forcing and greenhouse effect derived from models and observations may differ systematically over warm tropical oceans. A comparison of ERBE fluxes with radiative calculations using coincident balloon-sonde atmospheric profiles indicates negligible systematic bias in the observations. The effect of convection on the clear sky fluxes may be localized to the edges of individual cloud systems. Second, the balance between shortwave and cloud forcing and longwave forcing is a persistent feature of tropical cloud systems in ERBE data. This cancellation effect has been used to diagnose problems in GCM (General Circulation Model) cloud fields on seasonal time scales. The daily record of net cloud radiative forcing is analyzed to determine the smallest spatial and temporal scales for the balance. The results show cancellation on periods as short as three days for regions smaller than 2.5 by 2.5 deg. The analysis indicates that the balance is primarily a local phenomenon characteristic of tropical convection. This is consistent with findings that the small cloud radiative forcing is due primarily to thick tropical cirrus. These results represent a particularly stringent test of convective parameterizations in GCM's with interactive ocean surfaces.
Various modeling techniques are used to predict the capacity fade of Li-ion batteries. Algebraic reduced-order models, which are inherently interpretable and computationally fast, are ideal for use in battery controllers, technoeconomic models, and multi-objective optimizations. For Li-ion batteries with graphite anodes, solid-electrolyte-interphase (SEI) growth on the graphite surface dominates fade. This fade is often modeled using physically informed equations, such as square-root of time for predicting solvent-diffusion limited SEI growth, and Arrhenius and Tafel-like equations predicting the temperature and state-of-charge rate dependencies. In some cases, completely empirical relationships are proposed. However, statistical validation is rarely conducted to evaluate model optimality, and only a handful of possible models are usually investigated. This article demonstrates a novel procedure for automatically identifying reduced-order degradation models from millions of algorithmically generated equations via bi-level optimization and symbolic regression. Identified models are statistically validated using cross-validation, sensitivity analysis, and uncertainty quantification via bootstrapping. On a LiFePO 4 /Graphite cell calendar aging data set, automatically identified models utilizing square-root, power law, stretched exponential, and sigmoidal functions result in greater accuracy and lower uncertainty than models identified by human experts, and demonstrate that previously known physical relationships can be empirically "rediscovered" using machine learning.
In the process of model validation, models are often declared valid when the differences between model predictions and experimental data sets are satisfactorily small. However, little consideration is given to the effectiveness of a model using parameters that deviate slightly from those that were fitted to data, such as a higher load level. Furthermore, few means exist to compare and choose between two or more models that reproduce data equally well. These issues can be addressed by analyzing model form error, which is the error associated with the differences between the physical phenomena captured by models and that of the real system. This report presents a new quantitative method for model form error analysis and applies it to data taken from experiments on tape joint bending vibrations. Two models for the tape joint system are compared, and suggestions for future improvements to the method are given. As the available data set is too small to draw any statistical conclusions, the focus of this paper is the development of a methodology that can be applied to general problems.
The need for simple, but accurate performance models of wind turbine generators (WTGs), photovoltaic (PV) plants, and battery energy storage systems (BESS) for various hybrid power plant (HPP) studies motivates the present work. Particularly, the development and verification stage of HPP controls requires reduced-order models to minimize the complexity and computation effort of simulation platforms. In this paper, such models are proposed, and the most essential parts of the models are validated through field measurements. The models target power system integration studies involving active and reactive power, as well as frequency and voltage regulation where detailed models, as proposed in the standards, can be cumbersome. Field measurements of two Vestas WTGs, one 1-MW PV plant, and one 1-MW/1-MWh BESS are used for model validation. The results show that the WTG and PV performance models correctly estimate the power generation variability according to fluctuations in wind speed and solar irradiance. The BESS performance model provides satisfactory results related to grid-forming control performance and estimation of state-of-charge. The presented validation work enables using the proposed performance models for power system studies and HPP control design in all model-based design stages, that is, preliminary analysis, design, verification, and validation with a high level of confidence.
Models predicting rear irradiance for bifacial systems are critical to establish accurate estimates of energy yield. In this paper, we compare five published bifacial irradiance models, varying clearance, row spacing, tilt, and albedo to measure the sensitivity to these parameters. Bifacial energy gains (BG E ) as high as 20% are predicted for some configurations. Model agreement is generally good for low ground clearance (clearance heights lower than 0.75 times the collector width), but at higher clearances, finite system size and edge effects become a significant factor in simulations, stretching assumptions of infinite system extent made in some models. In addition, rear irradiance uniformity is improved at high ground clearance, as expected. A test-bed construction and results are described for comparison between modeled and measured data in Golden, CO, USA. Finally, the investigations indicate that model agreement for BG E calculation is better than 2% (absolute) when compared with measured results, depending on the system configuration.
Chemically reacting flow models generally involve inputs and parameters that are determined from empirical measurements, and therefore exhibit a certain degree of uncertainty. Estimating the propagation of this uncertainty into computational model output predictions is crucial for purposes of reacting flow model validation, model exploration, as well as design optimization. Recent years have seen great developments in probabilistic methods and tools for efficient uncertainty quantification (UQ) in computational models. These tools are grounded in the use of Polynomial Chaos (PC) expansions for representation of random variables. The utility and effectiveness of PC methods have been demonstrated in a range of physical models, including structural mechanics, transport in porous media, fluid dynamics, aeronautics, heat transfer, and chemically reacting flow. While high-dimensionality remains nominally an ongoing challenge, great strides have been made in dealing with moderate dimensionality along with non-linearity and oscillatory dynamics. In this talk, I will give an overview of UQ in chemical systems. I will cover both: (1) the estimation of uncertain input parameters from empirical data, and (2) the forward propagation of parametric uncertainty to model outputs. I will cover the basics of forward PC UQ methods with examples of their use. I will also highlight the need for accurate estimation of the joint probability density over the uncertain parameters, in order to arrive at meaningful estimates of model output uncertainties. Finally, I will discuss recent developments on the inference of this density given partial information from legacy experiments, in the absence of raw data.
Chemically reacting flow models generally involve inputs and parameters that are determined from empirical measurements, and therefore exhibit a certain degree of uncertainty. Estimating the propagation of this uncertainty into computational model output predictions is crucial for purposes of reacting flow model validation, model exploration, as well as design optimization. Recent years have seen great developments in probabilistic methods and tools for efficient uncertainty quantification (UQ) in computational models. These tools are grounded in the use of Polynomial Chaos (PC) expansions for representation of random variables. The utility and effectiveness of PC methods have been demonstrated in a range of physical models, including structural mechanics, transport in porous media, fluid dynamics, aeronautics, heat transfer, and chemically reacting flow. While high-dimensionality remains nominally an ongoing challenge, great strides have been made in dealing with moderate dimensionality along with non-linearity and oscillatory dynamics. In this talk, I will give an overview of UQ in chemical systems. I will cover both: (1) the estimation of uncertain input parameters from empirical data, and (2) the forward propagation of parametric uncertainty to model outputs. I will cover the basics of forward PC UQ methods with examples of their use. I will also highlight the need for accurate estimation of the joint probability density over the uncertain parameters, in order to arrive at meaningful estimates of model output uncertainties. Finally, I will discuss recent developments on the inference of this density given partial information from legacy experiments, in the absence of raw data.
Microgrids, consisting of distributed generators (DGs), loads, and energy storage systems, play an important role in future smart grids. In order to evaluate how microgrids operate and their dynamic response, it is necessary to develop accurate dynamic models of DGs in microgrids. This paper proposes a systematic method to calibrate synchronous-generator-interfaced DG models. Challenges associated with model validation in microgrids are analyzed. Underlying parameters are categorized into two groups, i.e., the steady-state parameters and time constants, which are estimated in two successive stages using multiple event data. This methodology ensures that the validated model with the estimated parameters are applicable for all events. A modified unscented Kalman filter (UKF) is proposed to deal with the cases in which measurement of power angle is unavailable. The effectiveness of the proposed method is validated using field test data of a 2.6 MVA diesel generator in a real microgrid.