Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

Evaluating turbulence models at high energy densities

The RESHOCK campaign at Lawrence Livermore National Laboratory has been working on increasing our understanding of the evolution of turbulent, unstable plasma interfaces which are applicable in various HED applications including ICF implosions. A common approach to model such interfaces involves the use of Renolds-Averaged-Navier-Stokes (RANS) models whose parameters have been constrained by theory and experiments. These parameter sets are not unique and the applicability of one set of tuned model parameters to systems with differing shock strengths or material densities has not been well tested. Our new NIF experiments, along with related studies of turbulent shear flows, provide data and model validation for plasma interfaces in the HED regime. Our experiments measure mixing-layer width at an unstable interface in the plasma regime and are specifically designed to challenge mix models of the Reynolds-Averaged-Navier-Stokes (RANS) type where turbulence is assumed to be fully developed, i.e. the flow exhibits a broad spectrum of length scales without memory of the initial condition.4 We utilize precise control of the initial interface conditions (densities on both sides of the interface, an initial rippled perturbation pattern consisting of equally weighted wavelengths between 10 and 20 microns, and initial material compositions) along with a repeatable drive history

36 MATERIALS SCIENCE↗

Evaluation of Models and Measurements to Estimate Solar Radiation for 1-Axis Tracking Modules at NREL's SRRL: Preprint

Solar radiation reaching photovoltaic (PV) modules on a 1-axis tracking system can be measured by reference cells or thermopiles. The former is often biased due to the reflection of solar radiation by the glass cover of the PV. The uncertainty can be moderated by applying a correction factor, as a function of solar incident angle and the refractive index of the glass, to the measurement. On the other hand, solar radiation on the inclined PV panels can be computed by transposition models using global horizontal irradiance (GHI) observations from thermopiles. This study examines the models and measurements to estimate solar radiation for 1-axis tracking modules at National Renewable Energy Laboratory’s (NREL’s) Solar Radiation Research Laboratory (SRRL). The 1-minute plane-of-array (POA) irradiances in 2019 are computed using the observed GHIs and a transposition model developed by Perez et al. The POA irradiances are compared with the observation by an IMT reference cell and a Kipp & Zonen CM Pyranometer 22 (CMP22) thermopile. For SRRL’s 1-axis tracking system with the annual solar energy of 2323.9 kWh/m2, the POA irradiance is overestimated by ~70 kWh/m2 using the transposition model. This bias is reduced by more than 50% using the IMT measurements calibrated by a correction factor for a PV surface of antireflection coated glass.

41 EE - Solar Energy Technologies Office (EE-4S)↗

Preliminary Results for Using Uncertainty and Out-of-distribution Detection to Identify Unreliable Predictions.

As machine learning (ML) models are deployed into an ever-diversifying set of application spaces, ranging from self-driving cars to cybersecurity to climate modeling, the need to carefully evaluate model credibility becomes increasingly important. Uncertainty quantification (UQ) provides important information about the ability of a learned model to make sound predictions, often with respect to individual test cases. However, most UQ methods for ML are themselves data-driven and therefore susceptible to the same knowledge gaps as the models themselves. Specifically, UQ helps to identify points near decision boundaries where the models fit the data poorly, yet predictions can score as certain for points that are under-represented by the training data and thus out-of-distribution (OOD). One method for evaluating the quality of both ML models and their associated uncertainty estimates is out-of-distribution detection (OODD). We combine OODD with UQ to provide insights into the reliability of the individual predictions made by an ML model.

97 MATHEMATICS AND COMPUTING↗

Large eddy simulation of a boundary layer with concave streamwise curvature

One of the most exciting recent developments in the field of large eddy simulation (LES) is the dynamic subgrid-scale model. The dynamic model concept is a general procedure for evaluating model constants by sampling a band of the smallest scales actually resolved in the simulation. To date, the procedure has been used primarily in conjunction with the Smagorinsky model. The dynamic procedure has the advantage that the value of the model constant need not be specified a priori, but rather is calculated as a function of space and time as the simulation progresses. This feature makes the dynamic model especially attractive for flows in complex geometries where it is difficult or impossible to calibrate model constants. The dynamic model was highly successful in benchmark tests involving homogeneous and channel flows. Having demonstrated the potential of the dynamic model in these simple flows, the overall direction of the LES effort at CTR shifted toward an evaluation of the model in more complex situations. The current test cases are basic engineering-type flows for which Reynolds averaged approaches were unable to model the turbulence to within engineering accuracy. Flows currently under investigation include a backward-facing step, wake behind a circular cylinder, airfoil at high angles of attack, separated flow in a diffuser, and boundary layer over a concave surface. Preliminary results from the backward-facing step and cylinder wake simulations are encouraging. Progress on the LES of a boundary layer on a concave surface is discussed. Although the geometry of a concave wall is not very complex, the boundary layer that develops on its surface is difficult to model due to the presence of streamwise Taylor-Gortler vortices. These vortices arise as a result of a centrifugal instability associated with the convex curvature.

Lund, Thomas S.↗

Vertical structure and physical processes of the Madden-Julian oscillation: Linking hindcast fidelity to simulated diabatic heating and moistening

Here, many theories for the Madden-Julian oscillation (MJO) focus on diabatic processes, particularly the evolution of vertical heating and moistening. Poor MJO performance in weather and climate models is often blamed on biases in these processes and their interactions with the large-scale circulation. We introduce one of the three components of a model evaluation project, which aims to connect MJO fidelity in models to their representations of several physical processes, focusing on diabatic heating and moistening. This component consists of 20 day hindcasts, initialized daily during two MJO events in winter 2009–2010. The 13 models exhibit a range of skill: several have accurate forecasts to 20 days lead, while others perform similarly to statistical models (8–11 days). Models that maintain the observed MJO amplitude accurately predict propagation, but not vice versa. We find no link between hindcast fidelity and the precipitation-moisture relationship, in contrast to other recent studies. There is also no relationship between models' performance and the evolution of their diabatic heating profiles with rain rate. A more robust association emerges between models' fidelity and net moistening: the highest-skill models show a clear transition from low-level moistening for light rainfall to midlevel moistening at moderate rainfall and upper level moistening for heavy rainfall. The midlevel moistening, arising from both dynamics and physics, may be most important. Accurately representing many processes may be necessary but not sufficient for capturing the MJO, which suggests that models fail to predict the MJO for a broad range of reasons and limits the possibility of finding a panacea.

58 GEOSCIENCES↗

Practical Application of Parallel Coordinates for Climate Model Analysis

The determination of relationships between climate variables and the identification of the most significant associations between them in various geographic regions is an important aspect of climate model evaluation. The EDEN visual analytics toolkit has been developed to aid such analysis by facilitating the assessment of multiple variables with respect to the amount of variability that can be attributed to specific other variables. EDEN harnesses the parallel coordinates visualization technique and is augmented with graphical indicators of key descriptive statistics. A case study is presented in which the focus on the Harvard Forest site (42.5378N Lat, 72.1715W Lon) and the Community Land Model Version 4 (CLM4) is evaluated. It is shown that model variables such as land water runoff are more sensitive to a particular set of environmental variables than a suite of other inputs in the 88 variable analysis conducted. The approach presented here allows climate-domain scientists to focus on the most important variables in the model evaluations.

54 ENVIRONMENTAL SCIENCES↗

A Modeling Framework for Evaluating the Drought Resilience of a Surface Water Supply System under Non-stationarity

The future resilience of water supply systems is unprecedentedly challenged by non-stationary processes, such as fast population growth and a changing climate. A thorough understanding of how these non-stationarities impact water supply resilience is vital to support sustainable decision making, particularly for large cities in arid and/or semi-arid regions. In this study, a novel modeling framework, which integrates hydrological processes and water management, was established over a representative water limited metropolitan area to evaluate the impacts of water availability and water demand on reservoir storage and water supply reliability. In this framework, climate change induced drought events were selected from statistically downscaled Coupled Model Intercomparison Project Phase 5 outputs under the Representative Concentration Pathway 8.5 scenario, while future water demand was estimated by the product of projected future population and per capita water use. Compared with the first half of the 21st century (2000–2049), reservoir storage and water supply reliability during the second half century (2050–2099) are projected to reduce by 16.1% and 14.2%, respectively. While both future multi-year droughts and population growth will lower water supply resilience, the uncertainty associated with future climate projection is larger than that associated with urbanization. To reduce the drought risks, a combination of mitigation strategies (e.g., additional conservation, integrating new water sources, and water use redistribution) was found to be the most efficient approach and can significantly improve water supply reliability by as much as 15.9%.

Zhao, Gang (ORCID:0000000327370530)↗

Data-based filtered dissipation rate modelling for multi-modal turbulent combustion: evaluating a priori model generalizability

Manifold-based models offer a computationally efficient alternative to directly transporting the thermochemical state in computational simulations of turbulent reacting flows, projecting the high-dimensional thermochemical state-space onto a low-dimensional manifold. Recent efforts have yielded a manifold-based model applicable to multi-modal combustion, enabling reconstruction of the thermochemical state from solutions to two-dimensional manifold equations in mixture fraction and generalized progress variable that are parameterised by three scalar dissipation rates. In coarse-grained simulations such as Large Eddy Simulation (LES), closure of the multi-modal manifold equations and subfilter variances/covariance requires closure of three filtered scalar dissipation rates. Here, the present work adopts a data-based approach, providing closure for the three filtered scalar dissipation rates via deep neural networks (DNNs). High-fidelity datasets corresponding to an autoigniting n-dodecane jet flame and a bluff body swirl-stabilized confined lifted spray flame of two aviation fuels (Jet-A and C1) with different ignition propensities are leveraged to generate training data that spans a diverse range of thermodynamic conditions and combustion modes, including low- and high-temperature ignition regimes in addition to premixed and nonpremixed behaviour. A final DNN model is trained to enforce inherent physical constraints by learning nonlinear functional transformations of the three filtered scalar dissipation rates. The generalizability of this constrained DNN model is demonstrated a priori via conditional statistics evaluated on the lifted spray flame with C1–a configuration that had not been included in the training data. Excellent DNN agreement with conditional DNS statistics is observed, and integrated gradients are computed to identify the most sensitive input variables. The similarity of the marginal PDFs of the most informative input variables and outputs across configurations are quantified via the Wasserstein metric, demonstrating that data-based models may successfully generalize to unseen parametric conditions so long as the most informative input variables share similar distributions across training and testing datasets.

Data-based modelling↗

Evaluation of Combustion Models for CFD Simulation of Pre-Chamber Ignition in a Natural Gas Engine

Natural gas is considered an attractive alternative fossil fuel as it produces 25% less carbon dioxide and emits fewer amounts of toxic pollutants compared to gasoline and diesel. A pre-chamber type ignition device in a natural gas engine is capable of extending the lean operating limit, and hence improving thermal efficiency, by providing spatially distributed ignition with multiple turbulent flame-jets and accelerating the combustion rate. However, the process of ignition in the main chamber triggered by turbulent jets emanating from a pre-chamber is very sensitive to the pre-chamber geometry and fueling strategy, and any effort to optimize these parameters requires fundamental understanding of the underlying physical and thermodynamic phenomena, which can be studied by a multi-dimensional numerical simulation approach. In this study, three turbulent combustion models widely adopted in the multi-dimensional engine modeling community were preliminary evaluated to simulate the combustion process in a natural gas engine with a pre-chamber spark-ignition system. One multi-zone well-stirred reactor model, and two flamelet-based combustion models, G-equation and Extended Coherent Flamelet Model, were employed for this assessment. Under a fuel-lean condition, the three combustion models were used to match experimental data and analyze the differences in terms of in-cylinder pressure and heat release rate.

combustion model evaluation↗

Quasi-stationary Planetary Scale Waves in Modern Climate Models

We evaluate climate model simulations of geopotential height at the largest space-time scales, considering the first Fourier components (in longitude) of long-term climatological means. Quasi-stationary planetary scale waves emerge from this analysis. Variations of these waves are important components of climate change, for example associated with regional drought. Systematic examination of their long-term climatology provides context for understanding their evolution in time. We compare four reanalysis datasets with “historical runs” from the latest version of the Coupled Model Intercomparison Project (CMIP) and find general agreement, but we also identify model errors outside the range of both observational uncertainty and the uncertainty implied by interannual variability. Errors in wave phase have possible implications for climate model projections of regional climate and of shifting weather patterns. One model in the set we examine, originally created nearly 20 years before the others, exhibits noticeably greater errors.

54 ENVIRONMENTAL SCIENCES↗

The AeroCom evaluation and intercomparison of organic aerosol in global models

This paper evaluates the current status of global modeling of the organic aerosol (OA) in the troposphere and analyzes the differences between models as well as between models and observations. Thirty-one global chemistry transport models (CTMs) and general circulation models (GCMs) have participated in this intercomparison, in the framework of AeroCom phase II. The simulation of OA varies greatly between models in terms of the magnitude of primary emissions, secondary OA (SOA) formation, the number of OA species used (2 to 62), the complexity of OA parameterizations (gas-particle partitioning, chemical aging, multiphase chemistry, aerosol microphysics), and the OA physical, chemical and optical properties. The diversity of the global OA simulation results has increased since earlier AeroCom experiments, mainly due to the increasing complexity of the SOA parameterization in models, and the implementation of new, highly uncertain, OA sources. Diversity of over one order of magnitude exists in the modeled vertical distribution of OA concentrations that deserves a dedicated future study. Furthermore, although the OA / OC ratio depends on OA sources and atmospheric processing, and is important for model evaluation against OA and OC observations, it is resolved only by a few global models. The median global primary OA (POA) source strength is 56 Tg a –1 (range 34–144 Tg a −1 ) and the median SOA source strength (natural and anthropogenic) is 19 Tg a –1 (range 13–121 Tg a −1 ). Among the models that take into account the semi-volatile SOA nature, the median source is calculated to be 51 Tg a –1 (range 16–121 Tg a −1 ), much larger than the median value of the models that calculate SOA in a more simplistic way (19 Tg a –1 ; range 13–20 Tg a –1 , with one model at 37 Tg a −1 ). The median atmospheric burden of OA is 1.4 Tg (24 models in the range of 0.6–2.0 Tg and 4 between 2.0 and 3.8 Tg), with a median OA lifetime of 5.4 days (range 3.8–9.6 days). In models that reported both OA and sulfate burdens, the median value of the OA/sulfate burden ratio is calculated to be 0.77; 13 models calculate a ratio lower than 1, and 9 models higher than 1. For 26 models that reported OA deposition fluxes, the median wet removal is 70 Tg a –1 (range 28–209 Tg a −1 ), which is on average 85% of the total OA deposition. Fine aerosol organic carbon (OC) and OA observations from continuous monitoring networks and individual field campaigns have been used for model evaluation. At urban locations, the model–observation comparison indicates missing knowledge on anthropogenic OA sources, both strength and seasonality. The combined model–measurements analysis suggests the existence of increased OA levels during summer due to biogenic SOA formation over large areas of the USA that can be of the same order of magnitude as the POA, even at urban locations, and contribute to the measured urban seasonal pattern. Global models are able to simulate the high secondary character of OA observed in the atmosphere as a result of SOA formation and POA aging, although the amount of OA present in the atmosphere remains largely underestimated, with a mean normalized bias (MNB) equal to –0.62 (–0.51) based on the comparison against OC (OA) urban data of all models at the surface, –0.15 (+0.51) when compared with remote measurements, and –0.30 for marine locations with OC data. The mean temporal correlations across all stations are low when compared with OC (OA) measurements: 0.47 (0.52) for urban stations, 0.39 (0.37) for remote stations, and 0.25 for marine stations with OC data. The combination of high (negative) MNB and higher correlation at urban stations when compared with the low MNB and lower correlation at remote sites suggests that knowledge about the processes that govern aerosol processing, transport and removal, on top of their sources, is important at the remote stations. There is no clear change in model skill with increasing model complexity with regard to OC or OA mass concentration. As a result, the complexity is needed in models in order to distinguish between anthropogenic and natural OA as needed for climate mitigation, and to calculate the impact of OA on climate accurately.

54 ENVIRONMENTAL SCIENCES↗

Regional and Model-Specific Response Types in A Global Gridded Crop Model Ensemble

Crop models are often employed to project crop yields under changing conditions such as global warming and associated management change for adaptation. Multi-model ensembles are promoted to enhance the robustness of projections, but questions remain on what causes often large differences between projections of individual models. Global Gridded Crop Models (GGCMs) are especially exposed to this question when applied for assessing climate change impacts, adaptation, environmental impacts of agricultural production, because their results are used in downstream analyses, such as in integrated assessment or economic modeling for projecting future land-use change. Even though global gridded crop models are often based on detailed field-scale models or have implemented similar modeling principles in other ecosystem models, global-scale models are subject to substantial uncertainties from both model structure and parametrization as well as from calibration and input data quality. AgMIP’s Global Gridded Crop Model Intercomparison (GGCMI) has thus set out to intercompare GGCMs in order to evaluate model performance, describe model uncertainties, identify inconsistencies within the ensemble and underlying reasons, and to ultimately improve models and modeling capacities. In phase 2 of the GGCMI activities, 12 modeling groups followed a modeling protocol that asked for up to 1404 31-year global simulations at 0.5 arc-degree spatial resolution to assess models’ sensitivities to changes in carbon dioxide (C; 4 different levels) temperature (T; 7 different offset levels), water supply (W; 9 levels), and nitrogen (N; 3 levels), the so-called CTWN experiment (Franke et al. 2020; http://dx.doi.org/10.5194/gmd-13-2315-2020). We here present analyses of model response types using impact response surfaces along the C, T, W, and N dimensions, respectively and collectively. Doing so, we can understand differences in simulated responses per driver rather than aggregated changes in yields. We find that models’ sensitivities to the individual driver dimensions are substantially different and often more different across models than across regions. A cluster analysis finds regional and model-specific patterns. There is some agreement across models with respect to the spatial patterns of response types but strong differences in the distribution of response type clusters across models suggests that models need to undergo further scrutiny. We suggest establishing standards in model process evaluation not only against historical dynamics but also against dedicated experiments across the CTWN dimensions.

crop models↗

Comprehensive system models: Strategies for evaluation

The task of evaluating comprehensive earth system models is vast involving validations of every model component at every scale of organization, as well as tests of all the individual linkages. Even the most detailed evaluation of each of the component processes and the individual links among them should not, however, engender confidence in the performance of the whole. The integrated earth system is so rich with complex feedback loops, often involving components of the atmosphere, oceans, biosphere, and cryosphere, that it is certain to exhibit emergent properties very difficult to predict from the perspective of a narrow focus on any individual component of the system. Therefore, a substantial share of the task of evaluating comprehensive earth system models must reside at the level of whole system evaluations. Since complete, integrated atmosphere/ ocean/ biosphere/ hydrology models are not yet operational, questions of evaluation must be addressed at the level of the kinds of earth system processes that the models should be competent to simulate, rather than at the level of specific performance criteria. Here, we have tried to identify examples of earth system processes that are difficult to simulate with existing models and that involve a rich enough suite of feedbacks that they are unlikely to be satisfactorily described by highly simplified or toy models. Our purpose is not to specify a checklist of evaluation criteria but to introduce characteristics of the earth system that may present useful opportunities for model testing and, of course, improvement.

Field, Christopher↗

A climate model projection weighting scheme accounting for performance and interdependence: Model Projection Weighting Scheme

Uncertainties of climate projections are routinely assessed by considering simulations from different models. Observations are used to evaluate models, yet there is a debate about whether and how to explicitly weight model projections by agreement with observations. Here we present a straightforward weighting scheme that accounts both for the large differences in model performance and for model interdependencies, and we test reliability in a perfect model setup. We provide weighted multimodel projections of Arctic sea ice and temperature as a case study to demonstrate that, for some questions at least, it is meaningless to treat all models equally. The constrained ensemble shows reduced spread and a more rapid sea ice decline than the unweighted ensemble. We argue that the growing number of models with different characteristics and considerable interdependence finally justifies abandoning strict model democracy, and we provide guidance on when and how this can be achieved robustly.

54 ENVIRONMENTAL SCIENCES↗