Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Incorporating an advanced aerosol activation parameterization into WRF-CAM5: Model evaluation and parameterization intercomparison

Aerosol activation into cloud droplets is an important process that governs aerosol indirect effects. The advanced treatment of aerosol activation by Fountoukis and Nenes (2005) and its recent updates, collectively called the FN series, have been incorporated into a newly developed regional coupled climate-air quality model based on the Weather Research and Forecasting model with the physics package of the Community Atmosphere Model version 5 (WRF-CAM5) to simulate aerosol-cloud interactions in both resolved and convective clouds. The model is applied to East Asia for two full years of 2005 and 2010. A comprehensive model evaluation is performed for model predictions of meteorological, radiative, and cloud variables, chemical concentrations, and column mass abundances against satellite data and surface observations from air quality monitoring sites across East Asia. The model performs overall well for major meteorological variables including near-surface temperature, specific humidity, wind speed, precipitation, cloud fraction, precipitable water, downward shortwave and longwave radiation, and column mass abundances of CO, SO 2 , NO 2 , HCHO, and O 3 in terms of both magnitudes and spatial distributions. Larger biases exist in the predictions of surface concentrations of CO and NO x at all sites and SO 2 , O 3 , PM 2.5 , and PM 10 concentrations at some sites, aerosol optical depth, cloud condensation nuclei over ocean, cloud droplet number concentration (CDNC), cloud liquid and ice water path, and cloud optical thickness. Compared with the default Abdul-Razzack Ghan (2000) parameterization, simulations with the FN series produce ~107–113% higher CDNC, with half of the difference attributable to the higher aerosol activation fraction by the FN series and the remaining half due to feedbacks in subsequent cloud microphysical processes. With the higher CDNC, the FN series are more skillful in simulating cloud water path, cloud optical thickness, downward shortwave radiation, shortwave cloud forcing, and precipitation. Here, the model evaluation identifies several areas of improvements including emissions and their vertical allocation as well as model formulations such as aerosol formation, cloud droplet nucleation, and ice nucleation.

54 ENVIRONMENTAL SCIENCES↗

Data–Driven Velocity Model Evaluation Using K–Means Clustering

In this work, we develop a data-driven clustering method to evaluate a velocity model using surface wave velocity dispersion. This is done by first computing theoretical dispersion curves for 1-D velocity profiles of all the grid locations and then splitting the resulting dispersion curves into a certain number of groups via the K-means clustering. The observed dispersion curves are also clustered following the same procedure and the velocity model is assessed by comparing the spatial patterns obtained for the observed and synthetic data sets. The method is applied to evaluate two community velocity models in southern California, CVM-S4.26 and CVM-H15.1, using phase velocity maps derived for 3–16 s Rayleigh waves. We found a good correlation in the spatial distribution of clusters between the result of CVM-S4.26 and that of the observed data, suggesting that the CVM-S4.26 fits the observed dispersion maps better than the CVM-H15.1 in terms of features extracted from the clustering analysis.

58 GEOSCIENCES↗

Towards improved and more routine Earth system model evaluation in CMIP

The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. More research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well established and considered an essential part of systematic evaluation but have been implemented ad hoc with little community coordination. Given the diversity and complexity of ESM analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently and consistently. We provide a perspective and viewpoint on how a more systematic, open, and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP can be achieved, and announce our intention to implement such a system for CMIP6. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop and apply evaluation tools that are based on the latest scientific knowledge and observational reference, are well suited for routine use, and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output and documentation accessible via the ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for model intercomparison projects using the same data structure and organization as the ESM output. This largely facilitates routine evaluation of the ESMs, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. And, at the same time, we encourage the community to contribute observations and reanalyses for model evaluation to the obs4MIPs and ana4MIPs archives. The intention is to produce through the ESGF a widely accepted quasi-operational evaluation framework for CMIP6 that would routinely execute a series of standardized evaluation tasks. Over time, as this capability matures, we expect to produce an increasingly systematic characterization of models which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also reveal whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. Finally, this framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community.

54 ENVIRONMENTAL SCIENCES↗

The Fire and Smoke Model Evaluation Experiment—A Plan for Integrated, Large Fire–Atmosphere Field Campaigns

The Fire and Smoke Model Evaluation Experiment (FASMEE) is designed to collect integrated observations from large wildland fires and provide evaluation datasets for new models and operational systems. Wildland fire, smoke dispersion, and atmospheric chemistry models have become more sophisticated, and next-generation operational models will require evaluation datasets that are coordinated and comprehensive for their evaluation and advancement. Integrated measurements are required, including ground-based observations of fuels and fire behavior, estimates of fire-emitted heat and emissions fluxes, and observations of near-source micrometeorology, plume properties, smoke dispersion, and atmospheric chemistry. To address these requirements the FASMEE campaign design includes a study plan to guide the suite of required measurements in forested sites representative of many prescribed burning programs in the southeastern United States and increasingly common high-intensity fires in the western United States. Here we provide an overview of the proposed experiment and recommendations for key measurements. The FASMEE study provides a template for additional large-scale experimental campaigns to advance fire science and operational fire and smoke models.

54 ENVIRONMENTAL SCIENCES↗

CrossCheck: Rapid, Reproducible, and Interpretable Model Evaluation

Evaluation beyond aggregate performance metrics, e.g. F1-score, is crucial to both establish an appropriate level of trust in machine learning models and identify future model improvements. In this paper we demonstrate CrossCheck, an interactive visualization tool for rapid crossmodel comparison and reproducible error analysis. We describe the tool and discuss design and implementation details. We then present three use cases (named entity recognition, reading comprehension, and clickbait detection) that show the benefits of using the tool for model evaluation. CrossCheck allows data scientists to make informed decisions to choose between multiple models, identify when the models are correct and for which examples, investigate whether the models are making the same mistakes as humans, evaluate models’ generalizability and highlight models’ limitations, strengths and weaknesses. Furthermore, CrossCheck is implemented as a Jupyter widget, which allows rapid and convenient integration into data scientists’ model development workflows.

Arendt, Dustin L.↗

Regime-oriented causal model evaluation of Atlantic–Pacific teleconnections in CMIP6

The climate system and its spatio-temporal changes are strongly affected by modes of long-term internal variability, like the Pacific decadal variability (PDV) and the Atlantic multidecadal variability (AMV). As they alternate between warm and cold phases, the interplay between PDV and AMV varies over decadal to multidecadal timescales. Here, we use a causal discovery method to derive fingerprints in the Atlantic–Pacific interactions and to investigate their phase-dependent changes. Dependent on the phases of PDV and AMV, different regimes with characteristic causal fingerprints are identified in reanalyses in a first step. In a second step, a regime-oriented causal model evaluation is performed to evaluate the ability of models participating in the Coupled Model Intercomparison Project Phase 6 (CMIP6) in representing the observed changing interactions between PDV, AMV and their extra-tropical teleconnections. The causal graphs obtained from reanalyses detect a direct opposite-sign response from AMV to PDV when analyzing the complete 1900–2014 period and during several defined regimes within that period, for example, when AMV is going through its negative (cold) phase. Reanalyses also demonstrate a same-sign response from PDV to AMV during the cold phase of PDV. Historical CMIP6 simulations exhibit varying skill in simulating the observed causal patterns. Generally, large-ensemble (LE) simulations showed better network similarity when PDV and AMV were out of phase compared to other regimes. Also, the two largest ensembles (in terms of number of members) were found to contain realizations with similar causal fingerprints to observations. For most regimes, these same models showed higher network similarity when compared to each other. This work shows how causal discovery on LEs complements the available diagnostics and statistical metrics of climate variability to provide a powerful tool for climate model evaluation.

54 ENVIRONMENTAL SCIENCES↗

Model Evaluation Report for Corrective Action Unit 98: Frenchman Flat, Nevada National Security Site, Nye County, Nevada, Revision 1

Model evaluation focused solely on the PIN STRIPE and MILK SHAKE underground nuclear tests’ contaminant boundaries (CBs) because they had the largest extent, uncertainty, and potential consequences. The CAMBRIC radionuclide migration experiment also had a relatively large CB, but because it was constrained by transport data (notably Well UE-5n), there was little uncertainty, and radioactive decay reduced concentrations before much migration could occur. Each evaluation target and the associated data-collection activity were assessed in turn to determine whether the new data support, or demonstrate conservatism of, the CB forecasts. The modeling team—in this case, the same team that developed the Frenchman Flat geologic, source term, and groundwater flow and transport models—analyzed the new data and presented the results to a PER committee. Existing site understanding and its representation in numerical groundwater flow and transport models was evaluated in light of the new data and the ability to proceed to the CR stage of long-term monitoring and institutional control.

54 ENVIRONMENTAL SCIENCES↗

Greenland Surface Mass Balance as Simulated by the Community Earth System Model. Part I: Model Evaluation and 1850–2005 Results

The modeling of the surface mass balance (SMB) of the Greenland Ice Sheet (GIS) requires high-resolution models in order to capture the observed large gradients in the steep marginal areas. Until now, global climate models have not been considered suitable to model ice sheet SMB owing to model biases and insufficient resolution. This study analyzes the GIS SMB simulated for the period 1850–2005 by the Community Earth System Model (CESM), which includes a new ice sheet component with multiple elevation classes for SMB calculations. The model is evaluated against observational data and output from the regional model Regional Atmospheric Climate Model version 2 (RACMO2). Because of a lack of major climate biases, a sophisticated calculation of snow processes (including surface albedo evolution) and an adequate downscaling technique, CESM is able to realistically simulate GIS surface climate and SMB. CESM SMB agrees reasonably well with in situ data from 475 locations ($r$ = 0.80) and output from RACMO2 ($r$ = 0.79). The simulated mean SMB for 1960–2005 is 359 ± 120 Gt yr -1 in the range of estimates from regional climate models. The simulated seasonal mass variability is comparable with mass observations from the Gravity Recovery and Climate Experiment (GRACE), with synchronous annual maximum (May) and minimum (August–September) and similar amplitudes of the seasonal cycle. CESM is able to simulate the bands of precipitation maxima along the southeast and northwest margins, but absolute precipitation rates are underestimated along the southeastern margin and overestimated in the high interior. The model correctly simulates the major ablation areas. Total refreezing represents 35% of the available liquid water (the sum of rain and melt).

54 ENVIRONMENTAL SCIENCES↗

Geothermal Electricity Technologies Evaluation Model DOE Tool for Assessing Impact of Research on Cost of Power

The U.S. Department of Energy (DOE) has developed a spreadsheet model to provide insight as to how its research activities can impact of cost of producing power from geothermal energy. This model is referred to as GETEM, which stands for “Geothermal Electricity Technologies Evaluation Model”. Based on user input, the model develops estimates of costs associated with exploration, well field development, and power plant construction that are used along with estimated operating costs to provide a predicted power generation cost. The model allows the user to evaluate how reductions in cost, or increases in performance or productivity will impact the predicted power generation cost. This feature provides a means of determining how specific technology improvements can impact generation costs, and as such assists DOE in both prioritizing research areas and identifying where research is needed.

15 GEOTHERMAL ENERGY↗

Process-level model evaluation: a snow and heat transfer metric

Land models require evaluation in order to understand results and guide future development. Examining functional relationships between model variables can provide insight into the ability of models to capture fundamental processes and aid in minimizing uncertainties or deficiencies in model forcing. This study quantifies the proficiency of land models to appropriately transfer heat from the soil through a snowpack to the atmosphere during the cooling season (Northern Hemisphere: October–March). Using the basic physics of heat diffusion, we investigate the relationship between seasonal amplitudes of soil versus air temperatures due to insulation from seasonal snow. Observations demonstrate the anticipated exponential relationship of attenuated soil temperature amplitude with increasing snow depth and indicate that the marginal influence of snow insulation diminishes beyond an effective snow depth of about 50 cm. A snow and heat transfer metric (SHTM) is developed to quantify model skill compared to observations. Land models within the CMIP5 experiment vary widely in SHTM scores, and deficiencies can often be traced to model structural weaknesses. The SHTM value for individual models is stable over 150 years of climate, 1850–2005, indicating that the metric is insensitive to climate forcing and can be used to evaluate each model's representation of the insulation process.

58 GEOSCIENCES↗

The winter central Arctic surface energy budget: A model evaluation using observations from the MOSAiC campaign

This study evaluates the simulation of wintertime (15 October, 2019, to 15 March, 2020) statistics of the central Arctic near-surface atmosphere and surface energy budget observed during the MOSAiC campaign with short-term forecasts from 7 state-of-the-art operational and experimental forecast systems. Five of these systems are fully coupled ocean-sea ice-atmosphere models. Forecast systems need to simultaneously simulate the impact of radiative effects, turbulence, and precipitation processes on the surface energy budget and near-surface atmospheric conditions in order to produce useful forecasts of the Arctic system. This study focuses on processes unique to the Arctic, such as, the representation of liquid-bearing clouds at cold temperatures and the representation of a persistent stable boundary layer. It is found that contemporary models still struggle to maintain liquid water in clouds at cold temperatures. Given the simple balance between net longwave radiation, sensible heat flux, and conductive ground flux in the wintertime Arctic surface energy balance, a bias in one of these components manifests as a compensating bias in other terms. This study highlights the different manifestations of model bias and the potential implications on other terms. Three general types of challenges are found within the models evaluated: representing the radiative impact of clouds, representing the interaction of atmospheric heat fluxes with sub-surface fluxes (i.e., snow and ice properties), and representing the relationship between stability and turbulent heat fluxes.

54 ENVIRONMENTAL SCIENCES↗

Model Evaluation Report for Corrective Action Unit 97: Yucca Flat/Climax Mine, Nevada National Security Site, Nye County, Nevada (Revision 1)

This Model Evaluation Report (MER) has been prepared for Corrective Action Unit (CAU) 97, Yucca Flat/Climax Mine (YF/CM), Nevada National Security Site (NNSS), Nevada. The YF/CM CAU is located in the northeastern portion of the NNSS and comprises 720 corrective action sites. A total of 747 underground nuclear detonations took place within this CAU between 1957 and 1992 and resulted in the release of radionuclides (RNs) in the subsurface in the vicinity of the detonation cavities.

54 ENVIRONMENTAL SCIENCES↗

R&D for computational cognitive and social models : foundations for model evaluation through verification and validation (final LDRD report).

Sandia National Laboratories is investing in projects that aim to develop computational modeling and simulation applications that explore human cognitive and social phenomena. While some of these modeling and simulation projects are explicitly research oriented, others are intended to support or provide insight for people involved in high consequence decision-making. This raises the issue of how to evaluate computational modeling and simulation applications in both research and applied settings where human behavior is the focus of the model: when is a simulation 'good enough' for the goals its designers want to achieve? In this report, we discuss two years' worth of review and assessment of the ASC program's approach to computational model verification and validation, uncertainty quantification, and decision making. We present a framework that extends the principles of the ASC approach into the area of computational social and cognitive modeling and simulation. In doing so, we argue that the potential for evaluation is a function of how the modeling and simulation software will be used in a particular setting. In making this argument, we move from strict, engineering and physics oriented approaches to V&V to a broader project of model evaluation, which asserts that the systematic, rigorous, and transparent accumulation of evidence about a model's performance under conditions of uncertainty is a reasonable and necessary goal for model evaluation, regardless of discipline. How to achieve the accumulation of evidence in areas outside physics and engineering is a significant research challenge, but one that requires addressing as modeling and simulation tools move out of research laboratories and into the hands of decision makers. This report provides an assessment of our thinking on ASC Verification and Validation, and argues for further extending V&V research in the physical and engineering sciences toward a broader program of model evaluation in situations of high consequence decision-making.

97 MATHEMATICS AND COMPUTING↗

p-values for model evaluation

Deciding whether a model provides a good description of data is often based on a goodness-of-fit criterion summarized by a p-value. Although there is considerable confusion concerning the meaning of p-values, leading to their misuse, they are nevertheless of practical importance in common data analysis tasks. We motivate their application using a Bayesian argumentation. We then describe commonly and less commonly known discrepancy variables and how they are used to define p-values. The distribution of these are then extracted for examples modeled on typical data analysis tasks, and comments on their usefulness for determining goodness-of-fit are given.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

The Single Column Atmosphere Model Version 6 (SCAM6): Not a Scam but a Tool for Model Evaluation and Development

The Single Column Atmosphere Model (SCAM) is a single column model version of the Community Atmosphere Model (CAM). Here we describe the functionality and features of SCAM6, available as part of CAM6 in the Community Earth System Model, version 2 (CESM2). SCAM6 features a wide selection of standard cases, as well as the ability to easily configure a case specified by the user based on a particular point in a CAM 3-D simulation. This work illustrates how SCAM6 reproduces CAM6 results for physical parameterizations, mostly of moisture and clouds. We demonstrate how SCAM6 can be used for model development through different physics selections, as well as with parameter sweep experiments to highlight the sensitivity of cloud properties to the specification of the vapor deposition process in the cloud microphysics. Furthermore, we use SCAM6 to illustrate the sensitivity of CAM6 cloud radiative properties and precipitation to variable drop number (cloud microphysics properties). Finally, we illustrate how SCAM6 can be used to explore critical emergent processes such as cloud feedbacks and show that CAM6 cloud responses to surface warming in stratus and stratocumulus regimes are similar to those in CAM5. CAM6 has a larger response in the shallow cumulus regime than CAM5. CAM6 cloud feedbacks in the shallow cumulus regime are sensitive to turbulence parameters. SCAM6 is thus a valuable tool for model development, evaluation, and scientific analy sis and an important part of the model hierarchy in Community Earth System Model, version 2.

58 GEOSCIENCES↗

PCMDI - An Earth System Model Evaluation Project Lawrence Livermore National Laboratory Annual Report

This annual report describes the activities of PCMDI – An Earth System Model Evaluation Project for the period November 1, 2019 – June 30, 2020. PCMDI is a project at Lawrence Livermore National Laboratory that is funded by the Climate and Environmental Sciences Division (CESD) of the Department of Energy. PCMDI is funded by two efforts within CESD: The Regional and Global Model Analysis program area (Dr. Renu Joseph, program manager) and the Atmospheric Systems Research program (Dr. Shaima Nasiri and Jeffrey Stehr, program managers). As instructed by Drs. Joseph, Nasiri, and Stehr, we provide an annual report addressing sections 4 (Progress and Plans) and 5 (Staffing and Budget Summary) of the SFA Program Management and Performance Reporting Criteria.

54 ENVIRONMENTAL SCIENCES↗

On the Forward Modeling of Radar Doppler Spectrum Width From LES: Implications for Model Evaluation

Large-eddy simulations of an observed single-layer Arctic mixed-phase cloud are analyzed to study the value of forward modeling of profiling millimeter wave cloud radar Doppler spectral width for model evaluation. Individual broadening terms and their uncertainties are quantified for the observed spectral width and compared to modeled broadening terms. Modeled turbulent broadening is narrower than the observed values when the turbulent kinetic energy dissipation rate from the subgrid scale model is used in the forward model. The total dissipation rates, estimated with the subgrid scale dissipation rates and the numerical dissipation rates, agree much better with both the retrieved dissipation rates and those inferred from the power spectra of the simulated vertical air velocity. The comparison of the microphysical broadening provides another evaluative measure of the ice properties in the simulation. To accurately retrieve dissipation rates as well as each broadening term from the observations, we suggest a few modifications to previously presented techniques. First, we show that the inertial subrange spectrum filtered with the radar sampling volume is a better underlying model than the unfiltered -5/3 law for the retrieval of the dissipation rate from the power spectra of the mean Doppler velocity. Second, we demonstrate that it is important to filter out turbulence and remove the layer-mean reflectivity-weighted mean fall speed from the observed mean Doppler velocity to avoid overestimation of shear broadening. Finally, we provide a method to quantify the uncertainty in the retrieved dissipation rates, which eventually propagates to the uncertainty in the microphysical broadening.

54 ENVIRONMENTAL SCIENCES↗