Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

Nevada National Security Site (NNSS) Environmental Remediation Using EPA's Model Evaluation Guidance to Move to Closure - 15588

The National Nuclear Security Administration Nevada Field Office (NNSA/NFO) has adapted the Environmental Protection Agency's guidance on environmental model evaluation as a framework for evaluating corrective action investigation models. Model evaluation is an important confidence building activity, which is needed to support closure of the Nevada National Security Site (NNSS) Corrective Action Units (CAUs) associated with underground nuclear testing. Through this process, site-investigation models of groundwater flow and contaminant transport are reviewed and compared to new site-specific data to determine if the results can be used to support regulatory decisions required for closure and to achieve NNSA/NFO site management objectives. The regulatory decisions include identifying and establishing CAU regulatory boundaries; identifying institutional controls, including use-restriction boundaries; and developing a long-term closure monitoring program. A successful case study from Frenchman Flat, Nevada is used to illustrate the model evaluation process. Groundwater flow and transport from two of the 10 underground nuclear tests in Frenchman Flat had the largest potential for contaminating groundwater, and were selected for evaluation. Wells were drilled, logged, hydraulically tested, sampled, and analyzed for radionuclides and the data was evaluated with respect to previous site conceptualization and numerical model results. The approach and results of this process as applied to the PIN STRIPE test and how it helped build confidence to close the Frenchman Flat CAU is described in this case study. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2011

Ten Years of Cloud Properties from MODIS: Global Statistics and Use in Climate Model Evaluation

The NASA Moderate Resolution Imaging Spectroradiometer (MODIS), launched onboard the Terra and Aqua spacecrafts, began Earth observations on February 24, 2000 and June 24,2002, respectively. Among the algorithms developed and applied to this sensor, a suite of cloud products includes cloud masking/detection, cloud-top properties (temperature, pressure), and optical properties (optical thickness, effective particle radius, water path, and thermodynamic phase). All cloud algorithms underwent numerous changes and enhancements between for the latest Collection 5 production version; this process continues with the current Collection 6 development. We will show example MODIS Collection 5 cloud climatologies derived from global spatial . and temporal aggregations provided in the archived gridded Level-3 MODIS atmosphere team product (product names MOD08 and MYD08 for MODIS Terra and Aqua, respectively). Data sets in this Level-3 product include scalar statistics as well as 1- and 2-D histograms of many cloud properties, allowing for higher order information and correlation studies. In addition to these statistics, we will show trends and statistical significance in annual and seasonal means for a variety of the MODIS cloud properties, as well as the time required for detection given assumed trends. To assist in climate model evaluation, we have developed a MODIS cloud simulator with an accompanying netCDF file containing subsetted monthly Level-3 statistical data sets that correspond to the simulator output. Correlations of cloud properties with ENSO offer the potential to evaluate model cloud sensitivity; initial results will be discussed.

Platnick, Steven E.↗
DOE OSTI2023

Incorporation of aerosol into the COSPv2 satellite lidar simulator for climate model evaluation

Abstract. Atmospheric aerosol has substantial impacts on climate, air quality and biogeochemical cycles, and its concentrations are highly variable in space and time. A key variability to evaluate within models that simulate aerosol is the vertical distribution, which influences atmospheric heating profiles and aerosol–cloud interactions, to help constrain aerosol residence time and to better represent the magnitude of simulated impacts. To ensure a consistent comparison between modeled and observed vertical distribution of aerosol, we implemented an aerosol lidar simulator within the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package version 2 (COSPv2). We assessed the attenuated total backscattered (ATB) signal and the backscatter ratios (SRs) at 532 nm in the U.S. Department of Energy's Energy Exascale Earth System Model version 1 (E3SMv1). The simulator performs the computations at the same vertical resolution as the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP), making use of aerosol optics from the E3SMv1 model as inputs and assuming that aerosol is uniformly distributed horizontally within each model grid box. The simulator applies a cloud masking and an aerosol detection threshold to obtain the ATB and SR profiles that would be observed above clouds by CALIOP with its aerosol detection capability. Our analysis shows that the aerosol distribution simulated at a seasonal timescale is generally in good agreement with observations. Over the Southern Ocean, however, the model does not produce the SR maximum as observed in the real world. Comparison between clear-sky and all-sky SRs shows little differences, indicating that the cloud screening by potentially incorrect model clouds does not affect the mean aerosol signal averaged over a season. This indicates that the differences between observed and simulated SR values are due not to sampling errors, but to deficiencies in the representation of aerosol in models. Finally, we highlight the need for future applications of lidar observations at multiple wavelengths to provide insights into aerosol properties and distribution and their representation in Earth system models.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2017

Global Gridded Crop Model Evaluation: Benchmarking, Skills, Deficiencies and Implications.

Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.

wheat↗
NASA NTRS2012

Evaluating Models of Human Performance: Safety-Critical Systems Applications

This presentation is part of panel discussion on Evaluating Models of Human Performance. The purpose of this panel is to discuss the increasing use of models in the world today and specifically focus on how to describe and evaluate models of human performance. My presentation will focus on discussions of generating distributions of performance, and the evaluation of different strategies for humans performing tasks with mixed initiative (Human-Automation) systems. I will also discuss issues with how to provide Human Performance modeling data to support decisions on acceptability and tradeoffs in the design of safety critical systems. I will conclude with challenges for the future.

Feary, Michael S.↗
DOE OSTI2015

Incorporating an advanced aerosol activation parameterization into WRF-CAM5: Model evaluation and parameterization intercomparison

Aerosol activation into cloud droplets is an important process that governs aerosol indirect effects. The advanced treatment of aerosol activation by Fountoukis and Nenes (2005) and its recent updates, collectively called the FN series, have been incorporated into a newly developed regional coupled climate-air quality model based on the Weather Research and Forecasting model with the physics package of the Community Atmosphere Model version 5 (WRF-CAM5) to simulate aerosol-cloud interactions in both resolved and convective clouds. The model is applied to East Asia for two full years of 2005 and 2010. A comprehensive model evaluation is performed for model predictions of meteorological, radiative, and cloud variables, chemical concentrations, and column mass abundances against satellite data and surface observations from air quality monitoring sites across East Asia. The model performs overall well for major meteorological variables including near-surface temperature, specific humidity, wind speed, precipitation, cloud fraction, precipitable water, downward shortwave and longwave radiation, and column mass abundances of CO, SO 2 , NO 2 , HCHO, and O 3 in terms of both magnitudes and spatial distributions. Larger biases exist in the predictions of surface concentrations of CO and NO x at all sites and SO 2 , O 3 , PM 2.5 , and PM 10 concentrations at some sites, aerosol optical depth, cloud condensation nuclei over ocean, cloud droplet number concentration (CDNC), cloud liquid and ice water path, and cloud optical thickness. Compared with the default Abdul-Razzack Ghan (2000) parameterization, simulations with the FN series produce ~107–113% higher CDNC, with half of the difference attributable to the higher aerosol activation fraction by the FN series and the remaining half due to feedbacks in subsequent cloud microphysical processes. With the higher CDNC, the FN series are more skillful in simulating cloud water path, cloud optical thickness, downward shortwave radiation, shortwave cloud forcing, and precipitation. Here, the model evaluation identifies several areas of improvements including emissions and their vertical allocation as well as model formulations such as aerosol formation, cloud droplet nucleation, and ice nucleation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Data–Driven Velocity Model Evaluation Using K–Means Clustering

In this work, we develop a data-driven clustering method to evaluate a velocity model using surface wave velocity dispersion. This is done by first computing theoretical dispersion curves for 1-D velocity profiles of all the grid locations and then splitting the resulting dispersion curves into a certain number of groups via the K-means clustering. The observed dispersion curves are also clustered following the same procedure and the velocity model is assessed by comparing the spatial patterns obtained for the observed and synthetic data sets. The method is applied to evaluate two community velocity models in southern California, CVM-S4.26 and CVM-H15.1, using phase velocity maps derived for 3–16 s Rayleigh waves. We found a good correlation in the spatial distribution of clusters between the result of CVM-S4.26 and that of the observed data, suggesting that the CVM-S4.26 fits the observed dispersion maps better than the CVM-H15.1 in terms of features extracted from the clustering analysis.

58 GEOSCIENCES↗
DOE OSTI2016

Towards improved and more routine Earth system model evaluation in CMIP

The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. More research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well established and considered an essential part of systematic evaluation but have been implemented ad hoc with little community coordination. Given the diversity and complexity of ESM analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently and consistently. We provide a perspective and viewpoint on how a more systematic, open, and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP can be achieved, and announce our intention to implement such a system for CMIP6. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop and apply evaluation tools that are based on the latest scientific knowledge and observational reference, are well suited for routine use, and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output and documentation accessible via the ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for model intercomparison projects using the same data structure and organization as the ESM output. This largely facilitates routine evaluation of the ESMs, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. And, at the same time, we encourage the community to contribute observations and reanalyses for model evaluation to the obs4MIPs and ana4MIPs archives. The intention is to produce through the ESGF a widely accepted quasi-operational evaluation framework for CMIP6 that would routinely execute a series of standardized evaluation tasks. Over time, as this capability matures, we expect to produce an increasingly systematic characterization of models which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also reveal whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. Finally, this framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

The Fire and Smoke Model Evaluation Experiment—A Plan for Integrated, Large Fire–Atmosphere Field Campaigns

The Fire and Smoke Model Evaluation Experiment (FASMEE) is designed to collect integrated observations from large wildland fires and provide evaluation datasets for new models and operational systems. Wildland fire, smoke dispersion, and atmospheric chemistry models have become more sophisticated, and next-generation operational models will require evaluation datasets that are coordinated and comprehensive for their evaluation and advancement. Integrated measurements are required, including ground-based observations of fuels and fire behavior, estimates of fire-emitted heat and emissions fluxes, and observations of near-source micrometeorology, plume properties, smoke dispersion, and atmospheric chemistry. To address these requirements the FASMEE campaign design includes a study plan to guide the suite of required measurements in forested sites representative of many prescribed burning programs in the southeastern United States and increasingly common high-intensity fires in the western United States. Here we provide an overview of the proposed experiment and recommendations for key measurements. The FASMEE study provides a template for additional large-scale experimental campaigns to advance fire science and operational fire and smoke models.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2014

Temperature and Humidity Profiles in the TqJoint Data Group of AIRS Version 6 Product for the Climate Model Evaluation

The Atmospheric Infrared Sounder (AIRS) mission is entering its 13th year of global observations of the atmospheric state, including temperature and humidity profiles, outgoing long-wave radiation, cloud properties, and trace gases. Thus AIRS data have been widely used, among other things, for short-term climate research and observational component for model evaluation. One instance is the fifth phase of the Coupled Model Intercomparison Project (CMIP5) which uses AIRS version 5 data in the climate model evaluation. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is the home of processing, archiving, and distribution services for data from the AIRS mission. The GES DISC, in collaboration with the AIRS Project, released data from the version 6 algorithm in early 2013. The new algorithm represents a significant improvement over previous versions in terms of greater stability, yield, and quality of products. The ongoing Earth System Grid for next generation climate model research project, a collaborative effort of GES DISC and NASA JPL, will bring temperature and humidity profiles from AIRS version 6. The AIRS version 6 product adds a new "TqJoint" data group, which contains data for a common set of observations across water vapor and temperature at all atmospheric levels and is suitable for climate process studies. How different may the monthly temperature and humidity profiles in "TqJoint" group be from the "Standard" group where temperature and water vapor are not always valid at the same time? This study aims to answer the question by comprehensively comparing the temperature and humidity profiles from the "TqJoint" group and the "Standard" group. The comparison includes mean differences at different levels globally and over land and ocean. We are also working on examining the sampling differences between the "TqJoint" and "Standard" group using MERRA data.

climate model evaluation↗
DOE OSTI2021

CrossCheck: Rapid, Reproducible, and Interpretable Model Evaluation

Evaluation beyond aggregate performance metrics, e.g. F1-score, is crucial to both establish an appropriate level of trust in machine learning models and identify future model improvements. In this paper we demonstrate CrossCheck, an interactive visualization tool for rapid crossmodel comparison and reproducible error analysis. We describe the tool and discuss design and implementation details. We then present three use cases (named entity recognition, reading comprehension, and clickbait detection) that show the benefits of using the tool for model evaluation. CrossCheck allows data scientists to make informed decisions to choose between multiple models, identify when the models are correct and for which examples, investigate whether the models are making the same mistakes as humans, evaluate models’ generalizability and highlight models’ limitations, strengths and weaknesses. Furthermore, CrossCheck is implemented as a Jupyter widget, which allows rapid and convenient integration into data scientists’ model development workflows.

Arendt, Dustin L.↗
DOE OSTI2023

Regime-oriented causal model evaluation of Atlantic–Pacific teleconnections in CMIP6

The climate system and its spatio-temporal changes are strongly affected by modes of long-term internal variability, like the Pacific decadal variability (PDV) and the Atlantic multidecadal variability (AMV). As they alternate between warm and cold phases, the interplay between PDV and AMV varies over decadal to multidecadal timescales. Here, we use a causal discovery method to derive fingerprints in the Atlantic–Pacific interactions and to investigate their phase-dependent changes. Dependent on the phases of PDV and AMV, different regimes with characteristic causal fingerprints are identified in reanalyses in a first step. In a second step, a regime-oriented causal model evaluation is performed to evaluate the ability of models participating in the Coupled Model Intercomparison Project Phase 6 (CMIP6) in representing the observed changing interactions between PDV, AMV and their extra-tropical teleconnections. The causal graphs obtained from reanalyses detect a direct opposite-sign response from AMV to PDV when analyzing the complete 1900–2014 period and during several defined regimes within that period, for example, when AMV is going through its negative (cold) phase. Reanalyses also demonstrate a same-sign response from PDV to AMV during the cold phase of PDV. Historical CMIP6 simulations exhibit varying skill in simulating the observed causal patterns. Generally, large-ensemble (LE) simulations showed better network similarity when PDV and AMV were out of phase compared to other regimes. Also, the two largest ensembles (in terms of number of members) were found to contain realizations with similar causal fingerprints to observations. For most regimes, these same models showed higher network similarity when compared to each other. This work shows how causal discovery on LEs complements the available diagnostics and statistical metrics of climate variability to provide a powerful tool for climate model evaluation.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1999

Human Modeling Evaluations in Microgravity Workstation and Restraint Development

The International Space Station (ISS) will provide long-term missions which will enable the astronauts to live and work, as well as, conduct research in a microgravity environment. The dominant factor in space affecting the crew is "weightlessness" which creates a challenge for establishing workstation microgravity design requirements. The crewmembers will work at various workstations such as Human Research Facility (HRF), Microgravity Sciences Glovebox (MSG) and Life Sciences Glovebox (LSG). Since the crew will spend considerable amount of time at these workstations, it is critical that ergonomic design requirements are integral part of design and development effort. In order to achieve this goal, the Space Human Factors Laboratory in the Johnson Space Center Flight Crew Support Division has been tasked to conduct integrated evaluations of workstations and associated crew restraints. Thus, a two-phase approach was used: 1) ground and microgravity evaluations of the physical dimensions and layout of the workstation components, and 2) human modeling analyses of the user interface. Computer-based human modeling evaluations were an important part of the approach throughout the design and development process. Human modeling during the conceptual design phase included crew reach and accessibility of individual equipment, as well as, crew restraint needs. During later design phases, human modeling has been used in conjunction with ground reviews and microgravity evaluations of the mock-ups in order to verify the human factors requirements. (Specific examples will be discussed.) This two-phase approach was the most efficient method to determine ergonomic design characteristics for workstations and restraints. The real-time evaluations provided a hands-on implementation in a microgravity environment. On the other hand, only a limited number of participants could be tested. The human modeling evaluations provided a more detailed analysis of the setup. The issues identified during the real-time testing were investigated in the human modeling analyses. In some cases, the opposite was true where preliminary human modeling analyses provided the design engineers with critical issues that needed to be addressed further. This extensive approach provided an effective means to fully address ergonomic design considerations and accurately identify critical issues.

Whitmore, Mihriban↗
NASA NTRS1997

Comparative Model Evaluation Studies of Biogenic Trace Gas Fluxes in Tropical Forests

Simulation modeling can play a number of important roles in large-scale ecosystem studies, including synthesis of patterns and changes in carbon and nutrient cycling dynamics, scaling up to regional estimates, and formulation of testable hypotheses for process studies. Recent comparative studies have shown that ecosystem models of soil trace gas exchange with the atmosphere are evolving into several distinct simulation approaches. Different levels of detail exist among process models in the treatment of physical controls on ecosystem nutrient fluxes and organic substrate transformations leading to gas emissions. These differences are is in part from distinct objectives of scaling and extrapolation. Parameter requirements for initialization scalings, boundary conditions, and time-series driven therefore vary among ecosystem simulation models, such that the design of field experiments for integration with modeling should consider a consolidated series of measurements that will satisfy most of the various model requirements. For example, variables that provide information on soil moisture holding capacity, moisture retention characteristics, potential evapotranspiration and drainage rates, and rooting depth appear to be of the first order in model evaluation trials for tropical moist forest ecosystems. The amount and nutrient content of labile organic matter in the soil, based on accurate plant production estimates, are also key parameters that determine emission model response. Based on comparative model results, it is possible to construct a preliminary evaluation matrix along categories of key diagnostic parameters and temporal domains. Nevertheless, as large-scale studied are planned, it is notable that few existing models age designed to simulate transient states of ecosystem change, a feature which will be essential for assessment of anthropogenic disturbance on regional gas budgets, and effects of long-term climate variability on biosphere-atmosphere exchange.

Potter, C. S.↗
DOE OSTI2014

Model Evaluation Report for Corrective Action Unit 98: Frenchman Flat, Nevada National Security Site, Nye County, Nevada, Revision 1

Model evaluation focused solely on the PIN STRIPE and MILK SHAKE underground nuclear tests’ contaminant boundaries (CBs) because they had the largest extent, uncertainty, and potential consequences. The CAMBRIC radionuclide migration experiment also had a relatively large CB, but because it was constrained by transport data (notably Well UE-5n), there was little uncertainty, and radioactive decay reduced concentrations before much migration could occur. Each evaluation target and the associated data-collection activity were assessed in turn to determine whether the new data support, or demonstrate conservatism of, the CB forecasts. The modeling team—in this case, the same team that developed the Frenchman Flat geologic, source term, and groundwater flow and transport models—analyzed the new data and presented the results to a PER committee. Existing site understanding and its representation in numerical groundwater flow and transport models was evaluated in light of the new data and the ability to proceed to the CR stage of long-term monitoring and institutional control.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1974

Solid rocket booster performance evaluation model. Volume 3: Sample case

The solid rocket booster performance evaluation model (SRB-11) is used to predict internal ballistics in a sample motor. This motor contains a five segmented grain. The first segment has a 14 pointed star configuration with a web which wraps partially around the forward dome. The other segments are circular in cross-section and are tapered along the interior burning surface. Two of the segments are inhibited on the forward face. The nozzle is not assumed to be submerged. The performance prediction is broken into two simulation parts: the delivered end item specific impulse and the propellant properties which are required as inputs for the internal ballistics module are determined; and the internal ballistics for the entire burn duration of the motor are simulated.

Source record↗