Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Human Thermal Model Evaluation Using the JSC Human Thermal Database

Human thermal modeling has considerable long term utility to human space flight. Such models provide a tool to predict crew survivability in support of vehicle design and to evaluate crew response in untested space environments. It is to the benefit of any such model not only to collect relevant experimental data to correlate it against, but also to maintain an experimental standard or benchmark for future development in a readily and rapidly searchable and software accessible format. The Human thermal database project is intended to do just so; to collect relevant data from literature and experimentation and to store the data in a database structure for immediate and future use as a benchmark to judge human thermal models against, in identifying model strengths and weakness, to support model development and improve correlation, and to statistically quantify a model s predictive quality. The human thermal database developed at the Johnson Space Center (JSC) is intended to evaluate a set of widely used human thermal models. This set includes the Wissler human thermal model, a model that has been widely used to predict the human thermoregulatory response to a variety of cold and hot environments. These models are statistically compared to the current database, which contains experiments of human subjects primarily in air from a literature survey ranging between 1953 and 2004 and from a suited experiment recently performed by the authors, for a quantitative study of relative strength and predictive quality of the models.

Bue, Grant↗

Predicting the mineral composition of dust aerosols – Part 2: Model evaluation and identification of key processes with observations

A global compilation of nearly sixty measurement studies is used to evaluate two methods of simulating the mineral composition of dust aerosols in an Earth system model. Both methods are based upon a Mean Mineralogical Table (MMT) that relates the soil mineral fractions to a global atlas of arid soil type. The Soil Mineral Fraction (SMF) method assumes that the aerosol mineral fractions match the fractions of the soil. The MMT is based upon soil measurements after wet sieving, a process that destroys aggregates of soil particles that would have been emitted from the original, undisturbed soil. The second method approximately reconstructs the emitted aggregates. This model is referred to as the Aerosol Mineral Fraction (AMF) method because the mineral fractions of the aerosols differ from those of the wet-sieved parent soil, partly due to reaggregation. The AMF method remedies some of the deficiencies of the SMF method in comparison to observations. Only the AMF method exhibits phyllosilicate mass at silt sizes, where they are abundant according to observations. In addition, the AMF quartz fraction of silt particles is in better agreement with measured values, in contrast to the overestimated SMF fraction. Measurements at distinct clay and silt particle sizes are shown to be more useful for evaluation of the models, in contrast to the sum over all particles sizes that is susceptible to compensating errors, as illustrated by the SMF experiment. Model errors suggest that allocation of the emitted silt fraction of each mineral into the corresponding transported size categories is an important remaining source of uncertainty. Evaluation of both models and the MMT is hindered by the limited number of size-resolved measurements of mineral content that sparsely sample aerosols from the major dust sources. In conclusion, the importance of climate processes dependent upon aerosol mineral composition shows the need for global and routine mineral measurements.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Predicting the Mineral Composition of Dust Aerosols: Model Evaluation and Identification of Key Processes with Observations - Part 2

A global compilation of nearly sixty measurement studies is used to evaluate two methods of simulating the mineral composition of dust aerosols in an Earth system model. Both methods are based upon a Mean Mineralogical Table (MMT) that relates the soil mineral fractions to a global atlas of arid soil type. The Soil Mineral Fraction (SMF) method assumes that the aerosol mineral fractions match the fractions of the soil. The MMT is based upon soil measurements after wet sieving, a process that destroys aggregates of soil particles that would have been emitted from the original, undisturbed soil. The second method approximately reconstructs the emitted aggregates. This model is referred to as the Aerosol Mineral Fraction (AMF) method because the mineral fractions of the aerosols differ from those of the wet-sieved parent soil, partly due to reaggregation. The AMF method remedies some of the deficiencies of the SMF method in comparison to observations. Only the AMF method exhibits phyllosilicate mass at silt sizes, where they are abundant according to observations. In addition, the AMF quartz fraction of silt particles is in better agreement with measured values, in contrast to the overestimated SMF fraction. Measurements at distinct clay and silt particle sizes are shown to be more useful for evaluation of the models, in contrast to the sum over all particles sizes that is susceptible to compensating errors, as illustrated by the SMF experiment. Model errors suggest that allocation of the emitted silt fraction of each mineral into the corresponding transported size categories is an important remaining source of uncertainty. Evaluation of both models and the MMT is hindered by the limited number of size-resolved measurements of mineral content that sparsely sample aerosols from the major dust sources. The importance of climate processes dependent upon aerosol mineral composition shows the need for global and routine mineral measurements.

aerosols↗

Scenario Setup and Forcing Data for Impact Model Evaluation and Impact Attribution Within the Third Round of the Inter-Sectoral Model Intercomparison Project (ISIMIP3a)

This paper describes the rationale and the protocol of the first component of the third simulation round of the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP3a, http://www.isimip.org, last access: 2 November 2023) and the associated set of climate-related and direct human forcing data (CRF and DHF, respectively). The observation-based climate-related forcings for the first time include high-resolution observational climate forcings derived by orographic downscaling, monthly to hourly coastal water levels, and wind fields associated with historical tropical cyclones. The DHFs include land use patterns, population densities, information about water and agricultural management, and fishing intensities. The ISIMIP3a impact model simulations driven by these observation-based climate-related and direct human forcings are designed to test to what degree the impact models can explain observed changes in natural and human systems. In a second set of ISIMIP3a experiments the participating impact models are forced by the same DHFs but a counterfactual set of atmospheric forcings and coastal water levels where observed trends have been removed. These experiments are designed to allow for the attribution of observed changes in natural, human, and managed systems to climate change, rising CH 4 and CO 2 concentrations, and sea level rise according to the definition of the Working Group II contribution to the IPCC AR6.

Inter Sectoral Impact Model Intercomparison Projec↗

New Measurements to Resolve Discrepancies in Evaluated Model Parameters of 181 Ta

To resolve discrepancies in evaluated cross sections among major nuclear data libraries, energy-differential neutron transmission and radiative capture yield of 181 Ta were measured from 0.15 to 100 keV using multiple sample thicknesses. The new measurements provide resolution such that the resolved resonance region (RRR) can be evaluated up to at least 2.5 keV and the unresolved resonance region can be evaluated up to at least 100 keV. The transmission and capture yield measurements were modeled using resonance parameters from three major libraries to assess the predictive capability of each. It was found that JENDL-5.0 performed best in the RRR. Because of the poor performance of the U.S. ENDF/B evaluation, it is recommended that ENDF/B be reevaluated for 181 Ta.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Model Evaluation of Short-Lived Climate Forcers for the Arctic Monitoring and Assessment Programme: A Multi-Species, Multi-Model Study

While carbon dioxide is the main cause for global warming, modeling short-lived climate forcers (SLCFs) such as methane, ozone, and particles in the Arctic allows us to simulate near-term climate and health impacts for a sensitive, pristine region that is warming at 3 times the global rate. Atmospheric modeling is critical for understanding the long-range transport of pollutants to the Arctic, as well as the abundance and distribution of SLCFs throughout the Arctic atmosphere. Modeling is also used as a tool to determine SLCF impacts on climate and health in the present and in future emissions scenarios. In this study, we evaluate 18 state-of-the-art atmospheric and Earth system models by assessing their representation of Arctic and Northern Hemisphere atmospheric SLCF distributions, considering a wide range of different chemical species (methane, tropospheric ozone and its precursors, black carbon, sulfate, organic aerosol, and particulate matter) and multiple observational datasets. Model simulations over 4 years (2008-2009 and 2014-2015) conducted for the 2022 Arctic Monitoring and Assessment Programme (AMAP) SLCF assessment report are thoroughly evaluated against satellite, ground, ship, and aircraft-based observations. The annual means, seasonal cycles, and 3-D distributions of SLCFs were evaluated using several metrics, such as absolute and percent model biases and correlation coefficients. The results show a large range in model performance, with no one particular model or model type performing well for all regions and all SLCF species. The multi-model mean (mmm) was able to represent the general features of SLCFs in the Arctic and had the best overall performance. For the SLCFs with the greatest radiative impact (CH4, 03, BC, and SO(sup 2-)(sub 4)), the mmm was within ±25 % of the measurements across the Northern Hemisphere. Therefore, we recommend a multi-model ensemble be used for simulating climate and health impacts of SLCFs. Of the SLCFs in our study, model biases were smallest for C"4 and greatest for OA. For most SLCFs, model biases skewed from positive to negative with increasing latitude. Our analysis suggests that vertical mixing, long-range transport, deposition, and wildfires remain highly uncertain processes. These processes need better representation within atmospheric models to improve their simulation of SLCFs in the Arctic environment. As model development proceeds in these areas, we highly recommend that the vertical and 3-D distribution of SLCFs be evaluated, as that information is critical to improving the uncertain processes in models.

Arctic Monitoring and Assessment Programme↗

Human Thermal Model Evaluation Using the JSC Human Thermal Database

The human thermal database developed at the Johnson Space Center (JSC) is used to evaluate a set of widely used human thermal models. This database will facilitate a more accurate evaluation of human thermoregulatory response using in a variety of situations, including those situations that might otherwise prove too dangerous for actual testing--such as extreme hot or cold splashdown conditions. This set includes the Wissler human thermal model, a model that has been widely used to predict the human thermoregulatory response to a variety of cold and hot environments. These models are statistically compared to the current database, which contains experiments of human subjects primarily in air from a literature survey ranging between 1953 and 2004 and from a suited experiment recently performed by the authors, for a quantitative study of relative strength and predictive quality of the models. Human thermal modeling has considerable long term utility to human space flight. Such models provide a tool to predict crew survivability in support of vehicle design and to evaluate crew response in untested environments. It is to the benefit of any such model not only to collect relevant experimental data to correlate it against, but also to maintain an experimental standard or benchmark for future development in a readily and rapidly searchable and software accessible format. The Human thermal database project is intended to do just so; to collect relevant data from literature and experimentation and to store the data in a database structure for immediate and future use as a benchmark to judge human thermal models against, in identifying model strengths and weakness, to support model development and improve correlation, and to statistically quantify a model s predictive quality.

Cognata, T.↗

An evaluative model of system performance in manned teleoperational systems

Manned teleoperational systems are used in aerospace operations in which humans must interact with machines remotely. Manual guidance of remotely piloted vehicles, controling a wind tunnel, carrying out a scientific procedure remotely are examples of teleoperations. A four input parameter throughput (Tp) model is presented which can be used to evaluate complex, manned, teleoperations-based systems and make critical comparisons among candidate control systems. The first two parameters of this model deal with nominal (A) and off-nominal (B) predicted events while the last two focus on measured events of two types, human performance (C) and system performance (D). Digital simulations showed that the expression A(1-B)/C+D) produced the greatest homogeneity of variance and distribution symmetry. Results from a recently completed manned life science telescience experiment will be used to further validate the model. Complex, interacting teleoperational systems may be systematically evaluated using this expression much like a computer benchmark is used.

Haines, Richard F.↗

User's appraisal of yield model evaluation criteria

The five major potential USDA users of AgRISTAR crop yield forecast models rated the Yield Model Development (YMD) project Test and Evaluation Criteria by the importance placed on them. These users were agreed that the "TIMELINES" and "RELIABILITY" of the forecast yields would be of major importance in determining if a proposed yield model was worthy of adoption. Although there was considerable difference of opinion as to the relative importance of the other criteria, "COST", "OBJECTIVITY", "ADEQUACY", AND "MEASURES OF ACCURACY" generally were felt to be more important that "SIMPLICITY" and "CONSISTENCY WITH SCIENTIFIC KNOWLEDGE". However, some of the comments which accompanied the ratings did indicate that several of the definitions and descriptions of the criteria were confusing.

Warren, F. B.↗

Pumped Storage Hydro Model Evaluation Report Task 1: Tool Assessment

Pumped storage hydro (PSH) projects support various aspects of power system operations. However, determining the value of PSH projects and the many services and contributions to the system they provide can be a challenge. The PSH techno-economic studies being carried out by a 5-lab consortium are defining extensive modeling approaches for evaluating the economic benefits of PSH projects. To enhance the impact of the valuation approach and follow-on techno-economic studies, Pacific Northwest National Laboratory (PNNL) will be developing a set of online tools that industry could adopt and use to advance PSH projects in the US. The problem is that the research team is not entirely clear on the features of the tools that would offer the most significant impact to industry. Thus, year one of this effort is designed to first define the needs of industry before developing the PSH valuation tool. The first step in defining industry needs is to evaluate existing tool packages, including those created by DOE and those by industry groups, to both measure usage levels and define tool attributes that maximize industry impact. The second step in this process will be to hold a workshop designed to elicit direct industry input. Given the breadth and nature of the task, which is to define ideal model attributes, the research team did not constrain the assessment to storage valuation tools. Several tools were assessed, and their characteristics compared against a list of “ideal” attributes that included ease of use, generalizability of results, and analytical flexibility. Information regarding usage and impact were then compared to specific model attributes to define which attributes appear to drive success. The following key lessons and implications can be drawn from the analysis: Models with high ease of use were often the most adopted by industry, though they did not necessarily have high publication reference records, meaning a published article made direct reference to the tool. For example, REScheck TM and COMcheck TM , two tools which share a site, have seen a high amount of web traffic and usage. Similarly, Storage Value Estimate Tool (StorageVET®), an energy storage valuation model, has a large number of organizations using it despite a low publication reference record. A common feature between these models is that they are web-hosted, theoretically making them more easily accessible with graphic user interfaces (GUIs) that are more navigable; Models with higher usage statistics were typically free and/or open source. While it may be unsurprising that this attribute could be considered a draw for users, there is the potential for overestimating the tool's value based on popularity due to the fact that some users may have used the model simply because it was free and not because it is the best model; Models that offer more in-depth analysis and a wide coverage of applications (e.g. VOLTTRON™, GridLAB-D™, and Integrated Valuation of Ecosystem Services and Tradeoffs (InVest)) have shown a high number of publication references as well as showing a high number of downloads in the case of GridLAB-D™. It should be noted that the figure below does not include the model with the highest number of publication references (EnergyPlus™) as it has been removed to show clearer distinction between the remaining tools; The most popular model within the evaluation list was EnergyPlus™. This tool has the highest number of publication references (2,034) of all models considered and contains both a high ease of use level as well as broad application flexibility. Its high usage may be correlated, however, with the presumed wider audience conducting energy consumption modeling compared to the audiences for some of the other tools in this list; Models that were low in both general usage and publication reference generally did not exhibit characteristics considered ideal by the research team. Some of these models offered a narrower range of analytical applications or lower generalizability of results. This would automatically restrict the pool of potential users to those who fall within a narrow range of specific technologies or applications. This report also explores two model structures for the PSH tool: decision tree-based, which would guide the user through the valuation methods defined in the PSH techno-economic studies currently being carried out by DOE, and an all-inclusive comprehensive model. The strengths of each were explored and a hybrid approach was defined that would embed a price taker model into the decision tree approach. Such an approach would provide a comprehensive approach for modeling small-scale PSH while recognizing the challenges associated with implementing a price-maker approach in an all-inclusive model. Insight gained in this analysis of other models can aid in narrowing down the list of effective and sought-after attributes, and in defining model structure. Next steps toward developing a model for storage valuation will be to gain direct industry input through workshops.

13 HYDRO ENERGY↗

Evaluating Model Parameterizations of Submicron Aerosol Scattering and Absorption with in situ Data from ARCTAS 2008

Accurate modeling of the scattering and absorption of ultraviolet and visible radiation by aerosols is essential for accurate simulations of atmospheric chemistry and climate. Closure studies using in situ measurements of aerosol scattering and absorption can be used to evaluate and improve models of aerosol optical properties without interference from model errors in aerosol emissions, transport, chemistry, or deposition rates. Here we evaluate the ability of four externally mixed, fixed size distribution parameterizations used in global models to simulate submicron aerosol scattering and absorption at three wavelengths using in situ data gathered during the 2008 Arctic Research of the Composition of the Troposphere from Aircraft and Satellites (ARCTAS) campaign. The four models are the NASA Global Modeling Initiative (GMI) Combo model, GEOS-Chem v9- 02, the baseline configuration of a version of GEOS-Chem with online radiative transfer calculations (called GC-RT), and the Optical Properties of Aerosol and Clouds (OPAC v3.1) package. We also use the ARCTAS data to perform the first evaluation of the ability of the Aerosol Simulation Program (ASP v2.1) to simulate submicron aerosol scattering and absorption when in situ data on the aerosol size distribution are used, and examine the impact of different mixing rules for black carbon (BC) on the results. We find that the GMI model tends to overestimate submicron scattering and absorption at shorter wavelengths by 10-23 percent, and that GMI has smaller absolute mean biases for submicron absorption than OPAC v3.1, GEOS-Chem v9-02, or GC-RT. However, the changes to the density and refractive index of BC in GCRT improve the simulation of submicron aerosol absorption at all wavelengths relative to GEOS-Chem v9-02. Adding a variable size distribution, as in ASP v2.1, improves model performance for scattering but not for absorption, likely due to the assumption in ASP v2.1 that BC is present at a constant mass fraction throughout the aerosol size distribution. Using a core-shell mixing rule in ASP overestimates aerosol absorption, especially for the fresh biomass burning aerosol measured in ARCTAS-B, suggesting the need for modeling the time-varying mixing states of aerosols in future versions of ASP.

Alvarado, Matthew J.↗

Exploring the use of dynamic linear panel data models for evaluating energy/economy/environment models — an application for the transportation sector

This paper evaluates the RoSE transportation sector scenarios of the GCAM and REMIND models for the U.S. region by comparing these models’ projected future final energy use levels (1) to historical trends and (2) across each other. To estimate and compare the intrinsic model elasticities, we explore the use of dynamic linear panel data models. On the basis of 26 scenarios (panels) from each of the two energy system models, our analysis suggests that most intrinsic model elasticities with respect to final energy use are roughly comparable to each other and to those observed historically. The key difference is the income elasticity of final energy use, which is significantly smaller for the two energy models than that derived from the historical development. This difference and other minor inconsistencies are interpreted through key assumptions underlying both energy system models.

54 ENVIRONMENTAL SCIENCES↗

An AeroCom–AeroSat study: intercomparison of satellite AOD datasets for aerosol model evaluation

To better understand and characterize current uncertainties in the important observational constraint of climate models of aerosol optical depth (AOD), we evaluate and intercompare 14 satellite products, representing nine different retrieval algorithm families using observations from five different sensors on six different platforms. The satellite products (super-observations consisting of 1°×1° daily aggregated retrievals drawn from the years 2006, 2008 and 2010) are evaluated with AErosol RObotic NETwork (AERONET) and Maritime Aerosol Network (MAN) data. Results show that different products exhibit different regionally varying biases (both under- and overestimates) that may reach ±50 %, although a typical bias would be 15 %–25 % (depending on the product). In addition to these biases, the products exhibit random errors that can be 1.6 to 3 times as large. Most products show similar performance, although there are a few exceptions with either larger biases or larger random errors. The intercomparison of satellite products extends this analysis and provides spatial context to it. In particular, we show that aggregated satellite AOD agrees much better than the spatial coverage (often driven by cloud masks) within the 1°×1° grid cells. Up to ∼50 % of the difference between satellite AOD is attributed to cloud contamination. The diversity in AOD products shows clear spatial patterns and varies from 10 % (parts of the ocean) to 100 % (central Asia and Australia). More importantly, we show that the diversity may be used as an indication of AOD uncertainty, at least for the better performing products. This provides modellers with a global map of expected AOD uncertainty in satellite products, allows assessment of products away from AERONET sites, can provide guidance for future AERONET locations and offers suggestions for product improvements. We account for statistical and sampling noise in our analyses. Sampling noise, variations due to the evaluation of different subsets of the data, causes important changes in error metrics. The consequences of this noise term for product evaluation are discussed.

Aerosol↗

Evaluation of the cytotoxic and cellular proteome impacts of food-grade TiO2 (E171) using simulated gastrointestinal digestions and a tri-culture small intestinal epithelial model

Engineered nanomaterials (ENMs) are widely used in the food industry; however, regulations for ENMs in food are still in the early stages of development due to insufficient health data. This study investigated the cytotoxicity and changes to the proteomic profile in an in vitro small intestinal epithelium model after exposure to digested food models containing the ubiquitous engineered particulate food additive, TiO2 (E171). TiO2 at 0.75% or 1.5% (w/w) concentrations in either a fasting food model (FFM) or a standardized food model (SFM) based on American diet were digested using an in vitro oral-gastric-small intestinal simulator, and the resulting digestas were applied to a small intestinal epithelium tri-culture cellular model. Effects on cell layer integrity, cytotoxicity, and oxidative stress were assessed. In order to explore impact on cellular processes beyond basic cytotoxicity, mass spectrometrybased quantitative proteomic analyses of control and exposed tri-culture cells was performed. TiO2 in FFM, but not in SFM, produced significant, dose-dependent cytotoxicity, and at the higher dose caused significant oxidative stress, indicative of a food matrix effect. No significant perturbations of the cellular proteome were observed with TiO2 in either FFM or SFM food models. However, proteins involved in the tricarboxylic acid (TCA) cycle, gluconeogenesis, and translation were up-regulated by digestas from SFM compared to those from FFM, also indicative of food matrix effects in the case of food models alone. Interestingly, the differences in profiles between the two food models was more pronounced in the presence of TiO2 indicative of food matrix effects. Together, these results indicate that TiO2 in a fasting diet may be slightly cytotoxic, and that ingested TiO2 does not significantly alter the epithelial proteome, whereas the food matrix alone can have a dramatic effect on the proteome.

Cao, Xiaoqiong↗

Simplified model evaluation of cooling rates for glass-containing lunar compositions

The simplified model of glass formation and the development of partial crystallinity in cooled bodies has been applied to lunar compositions 10060, 15028, 15086, 15101, 15286, 15301, 15498, 15499, 60255, 65016, 77017, Apollo 15 green glass and LUNA 24 highland basalt. The critical cooling rates for glass formation predicted by the simplified model are found to be in good agreement (to within an order of magnitude) with those predicted by the exact treatment of crystallization statistics. These predicted critical cooling rates are in even better agreement (a factor of 2) with measured values of the rates required to form glasses of the materials.

Uhlmann, D. R.↗

Assessment of the ParFlow–CLM CONUS 1.0 integrated hydrologic model: evaluation of hyper-resolution water balance components across the contiguous United States

Abstract. Recent advancements in computational efficiency and Earth system modeling have awarded hydrologists with increasingly high-resolution models of terrestrial hydrology, which are paramount to understanding and predicting complex fluxes of moisture and energy. Continental-scale hydrologic simulations are, in particular, of interest to the hydrologic community for numerous societal, scientific, and operational benefits. The coupled hydrology–land surface model ParFlow–CLM configured over the continental United States (PFCONUS) has been employed in previous literature to study scale-dependent connections between water table depth, topography, recharge, and evapotranspiration, as well as to explore impacts of anthropogenic aquifer depletion to the water and energy balance. These studies have allowed for an unprecedented process-based understanding of the continental water cycle at high resolution. Here, we provide the most comprehensive evaluation of PFCONUS version 1.0 (PFCONUSv1) performance to date by comparing numerous modeled water balance components with thousands of in situ observations and several remote sensing products and using a range of statistical performance metrics for evaluation. PFCONUSv1 comparisons with these datasets are a promising indicator of model fidelity and ability to reproduce the continental-scale water balance at high resolution. Areas for improvement are identified, such as a positive streamflow bias at gauges in the eastern Great Plains, a shallow water table bias over many areas of the model domain, and low bias in seasonal total water storage amplitude, especially for the Ohio, Missouri, and Arkansas River basins. We discuss several potential sources for model bias and suggest that minimizing error in topographic processing and meteorological forcing would considerably improve model performance. Results here provide a benchmark and guidance for further PFCONUS model development, and they highlight the importance of concurrently evaluating all hydrologic components and fluxes to provide a multivariate, holistic validation of the complete modeled water balance.

O'Neill, Mary M. F. (ORCID:0000000221455165)↗