Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Mid-Atlantic US observations of radiocarbon in CO 2 : fossil and biogenic source partitioning and model evaluation

Accurately quantifying regional anthropogenic CO 2 fluxes is fundamental to improving our understanding of the carbon cycle and for creating effective carbon mitigation policies, and the radiocarbon to total carbon ratio in atmospheric CO 2 (Δ 14 CO 2 ) is a robust tracer of fossil fuel CO 2 that can discriminate between biogenic and fossil fuel CO 2 sources. NASA's Atmospheric Carbon and Transport-America (ACT-America) airborne mission between 2016 and 2019 aimed to improve the accuracy of regional greenhouse gas flux estimates, through refining our understanding and characterization of fluxes and flux uncertainties in models. Δ 14 CO 2 observations from 26 flights are presented for examining seasonal CO 2 source partitioning in the Mid-Atlantic USA. Observed variability in boundary layer CO 2 at timescales ranging from intra-day to seasonal was largely driven by biogenic CO 2 (CO 2bio ) variability that ranged from −19.7 ppm in summer to 16.2 ppm in fall, while fossil fuel CO 2 (CO 2ff ) variability remained at 3.3±2.0 ppm. Carbonyl sulfide uptake was well-correlated with CO 2bio uptake, and examining this relationship, as well as that between CO 2 and CO 2bio variability reinforces the seasonal extent of gross primary productivity response throughout ACT-America. We use airborne Δ 14 CO 2 flask sampling alongside in situ carbon monoxide measurements to calculate high-frequency CO 2ff and evaluate the magnitude and diurnal variability of modeled CO 2ff , deducing likely transport errors in an example flight. Although ACT-America CO 2ff signals were attenuated due to the broad source regions sampled, results illustrate the value of Δ 14 CO 2 sampling and observation-based methodologies for regional CO 2 flux attribution, evaluation and improvement of modeled CO 2 .

Baier, Bianca C. [National Oceanic and Atmospheric↗

Evaluating model accuracy for model-based reasoning

Described here is an approach to automatically assessing the accuracy of various components of a model. In this approach, actual data from the operation of a target system is used to drive statistical measures to evaluate the prediction accuracy of various portions of the model. We describe how these statistical measures of model accuracy can be used in model-based reasoning for monitoring and design. We then describe the application of these techniques to the monitoring and design of the water recovery system of the Environmental Control and Life Support System (ECLSS) of Space Station Freedom.

Chien, Steve↗

Carbon dioxide pipeline network transportation cost model: evaluating economic and geographic factors for efficient carbon capture, storage, and utilization

This study presents a comprehensive pipeline network modeling framework to estimate the CO 2 delivery cost for CO 2 utilization and geologic CO 2 storage across the United States. We developed a Python-based CO 2 pipeline transportation cost model leveraging Argonne National Laboratory’s pipeline engineering expertise and detailed natural gas transmission pipeline cost data across U.S. regions. Using existing road corridors as practical routing guides, the model designs pipeline networks that aggregate CO 2 from one or multiple sources and deliver it to selected destinations. It then minimizes the total transportation cost by optimizing pipeline diameters and incorporating booster pumps. A key contribution is the incorporation of up-to-date, region-specific cost factors with itemized components for materials, labor, miscellaneous construction expenses, and right-of-way acquisition. Results emphasize that regional variation and economies of scale associated with CO 2 pipeline costs are significant and should be explicitly accounted for in screening and planning studies. By combining realistic routing constraints with regionalized cost inputs, the model provides transparent design methodology and location-specific insights into source–destination delivery costs, including the effects of routing complexity along existing road networks. We demonstrate the model with two illustrative case studies – one for CO 2 storage and one for CO 2 utilization – in which the model designs pipeline networks spanning hundreds of miles across the states, collecting CO 2 from multiple sources and delivering it to designated endpoints while minimizing levelized cost of delivery via diameter and compression optimization. The model offers a practical, scalable approach for alternative design option screening and early-stage CO 2 transportation planning.

CCS↗

Atmospheric Composition Forecast Model Evaluation Using Ozone Measurements Collected by the Langley Mobile Ozone Lidar

The Langley Mobile Ozone Lidar (LMOL) is a mobile ground based lidar system based at NASA Langley in Hampton, Virginia. Between 2022 and 2024, LMOL collected over 2500 hours of ozone measurements for a range of different atmospheric conditions, including calm days, stratospheric intrusions, surface frontal passages, and long-range transported wildfire smoke plumes. Here, the data is used to evaluate the forecast accuracy of NASA’s Global GEOS Composition Forecasting (GEOS-CF) model. GEOS-CF makes daily three-dimensional forecasts of trace gases and aerosol species. Overall, for calm periods, the forecast model predicts lower tropospheric ozone at NASA Langley with reasonable accuracy (within 20%). The model best predicts the timing and extent of stratospheric intrusions but often vary in the magnitude of the ozone mixing ratio. Among the other types of atmospheric conditions, there is more variability in the model forecasts. Based on this analysis, model forecasts are utilized to determine future data acquisition opportunities with the goal of providing feedback to the modeling teams, thereby enabling them to better understand the model biases and improve the model forecasts of ozone during these different atmospheric conditions.

Daniel B Phoenix↗

A novel network-based approach to determining measurement representation error for model evaluation of aerosol microphysical properties

Atmospheric aerosol size and abundance influence radiative effects and climate change. To date, efforts to constrain global climate models’ radiative forcing with in situ aerosol observations have been hamstrung by uncertainty. One source of error, the regional “representation error,” arises when accurate but sparse single-point measurements of atmospheric aerosol distributions are compared with a model value, assuming that the single-point measurement is representative of the model domain. The Portable Optical Particle Spectrometer network in the Southern Great Plains (POPSnet-SGP) campaign has demonstrated that a network of nearly autonomous aerosol instruments operating at ambient temperature and relative humidity (with low measurement error) may be used to quantify measurement representation error and investigate the factors introducing heterogeneity in aerosol distributions across a rural, continental background region. Measurements were made using Portable Optical Particle Spectrometer (POPS) instruments at several sites for five months across the Department of Energy’s Aerosol Radiation Measurement Southern Great Plains (ARM-SGP) User Facility in the central USA. Measurement representation error decreased with longer averaging periods (20-40 % between 1 sec and 1 day), varied between sites by 10 – 20 % for aerosol concentration 140 – 2500 nm in diameter (N_140), and was higher for aerosols > 400 nm in diameter (N_400). Our measurements also show the influence of local meteorology on aerosol surface area (A_140) and size distributions: A_140 is positively correlated with wind speed and relative humidity, negatively correlated with precipitation, and lower given westerly winds. Based on this study, we conclude that the POPSnet approach provides considerably more insight into the spatial variability in the aerosol population that can be used to constrain climate models than would be available from similar networks of PM 2.5 monitors.

54 ENVIRONMENTAL SCIENCES↗

Ice processes in Antarctica: identification via multi-wavelength active and passive measurements and model evaluation (Final Report)

This project is based on new multi-frequency radar observations collected during the AWARE field campaign in 2016 at McMurdo Station by the ARM Mobile facility. Three scientific goals have been achieved. We have developed ice and mixed-phase cloud microphysics retrievals from multi-wavelength radar observations and ancillary observations, generally applicable at ARM sites. These technique generally better characterize the size of the ice crystals, their equivalent water content and the degree of riming better than using single wavelength techniques. We have characterized ice microphysical properties and ice processes via multi-frequency signatures. We have identified a unique mode in the McMurdo dataset appearing at quite cold temperature that we have attributed to riming processes. Triple frequency radar measurements unfortunately were available only for a short period but our work lays the foundation for building climatological and testing their replicability in models. Simultaneously case studies from the AWARE campaign have been exploited to evaluate and improve the NASA GISS ModelE3 climate model, including selection of case studies for combined goals of (i) running side-by-side using large-eddy simulation (LES) and ModelE3 in single-column model (SCM) mode and (ii) fingerprinting cloud processes in multi-wavelength radar observations. The different components of this work have contributed to 15 published papers with two currently under review.

54 ENVIRONMENTAL SCIENCES↗

Evaluating Model Robustness for Defect Identification and Classification in a Composite Aerostructure Material

Aircraft structures are required to have a high level of quality to satisfy their need for light weight, efficient flight, and withstanding high loads over their lifespan. These aerostructures are typically made from a composite material due to their good tensile strength and resistance to compression. To ensure their structural integrity, the composite material requires inspection for common flaws such as porosity, delaminations, voids, foreign object debris, and other defects. Ultrasonic testing (UT) is a popular non-destructive inspection (NDI) technique used for effectively evaluating the composite material. Current inspection methods rely heavily on human experience and are extremely time consuming. Therefore, there is a need for the development of techniques to reduce the manual inspection time. This work compares the performance of different deep learning-based methods in the identification and classification of defects. Deep learning has shown great promise in numerous fields, and we show its effectiveness in the evaluation of the composite aerostructure material. The methods developed here are both highly reliable with a top recall value of 98.64% as well as extremely efficient requiring an average of 4 s during the inferencing stage to evaluate new composites. Lastly, we investigate model robustness to concept drift by measuring its performance over time.

36 MATERIALS SCIENCE↗

Pan-spectral observing system simulation experiments of shortwave reflectance and long-wave radiance for climate model evaluation

Abstract. Top-of-atmosphere (TOA) spectrally resolved shortwave reflectances and long-wave radiances describe the response of the Earth's surface and atmosphere to feedback processes and human-induced forcings. In order to evaluate proposed long-duration spectral measurements, we have projected 21st Century changes from the Community Climate System Model (CCSM3.0) conducted for the Intergovernmental Panel on Climate Change (IPCC) A2 Emissions Scenario onto shortwave reflectance spectra from 300 to 2500 nm and long-wave radiance spectra from 2000 to 200 cm –1 at 8 nm and 1 cm –1 resolution, respectively. The radiative transfer calculations have been rigorously validated against published standards and produce complementary signals describing the climate system forcings and feedbacks. Additional demonstration experiments were performed with the Model for Interdisciplinary Research on Climate (MIROC5) and Hadley Centre Global Environment Model version 2 Earth System (HadGEM2-ES) models for the Representative Concentration Pathway 8.5 (RCP8.5) scenario. The calculations contain readily distinguishable signatures of low clouds, snow/ice, aerosols, temperature gradients, and water vapour distributions. The goal of this effort is to understand both how climate change alters reflected solar and emitted infrared spectra of the Earth and determine whether spectral measurements enhance our detection and attribution of climate change. This effort also presents a path forward to understand the characteristics of hyperspectral observational records needed to confront models and inline instrument simulation. Such simulation will enable a diverse set of comparisons between model results from coupled model intercomparisons and existing and proposed satellite instrument measurement systems.

54 ENVIRONMENTAL SCIENCES↗

ECAR-7932 Rev 0 Large Eddy Simulation of MARVEL Reactor Core Subchannel to Evaluate Model Uncertainty of Reynolds-Averaged Navier-Stokes Equation Based Computational Fluid Dynamics Analysis

In the previous work (ECAR-7210), the peak cladding temperature of the MARVEL microreactor has been evaluated by steady-state Reynolds-Averaged Navier-Stokes (RANS) based computational fluid dynamics (CFD) simulations. Although numerical uncertainties of RANS-based CFD simulations has been assessed in ECAR-7210, the model uncertainty of RANS turbulence models must be investigated to resolve the issues related to inaccurate prediction of turbulent heat flux and flow pulsation in a tight lattice rod bundle using the steady-state RANS simulations. Consequently, this ECAR conducted a high-fidelity CFD analysis utilizing Large Eddy Simulation (LES) to generate reference data and investigated the model uncertainty of RANS-based CFD simulations.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗

Towards a Consolidated Approach for the Assessment of Evaluation Models of Nuclear Power Reactors

The STARS project at the Paul Scherrer Institut (PSI) has adopted the TRACE thermal-hydraulic (T-H) code for best-estimate system transient simulations of the Swiss Light Water Reactors (LWRs). For analyses involving interactions between system and core, a coupling of TRACE with the SIMULATE-3K (S3K) LWR core simulator has also been developed. In this configuration, the TRACE code and associated nuclear power reactor simulation models play a central role to achieve a comprehensive safety analysis capability. Thus, efforts have now been undertaken to consolidate the validation strategy by implementing a more rigorous and structured assessment approach for TRACE applications involving either only system T-H evaluations or requiring interfaces to e.g. detailed core or fuel behavior models. The first part of this paper presents the preliminary concepts of this validation strategy. The principle is to systematically track the evolution of a given set of predicted physical Quantities of Interest (QoIs) over a multidimensional parametric space where each of the dimensions represent the evolution of specific analysis aspects, including e.g. code version, transient specific simulation methodology and model "nodalisation". If properly set up, such environment should provide code developers and code users with persistent (less affected by user effect) and quantified information (sensitivity of QoIs) on the applicability of a simulation scheme (codes, input models, methodology) for steady state and transient analysis of full LWR systems. Through this, for each given transient/accident, critical paths of the validation process can be identified that could then translate into defining reference schemes to be applied for downstream predictive simulations. In order to illustrate this approach, the second part of this paper presents a first application of this validation strategy to an inadvertent blowdown event that occurred in a Swiss BWR/6. The transient was initiated by the spurious actuation of the Automatic Depressurization System (ADS). The validation approach progresses through a number of dimensions here: First, the same BWR system simulation model is assessed for different versions of the TRACE code, up to the most recent one. The second dimension is the "nodalisation" dimension, where changes to the input model are assessed. The third dimension is the "methodology" dimension. In this case imposed power and an updated TRACE core model are investigated. For each step in each validation dimension, a common set of QoIs are investigated. For the steady-state results, these include fuel temperatures distributions. For the transient part of the present study, the evaluated QoIs include the system pressure evolution and water carry-over into the steam line.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation

A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility Infrastructure Team at Lawrence Livermore National Laboratory to facilitate the use of long-term, high-frequency measurements from ARM in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data.

54 ENVIRONMENTAL SCIENCES↗

Performance Evaluation Modeling of Network Sensors

Substantial benefits are promised by operating many spatially separated sensors collectively. Such systems are envisioned to consist of sensor nodes that are connected by a communications network. A simulation tool is being developed to evaluate the performance of networked sensor systems, incorporating such metrics as target detection probabilities, false alarms rates, and classification confusion probabilities. The tool will be used to determine configuration impacts associated with such aspects as spatial laydown, and mixture of different types of sensors (acoustic, seismic, imaging, magnetic, RF, etc.), and fusion architecture. The QualNet discrete-event simulation environment serves as the underlying basis for model development and execution. This platform is recognized for its capabilities in efficiently simulating networking among mobile entities that communicate via wireless media. We are extending QualNet's communications modeling constructs to capture the sensing aspects of multi-target sensing (analogous to multiple access communications), unimodal multi-sensing (broadcast), and multi-modal sensing (multiple channels and correlated transmissions). Methods are also being developed for modeling the sensor signal sources (transmitters), signal propagation through the media, and sensors (receivers) that are consistent with the discrete event paradigm needed for performance determination of sensor network systems. This work is supported under the Microsensors Technical Area of the Army Research Laboratory (ARL) Advanced Sensors Collaborative Technology Alliance.

probability of detection↗

VIC-Global Parameter Dataset Sensitivity with the Variable Infiltration Capacity Model: Evaluating the importance of dynamic land surface parameters when using the VIC-Global parameter dataset

Accurate prediction of runoff is essential to water resources management, flood risk assessment, and ecosystem protection. However, many hydrological models still have relatively substantial limitations when representing the influence of land use and land cover (LULC) on runoff generation and routing. Changes in LULC, such as deforestation, urban expansion, agricultural intensification, and wetland loss, have been shown to alter the water balance at the land surface through fundamental hydrologic processes (e.g., interception, infiltration, evapotranspiration, and soil storage). However, it remains an open question what the exact magnitude and timing of these impacts are for the spatial and temporal scales commonly used in engineering applications. In this analysis we focus on one aspect of recent LULC change for assessing human impacts, which is urbanization. Specifically we seek to determine the impacts of urbanization on the magnitude and timing of surface runoff and baseflow in HUC-12 basins in Clark County, Nevada which has experienced rapid urbanization. We use the Variable Infiltration Capacity (VIC) hydrology model with a widely used off-the-shelf dataset of land surface parameters, VIC-Global, both of which have been commonly used in the past for water and energy balance modeling for large scale hydrologic studies. We examine two scenarios where the first scenario removes all urbanized land cover and parameterizes those areas of the basins as barren or open shrubland. The second scenario tests the opposite case where all areas of the basins are classified as urban regardless of their present classification. The results from the VIC model show there is a low sensitivity for daily surface runoff between scenarios. The daily baseflow values indicate similar low sensitivity to the classification change during specific periods, but then have substantial differences during other period when large precipitation events are occurring. This is likely due to the assumed parameter values for the urban land cover classification made by the VIC-Global dataset. Using a static land cover parameterization is reasonable for large domain hydrology models that are being used for near-term planning horizons (<30 years). However, longer planning horizons where feedbacks between the atmosphere and land surface are important, especially in transient climate situations, considerations for how to update land surface parameters should be incorporated.

42 ENGINEERING↗

AeroCom Phase III Multi-Model Evaluation of the Aerosol Life Cycle and Optical Properties Using Ground and Space-Based Remote Sensing as Well as Surface In Situ Observations

Within the framework of the AeroCom (Aerosol Comparisons between Observations and Models) initiative, the state-of-the-art modelling of aerosol optical properties is assessed from 14 global models participating in the phase III control experiment (AP3). The models are similar to CMIP6/AerChemMIP Earth System Models (ESMs) and provide a robust multi-model ensemble. Inter-model spread of aerosol species lifetimes and emissions appears to be similar to that of mass extinction coefficients (MECs), suggesting that aerosol optical depth (AOD) uncertainties are associated with a broad spectrum of parameterised aerosol processes. Total AOD is approximately the same as in AeroCom phase I (AP1) simulations. However, we find a 50 % decrease in the optical depth (OD) of black carbon (BC), attributable to a combination of decreased emissions and lifetimes. Relative contributions from sea salt (SS) and dust (DU) have shifted from being approximately equal in AP1 to SS contributing about 2∕3 of the natural AOD in AP3. This shift is linked with a decrease in DU mass burden, a lower DU MEC, and a slight decrease in DU lifetime, suggesting coarser DU particle sizes in AP3 compared to AP1. Relative to observations, the AP3 ensemble median and most of the participating models underestimate all aerosol optical properties investigated, that is, total AOD as well as fine and coarse AOD (AODf, AODc), Ångström exponent (AE), dry surface scattering (SCdry), and absorption (ACdry) coefficients. Compared to AERONET, the models underestimate total AOD by ca. 21 % ± 20 % (as inferred from the ensemble median and interquartile range). Against satellite data, the ensemble AOD biases range from −37 % (MODIS-Terra) to −16 % (MERGED-FMI, a multi-satellite AOD product), which we explain by differences between individual satellites and AERONET measurements themselves. Correlation coefficients (R) between model and observation AOD records are generally high (R>0.75), suggesting that the models are capable of capturing spatio-temporal variations in AOD. We find a much larger underestimate in coarse AODc (∼ −45 % ± 25 %) than in fine AODf (∼ −15 % ± 25 %) with slightly increased inter-model spread compared to total AOD. These results indicate problems in the modelling of DU and SS. The AODc bias is likely due to missing DU over continental land masses (particularly over the United States, SE Asia, and S. America), while marine AERONET sites and the AATSR SU satellite data suggest more moderate oceanic biases in AODc. Column AEs are underestimated by about 10 % ± 16 %. For situations in which measurements show AE > 2, models underestimate AERONET AE by ca. 35 %. In contrast, all models (but one) exhibit large overestimates in AE when coarse aerosol dominates (bias ca. +140 % if observed AE < 0.5). Simulated AE does not span the observed AE variability. These results indicate that models overestimate particle size (or underestimate the fine-mode fraction) for fine-dominated aerosol and underestimate size (or overestimate the fine-mode fraction) for coarse-dominated aerosol. This must have implications for lifetime, water uptake, scattering enhancement, and the aerosol radiative effect, which we can not quantify at this moment. Comparison against Global Atmosphere Watch (GAW) in situ data results in mean bias and inter-model variations of −35 % ± 25 % and −20 % ± 18 % for SCdry and ACdry, respectively. The larger underestimate of SCdry than ACdry suggests the models will simulate an aerosol single scattering albedo that is too low. The larger underestimate of SCdry than ambient air AOD is consistent with recent findings that models overestimate scattering enhancement due to hygroscopic growth. The broadly consistent negative bias in AOD and surface scattering suggests an underestimate of aerosol radiative effects in current global aerosol models. Considerable inter-model diversity in the simulated optical properties is often found in regions that are, unfortunately, not or only sparsely covered by ground-based observations. This includes, for instance, the Sahara, Amazonia, central Australia, and the South Pacific. This highlights the need for a better site coverage in the observations, which would enable us to better assess the models, but also the performance of satellite products in these regions. Using fine-mode AOD as a proxy for present-day aerosol forcing estimates, our results suggest that models underestimate aerosol forcing by ca. −15 %, however, with a considerably large interquartile range, suggesting a spread between −35 % and +10 %.

space-based remote sensing↗

Model Evaluation and Intercomparison of Marine Warm Low Cloud Fractions With Neural Network Ensembles

Abstract Low cloud fractions (LCFs) and meteorological factors (MFs) over an oceanic region containing multiple cloud regimes are examined for three data sets: one Energy Exascale Earth System Model (E3SM) simulation with the default 72‐layer vertical grid (E3SM72), another one with 8‐times vertical resolution via the Framework for Improvement by Vertical Enhancement (E3SM 8), and one with MFs from ERA5 reanalysis and LCFs from the CERES SSF product (ERA5‐SSF). Neural networks (NNs) are trained to capture the relationship between MFs and LCF and to select the best‐performing MF subsets for predicting LCF. NN ensembles are used to (a) confirm the performance of selected MF subsets, (b) to serve as proxy models for each data set to predict LCFs for MFs from all data sets, and (c) to classify MFs into those in shared and uniquely occupied MF subspaces. Overall, E3SM72 and E3SM 8 have large fractions of MFs in shared MF subspace, but less so near the Californian and Peruvian stratocumulus decks. E3SM 8 and ERA5 have small fractions of MFs in shared MF subspace but greater than E3SM72 and ERA5, especially in the Southeast Pacific. The differences in LCFs between three pairs of data sets are decomposed into those associated with the differences in the LCF‐MF relationship and those involving different MFs. Given the same MFs, LCFs produced by E3SM 8 are greater than those produced by E3SM72 but are still different from those in ERA5‐SSF. In general, the shift in MFs dominates the difference in the LCFs.

54 ENVIRONMENTAL SCIENCES↗