Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

Thermal sensing of cryogenic wind tunnel model surfaces Evaluation of silicon diodes

Different sensors and installation techniques for surface temperature measurement of cryogenic wind tunnel models were investigated. Silicon diodes were selected for further consideration because of their good inherent accuracy. Their average absolute temperature deviation in comparison tests with standard platinum resistance thermometers was found to be 0.2 K in the range from 125 to 273 K. Subsurface temperature measurement was selected as the installation technique in order to minimize aerodynamic interference. Temperature distortion caused by an embedded silicon diode was studied numerically.

Daryabeigi, K.↗

Thermal sensing of cryogenic wind tunnel model surfaces - Evaluation of silicon diodes

Different sensors and installation techniques for surface temperature measurement of cryogenic wind tunnel models were investigated. Silicon diodes were selected for further consideration because of their good inherent accuracy. Their average absolute temperature deviation in comparison tests with standard platinum resistance thermometers was found to be 0.2 K in the range from 125 to 273 K. Subsurface temperature measurement was selected as the installation technique in order to minimize aerodynamic interference. Temperature distortion caused by an embedded silicon diode was studied numerically.

Daryabeigi, Kamran↗

Using Remotely Sensed Data and Watershed and Hydrodynamic Models to Evaluate the Effects of Land Cover Land Use Change on Aquatic Ecosystems in Mobile Bay, AL

Alabama coastal systems have been subjected to increasing pressure from a variety of activities including urban and rural development, shoreline modifications, industrial activities, and dredging of shipping and navigation channels. The impacts on coastal ecosystems are often observed through the use of indicator species. One such indicator species for aquatic ecosystem health is submerged aquatic vegetation (SAV). Watershed and hydrodynamic modeling has been performed to evaluate the impact of land cover land use (LCLU) change in the two counties surrounding Mobile Bay (Mobile and Baldwin) on SAV stressors and controlling factors (temperature, salinity, and sediment) in the Mobile Bay estuary. Watershed modeling using the Loading Simulation Package in C++ (LSPC) was performed for all watersheds contiguous to Mobile Bay for LCLU scenarios in 1948, 1992, 2001, and 2030. Remotely sensed Landsat-derived National Land Cover Data (NLCD) were used in the 1992 and 2001 simulations after having been reclassified to a common classification scheme. The Prescott Spatial Growth Model was used to project the 2030 LCLU scenario based on current trends. The LSPC model simulations provided output on changes in flow, temperature, and sediment for 22 discharge points into the estuary. These results were inputted in the Environmental Fluid Dynamics Computer Code (EFDC) hydrodynamic model to generate data on changes in temperature, salinity, and sediment on a grid throughout Mobile Bay and adjacent estuaries. The changes in the aquatic ecosystem were used to perform an ecological analysis to evaluate the impact on SAV habitat suitability. This is the key product benefiting the Mobile Bay coastal environmental managers that integrates the influences of temperature, salinity, and sediment due to LCLU driven flow changes with the restoration potential of SAVs. Data products and results are being integrated into NOAA s EcoWatch and Gulf of Mexico Data Atlas online systems for dissemination to coastal resource managers and stakeholders. Objective 1: Develop and utilize Land Use scenarios for Mobile and Baldwin Counties, AL as input to models to predict the affects on water properties (temperature,salinity,)for Mobile Bay through 2030. Objective 2: Evaluate the impact of land use change on seagrasses and SAV in Mobile Bay. Hypothesis: Urbanization will significantly increase surface flows and impact salinity and temperature variables that effect seagrasses and SAVs.

Al-Hamdan, Mohammad↗

Documentation of a Linked-Population Viability Model to Evaluate Recovery Options for Snake River Fall Chinook Salmon

We developed an age- and life-history based population viability analysis model for linked populations of fall Chinook salmon (LPVA). LPVA was designed to quantify the costs and benefits of re-establishing access to an upstream spawning area using managed reconnection options such as hatchery stocking and translocation. The model incorporates stochasticity in the age at which salmon return to spawn, sex ratio, juvenile life history, and survival rates. Response variables include the proportion of replicate populations that persisted for 80 y (roughly 20 generations), the linked population’s growth rate, λ, time to extinction and summaries of population sizes including average female spawner abundance, final female spawner abundance, and total female abundance. This report attempts to provide a complete model description, including simulation of management options and options for simulating forward with a joint distribution of parameters based on their likelihoods with respect to historical data. Intermediate results illustrating the Bayesian approach to identifying poorly-known parameters are reported here. LPVA simulates reconnection of linked populations by representing management processes such as trapping of adults migrating upstream at two dams and juveniles migrating downstream, and hatchery operations for one or two hatcheries. The purpose of the model is to support efforts to recover the threatened Snake River fall Chinook salmon ESU by evaluating population viability using existing spawning areas with and without a hatchery and to assess the potential costs and benefits of programs to re-establish an upstream spawning area.

Jager, Henriette (Yetta) I.↗

Assessing Mountains as Natural Reservoirs With a Multimetric Framework

Anthropogenic climate change will continue to diminish the unique role that mountains perform as natural reservoirs and alter long‐held assumptions of water management. Climate models are important tools to help constrain uncertainty and understand processes that shape this decline. To ensure that climate model estimates provide stakeholder relevant information, the formulation of multimetric model evaluation frameworks informed by stakeholder interactions are critical. In this study, we present one such multimetric framework to evaluate snowpack data sets in the California Sierra Nevada: the snow water equivalent (SWE) triangle. SWE triangle metrics help to describe snowpack characteristics associated with total water volume buildup, peak water availability, and the rate of water release. This approach highlights compensating errors that would not be reflected in conventional large‐scale spatiotemporal analysis. To test our multimetric evaluation framework, we evaluate several publicly available snow products including the Sierra Nevada Snow Reanalysis, Livneh (L15), and the North American Land Data Assimilation System version 2 data sets. We then evaluate regional climate model skill within the North American Coordinated Regional Climate Downscaling Experiment. All data sets analyzed show variation across the various SWE triangle metrics, even within observationally constrained snow products. This spread was especially shown in spring season melt rates. Melt rate biases were prevalent throughout most regional climate model simulations, regardless of snow accumulation dynamics, and will need to be addressed to improve their utility for water stakeholders.

54 ENVIRONMENTAL SCIENCES↗

Multilevel and multifidelity uncertainty quantification for cardiovascular hemodynamics

Standard approaches for uncertainty quantification in cardiovascular modeling pose challenges due to the large number of uncertain inputs and the significant computational cost of realistic three-dimensional simulations. Here, we propose an efficient uncertainty quantification framework utilizing a multilevel multifidelity Monte Carlo (MLMF) estimator to improve the accuracy of hemodynamic quantities of interest while maintaining reasonable computational cost. This is achieved by leveraging three cardiovascular model fidelities, each with varying spatial resolution to rigorously quantify the variability in hemodynamic outputs. We employ two low-fidelity models (zero- and one-dimensional) to construct several different estimators. Our goal is to investigate and compare the efficiency of estimators built from combinations of these two low-fidelity model alternatives and our high-fidelity three-dimensional models. We demonstrate this framework on healthy and diseased models of aortic and coronary anatomy, including uncertainties in material property and boundary condition parameters. Our goal is to demonstrate that for this application it is possible to accelerate the convergence of the estimators by utilizing a MLMF paradigm. Therefore, we compare our approach to single fidelity Monte Carlo estimators and to a multilevel Monte Carlo approach based only on three-dimensional simulations, but leveraging multiple spatial resolutions. We demonstrate significant, on the order of 10 to 100 times, reduction in total computational cost with the MLMF estimators. We also examine the differing properties of the MLMF estimators in healthy versus diseased models, as well as global versus local quantities of interest. As expected, global quantities such as outlet pressure and flow show larger reductions than local quantities, such as those relating to wall shear stress, as the latter rely more heavily on the highest fidelity model evaluations. Similarly, healthy models show larger reductions than diseased models. In all cases, our workflow coupling Dakota’s MLMF estimators with the SimVascular cardiovascular modeling framework makes uncertainty quantification feasible for constrained computational budgets.

59 BASIC BIOLOGICAL SCIENCES↗

Toward equitable environmental exposure modeling through convergence of data, open, and citizen sciences: an example of air pollution exposure modeling amidst increasing wildfire smoke

Exposure modeling is critical in environmental epidemiology and human health but may face challenges (e.g., skewed data, unequal error, context-insensitive validation, and computational demands). Modeling decisions reflect the intended use of the models and the values that modelers prioritize. We aimed to provide a conceptual framework and machine learning (ML) modeling protocols that address these issues. With 500m-gridded hourly PM 2.5 and O 3 levels in Illinois before, during, and after the 2023 Canadian wildfire season as a motivating example, we conducted modeling experiments to evaluate modeling methods, guided by three domains we propose based on theories of science: 1) Data Diversity, leveraging open and citizen science data to enhance inclusivity, parsimony, and representativeness; 2) Equitable Accuracy, ensuring fairly distributed uncertainties across subpopulations; and 3) Sustainable Modeling, balancing accuracy with reducing computational demands to promote accessibility for under-resourced researchers. Here, we found that ML with publicly available data can achieve high accuracy. Depending on methods, performance may vary substantially, even with identical input data. Large but skewed data may reduce performance. Misuse of cross-validation protocols can underestimate prediction error; although we observed R 2 s of ∼98 %, the modeled estimates varied significantly, indicating the need for careful model validation. By using new modeling protocols including representativeness-considered training and validation data and a new loss function, we achieved high agreement between estimates and ground-based measurements (e.g., R 2 = ∼90 % for PM 2.5 ; ∼80 % for O 3 ), equally distributed errors across sociodemographic strata and urban–rural divides, and reduction in computation time—from several weeks or months to a few days.

Exposure assessment↗

A Hindcast Experiment Using the GCAM 3.0 Agriculture and Land-Use Module

We report results of a “hindcast” experiment focusing on the agricultural and land-use component of the Global Change Assessment Model (GCAM). We initialize GCAM to reproduce observed agriculture and land use in 1990 and forecast agriculture and land use patterns on 1-year time steps to 2010. We report overall model performance for 9 crops in 14 regions. We report areas where the hindcast is in relatively good agreement with observations and areas where the correspondence is poorer. We find that when given observed crop yields as input data, producers in GCAM implicitly have perfect foresight for yields leading to over compensation for year-to-year yield variation. We explore a simple model in which planting decisions are based on expectations but production depends on actual yields and find that this addresses the implicit perfect foresight problem. Second, while existing policies are implicitly calibrated into IAMs, changes in those policies over the period of analysis can have a dramatic effect on the fidelity model output. Third, we demonstrate that IAMs can employ techniques similar to those used by the climate modeling community to evaluate model skill. We find that hindcasting has the potential to yield substantial benefits to the IAM community.

hindcast, gc, agriculture and landuse modeling, va↗

Warm air intrusions reaching the MOSAiC expedition in April 2020—The YOPP targeted observing period (TOP)

In the spring period of the Multidisciplinary drifting Observatory for the Study of Arctic Climate (MOSAiC) expedition, an initiative was in place to increase the radiosounding frequency during warm air intrusions in the Atlantic Arctic sector. Two episodes with increased surface temperatures were captured during April 12–22, 2020, during a targeted observing period (TOP). The large-scale circulation efficiently guided the pulses of warm air into the Arctic and the observed surface temperature increased from -30°C to near melting conditions marking the transition to spring, as the temperatures did not return to values below -20°C. Back-trajectory analysis identifies 3 pathways for the transport. For the first temperature maximum, the circulation guided the airmass over the Atlantic to the northern Norwegian coast and then to the MOSAiC site. The second pathway was from the south, and it passed over the Greenland ice sheet and arrived at the observational site as a warm but dry airmass due to precipitation on the windward side. The third pathway was along the Greenland coast and the arriving airmass was both warm and moist. The back trajectories originating from pressure levels between 700 and 900 hPa line up vertically, which is somewhat surprising in this dynamically active environment. The processes acting along the trajectory originating from 800 hPa at the MOSAIC site are analyzed. Vertical profiles and surface energy exchange are presented to depict the airmass transformation based on ERA5 reanalysis fields. The TOP could be used for model evaluation and Lagrangian model studies to improve the representation of the small-scale physical processes that are important for airmass transformation. A comparison between MOSAiC observations and ERA5 reanalysis demonstrates challenges in the representation of small-scale processes, such as turbulence and the contributions to various terms of the surface energy budget, that are often misrepresented in numerical weather prediction and climate models.

54 ENVIRONMENTAL SCIENCES↗

Description of the Three-Dimensional Large-Scale Forcing Data from the 3D Constrained Variational Analysis (VARANAL3D)

This technical report introduces a Three-Dimensional Constrained Variational Analysis (3DCVA) (Tang and Zhang 2015) and its product of three-dimensional large-scale forcing data to drive single-column models (SCM), cloud-resolving models (CRM), and large-eddy simulation (LES) models, and to evaluate model results. The 3DCVA algorithm is an extension of the original 1D constrained variational analysis (1DCVA) (Zhang and Lin 1997, Zhang et al. 2001). The three-dimensional structure of the forcing data allows studies of spatial variation of the large-scale forcing fields and tests of physical parameterizations across scales. In the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility, the 3D forcing data are assigned the datastream name varanal3d. In this technical report, 3DCVA will be used to refer to the algorithm, while VARANAL3D will be used to refer to the data product.

54 ENVIRONMENTAL SCIENCES↗

Description of the Three-Dimensional Large-Scale Forcing Data from the 3D Constrained Variational Analysis (VARANAL3D)

This technical report introduces a Three-Dimensional Constrained Variational Analysis (3DCVA) (Tang and Zhang 2015) and its product of three-dimensional large-scale forcing data to drive single-column models (SCM), cloud-resolving models (CRM), and large-eddy simulation (LES) models, and to evaluate model results. The 3DCVA algorithm is an extension of the original 1D constrained variational analysis (1DCVA) (Zhang and Lin 1997, Zhang et al. 2001). The three-dimensional structure of the forcing data allows studies of spatial variation of the large-scale forcing fields and tests of physical parameterizations across scales. In the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility, the 3D forcing data are assigned the datastream name varanal3d. In this technical report, 3DCVA will be used to refer to the algorithm, while VARANAL3D will be used to refer to the data product.

54 ENVIRONMENTAL SCIENCES↗

A new process-based and scale-aware desert dust emission scheme for global climate models – Part I: Description and evaluation against inverse modeling emissions

Desert dust accounts for most of the atmosphere's aerosol burden by mass and produces numerous important impacts on the Earth system. However, current global climate models (GCMs) and land-surface models (LSMs) struggle to accurately represent key dust emission processes, in part because of inadequate representations of soil particle sizes that affect the dust emission threshold, surface roughness elements that absorb wind momentum, and boundary-layer characteristics that control wind fluctuations. Furthermore, because dust emission is driven by small-scale (~ 1 km or smaller) processes, simulating the global cycle of desert dust in GCMs with coarse horizontal resolutions (~ 100 km) presents a fundamental challenge. This representation problem is exacerbated by dust emission fluxes scaling nonlinearly with wind speed above a threshold wind speed that is sensitive to land-surface characteristics. Here, we address these fundamental problems underlying the simulation of dust emissions in GCMs and LSMs by developing improved descriptions of (1) the effect of soil texture on the dust emission threshold, (2) the effects of nonerodible roughness elements (both rocks and green vegetation) on the surface wind stress, and (3) the effects of boundary-layer turbulence on driving intermittent dust emissions. We then use the resulting revised dust emission parameterization to simulate global dust emissions in a standalone model forced by reanalysis meteorology and land-surface fields. We further propose (4) a simple methodology to rescale lower-resolution dust emission simulations to match the spatial variability of higher-resolution emission simulations in GCMs. The resulting dust emission simulation shows substantially improved agreement against regional dust emissions observationally constrained by inverse modeling. We thus find that our revised dust emission parameterization can substantially improve dust emission simulations in GCMs and LSMs.

54 ENVIRONMENTAL SCIENCES↗

Toward a Minimal Representation of Aerosols in Climate Models: Description and Evaluation in the Community Atmosphere Model CAM5

A modal aerosol module (MAM) has been developed for the Community Atmosphere Model version 5 (CAM5), the atmospheric component of the Community Earth System Model version 1 (CESM1). MAM is capable of simulating the aerosol size distribution and both internal and external mixing between aerosol components, treating numerous complicated aerosol processes and aerosol physical, chemical and optical properties in a physically based manner. Two MAM versions were developed: a more complete version with seven-lognormal modes (MAM7), and a three-lognormal mode version (MAM3) for the purpose of long-term (decades to centuries) simulations. Major approximations in MAM3 include assuming immediate mixing of primary organic matter (POM) and black carbon (BC) with other aerosol components, merging of the MAM7 fine dust and fine sea salt modes into the accumulation mode, merging of the MAM7 coarse dust and coarse sea salt modes into the single coarse mode, and neglecting the explicit treatment of ammonia and ammonium cycles. Simulated sulfate and secondary organic aerosol (SOA) mass concentrations are remarkably similar between MAM3 and MAM7 as most ({approx}90%) of these aerosol species are in the accumulation mode. Differences of POM and BC concentrations between MAM3 and MAM7 are also small (mostly within 10%) because of the assumed hygroscopic nature of POM, so that freshly emitted POM and BC are wet-removed before mixing internally with soluble aerosol species. Sensitivity tests with the POM assumed to be hydrophobic and with slower aging process increase the POM and BC concentrations, especially at high latitudes (by several times). The mineral dust global burden differs by 10% and sea salt burden by 30-40% between MAM3 and MAM7 mainly due to the different size ranges for dust and sea salt modes and different standard deviations of log-normal size distribution for sea salt modes between MAM3 and MAM7. The model is able to qualitatively capture the observed geographical and temporal variations of aerosol mass and number concentrations, size distributions, and aerosol optical properties. However, there are noticeable biases, e.g., simulated sulfate and mineral dust concentrations at surface over the oceans are too low. Simulated BC concentrations are significant low in the Arctic. There is a low bias in modeled aerosol optical depth on the global scale, especially in the developing counties. There biases in aerosol simulations clearly indicate the need for improvements of aerosol processes (e.g., emission fluxes of anthropogenic aerosols and precursor gases in developing countries, boundary layer nucleation) and properties (e.g., primary aerosol emission size, POM hygroscopicity). In addition the critical role of cloud properties (e.g., liquid water content, cloud fraction) responsible for the wet scavenging of aerosol is highlighted.

54 ENVIRONMENTAL SCIENCES↗

Stress evaluations under rolling/sliding contacts

The state of stress beneath traction drive type of contacts were analyzed. Computing stresses and stress reversals on various planes for points beneath the surface were examined. The effect of tangential and axial friction under gross slip conditions is evaluated with the models. Evaluations were performed on an RC (rolling contact) tester configuration and it is indicated that the classical fatigue stresses are not altered by friction forces typical of lubricated contact. Higher values of friction can result in surface shear reversal that exceeds the stresses at the depth of maximum shear reversal under rolling contact.

Kannel, J. W.↗

A theoretical and experimental study of turbulent nonevaporating sprays

Measurements and analysis limited to the dilute portions of turbulent nonevaporating sprays injected into a still air environment were completed. Mean and fluctuating velocities and Reynolds stress were measured in the continuous phase. Liquid phase measurements included liquid mass fluxes, drop sizes and drop size and velocity correlation. Initial conditions needed for model evaluation were measured at a location as close to the injector exit as possible. The test sprays showed significant effects of slip and turbulent dispersion of the discrete phase. The measurements were used to evaluate three typical models of these processes: (1) a locally homogenous flow (LHF) model, where slip between the phases were neglected; (2) a deterministic separated flow (DSF) model, where slip was considered but effects of drop dispersion by turbulence were ignored; and (3) a stochastic separated flow (SSF) model, where effects of interphase slip and turbulent dispersion were considered using random-walk computations for drop motion. The LHF and DSF models did not provide very satisfactory predictions for the present measurements. In contrast, the SSF model performed reasonably well with no modifications in the prescription of eddy properties from its original calibration. Some effects of drops on turbulence properties were observed near the dense regions of the sprays.

Solomon, A. S. P.↗

A theoretical and experimental study of turbulent evaporating sprays

Measurements and analysis limited to the dilute portions of turbulent evaporating sprays, injected into a still air environment were completed. Mean and fluctuating velocities and Reynolds stress were measured in the continuous phase. Liquid phase measurements included liquid mass fluxes, drop sizes and drop size and velocity correlation. Initial conditions needed for model evaluation were measured at a location as close to the injector exit as possible. The test sprays showed significant effects of slip and turbulent dispersion of the discrete phase. The measurements were used to evaluate three typical models of these processes: (1) a locally homogeneous flow (LHF) model, where slip between the phases were neglected; (2) a deterministic separated flow (DSF) model, where slip was considered but effects of drop dispersion by turbulence were ignored; and (3) a stochastic separated flow (SSF) model, where effects of interphase slip and turbulent dispersion were considered using random-walk computations for drop motion. For all three models, a k-epsilon model as used to find the properties of the continuous phase. The LHF and DSF models did not provide very satisfactory predictions for the present measurements. In contrast, the SSF model performed reasonably well--with no modifications in the prescription of eddy properties from its original calibration.

Solomon, A. S. P.↗

Dynamic Model Agnostic Reliability Evaluation of Machine-Learning Models Integrated in Instrumentation & Control Systems

In recent years, the field of machine learning (ML), specifically neural networks, has grown significantly and has spurred research in its applicability to digital instrumentation and control systems (DI&C). While ML models have shown promise in operational contexts, the trustworthiness of using such algorithms has not been adequately assessed. Failures of ML integrated systems are not well understood, and the lack of comprehensive risk modeling can degrade the trustworthiness in these systems. In recent reports by the National Institute for Standards and Technology (NIST) [1] and the Nuclear Regulatory Commission (NRC) [2], they indicate that trustworthiness in ML is a critical barrier and will play a vital role in the safe, accountable, and secure operation of intelligent systems. Thus, in this work, we demonstrate a dynamic model-agnostic method to quantify the relative reliability of AI/ML predictions by incorporating out-of-distribution (OOD) detection on the training dataset. It is well documented that most ML algorithms excel at interpolation (or near-interpolation) tasks but experience significant performance degradation at extrapolation. The method, referenced as the Laplacian distributed decay for reliability (LADDR), determines the difference between the operational and training datasets which can used to the relative reliability of AI/ML predictions. LADDR is then demonstrated on a feedforward neural network based digital twin used for the prediction of safety significant factors during a loss-of-flow transient. LADDR is used to demonstrate how training data can be used as evidence to support the relative reliability of ML/AI predictions enhancing the overall trustworthiness of the system.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Modeling and Evaluation of the Global Sea-Salt Aerosol Distribution: Sensitivity to Emission Schemes and Resolution Effects at Coastal/Orographic Sites

One of the major sources of uncertainty in model estimates of the global sea-salt aerosol distribution is the emission parameterization. We evaluate a new sea-salt aerosol life cycle module coupled to the online multi-scale chemical transport model NMMB/BSC-CTM. We compare 5 year global simulations using five state-of-the-art sea-salt open-ocean emission schemes with monthly averaged coarse aerosol optical depth (AOD) from selected AERONET sun photometers, surface concentration measurements from the University of Miami's Ocean Aerosol Network, and measurements from two NOAA/PMEL cruises (AEROINDOEX and ACE1). Model results are highly sensitive to the introduction of sea-surface-temperature (SST)-dependent emissions and to the accounting of spume particles production. Emission ranges from 3888 teragrams per year to 8114 teragrams per year, lifetime varies between 7.3 hours and 11.3 hours, and the average column mass load is between 5.0 teragrams and 7.2 teragrams. Coarse AOD is reproduced with an overall correlation of around 0.5 and with normalized biases ranging from +8.8 percent to +38.8 percent. Surface concentration is simulated with normalized biases ranging from minus 9.5 percent to plus 28 percent and the overall correlation is around 0.5. Our results indicate that SST-dependent emission schemes improve the overall model performance in reproducing surface concentrations. On the other hand, they lead to an overestimation of the coarse AOD at tropical latitudes, although it may be affected by uncertainties in the comparison due to the use of all-sky model AOD, the treatment of water uptake, deposition and optical properties in the model and/or an inaccurate size distribution at emission.

Bias↗