Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 685 records · Page 38

Modeling Radiation Detection Using a Mixed-Effects Statistical Model for Background Variation

In this paper, we consider radiation source localization in an urban environment as an inverse problem and use Bayesian parameter estimation techniques to infer the source location and intensity. Since parameter estimation generally requires a large number of model evaluations, we employ a simplified photon transport model, as in a 2017 paper by Stefanescu, et al., to reduce computational requirements. However, the model in the 2017 paper does not account for the physical reality that background radiation varies with position within an urban locale. We incorporate statistical mixed-effects modeling into the simplified transport equation from the Stefanescu, et al. paper to reflect individual variation among the background radiation measurements at multiple detector locations. With our new model, we use a Bayesian DRAM-within-Gibbs technique, which incorporates the Delayed Rejection Adaptive Metropolis (DRAM) algorithm, to estimate the mixed-effects parameters. We note that without adequate prior information, we were unable to simultaneously estimate the set of parameters corresponding to the source intensity, source location, and individual background radiation counts. By calibrating the background in the absence of a source, we obtained narrow prior distributions, which permitted us to jointly infer the source intensity and location and the background radiation distribution.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Ensemble-Based Characterization of Historical Storm Types in ERA5 over CONUS

We present Storm-Type Labeled Precipitation, a gridded dataset that classifies ERA5 precipitation over CONUS (20–50° N, 125–66° W) by storm type at 6-hourly resolution. Each grid cell/time step is assigned to one of five classes—mesoscale convective system (MCS), extratropical cyclone (ETC), hurricane (HUR), atmospheric river (AR), or other convective, with an “unidentified” code reserved for cases with no detected type. Two products are provided: (i) a TempestExtremes (TE)–only version, and (ii) a hybrid TRACK–TE version that uses TRACK for ETCs and HURs and TE for MCSs and ARs. Outputs are integer masks aligned to ERA5 precipitation, supporting storm-type attribution of rainfall, event compositing, trend analysis, and model evaluation. Future work will extend detection to high-resolution downscaled Earth system model projections to support evaluations of projected changes in hydrologic hazards that may threaten critical water and energy infrastructures.

Rastogi, Deeksha [ORNL] (ORCID:0000000204624027)↗

Fast Physics Testbed for the FASTER Project

This poster describes the Fast Physics Testbed for the new FAst-physics System Testbed and Research (FASTER) project. The overall objective is to provide a convenient and comprehensive platform for fast turn-around model evaluation against ARM observations and to facilitate development of parameterizations for cloud-related fast processes represented in global climate models. The testbed features three major components: a single column model (SCM) testbed, an NWP-Testbed, and high-resolution modeling (HRM). The web-based SCM-Testbed features multiple SCMs from major climate modeling centers and aims to maximize the potential of SCM approach to enhance and accelerate the evaluation and improvement of fast physics parameterizations through continuous evaluation of existing and evolving models against historical as well as new/improved ARM and other complementary measurements. The NWP-Testbed aims to capitalize on the large pool of operational numerical weather prediction products. Continuous evaluations of NWP forecasts against observations at ARM sites are carried out to systematically identify the biases and skills of physical parameterizations under all weather conditions. The highresolution modeling (HRM) activities aim to simulate the fast processes at high resolution to aid in the understanding of the fast processes and their parameterizations. A four-tier HRM framework is established to augment the SCM- and NWP-Testbeds towards eventual improvement of the parameterizations.

54 ENVIRONMENTAL SCIENCES↗

Diagnosing added value of convection-permitting regional models using precipitation event identification and tracking

Dynamical downscaling with high-resolution regional climate models may offer the possibility of realistically reproducing precipitation and weather events in climate simulations. As resolutions fall to order kilometers, the use of explicit rather than parametrized convection may offer even greater fidelity. However, these increased resolutions both allow and require increasingly complex diagnostics for evaluating model fidelity. Here we focus on precipitation evaluation and analyze five 2-month-long dynamically downscaled model runs over the continental United States that employ different convective and microphysics parameterizations, including one high-resolution convection-permitting simulation. All model runs use the Weather Research and Forecasting Model driven by National Center for Environmental Prediction reanalysis data. We show that employing a novel rainstorm identification and tracking algorithm that allocates essentially all rainfall to individual precipitation events allows new insights into model biases. Results include that, at least in these runs, model wet bias is driven by excessive areal extent of individual precipitating events, and that the effect is time-dependent, producing excessive diurnal cycle amplitude. This amplified cycle is driven not by new production of events but by excessive daytime enlargement of long-lived precipitation events. Here, we further show that in the domain average, precipitation biases appear best represented as additive offsets. Of all model configurations evaluated, convection-permitting simulations most consistently reduced biases in precipitation event characteristics.

54 ENVIRONMENTAL SCIENCES↗

Experimental and analytical studies of advanced air cushion landing systems

Several concepts are developed for air cushion landing systems (ACLS) which have the potential for improving performance characteristics (roll stiffness, heave damping, and trunk flutter), and reducing fabrication cost and complexity. After an initial screening, the following five concepts were evaluated in detail: damped trunk, filled trunk, compartmented trunk, segmented trunk, and roll feedback control. The evaluation was based on tests performed on scale models. An ACLS dynamic simulation developed earlier is updated so that it can be used to predict the performance of full-scale ACLS incorporating these refinements. The simulation was validated through scale-model tests. A full-scale ACLS based on the segmented trunk concept was fabricated and installed on the NASA ACLS test vehicle, where it is used to support advanced system development. A geometrically-scaled model (one third full scale) of the NASA test vehicle was fabricated and tested. This model, evaluated by means of a series of static and dynamic tests, is used to investigate scaling relationships between reduced and full-scale models. The analytical model developed earlier is applied to simulate both the one third scale and the full scale response.

Lee, E. G. S.↗

Prompt Phrase Ordering Using Large Language Models in HPC: Evaluating Prompt Sensitivity

Large language models (LLMs) have demonstrated effective performance in domain-specific tasks, often requiring a well-designed prompt to guide their responses. However, optimizing the right prompt is challenging due to prompt sensitivity—the phenomenon where small changes in the prompt can lead to significant variations in performance. In this study, we evaluate prompt performance by examining all permutations of independent phrases to investigate prompt sensitivity and robustness. We used two datasets: the GSM8k dataset, which assesses mathematical reasoning, and a custom template prompt for summarizing database metadata. Our goal was to evaluate the performance across all permutations of a sequence of prompt phrases. The study was conducted using the llama3-instruct- 7B model hosted on Ollama, with computations parallelized in a high-performance computing environment. By comparing the average index of phrases in the best and worst-performing prompts, we found that the order of independent phrases within a prompt significantly impacts LLM performance. Additionally, we used Hamming distance to assess changes between phrase orderings, concluding that prompt modifications can dramatically affect scores, often by almost random chance. These findings support existing research on prompt sensitivity. We discuss the challenges of prompt optimization, noting that altering phrases in a successful prompt does not always result in another successful prompt.

97 MATHEMATICS AND COMPUTING↗

SU-E-T-265: Development of Dose-To-Water Conversion Models for Pre-Treatment Verification with the New AS1200 Imager

Purpose: To develop and evaluate models for dose verification of flattened (FF) and flattening filter free (FFF) beams for the new Varian aS1200 backscatter-shielded electronic portal imaging device (EPID). Methods: The model converts EPID images to incident energy fluence using deconvolution of EPID scatter kernels and fluence to dose in water using convolution with dose-to-water kernels. Model parameters were optimized using non-transmission EPID images of varying jaw defined field sizes for energies of 6 and 10 MV FF and FFF beams. Energy fluence was obtained from the Acuros planning system and reference dose profiles and output factors were measured at depths of 5, 10, 15 and 20 cm in a water phantom. Images for 34 IMRT fields acquired at 6 and 10 MV FF energy were converted to dose at 10 cm depth in water and compared to treatment planning system dose plane calculations using gamma criteria. Results: Gamma evaluations for the IMRT fields had mean (1 standard deviation) pass rates of 99.4% (0.8%) and mean gamma scores of 0.32 (0.06) with 2%, 2 mm criteria and 10% of maximum dose threshold. Conclusion: The developed model has been shown to be highly accurate for pre-treatment verification with the new aS1200 imager which does not display support-arm backscatter artefact and has improved dosimetric properties. Further investigation of FFF modes is in progress. The model is currently being evaluated at sites for potential clinical release.

60 APPLIED LIFE SCIENCES↗

Scaling Outlooks on Integrated Energy Systems

The report provides research summaries of two developed theories: data projection to enable multi-scale facility digital interactions and introducing the Evaluation Model Development and Assessment Process (EMDAP) approach to Integrated Energy Systems (IES) design. For the data projection, a two-system scaling case including interactions between thermal storage and hydrogen production is provided. The corresponding facilities are the thermal energy distribution system (TEDS) facility and high temperature steam electrolysis (HTSE) facility at the Dynamic Energy Transport and Integration Laboratory (DETAIL) in Idaho National Laboratory (INL). For the EMDAP adaption to system design, a modified process of the EMDAP was developed and applied to indirect two-tank heat storage system design. System phenomena were identified, ranked, and scaled to discover helpful key relations that would determine the ideal component geometry.

08 HYDROGEN↗

Tropical and subtropical cloud transitions in weather and climate prediction models: the GCSS/WGNE Pacific Cross-Section Intercomparison (GPCI)

A model evaluation approach is proposed where weather and climate prediction models are analyzed along a Pacific Ocean cross-section, from the stratocumulus regions off the coast of California, across the shallow convection dominated trade-winds, to the deep convection regions of the ITCZ: the GCSS/WGNE Pacific Cross-section Intercomparison (GPCI). The main goal of GPCI is to evaluate, and help understand and improve the representation of tropical and sub-tropical cloud processes in weather and climate prediction models. In this paper, a detailed analysis of cloud regime transitions along the cross-section from the sub-tropics to the tropics for the season JJA of 1998 is presented. This GPCI study confirms many of the typical weather and climate prediction model problems in the representation of clouds: underestimation of clouds in the stratocumulus regime by most models with the corresponding consequences in terms of shortwave radiation biases; overestimation of clouds by the ECMWF Re-Analysis (ERA40) in the deep tropics (in particular) with the corresponding impact in the outgoing longwave radiation; large spread between the different models in terms of cloud cover, liquid water path and shortwave radiation; significant differences between the models in terms of vertical crosssections of cloud properties (in particular), vertical velocity and relative humidity. An alternative analysis of cloud cover mean statistics is proposed where sharp gradients in cloud cover along the GPCI transect are taken into account. This analysis shows that the negative cloud bias of some models and ERA40 in the stratocumulus regions (as compared to ISCCP) is associated not only with lower values of cloud cover in these regimes, but also with a stratocumulus-to-cumulus transition that occurs too early along the trade-wind Lagrangian trajectory. Histograms of cloud cover along the cross-section differ significantly between models. Some models exhibit a quasi-bimodal structure with cloud cover being either very large (close to 100%) or very small, while other models show a more continuous transition. The ISCCP observations suggest that reality is in-between these two extreme examples. These different patterns reflect the diverse nature of the cloud, boundary layer, and convection parameterizations in the participating weather and climate prediction models.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of predictive accuracy in structural dynamic models

The evaluation of the predictive accuracy of dynamic models for future large space structures is addressed. Mass and stiffness uncertainties derived from a comparison of analytical and experimental modes are used to evaluate the uncertainty of response predictions based on the analytical model.

Hasselman, T. K.↗

Cloud Properties Simulated by a Single-Column Model. Part II: Evaluation of Cumulus Detrainment and Ice-phase Microphysics Using a Cloud Resolving Model

This paper is the second in a series in which kilometer-scale-resolving observations from the Atmospheric Radiation Measurement program and a cloud-resolving model (CRM) are used to evaluate the single-column model (SCM) version of the National Centers for Environmental Prediction Global Forecast System model. Part I demonstrated that kilometer-scale cirrus properties simulated by the SCM significantly differ from the cloud radar observations while the CRM simulation reproduced most of the cirrus properties as revealed by the observations. The present study describes an evaluation, through a comparison with the CRM, of the SCM's representation of detrainment from deep cumulus and ice-phase microphysics in an effort to better understand the findings of Part I. It is found that detrainment occurs too infrequently at a single level at a time in the SCM, although the detrainment rate averaged over the entire simulation period is somewhat comparable to that of the CRM simulation. Relatively too much detrained ice is sublimated when first detrained. Snow falls over too deep of a layer due to the assumption that snow source and sink terms exactly balance within one time step in the SCM. These characteristics in the SCM parameterizations may explain many of the differences in the cirrus properties between the SCM and the observations (or between the SCM and the CRM). A possible improvement for the SCM consists of the inclusion of multiple cumulus cloud types as in the original Arakawa-Schubert scheme, prognostically determining the stratiform cloud fraction and snow mixing ratio. This would allow better representation of the detrainment from deep convection, better coupling of the volume of detrained air with cloud fraction, and better representation of snow field.

Luo, Yali↗

Evolution in Cloud Population Statistics of the MJO: From AMIE Field Observations to Global Cloud-Permiting Models

Motivated by the success of the AMIE/DYNAMO field campaign, which collected unprecedented observations of cloud and precipitation from the tropical Indian Ocean in Octber 2011 – March 2012, this project explored how such observations can be applied to assist the development of global cloud-permitting models through evaluating and correcting model biases in cloud statistics. The main accomplishment of this project were made in four categories: generating observational products for model evaluation, using AMIE/DYNAMO observations to validate global model simulations, using AMIE/DYNAMO observations in numerical studies of cloud-permitting models, and providing leadership in the field. Results from this project provide valuable information for building a seamless bridge between DOE ASR program’s component on process level understanding of cloud processes in the tropics and RGCM focus on global variability and regional extremes. In particular, experience gained from this project would be directly applicable to evaluation and improvements of ACME, especially as it transitions to a non-hydrostatic variable resolution model.

54 ENVIRONMENTAL SCIENCES↗

An evaluation of turbulence models for massively blown surfaces

In this study, an evaluation of the various turbulence models for massively blown as well as unblown viscous shock layers has been made. The direct impact of the variations introduced through the different turbulent models has been evaluated by employing a single numerical code for the flow-field computations. The cases analyzed include flows with low and high Reynolds numbers and peaked and uniform surface injection rates. It is observed that with appropriate mixing-length formulation, and the use of an intermittency factor, most of the turbulence models considered predict comparable results for both blown and unblown surfaces.

Gupta, R. N.↗

Evaluating Differences Among Crop Models in Simulating Soybean in-Season Growth

Crop models are useful tools for simulating agricultural systems that require continued model development and testing to increase their robustness and improve how they describe our current understanding of processes. Coordinated and “blind” evaluation of multiple models using same protocols and experimental datasets provides unique opportunities to further improve models and enhance their reliability. For soybean [Glycine max (L.) Merr.], there has been limited coordinated multi-model evaluations for the simulation of in-season plant growth dynamics. We evaluated ten dynamic soybean crop models for their simulation of in-season plant growth using data from five experiments conducted in Argentina, Brazil, France, and USA. We evaluated models after a Blind (using only phenology data) and a Full calibration (with in-season and end-of-season variables). Calibration reduced model uncertainty by reducing standard bias for the simulation of in-season variables (biomass, leaf, pod, and stem weights, and leaf area index, LAI). However, we found that most models had difficulty in reproducing leaf growth dynamics, with normalized root mean squared error (nRMSE) of 56% for leaf weight and 43% for LAI (across locations and models after Full calibration). Models with different levels of complexity and experience were capable of simulating final seed yield at maturity with reasonable accuracy (nRMSE of 8–31% after Full calibration). However, the nRMSE for pod weight (of 17–64% after Full calibration) was two-fold larger than that of seed yield. Moreover, the models differed in how they simulated timing from sowing to beginning seed growth (47–93 days) and effective seed filling period (18–54 days), owing to model structural differences in defining the reproductive developmental stages. Overall, we identified the following processes that can benefit from further model improvement: leaf expansion and senescence, reproductive phenology, and partitioning to reproductive growth. Simulation of pod wall tissue and individual seed cohorts is another aspect that many models currently lack. Model improvement can benefit from high-temporal resolution experimental datasets that concurrently account for phenology, plant growth, and partitioning. Further, we recommend collecting reproductive phenology in the field consistent with actual dry matter allocation to organs in the models and collecting multiple observations of seed and pod weight to aid model improvement for simulation of seed growth and yield formation.

Agricultural Model Intercomparison and Improvement↗

Surface Energy Balance Responses to Radiative Forcing in the Central Arctic From MOSAiC and Models

The Arctic surface energy budget (SEB) couples the atmosphere with the sea ice, making it useful for both studying surface processes as well as evaluating models. Improved understanding of atmosphere-ice interactions is required to improve models, requiring year-round observations to address seasonally dependent biases. This work uses novel observations from the MOSAiC expedition to quantify the responses of surface fluxes to radiative forcing over sea ice throughout a complete annual cycle. We identify two primary regimes of flux response: an ice growth regime in winter and an ice melt regime in summer. In the growth regime, changes in radiative forcing impact upwelling longwave, sensible heat, and subsurface heat fluxes, whereas in the melt regime changes in radiative forcing primarily alter the amount of melt and subsurface transmission because the surface temperature is fixed. These observed responses of surface fluxes to radiative forcing are used to evaluate seven weather forecast models during the ice growth regime. In most models, the responses of surface fluxes to radiative forcing do not match observations. Many models also have biased downwelling longwave. One model (the Coupled Arctic Forecast System; CAFS) adequately captures both the mean radiative forcing and the flux responses in winter. CAFS is further evaluated against observations spanning the full MOSAiC year, demonstrating sufficient agreement to provide a more generalized understanding of these SEB process relationships across the Arctic.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of Turbulence-Model Performance in Jet Flows

The importance of reducing jet noise in both commercial and military aircraft applications has made jet acoustics a significant area of research. A technique for jet noise prediction commonly employed in practice is the MGB approach, based on the Lighthill acoustic analogy. This technique requires as aerodynamic input mean flow quantities and turbulence quantities like the kinetic energy and the dissipation. The purpose of the present paper is to assess existing capabilities for predicting these aerodynamic inputs. Two modern Navier-Stokes flow solvers, coupled with several modern turbulence models, are evaluated by comparison with experiment for their ability to predict mean flow properties in a supersonic jet plume. Potential weaknesses are identified for further investigation. Another comparison with similar intent is discussed by Barber et al. The ultimate goal of this research is to develop a reliable flow solver applicable to the low-noise, propulsion-efficient, nozzle exhaust systems being developed in NASA focused programs. These programs address a broad range of complex nozzle geometries operating in high temperature, compressible, flows. Seiner et al. previously discussed the jet configuration examined here. This convergent-divergent nozzle with an exit diameter of 3.6 inches was designed for an exhaust Mach number of 2.0 and a total temperature of 1680 F. The acoustic and aerodynamic data reported by Seiner et al. covered a range of jet total temperatures from 104 F to 2200 F at the fully-expanded nozzle pressure ratio. The aerodynamic data included centerline mean velocity and total temperature profiles. Computations were performed independently with two computational fluid dynamics (CFD) codes, ISAAC and PAB3D. Turbulence models employed include the k-epsilon model, the Gatski-Speziale algebraic-stress model and the Girimaji model, with and without the Sarkar compressibility correction. Centerline values of mean velocity and mean temperature are compared with experimental data.

Woodruff, S. L.↗

Bayesian modeling of traffic-related air pollutants: A case study of urban transportation and air quality dynamics in Columbia, South Carolina

Traffic emissions significantly impact near-road air quality and public health. This research applies a Bayesian modeling framework to investigate these impacts using high-resolution traffic and air pollutant data from an urban corridor in Columbia, South Carolina. Despite a data collection period truncated by the COVID-19 lockdown, the Bayesian approach successfully identified significant predictors and quantified model uncertainty. Employing Bayesian Model Selection and Averaging enhanced prediction accuracy and evaluated model uncertainty. Findings indicate that higher temperatures and increased moisture levels elevate particulate matter (PM 1.0 , PM 2.5 , PM 10 ) concentrations, while traffic speed significantly affects nitrogen dioxide (NO 2 ) levels. Specifically, higher average traffic speeds (indicative of smoother flow) correspond to lower NO 2 concentrations, suggesting that less congested conditions reduce NO 2 emissions. This study highlights the robustness of Bayesian methods for generating reliable air quality insights even under data-constrained conditions. The findings underscore the importance of traffic flow management (e.g., reducing congestion) for mitigating near-road NO 2 exposure and provide a basis for developing targeted public health strategies.

54 ENVIRONMENTAL SCIENCES↗