Search NASA⌕ Search

SEARCH · Search NASA

Results for “quantitative evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Dispersion Validation for Flow Involving a Large Structure Revisited: 45 Degree Rotation

The atmospheric dispersion of contaminants in the wake of a large urban structure is a challenging fluid mechanics problem of interest to the scientific and engineering communities. Magnetic Resonance Velocimetry (MRV) and Magnetic Resonance Concentration (MRC) are relatively new techniques that leverage diagnostic equipment used primarily by the medical field to make 3D engineering measurements of flow and contaminant dispersal. SIERRA/Fuego, a computational fluid dynamics (CFD) code at Sandia National Labs is employed to make detailed comparisons to the dataset to evaluate the quantitative and qualitative accuracy of the model. This work is the second in a series of scenarios. In the prior work, a single large building in an array of similar buildings was considered with the wind perpendicular to a building face. In this work, the geometry is rotated by 45 degrees and improved studies are performed for simulation credibility. The comparison exercise shows conditionally good comparisons between the model and experiment. Model uncertainties are assessed through parametric variations. Various methods of quantifying the accuracy between experiments and data are examined Three-dimensional analysis of accuracy is performed. The effort helped identify deficiencies in the techniques used to make these comparisons, and further methods development therefore becomes one of the main recommendations for follow-on work.

42 ENGINEERING↗

Monte Carlo Simulation and Reconstruction: Assessment of Myocardial Perfusion Imaging of Tracer Dynamics With Cardiac Motion Due to Deformation and Respiration Using Gamma Camera With Continuous Acquisition

Purpose: Myocardial perfusion imaging (MPI) with single photon emission computed tomography (SPECT) is routinely used for stress testing in nuclear medicine. Recently, our group extended its potential going from 3D visual qualitative image analysis to 4D spatiotemporal reconstruction of dynamically acquired data to capture the time variation of the radiotracer concentration and the estimated myocardial blood flow (MBF) and coronary flow reserve (CFR). However, the quality of reconstructed image is compromised due to cardiac deformation and respiration. The work presented here develops an algorithm that reconstructs the dynamic sequence of separate respiratory and cardiac phases and evaluates the algorithm with data simulated with a Monte Carlo simulation for the continuous image acquisition and processing with a slowly rotating SPECT camera. Methods: A clinically realistic Monte Carlo (MC) simulation is developed using the 4D Extended Cardiac Torso (XCAT) digital phantom with respiratory and cardiac motion to model continuous data acquisition of dynamic cardiac SPECT with slowly rotating gamma cameras by incorporating deformation and displacement of the myocardium due to cardiac and respiratory motion. We extended our previously developed 4D maximum-likelihood expectation-maximization (MLEM) reconstruction algorithm for a data set binned from a continuous list mode (LM) simulation with cardiac and respiratory information. Our spatiotemporal image reconstruction uses splines to explicitly model the temporal change of the tracer for each cardiac and respiratory gate that delineates the myocardial spatial position as the tracer washes in and out. Unlike in a fully list-mode data acquisition and reconstruction the accumulated photons are binned over a specific but very short time interval corresponding to each cardiac and respiratory gate. Reconstruction results are presented showing the dynamics of the tracer in the myocardium as it continuously deforms. These results are then compared with the conventional 4D spatiotemporal reconstruction method that models only the temporal changes of the tracer activity. Mean Stabilized Activity (MSA), signal to noise ratio (SNR) and Bias for the myocardium activities for three different target-to-background ratios (TBRs) are evaluated. Dynamic quantitative indices such as wash-in (K1) and wash-out (k2) rates at each gate were also estimated. Results: The MSA and SNR are higher with higher TBRs while biases were improved with higher TBRs to less than 10%. The correlation between exhalation-inhalation sequence with the ground truth during respiratory cycle was excellent. Our reconstruction method showed better resolved myocardial walls during diastole to systole as compared to the ungated 4D image. Estimated values of K1 and k2 were also consistent with the ground truth. Conclusion: The continuous image acquisition for dynamic scan using conventional two-head gamma cameras can provide valuable information for MPI. Our study demonstrated the viability of using a continuous image acquisition method on a widely used clinical two-head SPECT system. Our reconstruction method showed better resolved myocardial walls during diastole to systole as compared to the ungated 4D image. Precise implementation of reconstruction algorithms, better segmentation techniques by generating images of different tissue types and background activity would improve the feasibility of the method in real clinical environment.

60 APPLIED LIFE SCIENCES↗

Ultrasonic Testing (UT) Reference Standard for Additive Manufacturing Quality Control

Many additive manufacturing (AM) reference standards for build quality verification concentrate primarily on external features. In contrast, EPRI proposes a pair of AM reference blocks that feature only internal and embedded forms. This report presents, collates, and discusses quantitative nondestructive evaluation (NDE) results from various techniques, including visual testing (VT), radiographic testing (RT), conventional ultrasonic testing (UT), and full matrix capture/total focusing method (FMC/TFM) scanning. The blocks are intended, as part of a larger series of blocks, to evaluate build quality and the relative performance of different NDE techniques in detecting various features. The limits of detectability and the closeness of the as-built shape to the intended form for certain features are quantified, facilitating direct comparison. Upon analysis of the results of this testing, it was found that FMC/TFM was consistently superior in detection, followed by conventional UT, then VT, and lastly RT.

36 MATERIALS SCIENCE↗

Evaluating Machine Learning-Based MRI Reconstruction Using Digital Image Quality Phantoms

Quantitative and objective evaluation tools are essential for assessing the performance of machine learning (ML)-based magnetic resonance imaging (MRI) reconstruction methods. However, the commonly used fidelity metrics, such as mean squared error (MSE), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR), often fail to capture fundamental and clinically relevant MR image quality aspects. To address this, we propose evaluation of ML-based MRI reconstruction using digital image quality phantoms and automated evaluation methods. Our phantoms are based upon the American College of Radiology (ACR) large physical phantom but created in k-space to simulate their MR images, and they can vary in object size, signal-to-noise ratio, resolution, and image contrast. Our evaluation pipeline incorporates evaluation metrics of geometric accuracy, intensity uniformity, percentage ghosting, sharpness, signal-to-noise ratio, resolution, and low-contrast detectability. We demonstrate the utility of our proposed pipeline by assessing an example ML-based reconstruction model across various training and testing scenarios. The performance results indicate that training data acquired with a lower undersampling factor and coils of larger anatomical coverage yield a better performing model. The comprehensive and standardized pipeline introduced in this study can help to facilitate a better understanding of the performance and guide future development and advancement of ML-based reconstruction algorithms.

47 OTHER INSTRUMENTATION↗

Evaluating long-term model-based scenarios of the energy system

Energy-economic models are used to provide science-based decision support in a variety of contexts. Analyses using these tools often involve defining a “reference” scenario, which serves as a counterfactual against which alternative scenarios are compared. Evaluating scenarios, including reference scenarios, is critical for establishing the credibility of the analyses these models support. We propose a framework for evaluating energy system scenarios which consists of three parts – a qualitative storyline, quantitative metrics, and evaluation criteria. We apply this framework to the reference scenario for GCAM-USA, a version of the global human-Earth system model GCAM (Global Change Assessment Model) with state-level detail in the United States, focusing on the evolution of the electric power sector. We develop new visual analytic tools to facilitate the evaluation of model outcomes in 51 sub-national regions, and demonstrate how scenario performance can be tracked and compared across four quantifications of the GCAM-USA reference scenario.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

The Technical, Economic, Risk, and Adoption Assessment for Evaluating Work Reduction Opportunities in the Nuclear Industry

Automation and cost-saving initiatives, such as process automation with advances in artificial intelligence, are gaining traction in modernization efforts across the nuclear industry. As these innovations are increasingly adopted, it becomes crucial to evaluate their impacts comprehensively. Various technical and economic attributes, along with risk and human readiness factors, must be achieved to ensure that innovative projects enabling automation and modernization are successful. However, no systematic or integrated framework exists that allows plants to evaluate these innovative projects. To address this gap, the Technical, Economic, Risk, and Adoption (TERA) assessment offers a structured method to evaluate innovative technologies, ensuring solutions meet both operational and safety standards. The TERA framework integrates the disparate perspectives to assess modernization opportunities in nuclear operations. It combines qualitative and quantitative models to evaluate the relationship between performance and business impacts, while also enabling continuous re-evaluation during project development. This approach helps plant owners identify high-priority opportunities, optimize cost savings, and minimize risks, ensuring projects remain on track to achieve desired returns. By providing a comprehensive, systematic methodology, TERA enables informed, data-driven decisions that support successful modernization efforts, enhancing efficiency, safety, and cost savings across nuclear operations. This paper explains the TERA framework and its benefits for the nuclear industry.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Applying queueing theory to evaluate wait-time-savings of triage algorithms

Abstract In the past decade, artificial intelligence (AI) algorithms have made promising impacts in many areas of healthcare. One application is AI-enabled prioritization software known as computer-aided triage and notification (CADt). This type of software as a medical device is intended to prioritize reviews of radiological images with time-sensitive findings, thus shortening the waiting time for patients with these findings. While many CADt devices have been deployed into clinical workflows and have been shown to improve patient treatment and clinical outcomes, quantitative methods to evaluate the wait-time-savings from their deployment are not yet available. In this paper, we apply queueing theory methods to evaluate the wait-time-savings of a CADt by calculating the average waiting time per patient image without and with a CADt device being deployed. We study two workflow models with one or multiple radiologists (servers) for a range of AI diagnostic performances, radiologist’s reading rates, and patient image (customer) arrival rates. To evaluate the time-saving performance of a CADt, we use the difference in the mean waiting time between the diseased patient images in the with-CADt scenario and that in the without-CADt scenario as our performance metric. As part of this effort, we have developed and also share a software tool to simulate the radiology workflow around medical image interpretation, to verify theoretical results, and to provide confidence intervals for the performance metric we defined. We show quantitatively that a CADt triage device is more effective in a busy, short-staffed reading setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the evaluation methodology presented in this paper can be applied to assess the time-saving performance of other types of algorithms that prioritize a subset of customers based on binary outputs.

Thompson, Yee Lam Elim (ORCID:0000000196537707)↗

Functional Relevance of CASP16 Nucleic Acid Predictions as Evaluated by Structure Providers

ABSTRACT Accurate biomolecular structure prediction enables the prediction of mutational effects, the speculation of function based on predicted structural homology, the analysis of ligand binding modes, experimental model building, and many other applications. Such algorithms to predict essential functional and structural features remain out of reach for biomolecular complexes containing nucleic acids. Here, we report a quantitative and qualitative evaluation of nucleic acid structures for the CASP16 blind prediction challenge by 12 of the experimental groups who provided nucleic acid targets. Blind predictions accurately model secondary structure and some aspects of tertiary structure, including reasonable global folds for some complex RNAs; however, predictions often lack accuracy in the regions of highest functional importance. All models have inaccuracies in non‐canonical regions where, for example, the nucleic‐acid backbone bends, deviating from an A‐form helix geometry, or a base forms a non‐standard hydrogen bond (not a Watson‐Crick base pair). These bends and non‐canonical interactions are integral to forming functionally important regions such as RNA enzymatic active sites. Additionally, the modeling of conserved and functional interfaces between nucleic acids and ligands, proteins, or other nucleic acids remains poor. For some targets, the experimental structures may not represent the only structure the biomolecular complex occupies in solution or in its functional life cycle, posing a future challenge for the community.

Biochemistry & Molecular Biology↗

LDRD23-0184: Resilience and Hazard Risk Assessment to Prioritize Security Operations for Decisions and Impacts (RHAPSODI)

Recent examples provide a significant concern for the resilience of the U.S. electric grid and represent a need for enhanced decision-making to address an increasingly wide range of complex system interactions and potential consequences. In response, this LDRD project produced a proof-of-concept evaluation called the Resilience and Hazard Assessment to Prioritize Security Operations for Decisions and Impacts (RHAPSODI) methodology as an agile and flexible analytic framework capable of addressing multiple, diverse threats to desired electric grid performance. After empirically grounding needs for the future of U.S. electric grid resilience, this project employed the systems-theoretic process analysis (STPA) to develop a systems engineering risk model. The results of a completed feasibility study of a notional high voltage transmission system demonstrate an improved ability to incorporate both spatial (e.g., geographically distributed) and temporal (e.g., dynamic and time-dependent) elements of security risk to the gird. The success of this LDRD project provides the foundation for further evolution of the systems engineering risk model for the grid; derivation of quantitative approaches to evaluate risk and resilience performance; facilitation of agile experimenting and grid sensitivity to a range of vulnerabilities; and development of tools to assist decision-makers in enhancing U.S. electrical grid resilience.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Development and Outcomes of Returning Polycyclic Aromatic Hydrocarbon Exposure Results in the Washington Heights, NYC Community

Report-back of research results (RBRR) is becoming standard practice for environmental health research studies. RBRR is thought to increase environmental health literacy (EHL), although standardized measurements are limited. For this study, we developed a report back document on exposure to air pollutants, Polycyclic Aromatic Hydrocarbons, during pregnancy through community engaged research and evaluated whether the report increased EHL. We used focus groups and surveys to gather feedback on the report document from an initial group of study participants (Group 1, n = 22) and then sent the revised report to a larger number of participants (Group 2, n = 168). We conducted focus groups among participants in Group 1 and discussed their suggested changes to the report and how those changes could be implemented. Participants in focus groups demonstrated multiple levels of EHL. While participant engagement critically informed report development, a survey comparing feedback from Group 1 (initial report) and Group 2 (revised report) did not show a significant difference in the ease of reading the report or knowledge gained about air pollutants. We acknowledge that our approach was limited by a lack of EHL tools that assess knowledge and behavior change, and a reliance on quantitative methodologies. Future approaches that merge qualitative and quantitative methodologies to evaluate RBRR and methodologies for assessing RBRR materials and subsequent changes in knowledge, attitudes, and behavior, may be necessary.

Children’s Environmental Health↗

A priori analysis on deep learning of subgrid-scale parameterizations for Kraichnan turbulence

In the present study, we investigate different data-driven parameterizations for large eddy simulation of two-dimensional turbulence in the a priori settings. These models utilize resolved flow field variables on the coarser grid to estimate the subgrid-scale stresses. We use data-driven closure models based on localized learning that employs a multilayer feedforward artificial neural network with point-to-point mapping and neighboring stencil data mapping, and convolutional neural network fed by data snapshots of the whole domain. The performance of these data-driven closure models is measured through a probability density function and is compared with the dynamic Smagorinksy model (DSM). The quantitative performance is evaluated using the cross-correlation coefficient between the true and predicted stresses. We analyze different frameworks in terms of the amount of training data, selection of input and output features, their characteristics in modeling with accuracy, and training and deployment computational time. We also demonstrate computational gain that can be achieved using the intelligent eddy viscosity model that learns eddy viscosity computed by the DSM instead of subgrid-scale stresses. Here, we detail the hyperparameters optimization of these models using the grid search algorithm.

42 ENGINEERING↗

Local bilinear computation of Jacobi sets

Abstract We propose a novel method for the computation of Jacobi sets in 2D domains. The Jacobi set is a topological descriptor based on Morse theory that captures gradient alignments among multiple scalar fields, which is useful for multi-field visualization. Previous Jacobi set computations use piecewise linear approximations on triangulations that result in discretization artifacts like zig-zag patterns. In this paper, we utilize a local bilinear method to obtain a more precise approximation of Jacobi sets by preserving the topology and improving the geometry. Consequently, zig-zag patterns on edges are avoided, resulting in a smoother Jacobi set representation. Our experiments show a better convergence with increasing resolution compared to the piecewise linear method. We utilize this advantage with an efficient local subdivision scheme. Finally, our approach is evaluated qualitatively and quantitatively in comparison with previous methods for different mesh resolutions and across a number of synthetic and real-world examples.

97 MATHEMATICS AND COMPUTING↗

Explaining word embeddings with perfect fidelity: a case study in predicting research impact

The best-performing approaches for scholarly document quality prediction are based on embedding models. In addition to their performance when used in classifiers, embedding models can also provide predictions even for words that were not contained in the labelled training data for the classification model, which is important in the context of the ever-evolving research terminology. Although model-agnostic explanation methods, such as Local interpretable model-agnostic explanations, can be applied to explain machine learning classifiers trained on embedding models, these produce results with questionable correspondence to the model. We introduce a new feature importance method, Self-Model Entities Rated (SMER), for logistic regression-based classification models trained on word embeddings. We show that SMER has theoretically perfect fidelity with the explained model, as the average of logits of SMER scores for individual words (SMER explanation) exactly corresponds to the logit of the prediction of the explained model. Quantitative and qualitative evaluation is performed through five diverse experiments conducted on 50,000 research articles (papers) from the CORD-19 corpus. In conclusion, through an AOPC curve analysis, we experimentally demonstrate that SMER produces better explanations than LIME, SHAP and global tree surrogates.

Coarse-grained models↗

Process-level cost analysis of hybrid manufacturing pathways for aerospace structural components

Hybrid manufacturing is a promising route for producing complex aerospace components, yet systematic cost benchmarking across multiple additive-subtractive pathways remains limited. This study presents a comprehensive process-based cost analysis of seven hybrid manufacturing routes, including laser powder bed fusion (L-PBF), powder- and wire-directed energy deposition (DED), wire arc additive manufacturing (WAAM), additive friction stir deposition (AFSD), metal binder jetting (MBJ), and agility forging, followed by scanning and finish machining. Parametric cost models incorporating direct material, labor, and energy costs were developed. L-PBF results are discussed in detail for a pickle fork component and directly compared with commercial pricing. Across all hybrid routes, labor emerged as the dominant cost driver, contributing more than 70% of total manufacturing cost in some cases. AFSD exhibited the lowest cost for aluminum components, with MBJ being its 316 L stainless steel counterpart, after accounting for geometric scaling. Benchmarking against industrial quotes suggests that hybrid manufacturing can achieve cost levels comparable to those of commercial services, although labor-intensive processes exhibit greater deviation. The analysis highlights automation of material handling, setup, and supervision as key opportunities for improving economic competitiveness. Overall, the proposed framework provides a quantitative basis for evaluating and optimizing hybrid manufacturing pathways for aerospace applications.

Baruah, Sweta [ORNL] (ORCID:0009000174256207)↗

Trajectory sampling and finite-size effects in first-principles stopping power calculations

Abstract Real-time time-dependent density functional theory (TDDFT) is presently the most accurate available method for computing electronic stopping powers from first principles. However, obtaining application-relevant results often involves either costly averages over multiple calculations or ad hoc selection of a representative ion trajectory. We consider a broadly applicable, quantitative metric for evaluating and optimizing trajectories in this context. This methodology enables rigorous analysis of the failure modes of various common trajectory choices in crystalline materials. Although randomly selecting trajectories is common practice in stopping power calculations in solids, we show that nearly 30% of random trajectories in an FCC aluminum crystal will not representatively sample the material over the time and length scales feasibly simulated with TDDFT, and unrepresentative choices incur errors of up to 60%. We also show that finite-size effects depend on ion trajectory via “ouroboros” effects beyond the prevailing plasmon-based interpretation, and we propose a cost-reducing scheme to obtain converged results even when expensive core-electron contributions preclude large supercells. This work helps to mitigate poorly controlled approximations in first-principles stopping power calculations, allowing 1–2 order of magnitude cost reductions for obtaining representatively averaged and converged results.

36 MATERIALS SCIENCE↗

Metrics for quantifying the efficiency of atmospheric CO 2 reduction by marine carbon dioxide removal (mCDR)

Abstract Marine carbon dioxide removal (mCDR) is gaining interest as a tool to meet global climate goals. Because the response of the ocean–atmosphere system to mCDR takes years to centuries, modeling is required to assess the impact of mCDR on atmospheric CO 2 reduction. Here, we use a coupled ocean–atmosphere model to quantify the atmospheric CO 2 reduction in response to a CDR perturbation. We define two metrics to characterize the atmospheric CO 2 response to both instantaneous ocean alkalinity enhancement (OAE) and direct air capture (DAC): the cumulative additionality ( α ) measures the reduction in atmospheric CO 2 relative to the magnitude of the CDR perturbation, while the relative efficiency ( ϵ ) quantifies the cumulative additionality of mCDR relative to that of DAC. For DAC, α is 100% immediately following CDR deployment, but declines to roughly 50% by 100 years post-deployment as the ocean degasses CO 2 in response to the removal of carbon from the atmosphere. For instantaneous OAE, α is zero initially and reaches a maximum of 40%–90% several years to decades later, depending on regional CO 2 equilibration rates and ocean circulation processes. The global mean ϵ approaches 100% after 40 years, showing that instantaneous OAE is nearly as effective as DAC after several decades. However, there are significant geographic variations, with ϵ approaching 100% most rapidly in the low latitudes while ϵ stays well under 100% for decades to centuries near deep and intermediate water formation sites. These metrics provide a quantitative framework for evaluating sequestration timescales and carbon market valuation that can be applied to any mCDR strategy.

Yamamoto, Kana (ORCID:0009000731519234)↗

InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations

We propose InSituNet, a deep learning based surrogate model to support parameter space exploration for ensemble simulations that are visualized in situ. In situ visualization, generating visualizations at simulation time, is becoming prevalent in handling large-scale simulations because of the I/O and storage constraints. However, in situ visualization approaches limit the flexibility of post-hoc exploration because the raw simulation data are no longer available. Although multiple image-based approaches have been proposed to mitigate this limitation, those approaches lack the ability to explore the simulation parameters. Our approach allows flexible exploration of parameter space for large-scale ensemble simulations by taking advantage of the recent advances in deep learning. Specifically, we design InSituNet as a convolutional regression model to learn the mapping from the simulation and visualization parameters to the visualization results. With the trained model, users can generate new images for different simulation parameters under various visualization settings, which enables in-depth analysis of the underlying ensemble simulations. We demonstrate the effectiveness of InSituNet in combustion, cosmology, and ocean simulations through quantitative and qualitative evaluations.

deep learning↗

Wildfire‐induced increases in photosynthesis in boreal forest ecosystems of North America

Abstract Observations of the annual cycle of atmospheric CO 2 in high northern latitudes provide evidence for an increase in terrestrial metabolism in Arctic tundra and boreal forest ecosystems. However, the mechanisms driving these changes are not yet fully understood. One proposed hypothesis is that ecological change from disturbance, such as wildfire, could increase the magnitude and change the phase of net ecosystem exchange through shifts in plant community composition. Yet, little quantitative work has evaluated this potential mechanism at a regional scale. Here we investigate how fire disturbance influences landscape‐level patterns of photosynthesis across western boreal North America. We use Alaska and Canadian large fire databases to identify the perimeters of wildfires, a Landsat‐derived land cover time series to characterize plant functional types (PFTs), and solar‐induced fluorescence (SIF) from the Orbiting Carbon Observatory‐2 (OCO‐2) as a proxy for photosynthesis. We analyze these datasets to characterize post‐fire changes in plant succession and photosynthetic activity using a space‐for‐time approach. We find that increases in herbaceous and sparse vegetation, shrub, and deciduous broadleaf forest PFTs during mid‐succession yield enhancements in SIF by 8–40% during June and July for 2‐ to 59‐year stands relative to pre‐fire controls. From the analysis of post‐fire land cover changes within individual ecoregions and modeling, we identify two mechanisms by which fires contribute to long‐term trends in SIF. First, increases in annual burning are shifting the stand age distribution, leading to increases in the abundance of shrubs and deciduous broadleaf forests that have considerably higher SIF during early‐ and mid‐summer. Second, fire appears to facilitate a long‐term shift from evergreen conifer to broadleaf deciduous forest in the Boreal Plain ecoregion. These findings suggest that increasing fire can contribute substantially to positive trends in seasonal CO 2 exchange without a close coupling to long‐term increases in carbon storage.

Biodiversity & Conservation↗