Search NASA⌕ Search

SEARCH · Search NASA

Results for “AI Evaluations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Multiscale Modeling of Reconstructed Tricalcium Silicate using NASA Multiscale Analysis Tool

To study microstructure characteristics of cementitious materials hydrated in space; previously, cement binder formations were processed under microgravity conditions and was further compared against ground-based experiments. For accurate estimation of process-structure-property linkage, particularly on samples hydrated in the microgravity environment, it is desired to have a high-fidelity volumetric representation of the microstructure. However, owing to small sample size and high porosity of the space-returned samples, conventional experimental characterization techniques are not viable. Hence, a deep learning-based reconstruction algorithm was employed to obtain high fidelity 3D volumes from sparse high resolution 2D Scanning Electron Microscopy (SEM) images, as inputs to micromechanics-based modeling. This machine learning-based reconstruction methodology validated against low-order statistical descriptors, captured the microstructural topology of both sample types (ground, 1g and microgravity, μg). Due to the lack of gravity, hydration products of the samples processed in space differed from those processed-on ground. Such AI-generated virtual samples were analyzed in a multiscale recursive micromechanics approach using the NASA Multiscale Analysis Tool (NASMAT). Here, we present a methodology to rapidly integrate and evaluate these AI-generated volumes in NASMAT. The synthesized microstructural volumes are directly employed as Representative Volume Elements (RVEs) to preserve the fidelity (1 pixel = 0.54 m). Invariably, analysis of such largescale problems (5123 voxels) requires huge amount of computational resources. By taking advantage of the NASMAT architecture, we also focused on systematic multiscale integration of these AI-reconstructed virtual volumes to reduce the computational demands. In this work, this methodology is demonstrated on the ground-based, 1g samples. The estimated stiffness value of 15.90 GPa is comparable to experimentally obtained modulus of hydrated tricalcium silicate sample. The workflow presented here paves the way for utilizing the NASMAT tool to perform multiscale analyses of other multi-phase material systems using either 3D virtual datasets synthesized using AI or obtained via micro-CT.

Machine Learning↗
DOE OSTI2025

Artificial Intelligence Benchmarking

AI benchmarking is a method for evaluating the effectiveness of an AI model using a set of standardized metrics, for example, high school-level math exams. These benchmarks and their results will enable ranking various AI models based on their effectiveness in performing a specific task.

Krishnan, Anjay [Fermilab]↗
DOE OSTI2025

Business Case Analysis for Artificial Intelligence-Large Language Model Technology Integration

AI-assisted processes are expected to enhance operational efficiency and improve decision-making, supporting the long-term economic viability of nuclear power plants. However, detailed business analyses of AI-generated cost savings are rarely performed. Given the recent industry interest in Large Language Model (LLM), the U.S. Department of Energy (DOE) Light Water Reactor Sustainability (LWRS) Program has conducted a comprehensive business case analysis of LLM Artificial Intelligence (AI) implementation in nuclear plant engineering workflows. The research employed three complementary business case approaches to evaluate impact of an LLM, using three representative engineering processes as use-cases: Boric Acid Corrosion (BAC) Evaluations, Maintenance Rule Evaluations, and 10 CFR 50.59 Screenings. Through detailed workload analyses and structured interviews, the study quantified significant efficiency improvements ranging from 11% to 59% across these processes. The research further considers how these efficiency gains could translate into tangible reliability improvements through enhanced engineering capacity. Analysis of historical plant trip data indicates that enabling engineers to focus on proactive reliability activities could provide substantial financial benefits through avoided outages, potentially generating greater value than the direct efficiency improvements alone. By documenting successful applications, implementation challenges, and strategic opportunities, this research provides nuclear utilities with a practical framework for evaluating the value of AI technology to support long-term operations through advanced digital technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Safety in Artificial Intelligence: Challenges and Opportunities for the U.S. National Labs and Beyond

This report discusses the importance of the critical and underexplored topic of artificial intelligence (AI) safety, as highlighted during the “Strategy Alignment on AI Safety” workshop convened at Lawrence Livermore National Laboratory (LLNL) in April 2024. Through a summary of keynote talks, panel discussions, and breakout sessions, world-leading AI safety experts from academic, industry, national labs, and government agencies clearly agree on the need for and importance of large-scale investments for research and capabilities in AI safety. With the field innovating at unprecedented rates, there is increasing urgency to develop novel evaluation methodologies that allow full considerations of risks/threats of AI technologies in different domains. Quantitative metrics and effective methodologies that can evaluate and audit the “safeness” of how a given AI technology is trained, deployed, or regulated are, at best, nascent for certain scenarios or, more commonly, nonexistent. This maturation gap presents the possibility of serious threats to national security, and further inaction may have serious consequences. Additionally, the gap between the public’s and research community’s perceptions of AI risks/rewards is significant. While numerous voices from the AI community have expressed concern that the risks could be so high that future AI systems could inflict extinction-level damage to humanity if deployed incorrectly, the public largely is aware only of risk in low-impact scenarios. This discrepancy highlights the crucial need for researchers to articulate to governmental bodies what, why, and when various AI risks matter as part of motivating funding requests. Thus, the call to action for this community is to pursue AI safety as a “Big Science” project on a scale comparable to the Manhattan Project. High risks and high payoffs are on the table, but safe AI is a fast-moving target, and large-scale investments are needed to guide development of this technology in a responsible way. We highlight the need for a multilayered solution combining the development of new methods and algorithmic approaches to mitigate threats with an active participation of the government(s) in setting high industry standards and regulations based on state-of-the-art technology. The U.S. Department of Energy (DOE) national laboratories have served as leading institutions for scientific innovation in the U.S. for more than 70 years. Drawing on their expertise in the AI community and their history of safeguarding critical and sensitive information, and as we look to the future, national labs are the best choice for evaluating and safeguarding AI technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Videos, photos, and AI-derived grain size data associated with “High-throughput AI Video Surveys Enable Reproducible Multiscale Sediment Size Mapping, with Implications for Hydrobiogeochemical Parameterization”

NOTE: The manuscript associated with this data package is currently in review. The data may be revised based on reviewer feedback. Upon manuscript acceptance, this data package will be updated with the final dataset and additional metadata. This data package is associated with the manuscript “High-throughput AI Video Surveys Enable Reproducible Multiscale Sediment Size Mapping, with Implications for Hydrobiogeochemical Parameterization” under review. This data package includes five data types: 1) raw photos and videos from drone survey and walking smartphone surveys; 2) images derived from raw videos; 3) manual labeling of reference scales; 4) metadata for all images and photo resolution derived from artificial intelligence (AI) models or manual labels, 5) grain size data obtained from AI models for all photos, 6) metadata and grain size data after quality control, 7) summaries of sample efficiency for all data, and 8) computational fluid dynamics (CFD) data used to support hydro-biogeochemical (HBGC) parameter estimation. Such data is used to 1) demonstrate significant improvements in accuracy, efficiency, and quality control for grain size data collection with the help of AI models, 2) study the spatial heterogeneity of grain size and observation reproducibility based on tens of thousands of data points generated by the AI models, and 3) evaluate the impacts of grain size heterogeneity on key HBGC parameters across sediment-to-reach and hourly-to-yearly scales. In particular, the data package contains 116 folders and 179696 files. The files include 41 videos in .mov format, 64047 photos in .jpg format, 13541 video-derived photos in .png format, 12747 segmentation mask data in .tif format, 12747 segmentation data in .json format, 24771 .csv files that with metadata and grain size for each individual photo as well as water depth and velocity data from CFD and observation, 51791 .txt files of raw AI predicted labels, and 11 flight record data in .srt format. The summary for all metadata and grain size statistics information is included in “Scales_V3_NG.csv” and “Statistics_V3_NG.csv”. The summary for data that pass data quality control (QC) level 0-2 is included in “QCStatistics_V3_NG.csv”. The QC level 0 represents photos whose photo resolution is positive, excluding photos that miss reference scale. The QC level 1 means reference scale circularity uncertainty is less than 5% for smartphone images while representing photo resolution is larger than 0.44 mm/pixel for drone images. The QC level 2 means excluding photos whose grain number is less than 100, a minimum number of grains recommended by classic literature. The summary for each video’s name, length, frame rates, survey area, grain number, survey efficiency, etc. can be found in “QCSummary_V3_NG.csv”. The summary for site name, GPS coordinates, and number of images at each site can be found in “SitesSummary_V3_*.csv” files. Overall computational efficiency summary is reported in Table 4 of accompanying manuscript. Additionally, the nitrate concentration data used in this work was downloaded from an existing dataset published on ESS-DIVE (Boat-Dragged Sensor Hanford Reach.csv; Conner A. et al., 2020). We thank the United States Forest Service, Washington Department of Fish and Wildlife, Washington Department of Natural Resources, Cowiche Canyon Conservatory, Port of Benton, and the Confederated Tribes and Bands of the Yakama Nation for access to field locations where the data were collected. We also thank the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate data collection and optimization of data usage according to their values and worldview.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Accelerating Next-Generation Cybersecurity R&D Using AI Workflows: BADGER Project Development

The Broadband Automation for Distributed Grid Efficiency and Resilience (BADGER) project aligns with national strategic priorities for integrating emerging wireless technologies and advancing AI-driven security. As critical infrastructure modernizes toward increasingly software-defined and interconnected systems, the ability to leverage 5G/NextG networks and AI-enabled control becomes essential. This report outlines work at the National Laboratory of the Rockies (NLR) to develop a NextG-native security architecture powered by AI-RAN concepts and evaluate workflows that enable efficient and reliable architectures. Together, these efforts position the laboratory to accelerate innovation while directly supporting national security and resilience objectives.

5G/6G↗
DOE OSTI2026

AI-Ready Data Pilot Project Report

The proliferation of artificial intelligence in scientific research has created an urgent need to define "AI-ready data" for researchers and, more importantly, provide resources to help them produce AI-ready data. At Pacific Northwest National Laboratory, we conducted a pilot study with three data scientists evaluating three CSV datasets from different scientific domains, followed by semi-structured interviews capturing assessment practices. Our findings reveal that AI-readiness evaluation is intuition-based, with practitioners asking "How fast can I go from raw data to my machine learning pipeline?" Data scientists consistently prioritized workflow efficiency, human interpretability, and quality stewardship signals. From these insights, we developed a practical evaluation framework comprising data requirements, metadata standards, and validation tests that provides actionable criteria for producing and curating AI-ready datasets, addressing the gap between theoretical understanding and practical implementation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

JACC.shared: Leveraging HPC Metaprogramming and Performance Portability for Computations That Use Shared Memory GPUs

In this work, we present JACC.shared, a new feature of Julia for ACCelerators (JACC), which is the performanceportable and metaprogramming model of the just-in-time and LLVM-based Julia language. This new feature allows JACC applications to leverage the high-performance computing (HPC) capabilities of high-bandwidth, on-chip GPU memory. Historically, exploiting high-bandwidth, shared-memory GPUs has not been a priority for high-level programming solutions. JACC.shared covers that gap for the first time, thereby providing a highlevel, portable, and easy-to-use solution for programmers to exploit this memory and supporting all current major accelerator architectures. Well-known HPC and AI workloads, such as multi/hyperspectral imaging and AI convolutions, have been used to evaluate JACC.shared on two exascale GPU architectures hosted by some of the most powerful US Department of Energy supercomputers: Perlmutter (NVIDIA A100) and Frontier (AMD MI250X). The performance evaluation reports speedup of up to 3.5× by adding only one line of code to the base codes, thus providing important accelerators in a simple, portable, and transparent way and elevating the programming productivity and performance-portability capabilities for Julia/JACC HPC, AI, and scientific applications.

Valero Lara, Pedro [ORNL] (ORCID:0000000214794310)↗
DOE OSTI2024

Outcomes of HPC User Support using a Science Gateway AI Assistant

High Performance Computing (HPC) is a vital resource for nuclear energy research, facilitating advanced simulations and complex modeling of the quantification and qualification of advanced reactor technology. However, a common gap in knowledge exists around utilizing HPC systems, particularly for nuclear energy researchers unfamiliar with specific HPC systems. A researcher may be well-versed in using one HPC system and understanding its associated processes. Yet, they might struggle when faced with a different HPC system and its unique processes. HPC support staff play a crucial role in addressing these challenges by providing educational resources and assisting users. However, they also face the challenge of maintaining these systems and ensuring they run efficiently for all users, a responsibility that can be challenging to scale effectively with the increasing demand and expansion of HPC systems. This paper addresses this knowledge gap with an artificial intelligence (AI) assistant that offers on-demand, site-specific HPC support for researchers. Idaho National Laboratory (INL) has deployed an AI assistant that is intended to supplement expert HPC support staff and assist nuclear energy researchers. This paper reports on a four-and-a-half-month study evaluating the integration of an AI assistant within a science gateway, with the goal of enhancing existing HPC support.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Transforming Science Prioritization Processes Using Artificial Intelligence

Artificial Intelligence (AI) and Machine Learning (ML) have potential to augment significantly the current labor-intensive processes of science prioritization, specifically by the National Academies’ Decadal Survey on behalf of NASA and NSF. Here we summarize what we believe to be the first exploratory demonstration-of-concept results from an application of AI/ML to Survey science prioritization. Specifically, we applied Latent Dirichlet Allocation (LDA) and Natural Language Processing (NLP) to reveal trends in published astrophysics research that may indicate science priorities and which could be applied to strategic planning. For the purpose of the work that we summarize here, AI/ML is able to analyze – that is, to “understand,” in a manner of speaking – a vast amount of text to reveal complex relationships among research topics, including the growth or decline of science community activities in those topics over time. We trained ourselves and AI/ML algorithms by using ~400,000 abstracts in the period 1998 to 2010 to “forecast” the Academies’ Astro2010 recommendations and compare with the solicited white papers. Comparing our results with actual Astro2010 recommendations allowed us to identify candidate metrics that better predicted the actual results of the Survey. We found, for example, that Compound Annual Growth Rate (CAGR) of papers published in a topic area is a good proxy measure for importance of this topic area of research. With this training complete, we identified candidate astrophysics astrophysics science priorities for the 2021+ period using the research during 2007 - 2019 . We conclude that appropriate application of AI can potentially significantly reduce the current workload of the Decadal Survey processes and reveal otherwise unrecognized characteristics in the body of astronomical research. We emphasize throughout the exploratory nature of our work, encouraging colleagues to pursue promising results further. Our most critical governing assumption was that increased (or decreased) research activity can be used to identify scientific or technology topic areas worthy of increased (or decreased) future emphasis. We discuss advantages, limitations, and recognize the “black box” nature of our technique. We note ethics issues associated, for example, with using AI/ML to reveal “hidden” meanings and biases in published work. Furthermore, inevitable improvements in AI may soon enable widespread and welcome identification of and advocacy for science and technology priorities by disparate and diverse groups and organizations. Consequently, we continue to urge a near-term, in-depth evaluation of appropriate applications of AI, including implications and consequences, as well as support for multiple follow-on assessments, of which ours is only a beginning.

Artificial Intelligence↗
DOE OSTI2026

Robust Explanations using Diverse Adversarially Trained Ensembles, Multi-Modal Contrastive Learning, and Attribution-based Confidence Metrics

The primary objective of this project is to strengthen the trustworthiness of AI systems by designing algorithms that make their internal decision-making processes more understandable to human users. This involves creating clear, interpretable explanations for AI decisions and developing metrics to assess these explanations' validity and reliability. Significant progress has been achieved through (i) developing symbolic explanations, (ii) generating meaningful interpretive insights, (iii) establishing accuracy and confidence metrics, and (iv) devising methods to evaluate the knowledge boundaries of AI models. To date, the research findings have been shared in peer-reviewed publications, with accompanying scientific and technical information (STI) detailed below.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Leveraging Artificial Intelligence in Federal Projects

Artificial intelligence (AI) has the potential to transform grid operations. As energy demand rises, weather patterns shift, and foreign threats to critical infrastructure grow, it is essential to harness advanced technology to modernize the grid and increase overall resiliency. This paper examines the integration of AI in federally funded grid infrastructure projects. By analyzing project data, it evaluates the penetration and use cases of AI technologies within key funding initiatives and explores opportunities and challenges associated with their deployment. This analysis categorizes AI adoption in recent federal grid investments, establishing a baseline for measuring near-term impacts and identifying promising AI applications. The research found that 16% of recent federal energy projects included AI integration, and approximately 75% of these projects used AI for more than one primary application.

24 - POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Data readiness pipeline patterns for scientific AI at scale: Insights from climate, fusion, life sciences, and materials

This article examines how data readiness for AI principles apply to large scientific datasets used to train foundation models. We analyze archetypal workflows across four representative domains—climate, nuclear fusion, life sciences, and materials—to identify common preprocessing patterns and domain‐specific constraints. We introduce a two‐dimensional readiness model that combines canonical preprocessing patterns with a five‐level operational readiness scale, both tailored to high‐performance computing (HPC) environments. This construct helps outline key challenges in transforming large‐scale scientific data into formats suitable for scalable AI training. Together, these dimensions form a conceptual maturity matrix that characterizes scientific data readiness and guides infrastructure development toward standardized, cross‐domain support for scalable and reproducible AI for science. Finally, we evaluate this maturity matrix in the context of case studies including ClimaX (climate), AFLOW (materials), OpenFold (proteomics), and DIII‐D fusion disruption‐prediction workflows, from which we distill lessons learned and provide recommendations to guide practitioners in developing robust AI‐readiness pipelines. Finally, we discuss remaining cross‐cutting challenges that persist across scientific domains.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Agentic AI vs ML-Based Autotuning: A Comparative Study for Loop Reordering Optimization

High Performance Computing (HPC) applications rely heavily on code optimizations to achieve good performance on modern CPU and GPU architectures. Traditional Machine Learning auto-tuning approaches have demonstrated success in exploring high-dimensional spaces, but they often require expensive compile-run evaluations and lack adaptability for large HPC applications. The recent advances in Large Language Models (LLMs) and Agentic AI systems raise intriguing questions about the potential of these approaches to address specific optimization methodologies. This work aims to answer an essential question for the HPC community: “How Agentic AI Systems Compare to Traditional ML Autotuning Techniques?” To address this question, we present a comparative analysis between a traditional ML-based optimization approach and an Agentic AI system, evaluating their respective capabilities and limitations for loop-level optimization. In addition, we introduced a new Agentic AI system named LoopGen-AI using three different Large Language Models: GPT-4.1, Claude 4.0, and Gemini 2.5. A key finding is that LoopGen-AI achieves competitive per-formance with only a few program runs, the reasoning logs from the agents revealed that their decisions rely heavily on the combination of semantic understanding of the target kernel with dynamic feedback from the environment, highlighting a promising new dimension in performance tuning. In contrast, ML-based autotuners focus on statistical exploration, and require orders of magnitude more runs to reach peak performance. Additionally, our analysis shows that prompt engineering, particularly using Persona + Context Manager patterns, significantly impacts the effectiveness of Agentic AI. Our results indicate that while Agentic AI systems are not yet a complete replacement for ML-based autotuners, it can effectively complement traditional methods.

Rosas, Miguel Romero↗
DOE OSTI2025

Moving beyond post hoc explainable artificial intelligence: a perspective paper on lessons learned from dynamical climate modeling

AI models are criticized as being black boxes, potentially subjecting climate science to greater uncertainty. Explainable artificial intelligence (XAI) has been proposed to probe AI models and increase trust. In this review and perspective paper, we suggest that, in addition to using XAI methods, AI researchers in climate science can learn from past successes in the development of physics-based dynamical climate models. Dynamical models are complex but have gained trust because their successes and failures can sometimes be attributed to specific components or sub-models, such as when model bias is explained by pointing to a particular parameterization. We propose three types of understanding as a basis to evaluate trust in dynamical and AI models alike: (1) instrumental understanding, which is obtained when a model has passed a functional test; (2) statistical understanding, obtained when researchers can make sense of the modeling results using statistical techniques to identify input–output relationships; and (3) component-level understanding, which refers to modelers' ability to point to specific model components or parts in the model architecture as the culprit for erratic model behaviors or as the crucial reason why the model functions well. We demonstrate how component-level understanding has been sought and achieved via climate model intercomparison projects over the past several decades. Such component-level understanding routinely leads to model improvements and may also serve as a template for thinking about AI-driven climate science. Currently, XAI methods can help explain the behaviors of AI models by focusing on the mapping between input and output, thereby increasing the statistical understanding of AI models. Yet, to further increase our understanding of AI models, we will have to build AI models that have interpretable components amenable to component-level understanding. We give recent examples from the AI climate science literature to highlight some recent, albeit limited, successes in achieving component-level understanding and thereby explaining model behavior. The merit of such interpretable AI models is that they serve as a stronger basis for trust in climate modeling and, by extension, downstream uses of climate model data.

54 ENVIRONMENTAL SCIENCES↗