Search NASA⌕ Search

SEARCH · Search NASA

Results for “tool evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Green-Field Geothermal/Solar Designs (Cooperative Research and Development Final Report)

Geothermal power (GEO) and concentrating solar power (CSP) can be integrated to obtain the best aspects of both systems. The green-field design will utilize the full potential of the higher-temperature CSP sector by having a topping solar steam turbine and a bottoming geothermal cycle. In the proposed hybrid cycle, geothermal energy can be converted into electricity at a higher cycle efficiency because of the high-temperature topping cycle during the daytime. CSP benefits from lower-cost solar collectors, a less expensive topping turbine, and virtual elimination of the heat rejection system cost. The green-field case study will be focused on a geothermal resource site which will deliver the 160 - 220 °C pressurized brine at the inlet of power block. While the case study will look at a single case study, the tools to be developed as part of the project will be available to model annual performance of GEO/CSP hybrids for new greenfield projects (may also be applied to certain types of brown-field retrofits). The evaluation tools will also calculate important economic parameters such as the levelized cost of energy (LCOE) and the internal rate of return (IRR).

14 SOLAR ENERGY↗

Retrofit Energy Analysis and Central Thermal modeling (REACT) v1.0

REACT is a website designed to simplify the analysis and decision-making process for retrofitting existing central plant heating and cooling systems with advanced heat pump technologies. The tool evaluates the technical and economic viability of replacing traditional boilers with various options including water-to-water or air-to-water heat pumps, which can provide efficient and lower-cost heating and cooling. It allows users to compare current central plant configurations with retrofit scenarios, assessing energy consumption, life-cycle costs, and environmental impact. Retrofitting traditional boiler and chiller systems with water-to-water or air-to-water heat pumps can significantly reduce energy consumption and lower lifecycle costs. The REACT provides: User-Friendly Tools: A user friendly web interface for quick, intuitive analysis accessible to non-experts. Advanced Modeling: A Python-powered engine for detailed parametric studies, optimization, and research applications. Comprehensive Analysis: Lifecycle cost evaluation, energy consumption modeling, and environmental impact assessment. Visual Insights: A variety of plots to visualize system performance and design trade-offs. The engine for the website (REACT) bases on several Python libraries, and the website will be hosted on an ETA server.

Kim, Donghun [Lawrence Berkeley National Laborator↗

Evaluating Machine Learning-Based MRI Reconstruction Using Digital Image Quality Phantoms

Quantitative and objective evaluation tools are essential for assessing the performance of machine learning (ML)-based magnetic resonance imaging (MRI) reconstruction methods. However, the commonly used fidelity metrics, such as mean squared error (MSE), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR), often fail to capture fundamental and clinically relevant MR image quality aspects. To address this, we propose evaluation of ML-based MRI reconstruction using digital image quality phantoms and automated evaluation methods. Our phantoms are based upon the American College of Radiology (ACR) large physical phantom but created in k-space to simulate their MR images, and they can vary in object size, signal-to-noise ratio, resolution, and image contrast. Our evaluation pipeline incorporates evaluation metrics of geometric accuracy, intensity uniformity, percentage ghosting, sharpness, signal-to-noise ratio, resolution, and low-contrast detectability. We demonstrate the utility of our proposed pipeline by assessing an example ML-based reconstruction model across various training and testing scenarios. The performance results indicate that training data acquired with a lower undersampling factor and coils of larger anatomical coverage yield a better performing model. The comprehensive and standardized pipeline introduced in this study can help to facilitate a better understanding of the performance and guide future development and advancement of ML-based reconstruction algorithms.

47 OTHER INSTRUMENTATION↗

SetGo: Metadata Readiness for Scientific AI Datasets

Scientific datasets intended for AI use require both computational readiness for model training and metadata readiness for discovery, sharing, and reuse. The Readiness Engine for Data Integration (REDI) addresses computational readiness, but no corresponding tool evaluates whether a dataset’s metadata are sufficiently complete, governed, and standards-compliant for publication and agent-based consumption. Existing FAIR assessors operate only on published repository records, and no single system covers FAIR compliance, licensing, provenance, governance, reproducibility, and catalog readiness together. We present SetGo, an open-source Python toolkit that assesses and repairs metadata readiness across these six dimensions before a dataset is published or archived. Applied to four scientific corpora, SetGo surfaces deficiencies that general-purpose tools do not detect: ERA5 climate metadata scores 4% on ACDD 1.3 compliance; materials datasets fail OPTIMADE species-definition requirements; and PDB-derived proteomics data carries licensing terms incompatible with standard SPDX identifiers. Guided enrichment raises overall FAIR scores from 52–57% to 81–91%, and a single setgo publish command pushes to Hugging Face Hub, CKAN, or OpenMetadata with ML Commons Croissant 1.0 metadata sidecars. To support interactive and automated workflows, SetGo integrates with coding agents powered by large language models (LLMs) through a /setgo skill that enables natural-language execution of the full assess–enrich–publish loop, with user involvement limited to supplying missing metadata values.

Wilkinson, Sean [ORNL] (ORCID:0000000214437479)↗

Hydropower Biological Evaluation Toolset (HBET) Version 3.0: User Guide

The Hydropower Biological Evaluation Tools (HBET) software package, developed by Pacific Northwest National Laboratory (PNNL), is designed to assemble, organize, and process data collected by Sensor Fish and live fish. HBET enable users to characterize the hydraulic conditions of hydropower structures and estimate fish injury and mortality rates from various stressors. Future updates of the software may support other technologies, such as bead tracking in physical models and computational fluid dynamics. The HBET program can be customized to analyze different hydraulic applications, including turbines, spillways, weirs, pumped storage, and other user-defined functions, and therefore, help researchers, turbine designers, hydropower operators, and regulators better evaluate hydropower structures regarding their environmental sustainability and cost-effectiveness. Added content to the user guide about the new feature for predicting absolute injury rates.

13 HYDRO ENERGY↗

Southwest Regional Partnership on Carbon Sequestration: Phase III (Final Scientific/Technical Report)

The Southwest Regional Partnership on Carbon Sequestration (SWP) is one of 7 regional partnerships formed in 2003 under the U.S. Department of Energy’s (DOE) Regional Carbon Sequestration Partnerships (RCSPs) initiative. The overall purpose of the initiative was to help determine and implement the technology, infrastructure, and regulations most appropriate to promote carbon storage in different regions of the country. Covering Arizona, Colorado, New Mexico, Oklahoma, Utah, and parts of Texas, Wyoming, and Kansas, the SWP evaluated regional carbon storage and utilization potential and focused on technologies and sites that could complement the region’s strong position in energy production. The project progressed through three phases: • Phase I (2003–2005): Characterized regional geologic formations and CO 2 sources, assessed sequestration potential, and identified pilot test sites. • Phase II (2005–2013): Conducted small-scale field tests to validate sequestration methods, including geologic and terrestrial projects. • Phase III (2008–2022): Demonstrated large-scale CO 2 injection at a commercial oil field to test monitoring, verification, and long-term storage strategies. This report covers Phase III. The final project site, the Farnsworth Unit (FWU) in Texas, provided real-world testing of reservoir characterization, monitoring, and risk evaluation tools and processes that could be used in any commercial scale carbon capture, utilization, and storage (CCUS) project. Extensive data collection and analysis helped refine best practices for reservoir characterization, injection monitoring, and storage verification. The SWP contributed to national databases, DOE best practice manuals, and regional geological assessments to support future sequestration efforts. Key lessons learned include the importance of robust data management, strategic site selection, regulatory navigation, and effective industry collaboration. The project’s findings will inform ongoing and future carbon storage initiatives. Task 1 (Regional Characterization) • The SWP continued to participate in national outreach efforts and NATCARB. • The SWP evaluated multiple potential sites before selecting the FWU as the primary field test location. Task 2 (Public Outreach and Education) • The SWP contributed to national databases, DOE best practice manuals, and regional geological assessments to support future sequestration efforts. Task 3 (Permitting and Regulatory Compliance) • The SWP ensured compliance with federal and state regulations, including National Environmental Policy Act (NEPA) requirements. • The SWP obtained all necessary permits for drilling, injection, and monitoring activities. Task 4 (Site Characterization and Planning) • The SWP developed work plans for four key activities: characterization, simulation, monitoring and verification, and risk evaluation. • The SWP collected and synthesized legacy data from multiple sources to build initial static geological models and dynamic reservoir models demonstrating project feasibility. • The SWP conducted an initial risk evaluation and developed mitigation plans. Task 5 (Field Operations and Data Collection) • The SWP drilled, logged, and cored three characterization wells to gather critical subsurface data. • The SWP conducted multiple geophysical surveys, including 3D seismic, crosswell seismic, and vertical seismic profiling, to improve reservoir characterization. Task 6 (Monitoring and Verification) • The SWP performed extensive geological characterization using data from characterization wells and seismic surveys. • The SWP established a surface monitoring network to track CO 2 flux in soil gas, groundwater chemistry, and near-surface atmospheric CO 2 levels. • The SWP built and refined reservoir models to study the effects of relative permeability on simulation behavior and improve calibration with experimental data. Task 7 (Risk Assessment and Model Refinement) • The SWP conducted multiple studies to evaluate reservoir integrity, predict CO 2 plume behavior and improve predictive modeling capabilities. • The SWP refined geological models and used them to enhance the accuracy of simulation models. • The SWP continued quantitative risk assessment of top-ranked risks and strengthened the link between qualitative and quantitative risk methodologies.

02 PETROLEUM↗

pyEF: A Python Framework for QM and QM/MM Atom-Wise Electric Field Analysis

We introduce pyEF, a software package for computing molecular electric fields, electrostatic interaction energies, and electrostatic potentials from quantum mechanical (QM) atom-centered multipole expansions with atom-wise decomposable contributions. We demonstrate the computational efficiency and accuracy of this QM-derived electric field evaluation tool through several tests. To assess the influence of the underlying QM method and charge partitioning scheme on these electrostatic quantities, we analyze over 250 configurations of an acetone solute molecule in five solvents of variable polarity. We find that electric field calculations are highly sensitive to the choice of charge partitioning method. Even among real-space charge schemes, acetone Stark tuning rates differ by up to a factor of 2. Benchmarking computed solvent dipole moments against experimental bulk values, we conclude that the CM5, ADCH, and Hirshfeld-I charge schemes most reliably capture solvent electrostatics and therefore provide a more faithful foundation for computing electric fields. When constructed from these real-space charges, electric fields are nearly insensitive to basis set size and monotonically increase in magnitude with higher Fock exchange. We also demonstrate efficient convergence of QM electrostatics when more distant molecules are represented solely by MM point charges, reducing computational overhead. Leveraging these findings, we demonstrate the use of pyEF to deduce environmental effects on a transition metal complex from a Ga 4 L 6 12– nanocage and quantify the dominant role of organic linkers in orchestrating electrostatic preorganization.

electric fields↗

pnnl/soda-benchmarks

The SODA-Benchmarks repository (currently hosted at: https://gitlab.pnnl.gov/sodalite/soda-benchmarks) provides a comprehensive benchmark suite for evaluating tools within the SODA (Software-Defined Accelerators) toolchain, which focuses on hardware/software co-design and accelerator generation for FPGAs and ASICs

Agostini, Nicolas [Pacific Northwest National Labo↗

Differential Privacy in Grid Kitchen: Implementation & Software Documentation

Sharing of power grid feeder models faces significant challenges due to the potential risk of exposing sensitive operational information. Traditional anonymization techniques have shown notable limitations in other sensitive domains, as evidenced by documented re-identification attacks that combine supposedly anonymized datasets with auxiliary information, raising concerns that similar vulnerabilities could affect power grid data. Consequently, there is a pressing need for a more rigorous privacy protection strategy that not only delivers formal mathematical guarantees but also preserves the analytical value of the shared models. To address this challenge, we have enhanced the Grid Kitchen framework by implementing differential privacy mechanisms within the distribution model dehydration pipeline. This implementation carefully calibrates and applies noise to sensitive attributes in feeder models according to configurable privacy levels—low, moderate, and high—each offering different balances between data utility and privacy protection. Our approach uses established noise functions (Gaussian for continuous data and Discrete Laplace for integer values) with parameters carefully calibrated so that the impact of individual data points is effectively masked in the final output. The integration leverages our Noise Catalog, which we developed to categorize feeder model properties by component type, data type, and sensitivity. This catalog guides the application of appropriate noise functions and privacy parameters ($\varepsilon$ and $\delta$) to each attribute, ensuring consistent privacy protection across the model while maintaining its structural integrity and analytical usefulness. This implementation also includes evaluation tools that allow model owners to assess the impact of privacy-preserving transformations before sharing data with external parties. This report provides documentation for the differential privacy capabilities added to the Grid Kitchen project. It includes a primer on differential privacy concepts and their importance in modern data sharing, details the architecture of our implementation, explains the privacy modes and parameter configurations, and offers practical guidance on using the code for applying differential privacy to grid feeder models. Through examples and code snippets, we demonstrate the effective application of these privacy-enhancing technologies, enabling utility operators and researchers to confidently share grid data while protecting sensitive information.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Evaluating Economic Impact: An Investment Tool for Large Language Model Integration in Workweek Management

This paper explores the development and application of an investment tool designed to quantify the costs and potential savings associated with integrating large language models (LLMs) into work week management optimization (WMO) within the nuclear industry. LLMs, with their advanced natural language processing capabilities, can significantly enhance various aspects of work management, such as problem identification, prioritization, planning, scheduling, information retrieval, and information summary. Our investment tool focuses on evaluating the return on investment (ROI) for LLM applications in WMO by considering four pivotal decision factors: model selection, application, user training, and hosting options. This paper details the development and implementation of the ROI model and illustrates its application through multiple case studies, analyzing the impact of different variables, such as work time saved, number of requests, and model performance, on the computed ROI over two years. The computed ROI is also compared over different hosting solutions. Our findings indicate that ROI increases with enhanced work time savings and optimal request load but can decline with high request volumes or increased model costs. This model aids decision-makers in the nuclear industry by providing a structured approach to assessing the economic viability and potential savings from integrating LLMs into WMO processes.

97 - MATHEMATICS AND COMPUTING↗

Evaluating Economic Impact: An Investment Tool for Large Language Model Integration in Workweek Management

This paper explores the development and application of an investment tool designed to quantify the costs and potential savings associated with integrating large language models (LLMs) into work week management optimization (WMO) within the nuclear industry. LLMs, with their advanced natural language processing capabilities, can significantly enhance various aspects of work management, such as problem identification, prioritization, planning, scheduling, information retrieval, and information summary. Our investment tool focuses on evaluating the return on investment (ROI) for LLM applications in WMO by considering four pivotal decision factors: model selection, application, user training, and hosting options. This paper details the development and implementation of the ROI model and illustrates its application through multiple case studies, analyzing the impact of different variables, such as work time saved, number of requests, and model performance, on the computed ROI over two years. The computed ROI is also compared over different hosting solutions. Our findings indicate that ROI increases with enhanced work time savings and optimal request load but can decline with high request volumes or increased model costs. This model aids decision-makers in the nuclear industry by providing a structured approach to assessing the economic viability and potential savings from integrating LLMs into WMO processes.

99 - GENERAL AND MISCELLANEOUS↗

CSP Systems Analysis 2022-2024 (Final Technical Report)

The CSP Systems Analysis project estimates the current market cost of CSP subsystems and technologies for use by the DOE Program and within NREL analysis tools. The project addresses upgrades to NREL's System Advisor Model (SAM) related to CSP, as well as development of new modules within SAM to expand the types of CSP systems that can be simulated and new tools for evaluating CSP subsystem performance or optimizing CSP system layouts, for example, SolarPILOT and SolTrace. The project evaluates the potential cost and performance of new CSP-relevant technologies and generally includes assessment of CSP technologies in support of the DOE subprogram. A primary goal of the CSP System Analysis project is to provide timely and accurate cost data to the DOE to assess the current state of CSP technologies as well as predict performance and cost for pre-commercial and emerging technologies that may impact the industry. Prior work in this task has produced technology cost reports and roadmaps for DOE and highlighted promising research paths that could reduce overall CSP cost, such as the supercritical CO2 power cycle and molten-salt trough systems. CSP Systems Analysis produces modeling tools to evaluate the performance of CSP systems, subsystems, or components, and journal articles, conference papers, technical reports and databases to disseminate information to the CSP industry. CSP technology costs are estimated and tracked using these tools as well as analysis of literature and industry sources. The CSP modeling tools and NREL analysis results are used by researchers; government, financial, and industry analysts; and other CSP stakeholders. The purpose of this final technical report is to summarize the key accomplishments of the past three years of this project. The project was divided into seven separate tasks, and in what follows, we describe the background, objectives, key accomplishments, and path forward for each task within the project, with one section of this report summarizing each task.

14 SOLAR ENERGY↗

Comparative Analysis of TCR and TCR-pMHC Complex Structure Prediction Tools

The rapid development of computational approaches for predicting the structures of T cell receptors (TCRs) and TCR-peptide-major histocompatibility (TCR-pMHC) complexes, accelerated by AI breakthroughs such as AlphaFold, has made it feasible to calculate these structures with increasing accuracy. Although these tools show great potential, their relative accuracy and limitations remain unclear due to the lack of standardized benchmarks. Here, we systematically evaluate seven tools for predicting isolated TCR structures together with six tools for predicting TCR-pMHC complex structures. The methods include homology-based approaches, general prediction tools using AlphaFold, TCR-specific tools derived from AlphaFold2, and the newly developed tFold-TCR model. The evaluation uses a post-training data set comprising 40 αβ TCRs and 27 TCR-pMHC complexes (21 Class I and 6 Class II). Model accuracy is assessed at global, local, and interface levels using a variety of metrics. We find that each tool offers distinct advantages in various aspects of its predictions. AlphaFold2, AlphaFold3, and tFold-TCR excel in overall accuracy of TCR structure prediction, and TCRmodel2 and AlphaFold2 perform well in overall accuracy of TCR-pMHC structure prediction. However, TCR-specific tools derived from AlphaFold2 show lower accuracy in the framework region than both homology-based methods and general-purpose tools such as AlphaFold, and challenges remain for all in modeling CDR3 loops, docking orientations, TCR-peptide interfaces, and Class II MHC-peptide interfaces. Furthermore, these findings will guide researchers in selecting appropriate tools, emphasize the importance of using multiple evaluation metrics to assess model performance, and offer suggestions for improving TCR and TCR-pMHC structure prediction tools.

Chemical structure↗

Data-driven Community-centered Resilient Assessment and Planning Toolkit for Nexus of Energy and Water (DCRAPT-NEW)

Urban areas, including Detroit and Pittsburgh, have suffered significant dual outages of the electrical and water infrastructure in the past decade due, in part, to the increasing number of extreme weather events. With increasing temperatures and rainfall intensity, these regions need to prepare for increasing extreme events through community-based energy and water resilience analysis, planning, and enhancement. This project developed a suite of open-source, open-access, community-centered, data-driven assessment and distributed energy resource (DER) and planning tools for energy and water resilience enhancement in urban areas. Through establishing a multi-level community awareness and engagement mechanism and a comprehensive collection of power outage and flooding data, an innovative group of community energy and water resilience assessment and planning tools have been developed for a wide range of users with differing and variable sets of data available to them. The developed tools include (1) DOE EAGLE-I data-driven, deep-learning assisted resilience assessment and DER planning tools at the county level with socioeconomic factors incorporated; (2) Utility annual power outage data-driven tools for long term resilience assessment and DER planning and 15-min power outage data-driven tools for short term resilience assessment and planning; (3) Detailed engineering tools for energy and water systems resilience assessment and planning when the system topology and component fragility curves are available; (4) Alternative Resiliency Metric Calculation that extracts and separates outage and restoration processes; and (5) Co-optimization tools that evaluate the resilience of the power and sewage system and allow users to conduct joint planning with energy and wastewater systems. The developed tools provide planners, decision-makers, and stakeholders with powerful capabilities to systematically evaluate system/community resilience and optimal and actionable guidance for enhancing resilience while prioritizing DER investments. The tools have been used and validated in Detroit and Pittsburgh and can be used in other areas of the nation. In addition, this project will (1) advance the knowledge and applications of machine-learning methods in analyzing and fusing different layers of information and generating meaningful data points such as generating rare weather events; (2) significantly improve the energy and water resilience of the identified communities in Detroit and Pittsburgh and prepare for more frequent and severe weather conditions; (3) help communities assess extreme weather event impacts and address short-term and long-term resilience-related issues The developed tools have been made public via GitHub and demonstrated to community stakeholders and utility companies via the two annual workshops and numerous community engagement meetings. The project outcomes are also disseminated through publications in various journals and conference proceedings, and presentations at top conferences.

13 HYDRO ENERGY↗

BLEECAM™ (Benchmarking Life Cycle Environmental, Economic, and Social Metrics for Critical and Advanced Minerals and Materials) [SWR-25-125]

The National Laboratory of the Rockies' (NLR) Benchmarking Life Cycle Environmental, Economic, and Social Metrics for Critical and Advanced Minerals and Materials (BLEECAM™) is an open-source, integrated decision-support tool for evaluating the impacts, risks, and trade-offs across U.S. and global materials supply chains. Funded by the U.S. Department of Energy, BLEECAM supports supply chain and market analysis. The tool integrates multi-objective supply chain optimization, system dynamics, network design, lifecycle assessment, techno-economic modeling, and social impact assessment methods to evaluate how supply chains evolve over time, geography, and deployment scenarios. BLEECAM also supports analysis related to energy infrastructure, data centers and digital infrastructure, advanced manufacturing, and other sectors that depend on critical materials.

Khalifa, SherifA. [National Laboratory of the Rock↗

REopt: Energy Decision Analysis Overview for NCSP+ States Collaborative [Slides]

This presentation offers an overview of NREL's REopt(R) web tool for the National Community Solar Partnership+ States Collaborative. REopt is a techno-economic modeling tool for evaluating distributed energy resources toward a site's cost savings and resilience goals. The presentation includes an overview of the REopt platform, a REopt web tool demonstration, and resources to learn more about REopt.

14 SOLAR ENERGY↗

Automated Signal Timing Plan Reconstruction Using High-Resolution Event-Based Controller Data for Digital Twins

Transportation digital twins are essential tools for evaluating emerging technologies such as connected and automated vehicles, adaptive traffic signal control, and mobility optimization strategies. Realistic digital twins require accurate emulation of real-world signal controllers and detailed signal timing plans. However, signal timing plans are often unavailable or difficult to access, forcing researchers and modelers to rely on assumed fixed timings or halt their analysis. To overcome this challenge, we present a method that directly estimates signal timing plan parameters using high-resolution, event-based data from traffic signal controllers. The proposed method extracts key parameters, including cycle length, offset, phase sequence, coordinated phases, phase-specific minimum and maximum green durations, vehicle extensions, and splits under coordination. A rule-based deterministic signal timing reconstruction algorithm based on traffic signal operation rules, such as those outlined in the Signal Timing Manual, is developed and validated. We evaluate this method, which uses high-resolution controller event logs and verified signal timing plans, on 94 signalized intersections in Nashville, Tennessee, demonstrating their ability to generate accurate, simulation-ready signal timing plans for tools such as SUMO and Vissim.

Saroj, Abhilasha [ORNL] (ORCID:0000000191178063)↗

Large language models for batteries

Large Language Models (LLMs) are advanced artificial intelligence systems capable of solving diverse tasks using language, reasoning, and external tools. Despite their growing deployment in academia and industry, their potential remains underexplored in battery research. This review presents a comprehensive overview of existing and emerging applications of LLMs in batterie field, addressing two critical questions: What can LLMs offer to support battery-related tasks, and how to develop more effective models for this purpose. We begin by outlining the principles of LLMs and criteria for selecting appropriate models and tools for battery research and development. We then explore their roles in text-mining, data interpretation, and the development of intelligent battery systems. In parallel, we discuss technical challenges, such as data standardizing and sharing, model evaluation, and tool integration. Lastly, we propose future research directions with short-, medium-, and long-term goals and highlight more broad perspectives for connecting experts and cross-disciplinary collaborations.

SoC↗