Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

In-Flight Evaluation of the Traffic Aware Planner on the NASA HU-25A Guardian Aircraft

NASA’s Traffic Aware Planner (TAP) software is a research-prototype decision support tool that provides pilots with time- and fuel-saving route recommendations that optimize their current trajectory. The software runs on a first-of-a-kind system architecture onboard three aircraft in revenue service conducting operational evaluations with a major domestic airline. Therefore, significant NASA-internal testing is required prior to releasing the software to the partner airline. This paper describes a flight test plan that exercises the functionality of the TAP software in a representative operational environment, describes the system architecture developed and implemented for the NASA Langley HU-25A Guardian aircraft to support the test objectives, presents outcomes of the flight test campaign, and discusses use cases that demonstrate the value of flight testing for this activity.Research into flight path optimization of transport aircraft conducted by the National Aeronautics and SpaceAdministration (NASA) has produced an operational concept known as Traffic Aware Strategic Aircrew Requests(TASAR) [1, 2]. This near-term concept [3] provides the aircrew with a flight deck decision support tool known asthe Traffic Aware Planner (TAP). The TAP software leverages a growing number of information sources on the flightdeck to make time- and fuel-saving route optimization recommendations to the aircrew while en route. The aircrewcan then use the suggestions provided by the tool to make route change requests with a greater likelihood of acceptanceby air traffic control (ATC). Since TASAR is a concept intended for the current operational environment, it isintentionally designed to have no safety-critical impact or require any changes to current Federal AviationAdministration (FAA) rules and procedures [4, 5].The research prototype TAP system [6–8], explained further in Section III.C, continually incorporates up-to-dateaircraft state data from onboard avionics, as well as the latest position of surrounding traffic, the most recent windforecast, and the most recent convective weather forecast, in order to calculate candidate trajectory modifications thatimprove upon the current active route. These trajectories account for user-selectable objective functions [3] of reducedfuel burn, reduced flight time, or an airline-derived combination of factors known as trip cost. Previous analyses andsimulations have estimated substantial savings for airlines employing this technique within the U.S. National AirspaceSystem (NAS) [9–11]. Operational evaluations with Alaska Airlines seek to validate these projected benefits usingmeasured data while simultaneously providing benefits to the airline [12, 13].The TAP software has undergone a number of human-in-the-loop simulations [14] and flight test activities[15–17] in order to validate the operational concept, evaluate human factors considerations (e.g., workload, usability,distraction, etc.), and to assess the ability of the software to function in a representative operational environment (e.g.,connected to live avionics data, using in-flight internet connectivity, etc.). However, these simulations and flight testcampaigns did not account for the hardware architecture implemented on the three aircraft for Alaska Airlines’operational evaluations of the TAP software. Therefore, a need was identified to thoroughly test the functionality ofthe software in a similar hardware architecture to that of the partner airline’s aircraft. Information regarding testapparatus and environments used to evaluate TAP prior to testing on the HU-25A can be found in reference [18].A campaign of flight trials on a NASA aircraft, the HU-25A Guardian, was conducted to ensure that the researchprototype TAP system functions well in a configuration similar to the Alaska Airlines aircraft prior to deployment.This airborne, networked environment enables an assessment of the operational factors unique to the flight environment. Additionally, this activity evaluated the effectiveness and benefit of new TAP functionality andoperation in a relevant flight environment while allowing the rapid prototyping of new concepts and features.This paper is organized as follows: Section II discusses the details of the flight test plan, flight profiles, and theduties of personnel involved with conducting flight operations. Section III describes the test platform, avionicsequipage, and system architecture. Section IV presents a discussion of results, and Section V contains concludingremarks.

Underwood, Matthew C.↗
NASA NTRS2009

Evaluation methodology guidance for stack packages

This report provides evaluation methodology guidance based on previous National Aeronautics and Space Administration (NASA) reports and literature surveys for 3D stack packages and assemblies. Two aspects of technology are covered: the package itself (guidance for functional packages) and package assembly manufacturing and reliability. This work was funded by the NASA Electronic Parts and Packaging (NEPP) Program. The objectives of this NEPP project are to: Perform a literature survey of 3D stack technology; perform a literature survey on the evaluation methodology for 3D package and assembly; combine the two aspects to provide evaluation methodology for both aspects with consideration of interactions between package and assembly; generate guidance on the evaluation methodology for 3D stack package integrity prior to and after assembly; provide recommendations on future experimental activities. The qualification and evaluation methodology guidelines will facilitate NASA projects in effectively evaluating the reliability of very dense and newly available high-density 3D stack packages, allowing more processing power in a smaller board footprint and lower system weight.

Gerke, David↗
NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or aerial wildfire operations reports to better understand the risks present. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement. These applications can benefit from the use of state-of-the-art natural language processing techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of adaptation of NLP tools to the aviation domain by fine-tuning transformer based models using aviation data sets. In 2018, a novel language model based on neural units (also called transformers) was created and became known as “Bidirectional Encoder Representations from Transformers” or BERT. This architecture combined with large amounts of English training data and innovative semi-supervised training tasks set the standard for what would later emerge as Large Language Models. The performance of these models was further improved by hyperparameter tuning and refinement of the semi-supervised training task and resulted in “Robustly Optimized BERT Pre-training Approach through hyperparameter tuning” or RoBERTa models. These pre-trained Large Language Models proved to be useful for a wide variety of natural language processing tasks such as text classification and question answering through a process called fine-tuning. The transformer architecture with pre-trained weights served as the basis with the last few layers replaced with layers fine-tuned to perform a new task e.g., a layer that provides a label for the entire input text. This process of fine-tuning can also be used to adapt the models to new domains; e.g., BioBERT started with the pre-trained BERT model and was completed by additional fine-tuning and training on biomedical documents. Transformer-based architectures can also be used to create rich representations of text called embeddings which can serve as the input to other machine learning models. This allows simpler algorithms such as logistic regression to use context-rich representations of the text while still remaining quick to train and evaluate. In the world of aviation, there is a growing demand for natural language processing and understanding but the domain presents unique challenges. Due to the technical content (and specialized language) of most aviation documents, fine-tuning pre-trained Large Language Models to specific tasks has not met the benchmark on natural language processing tasks set by simpler models trained from scratch on the data. To address this deficiency, this paper evaluates the improvements from fine-tuning a Large Language Model on a large set of aviation documents using the original semi-supervised training tasks before performing specific natural language tasks. In fine-tuning, a domain-specific dataset is used on the original training task but with the pre-trained Large Language Model instead of starting from a random initialization. This approach allows the model to be adapted to the specific domain language without discarding the information gained from training on general English data. This paper utilized two major dataset types to train and assess the RoBERTa fine-tuning performance. The first are 7,057 Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the semi-supervised fine-tuning. The second type is the 494 document classification labels to be used for evaluation. This down-stream evaluation aims to show the performance of the fine-tuned model, better understand how much data is needed for an effective fine-tuning, and how fine-tuning can be adapted for different applications in-the domain. After semi-supervised training, evaluation begins by encoding the documents for classification using the fine-tuned RoBERTa model. Then a logistic regression classifier is trained to label the document type and compared against our ground truth labels. This currently leads to a 82.8% accuracy on 10-fold cross validation showing improvement over baseline RoBERTa which achieved 81.0%. We plan to measure the improvements on additional tasks and it is expected that these improvements will lead to more robust models that can tackle the natural language processing challenges present in aviation datasets.

ATM↗
DOE OSTI2025

EVALUATION OF HRA METHODOLOGIES FOR APPLICATION IN SDP WORK

This study critically evaluates human reliability analysis (HRA) methodologies applicable to regulatory probabilistic safety assessment (PSA) model, with a particular focus on their role in supporting the significance determination process (SDP) in nuclear safety assessment. Firstly, three widely utilized HRA methods – IDHEAS-ECA, SPAR-H, and ASEP/THERP – were qualitatively and quantitatively assessed. Qualitative assessments were conducted using attributes from the NEA/CSNI/R(2015)1 report, while quantitative evaluations employed regression and correlation analyses to compare predicted human error probabilities (HEPs) against empirical data. Results reveal distinct strengths, for example, IDHEAS-ECA’s robust predictive accuracy and K-HRA’s alignment with operational practices. In addition, dependency analysis and recovery analysis were critically evaluated. For dependency analysis, the methods’ handling of inter-task dependencies and their impact on HEPs were examined, while recovery analysis highlighted strategies for mitigating failure events. Furthermore, strategies were proposed to evaluate performance-shaping factors under conditions of reduced human performance, such as stress, fatigue, or cognitive overload, addressing specific challenges faced in SDP evaluations. Human errors from KINS’s operational performance information system event reports were evaluated as a case study. This study identifies gaps and provides actionable insights to ensure their validity and applicability in SDP HRA applications. This paper is a part of research conducted by KINS, and it should be noted that this result does not represent the regulatory position of KINS.

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2024

Validating automated resonance evaluation with synthetic data

The integrity and precision of nuclear data are crucial for a broad spectrum of applications, from national security and nuclear reactor design to medical diagnostics, where the associated uncertainties can significantly impact outcomes. A substantial portion of uncertainty in nuclear data originates from the subjective biases in the evaluation process, a crucial phase in the nuclear data production pipeline. Recent advancements indicate that automation of certain routines can mitigate these biases, thereby standardizing the evaluation process and enhancing reproducibility. This research aims to provide a methodology, framework, and metrics for the validation of automated nuclear data evaluation software leveraging high-quality synthetic data that closely mimic real experimental observables. An introduced error metric provides a scale and intuitive measure of the evaluation quality by quantifying the estimate’s accuracy and performance across the specified energy range. Synthetic data provides access to experimental observables and underlying resonance parameters, enabling comparison of different evaluations. The methodology is demonstrated using Ta-181 isotope data in the resolved resonance region. The Automated Resonance Identification Subroutine (ARIS), which operates without prior resonance information, was used to test and showcase the framework’s capabilities utilizing the proposed error metrics. The results demonstrate the effectiveness of the proposed approach and framework for optimizing software parameters and testing hypotheses through “what-if” controlled experiments, such as modifying assumptions about experimental conditions or average resonance parameters.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Evaluating Machine Learning-Based MRI Reconstruction Using Digital Image Quality Phantoms

Quantitative and objective evaluation tools are essential for assessing the performance of machine learning (ML)-based magnetic resonance imaging (MRI) reconstruction methods. However, the commonly used fidelity metrics, such as mean squared error (MSE), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR), often fail to capture fundamental and clinically relevant MR image quality aspects. To address this, we propose evaluation of ML-based MRI reconstruction using digital image quality phantoms and automated evaluation methods. Our phantoms are based upon the American College of Radiology (ACR) large physical phantom but created in k-space to simulate their MR images, and they can vary in object size, signal-to-noise ratio, resolution, and image contrast. Our evaluation pipeline incorporates evaluation metrics of geometric accuracy, intensity uniformity, percentage ghosting, sharpness, signal-to-noise ratio, resolution, and low-contrast detectability. We demonstrate the utility of our proposed pipeline by assessing an example ML-based reconstruction model across various training and testing scenarios. The performance results indicate that training data acquired with a lower undersampling factor and coils of larger anatomical coverage yield a better performing model. The comprehensive and standardized pipeline introduced in this study can help to facilitate a better understanding of the performance and guide future development and advancement of ML-based reconstruction algorithms.

47 OTHER INSTRUMENTATION↗
NASA NTRS2003

A new method to evaluate human-robot system performance

One of the key issues in space exploration is that of deciding what space tasks are best done with humans, with robots, or a suitable combination of each. In general, human and robot skills are complementary. Humans provide as yet unmatched capabilities to perceive, think, and act when faced with anomalies and unforeseen events, but there can be huge potential risks to human safety in getting these benefits. Robots provide complementary skills in being able to work in extremely risky environments, but their ability to perceive, think, and act by themselves is currently not error-free, although these capabilities are continually improving with the emergence of new technologies. Substantial past experience validates these generally qualitative notions. However, there is a need for more rigorously systematic evaluation of human and robot roles, in order to optimize the design and performance of human-robot system architectures using well-defined performance evaluation metrics. This article summarizes a new analytical method to conduct such quantitative evaluations. While the article focuses on evaluating human-robot systems, the method is generally applicable to a much broader class of systems whose performance needs to be evaluated.

Space Flight/instrumentation↗
NASA NTRS1992

Utilization of potatoes for life support in space. V. Evaluation of cultivars in response to continuous light and high temperature

Twenty-four potato (Solanum tuberosum L.) cultivars from different regions of the world were evaluated in terms of their responses to continuous light (24 h photoperiod) and to high temperature (30 C) in two separate experiments under controlled environments. In each experiment, a first evaluation of the cultivars was made at day 35 after transplanting, at which time 12 cultivars exhibiting best growth and tuber initiation were selected. A final evaluation of the 12 cultivars was made after an additional 21 days of growth, at which time plant height, total dry weight, tuber dry weight, and tuber number were determined. In the continuous light evaluation, the 12 selected cultivars were Alaska 114, Atlantic, Bintje, Denali, Desiree, Haig, New York 81, Ottar, Rutt, Snogg, Snowchip, and Troll. In the high temperature evaluation, the 12 selected cultivars were Alpha, Atlantic, Bake King, Denali, Desiree, Haig, Kennebec, Norland, Russet Burbank, Rutt, Superior, and Troll. Among the cultivars selected under continuous irradiation, Desiree, Ottar, Haig, Rutt, Denali and Alaska showed the best potential for high productivity whereas New York 81 and Bintje showed the least production capability. Among the cultivars selected under high temperature, Rutt, Haig, Troll and Bake King had best performance whereas Atlantic, Alpha, Kennebec and Russet Burbank exhibited the least production potential. Thus, Haig and Rutt were the two cultivars that performed well under continuous irradiation and high temperature conditions, and could have maximum potential for adaptation to varying stress environments. These two cultivars may have the best potential for use in future space farming in which continuous light and/or high temperature conditions may exist. However, cultivar responses under combined conditions of continuous light and high temperature remains for further validation.

NASA Discipline Life Support Systems↗
NASA NTRS2017

Subject Matter Expert Evaluation of Multi-Flight Common Route Advisories

Traffic flow management seeks to balance the demand for National Airspace System (NAS) flight resources, such as airspace and airports, with the available supply. When forecasted weather blocks nominal air traffic routes, traffic managers must re-route affected flights for weather avoidance. Depending on the nature and scope of the weather, traffic managers may use pre-coordinated re-routes such as Playbook Routes or Coded Departure Routes, or may design ad hoc local re-routes. The routes of affected flights are modified accordingly. These weather avoidance routes will, of course, be less efficient than the nominal routes due to increased flight time and fuel burn. In current traffic management operations, the transition into a weather avoidance re-routing initiative is typically implemented more aggressively than the transition out of that initiative after the weather has dissipated or moved away. For example, strategic large-scale Playbook re-routes are sometimes left in place (as initially implemented) for many hours before being lifted entirely when the weather dissipates. There is an opportunity to periodically modify the re-routing plan as weather evolves, thereby attenuating its adverse impact on flight time and fuel consumption; this is called delay recovery. Multi-Flight Common Routes (MFCR) is a NASA-developed operational concept and associated decision support tool for delay recovery, designed to assist traffic managers to efficiently update weather avoidance traffic routes after the original re-routes have become stale due to subsequent evolution of the convective weather system. MFCR groups multiple flights to reduce the number of advisories that the traffic manager needs to evaluate, and also merges these flights on a common route segment to provide an orderly flow of re-routed traffic. The advisory is presented to the appropriate traffic manager who evaluates it and has the option to modify it using MFCRs graphical user interface. If the traffic manager finds the advisory to be operationally appropriate, he or she would coordinate with the Area Supervisor(s) of the sectors that currently control the flights in the advisory. When the traffic manager accepts the MFCR advisory via the user interface, the corresponding flight plan amendments would be sent to the displays of the appropriate sector controllers, using the Airborne Re-Routing (ABRR) capability which is scheduled for nationwide operation in 2017. The sector controllers would then offer this time-saving route modification to the pilots of the affected flights via datalink (or voice), and implement the corresponding flight plan amendment if the pilots accept it. MFCR is implemented as an application in the software environment of the Future Air traffic management Concepts Evaluation Tool (FACET). This paper focuses on an initial subject matter expert (SME) evaluation of MFCR. The evaluation covers MFCRs operational concept, algorithm, and user interface.

Human-in-the-loop Evaluation↗
NASA NTRS2017

Subject Matter Expert Evaluation of Multi-Flight Common Route Advisories

Traffic flow management seeks to balance the demand for National Airspace System (NAS) flight resources, such as airspace and airports, with the available supply. When forecasted weather blocks nominal air traffic routes, traffic managers must re-route affected flights for weather avoidance. Depending on the nature and scope of the weather, traffic managers may use pre-coordinated re-routes such as Playbook Routes or Coded Departure Routes, or may design ad hoc local re-routes. The routes of affected flights are modified accordingly. These weather avoidance routes will, of course, be less efficient than the nominal routes due to increased flight time and fuel burn. In current traffic management operations, the transition into a weather avoidance re-routing initiative is typically implemented more aggressively than the transition out of that initiative after the weather has dissipated or moved away. For example, strategic large-scale Playbook re-routes are sometimes left in place (as initially implemented) for many hours before being lifted entirely when the weather dissipates. There is an opportunity to periodically modify the re-routing plan as weather evolves, thereby attenuating its adverse impact on flight time and fuel consumption; this is called delay recovery. Multi-Flight Common Routes (MFCR) is a NASA-developed operational concept and associated decision support tool for delay recovery, designed to assist traffic managers to efficiently update weather avoidance traffic routes after the original re-routes have become stale due to subsequent evolution of the convective weather system. MFCR groups multiple flights to reduce the number of advisories that the traffic manager needs to evaluate, and also merges these flights on a common route segment to provide an orderly flow of re-routed traffic. The advisory is presented to the appropriate traffic manager who evaluates it and has the option to modify it using MFCRs graphical user interface. If the traffic manager finds the advisory to be operationally appropriate, he or she would coordinate with the Area Supervisor(s) of the sectors that currently control the flights in the advisory. When the traffic manager accepts the MFCR advisory via the user interface, the corresponding flight plan amendments would be sent to the displays of the appropriate sector controllers, using the Airborne Re-Routing (ABRR) capability which is scheduled for nationwide operation in 2017. The sector controllers would then offer this time-saving route modification to the pilots of the affected flights via datalink (or voice), and implement the corresponding flight plan amendment if the pilots accept it. MFCR is implemented as an application in the software environment of the Future Air traffic management Concepts Evaluation Tool (FACET). This paper focuses on an initial subject matter expert (SME) evaluation of MFCR. The evaluation covers MFCRs operational concept, algorithm, and user interface.

Traffic flow management↗
NASA NTRS2021

Technical Note: AQMEII4 Activity 1: Evaluation of Wet and Dry Deposition Schemes as an Integral Part of Regional-scale Air Quality Models

We present in this technical note the research protocol for phase 4 of the Air Quality Model Evaluation International Initiative (AQMEII4). This research initiative is divided into two activities, collectively having three goals: (i) to define the current state of the science with respect to representations of wet and especially dry deposition in regional models, (ii) to quantify the extent to which different dry deposition parameterizations influence retrospective air pollutant concentration and flux predictions, and (iii) to identify, through the use of a common set of detailed diagnostics, sensitivity simulations, model evaluation, and reduction of input uncertainty, the specific causes for the current range of these predictions. Activity 1 is dedicated to the diagnostic evaluation of wet and dry deposition processes in regional air quality models (described in this paper), and Activity 2 to the evaluation of dry deposition point models against ozone flux measurements at multiple towers with multiyear observations (to be described in future submissions as part of the special issue on AQMEII4). The scope of this paper is to present the scientific protocols for Activity 1, as well as to summarize the technical information associated with the different dry deposition approaches used by the participating research groups of AQMEII4. In addition to describing all common aspects and data used for this multi-model evaluation activity, most importantly, we present the strategy devised to allow a common process-level comparison of dry deposition obtained from models using sometimes very different dry deposition schemes. The strategy is based on adding detailed diagnostics to the algorithms used in the dry deposition modules of existing regional air quality models, in particular archiving diagnostics specific to land use–land cover (LULC) and creating standardized LULC categories to facilitate cross-comparison of LULC-specific dry deposition parameters and processes, as well as archiving effective conductance and effective flux as means for comparing the relative influence of different pathways towards the net or total dry deposition. This new approach, along with an analysis of precipitation and wet deposition fields, will provide an unprecedented process-oriented comparison of deposition in regional air quality models. Examples of how specific dry deposition schemes used in participating models have been reduced to the common set of comparable diagnostics defined for AQMEII4 are also presented.

Air Quality Model Evaluation International Initiat↗
DOE OSTI2025

Heuristic Evaluation Methods Applied to a Predictive Maintenance Chatbot

The need for an accessible iterative approach for evaluating prospective artificial intelligence (AI)/ML based technologies in the nuclear industry is needed, given the nature of algorithms and rapid advancements. This paper explores existing heuristic design principles for user-centered design and evaluates them based on their relevancy and usefulness for evaluating AI/ ML based technologies. Researchers at the Idaho National Laboratory (INL) have developed a machine learning software application called VIsualization for PrEdictive maintenance Recommendation (VIPER), which is used to help users understand and engage with the tool to learn more about work orders, data used, predictive maintenance, and machine learning (ML) algorithms. Early user research studies used to access VIPER’s technology readiness level have occurred; however, there is room for further improvement of the software through heuristic evaluations along with other methods and user testing. This work describes the applicability of heuristic evaluation methods and cognitive walkthroughs to help ensure human readiness for prospective AI/ ML based applications, using VIPER as a candidate use case. This work supports industry in ensuring that prospective AI/ML based technologies are usable and useful for plant personnel at nuclear power plants, ultimately leading to their safe, reliable, and efficient use.

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2025

The Technical, Economic, Risk, and Adoption Assessment for Evaluating Work Reduction Opportunities in the Nuclear Industry

Automation and cost-saving initiatives, such as process automation with advances in artificial intelligence, are gaining traction in modernization efforts across the nuclear industry. As these innovations are increasingly adopted, it becomes crucial to evaluate their impacts comprehensively. Various technical and economic attributes, along with risk and human readiness factors, must be achieved to ensure that innovative projects enabling automation and modernization are successful. However, no systematic or integrated framework exists that allows plants to evaluate these innovative projects. To address this gap, the Technical, Economic, Risk, and Adoption (TERA) assessment offers a structured method to evaluate innovative technologies, ensuring solutions meet both operational and safety standards. The TERA framework integrates the disparate perspectives to assess modernization opportunities in nuclear operations. It combines qualitative and quantitative models to evaluate the relationship between performance and business impacts, while also enabling continuous re-evaluation during project development. This approach helps plant owners identify high-priority opportunities, optimize cost savings, and minimize risks, ensuring projects remain on track to achieve desired returns. By providing a comprehensive, systematic methodology, TERA enables informed, data-driven decisions that support successful modernization efforts, enhancing efficiency, safety, and cost savings across nuclear operations. This paper explains the TERA framework and its benefits for the nuclear industry.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

Heuristic Evaluation Methods Applied to a Predictive Maintenance Chatbot

The need for an accessible iterative approach for evaluating prospective artificial intelligence (AI)/ML based technologies in the nuclear industry is needed, given the nature of algorithms and rapid advancements. This paper explores existing heuristic design principles for user-centered design and evaluates them based on their relevancy and usefulness for evaluating AI/ ML based technologies. Researchers at the Idaho National Laboratory (INL) have developed a machine learning software application called VIsualization for PrEdictive maintenance Recommendation (VIPER), which is used to help users understand and engage with the tool to learn more about work orders, data used, predictive maintenance, and machine learning (ML) algorithms. Early user research studies used to access VIPER?s technology readiness level have occurred; however, there is room for further improvement of the software through heuristic evaluations along with other methods and user testing. This work describes the applicability of heuristic evaluation methods and cognitive walkthroughs to help ensure human readiness for prospective AI/ ML based applications, using VIPER as a candidate use case. This work supports industry in ensuring that prospective AI/ML based technologies are usable and useful for plant personnel at nuclear power plants, ultimately leading to their safe, reliable, and efficient use. PowerPoint for conference that was reviewed in PRS and LRS PRS/CON-25-05379 and INL/CON-25-82946

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2026

A Benchmarking Framework for Evaluating Large Language Model Capabilities in Nuclear Reactor Safety Applications

Large language models (LLMs) are increasingly capable of answering technical questions, synthesizing domain knowledge, and supporting engineering workflows. For nuclear science and engineering, these capabilities require careful, domain-specific evaluation before they can be credibly incorporated into safety-related activities, regulatory review, or technical decision support. This paper presents preliminary results from benchmarking framework for evaluating LLM capabilities in nuclear contexts. The framework is organized into three evaluation categories: nuclear fundamentals, general dual-use knowledge, and plant specific knowledge. These categories are intended to distinguish general nuclear engineering competence from broader technical reasoning and more context-dependent nuclear knowledge. Initial evaluations focus on nuclear fundamentals using questions representative of the knowledge expected of a nuclear professional engineer. Results indicate that contemporary frontier models perform at a high level and substantially exceed the performance of older model generations, with some models approaching saturation of the current benchmark. These findings suggest both the rapid improvement of LLM capabilities in specialized technical domains and the need for more discriminating evaluation methods. The paper presents the benchmark structure, preliminary model-comparison results, and ongoing work. This work supports development of verifiable, responsible, and safety-conscious methods for assessing AI systems in nuclear engineering applications.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗
DOE OSTI2024

Applying queueing theory to evaluate wait-time-savings of triage algorithms

Abstract In the past decade, artificial intelligence (AI) algorithms have made promising impacts in many areas of healthcare. One application is AI-enabled prioritization software known as computer-aided triage and notification (CADt). This type of software as a medical device is intended to prioritize reviews of radiological images with time-sensitive findings, thus shortening the waiting time for patients with these findings. While many CADt devices have been deployed into clinical workflows and have been shown to improve patient treatment and clinical outcomes, quantitative methods to evaluate the wait-time-savings from their deployment are not yet available. In this paper, we apply queueing theory methods to evaluate the wait-time-savings of a CADt by calculating the average waiting time per patient image without and with a CADt device being deployed. We study two workflow models with one or multiple radiologists (servers) for a range of AI diagnostic performances, radiologist’s reading rates, and patient image (customer) arrival rates. To evaluate the time-saving performance of a CADt, we use the difference in the mean waiting time between the diseased patient images in the with-CADt scenario and that in the without-CADt scenario as our performance metric. As part of this effort, we have developed and also share a software tool to simulate the radiology workflow around medical image interpretation, to verify theoretical results, and to provide confidence intervals for the performance metric we defined. We show quantitatively that a CADt triage device is more effective in a busy, short-staffed reading setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the evaluation methodology presented in this paper can be applied to assess the time-saving performance of other types of algorithms that prioritize a subset of customers based on binary outputs.

Thompson, Yee Lam Elim (ORCID:0000000196537707)↗
DOE OSTI2025

Benchmark of numerical modeling approaches on the systematic performance evaluation of wave energy converters

Different numerical modeling methods have been developed and applied to evaluate a variety of performance indicators of wave energy converters (WECs), including the power performance, structural loads, levelized cost of energy, etc. Based on the modeling fidelity, the commonly used numerical modeling approaches can be classified as linear modeling, weakly nonlinear modeling and fully nonlinear modeling approaches. Each method differs in accuracy and computational efficiency, making them suitable for different stages of WEC design. However, the selection of modeling approach could significantly impact evaluation outcomes. For instance, simplified linear models may underestimate structural loads or overestimate energy production in some operational conditions, potentially leading to less cost-effective designs. Given the widespread utilization of these models, it is essential to understand the uncertainties brought by them in performance evaluations. This work is dedicated to benchmarking different linear-potential-flow-based numerical models for evaluating the systematic performance of WECs. Three representative numerical modeling approaches are considered in this work, including linear frequency-domain modeling, statistically linearized spectral-domain modeling and Cummins equation-based nonlinear time-domain modeling. A generic point absorber WEC is considered as the research reference in this work, and different sea sites are taken into account. The numerical models are utilized to predict critical performance indicators, including power performance, the annual energy production, the capacity factor, the levelized cost of energy and the PTO fatigue loads. By comparing the results, this work identifies the uncertainties associated with different modeling approaches in evaluating WEC performance.

Fatigue↗
DOE OSTI2026

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗