Search NASA⌕ Search

SEARCH · Search NASA

Results for “software testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

ON THE EFFECTIVENESS OF LLMS IN UNIT TEST GENERATION FOR STRUCTURED TEXT PROGRAMS

The reliability of industrial automation systems heavily depends on the correctness of Programmable Logic Controller (PLC) programs, which are often written in Structured Text (ST). While Large Language Models (LLMs) have shown promise in automating test generation for mainstream programming languages, their effectiveness for the syntactically strict ST language remains underexplored. This thesis presents a systematic empirical evaluation of three state-of-the-art LLMs—GPT-4o, Gemini 2.5 Pro, and Claude Sonnet 4.5—for generating ST unit tests. We examine three prompting strategies: Natural Language (NL), Code Language (CL), and Chain-of-Thought (CoT), across a curated set of 11 ST function blocks. The quality of the generated tests is assessed using Compilation Success Rate (CSR), Statement Coverage (SC), and Branch Coverage (BC). In the zero-shot setting, Claude Sonnet 4.5 achieves the highest CSR, while Gemini 2.5 Pro consistently delivers the best statement and branch coverage, particularly under CL prompts. By incorporating a one-shot CL prompt, all models exhibit substantial improvements—most notably GPT-4o, whose CSR increases from 45.45% to 90.91%, with substantial gains in both SC and BC. To further contextualize these findings, we compare GPT-4o’s one-shot results with PLCAutoTester, a state-ofthe- art ST unit test generation tool, on an additional benchmark dataset. While LLMgenerated tests approach competitive coverage levels, PLCAutoTester maintains significantly higher and more stable coverage across programs. This study provides the first comprehensive benchmark of modern LLMs for ST unit testing, highlighting their strengths, limitations, and improvements through one-shot prompting, and positioning their performance relative to specialized automated testing tools in industrial automation.

42 ENGINEERING↗
DOE OSTI2025

Exploring the Nature of Neutrinos with the Deep Underground Neutrino Experiment (DUNE)

The Deep Underground Neutrino Experiment (DUNE) is a next-generation experimental program designed to study the behaviour of neutrino oscillation. DUNE will utilize a neutrino beam originating at Fermilab, near Chicago, and will leverage a detector at Fermilab (Near Detector) and a detector 1300 km away in South Dakota (Far Detector), south of Saskatchewan. In the first phase of DUNE, its Far Detector will comprise of two 10,000 ton (fiducial) liquid argon (LAr) time-projection chamber (TPC) modules – powerful tracking calorimeter detectors – placed nearly a mile underground. With this large, sensitive, underground detector, DUNE aims to collect a high statistics and pure sample of neutrinos at the Far Detector. This setup also offers the potential to study non-beam physical processes via e.g. neutrinos produced in the atmosphere, supernova neutrino bursts, and/or solar neutrinos, etc. A second phase will aim to add more detector mass and expand the program. The Near Detector will consist of a LAr TPC module as well: critical to constraining systematic uncertainties in the oscillation analysis. However, this LAr TPC will have a novel design using a pixel-based readout instead of the traditional wire-based readout. This and the segmentation of the LAr TPC into multiple units are crucial in mitigating the high multiplicity of neutrino interactions expected in any readout window given its proximity to the beam. The Near Detector will feature additional components and capability beyond the LAr TPC, allowing one to deeply characterize the neutrino flux. Due to the complexity of this experimental program, several smaller-scale prototype detectors have been operating to test, validate, and improve both the technical designs and software for processing and analyzing events. By operating in charged particle test beams or neutrino beams, several of the prototypes are also capable of producing valuable results. Canadian institutions are involved in the realization of the DUNE through efforts with both the Near and Far Detectors and prototypes. DUNE is anticipated to begin operating near the end of this decade/the beginning of the next. This talk will focus on the overall DUNE program, for example its ultimate plans, status, and the efforts with prototypes.

Howard, Bruce [York U., Canada; Fermilab]↗
DOE OSTI2024

DEReliction: A Cybersecurity Vulnerability Assessment Methodology for Distributed Energy Resources

With the increasing integration of Distributed Energy Resources (DER) into the electric grid, maintaining grid reliability and resilience requires that these devices remain secure. This paper discusses a cybersecurity vulnerability assessment methodology that incorporates best practices from Sandia National Laboratories, SANS Institute, OWASP Foundation, and other web and Internet of Things (IoT) penetration testing (“pen testing”) programs, courses, and frameworks for assessing the security posture of devices. The methodology involves five sequential steps: (1) Collect Public Information, (2) Extract Hardware Details, (3) Inventory Software Components, (4) Identify Vulnerabilities, and (5) Test Vulnerabilities. Each step uncovers potential weaknesses in both hardware and software components of DER devices, considering adversary tactics, techniques, and procedures (TTPs), and potential attack vectors along the way. The results from the execution of this method on multiple residential- and small commercial-scale photovoltaic (PV) inverters reveled hardware and software vulnerabilities, which highlight the benefit of taking a methodical approach to discover vulnerabilities. While the specific vulnerability details are not shared here, a generalized overview of findings underscore the importance of robust security assessments for DER devices. Adoption of an assessment framework of this kind will identify and mitigate cybersecurity threats and bolster the resilience of DER-integrated electric grids.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Throughput Estimation of Data Transport Networks From Digital Twin Measurements

Digital twins of networked infrastructures, known as Virtual Infrastructure Twins (VITs), are increasingly used for software development, pre-deployment testing, and design space exploration. While VITs avoid the costs and potential disruptions associated with experiments on operational networks, their throughput measurements are typically not sufficiently accurate for performance profiling of wide-area networks that they emulate. Here, machine learning (ML) methods are developed to transform these inaccurate VIT network throughput measurements to closely match in peak and overall profile of those from a physical testbed or production network. First, a micro kernel network reflecting a physical network is utilized to collect one-time measurements on a host to support this ML transformation. Then, a generic multi-modal ML method is developed to learn a map that transforms measurements from subsequent VITs on the same host to match past, current and follow-on testbed and cloud networks. ML generalization equations are derived to establish its correctness and probabilistically guarantee its generalization accuracy. Experimental results are presented for a variety of VIT hosts with target testbed and cloud networks; they include a case study of a four-site science ecosystem wherein inaccurate convex VIT measurement profiles are transformed into accurate concave profiles of target networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Software-Defined Data Center Network Architecture using VXLAN-based BGP EVPN for Dynamic Workflows in a Supercomputing Environment (VXLAN-based BGP EVPN Fabric for HPC) v1

This software repository automates the deployment of a multi-vendor VXLAN-based BGP EVPN architecture, leveraging Containerlab to instantiate a stretched CLOS topology. It integrates Linux, Nokia SR Linux, and Arista cEOS, using BGP for underlay, overlay, and topology extension. The software enables rapid prototyping and testing of advanced network configurations. Its key advantage lies in providing a dynamic, programmable environment for research and development of critical technologies supporting dynamic workflows within supercomputing environments, surpassing the limitations of static, vendor-locked alternatives by fostering interoperability and agility.

Kumar, Ronal [Lawrence Berkeley National Laborator↗
DOE OSTI2024

Scalable DAQ system operating the CHIPS-5 neutrino detector

The CHIPS R&D project focuses on development of low-cost water Cherenkov neutrino detectors through novel design strategies and resourceful engineering. This work presents an end-to-end DAQ solution intended for a recent 5 kt CHIPS prototype, which is largely based on affordable mass-produced components. Much like the detector itself, the presented instrumentation is composed of modular arrays that can be scaled up and easily serviced. A single such array can carry up to 30 photomultiplier tubes (PMTs) accompanied by electronics that generate high voltage in-situ and deliver time resolution of up to 0.69 ns. In addition, the technology is compatible with the White Rabbit timing system, which can synchronize its elements to within 100 ps. While deployment issues did not permit the presented DAQ system to operate beyond initial evaluation, the presented hardware and software successfully passed numerous commissioning tests that demonstrated their viability for use in a large-scale neutrino detector, instrumented with thousands of PMTs.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

Wholesale Electricity Analysis via Simulation & Learning Experiments (WEASLE): Platform Development and Pilot Competition

This document reports the development of the Wholesale Electricity Analysis via Simulation and Learning Experiments (WEASLE) platform and the pilot competition that was conducted to test the platform. Due to the increasing reliance on variable renewable energy resources for bulk power, the pilot competition, called the Energy Storage Participation Algorithm Competition (ESPA-Comp), was used to test the effect of various market designs on storage utilization and market efficiency. Basic details of the platform are provided, including an overview of the market clearing engine, the battery dispatch and degradation models, electric grid topology and resource mix, and software architecture. Two market designs were tested: a two-settlement market analogous to typical ISO design today, and a multi-settlement market that allows additional forward-trading periods during the real-time market. Results from the pilot competition show that the storage bidding problem is nontrivial and is well suited for future challenges. We find that: 1) all four teams utilized different approaches to the bidding problem, 2) different methodological approaches led to substantially different offer behaviors, 3) resource profits are clustered by team and methodological approach, 4) simulated offers reduced market surplus by about 0.5%, 5) substantially different prices between two-settlement and multi-settlement markets albeit minimal difference in overall market surplus.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Fortran Compiler Test Suite v0.1.0

There is no currently available, open-source, easily adapted test suite for checking a compiler's conformance to the Fortran Standard. The main innovation of this software is in the framework to make the test suite apply to a new compiler. Making it open-source will enable community contributions of the test cases. Single organization construction of a comprehensive test suite is something that would be financially infeasible.

Richardson, Bradley↗
DOE OSTI2024

Cybersecurity Certification Requirements for Distributed Energy Resources: A Survey of SunSpec Alliance Standards

This survey paper explores the cybersecurity certification requirements defined by the SunSpec Alliance for Distributed Energy Resource (DER) devices, focusing on aspects such as software updates, device communications, authentication mechanisms, device security, logging, and test procedures. The SunSpec cybersecurity standards mandate support for remote and automated software updates, secure communication protocols, stringent authentication practices, and robust logging mechanisms to ensure operational integrity. Furthermore, the paper discusses the implementation of the SAE J3072 standard using the IEEE 2030.5 protocol, emphasizing the secure interactions between electric vehicle supply equipment (EVSE) and plug-in electric vehicles (PEVs) for functionalities like vehicle-to-grid (V2G) capabilities. This research also examines the SunSpec Modbus standard, which enhances the interoperability among DER system components, facilitating compliance with grid interconnection standards. This paper also analyzes the existing SunSpec Device Information Models, which standardize data exchange formats for DER systems across communication interfaces. Finally, this paper concludes with a detailed discussion of the energy storage cybersecurity specification and the blockchain cybersecurity requirements as proposed by SunSpec Alliance.

Tsikteris, Sean (ORCID:0009000524202250)↗
DOE OSTI2026

RAPIDS2: A SciDAC Institute for Computer Science, Data, and Artificial Intelligence (University of Delaware)

This report summarizes the activities, technical accomplishments, and outcomes of the RAPIDS2 Institute project at the University of Delaware (UD). The RAPIDS2 Institute was a large multi-institution project with the objective of assisting SciDAC and Office of Science application teams in the use of DOE supercomputing resources to achieve scientific breakthroughs. The UD team contributed to this effort through work on formal software verification. This thrust aims to reduce software developer time and effort, especially regarding debugging and testing, and to increase confidence in the correctness of the results computed by the software.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Generalizable, fast, and accurate DeepQSPR with fastprop

Abstract Quantitative Structure–Property Relationship studies (QSPR), often referred to interchangeably as QSAR, seek to establish a mapping between molecular structure and an arbitrary target property. Historically this was done on a target-by-target basis with new descriptors being devised to specifically map to a given target. Today software packages exist that calculate thousands of these descriptors, enabling general modeling typically with classical and machine learning methods. Also present today are learned representation methods in which deep learning models generate a target-specific representation during training. The former requires less training data and offers improved speed and interpretability while the latter offers excellent generality, while the intersection of the two remains under-explored. This paper introduces , a software package and general Deep-QSPR framework that combines a cogent set of molecular descriptors with deep learning to achieve state-of-the-art performance on datasets ranging from tens to tens of thousands of molecules. provides both a user-friendly Command Line Interface and highly interoperable set of Python modules for the training and deployment of feedforward neural networks for property prediction. This approach yields improvements in speed and interpretability over existing methods while statistically equaling or exceeding their performance across most of the tested benchmarks. is designed with Research Software Engineering best practices and is free and open source, hosted at github.com/jacksonburns/fastprop.

Burns, Jackson W. (ORCID:0000000206579426)↗
DOE OSTI2026

Best practices in software development for robust and reproducible geoscientific models based on insights from the Global Carbon Budget's dynamic vegetation models

Computational models play an increasingly vital role in scientific research by enabling the numerical simulation of complex processes. Such models are also fundamental in geosciences. For instance, they offer critical insights into the impacts of global change on the Earth system today and in the future. Beyond their value as research tools, models are also software products and should therefore adhere to certain established software engineering standards. However, scientists are rarely trained as software developers, which can lead to potential deficiencies in software quality like unreadable, inefficient, or erroneous code. The complexity of models, coupled with their integration into broader workflows, also often makes it challenging to reproduce results, evaluate processes, and build upon them. In this paper, we review the state and current practices of the development processes of the state-of-the-art land surface models used by the Global Carbon Budget. We combine the experience of modelers from the respective research groups with the expertise of software engineers from tech companies to outline key principles and tools for improving software quality in research. We explore four main areas: (1) model testing and validation, (2) scientific, technical, and user documentation, (3) version control, continuous integration, and code review, and (4) the portability and reproducibility of workflows. Our review reveals that while modeling communities are incorporating many best practices, significant room for improvement remains in areas such as automated testing, automated documentation, and reproducibility. Therefore, we here identify and promote essential software engineering practices, including numerous examples of practices from within the community that can serve as guidelines for other models and could help streamline processes across the entire community. We conclude with an open-source example implementation of these principles, demonstrating portable and reproducible data flows, a continuous integration setup, and web-based visualizations. This example may serve as a practical resource for model developers, users, and all scientists engaged in scientific programming.

Gregor, Konstantin [Technical Univ. of Munich (Ger↗
DOE OSTI2024

Practical Insights on Applying Simulation-Based Control Methods in Experimental Studies

Advanced nuclear reactors are crucial to the future of energy both in the United States and around the globe. In contrast to the current operating fleet, they are characterized as being deployable in remote locations and able to operate in semi-autonomous or autonomous fashion. This leap forward necessitates a new reactor control paradigm. Because advanced nuclear reactors are still under development in the United States, the creation of new control methods to achieve autonomous operations has been based on systems modeling and simulation. However, an important factor in successfully deploying these new control methods is the ability to seamlessly transition from simulation environments to real-world settings. Control methods tested in both simulation and experimental settings need to be investigated in the context of advanced reactor applications. This work developed a series of simple controllers for Idaho National Laboratory (INL)’s Microreactor Applications Research Validation and Evaluation (MARVEL) microreactor operating in load-following scenarios. These controllers were tested in both simulation and experimental settings, and a comparative performance analysis was performed. The simulation tests leveraged the Control and Optimization Modular Modeling Application for Nuclear Deployment (COMMAND) software developed in a previous stage of the current effort, along with the MARVEL Reactor Excursion and Leak Analysis Program (RELAP5-3D) and Monte Carlo N-Particle (MCNP) models. The experimental tests leveraged the COMMAND software, MARVEL models, and the U.S. Department of Energy Microreactor Program’s Microreactor Automated Control System (MACS). MACS was developed to serve as a control method testbed. It was customized to mirror the MARVEL microreactor, and COMMAND enabled MACS to emulate the physics of MARVEL. The load-following controller was developed using the simulation platform, with efforts to emulate real systems by introducing actuator saturation and noise. These factors were incrementally accounted for in the controller design. After finalizing the controller design, it was implemented with the experimental setup. The experimental conditions tested included an initial test under conditions similar to the final simulation test, and two additional scenarios. The first scenario introduced additional actuator saturation to account for equipment aging over time, which was unknown to the controller. The second scenario introduced sensor delay, a phenomenon anticipated with the use of remote operations or wireless communication in advanced reactors. These tests revealed several notable differences. While the controller performed well in simulation, it exhibited several limitations when transitioning to hardware. The main challenges involved maintaining the steady-state target power, as evidenced by larger error values between the true reactor power and setpoint power, as well as persistent oscillations in controlled reactor power. These issues could lead to unacceptable transient conditions in real reactor testing. Introducing actuator aging and stochastic delays in the experimental setup significantly impacted controller performance, resulting in increased overshoot and undershoot, and exacerbated error and oscillations previously mentioned. These findings underscore the importance of experimental testbeds for testing and validating control methods, as controllers developed using only theory and/or simulation may perform unexpectedly when applied to actual hardware. This research emphasizes the need for an experimental testbed for achieving such validation.

46 - INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AN↗
DOE OSTI2025

Testing convolutional neural network based deep learning systems: a statistical metamorphic approach

Machine learning technology spans many areas and today plays a significant role in addressing a wide range of problems in critical domains,i.e., healthcare, autonomous driving, finance, manufacturing, cybersecurity,etc. Metamorphic testing (MT) is considered a simple but very powerful approach in testing such computationally complex systems for which either an oracle is not available or is available but difficult to apply. Conventional metamorphic testing techniques have certain limitations in verifying deep learning-based models (i.e., convolutional neural networks (CNNs)) that have a stochastic nature (because of randomly initializing the network weights) in their training. In this article, we attempt to address this problem by using a statistical metamorphic testing (SMT) technique that does not require software testers to worry about fixing the random seeds (to get deterministic results) to verify the metamorphic relations (MRs). We propose seven MRs combined with different statistical methods to statistically verify whether the program under test adheres to the relation(s) specified in the MR(s). We further use mutation testing techniques to show the usefulness of the proposed approach in the healthcare space and test two CNN-based deep learning models (used for pneumonia detection among patients). The empirical results show that our proposed approach uncovers 85.71% of the implementation faults in the classifiers under test (CUT). Furthermore, we also propose an MRs minimization algorithm for the CUT, thus saving computational costs and organizational testing resources.

Computer Science↗
DOE OSTI2024

A general Bayesian algorithm for the autonomous alignment of beamlines

Autonomous methods to align beamlines can decrease the amount of time spent on diagnostics, and also uncover better global optima leading to better beam quality. The alignment of these beamlines is a high-dimensional expensive-to-sample optimization problem involving the simultaneous treatment of many optical elements with correlated and nonlinear dynamics. Bayesian optimization is a strategy of efficient global optimization that has proved successful in similar regimes in a wide variety of beamline alignment applications, though it has typically been implemented for particular beamlines and optimization tasks. In this paper, we present a basic formulation of Bayesian inference and Gaussian process models as they relate to multi-objective Bayesian optimization, as well as the practical challenges presented by beamline alignment. We show that the same general implementation of Bayesian optimization with special consideration for beamline alignment can quickly learn the dynamics of particular beamlines in an online fashion through hyperparameter fitting with no prior information. We present the implementation of a concise software framework for beamline alignment and test it on four different optimization problems for experiments on X-ray beamlines at the National Synchrotron Light Source II and the Advanced Light Source, and an electron beam at the Accelerator Test Facility, along with benchmarking on a simulated digital twin. We discuss new applications of the framework, and the potential for a unified approach to beamline alignment at synchrotron facilities.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

PROTOCALC, a W -band Polarized Calibrator for Cosmic Microwave Background Telescopes: Application to Simons Observatory and CLASS

Current- and next-generation cosmic microwave background (CMB) experiments will measure polarization anisotropies with unprecedented sensitivities. The need for high precision in these measurements underscores the importance of gaining a comprehensive understanding of instrument properties, with a particular emphasis on the study of the beam properties, and especially their polarization characteristics and the measurement of the polarization angle. In this context, a major challenge lies in the scarcity of millimeter polarized astrophysical sources with sufficient brightness and calibration knowledge to meet the stringent accuracy requirements of future CMB missions. This led to the development of a drone-borne calibration source designed for the frequency band centered on approximately 90 GHz, matching a commonly used channel in ground-based CMB measurements. The Prototype Calibrator for Cosmology, PROTOCALC, has undergone thorough in-lab testing, and its properties have been subsequently modeled through simulation software integrated into the standard Simons Observatory analysis pipeline. Moreover, the PROTOCALC system has been tested in the field, having been deployed twice on calibration campaigns with CMB telescopes in the Atacama Desert. The data collected constrain the roll angle of the source with a statistical accuracy of 0$^°_•$045.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

CACTI CSAPR2 Taranis Retrievals

Taranis is an end-to-end processing chain for radar data written in Python with C extension for computation performance. Features include: masking for quality control, specific differential phase (Kdp), attenuation correction for reflectivity factor (Z) and differential reflectivity (Zdr) in rain, and additional geophysical retrievals. Retrievals are mostly drawn from literature or open-source software when appropriate, and have been tested, tuned, and modified to work with one another cohesively rather than using isolated off-the-shelf algorithms. Incorporated algorithms include hydrometeor (echo) identification, rain water content, raindrop mass-weighted mean diameter (gamma size distribution assumption), and rainfall rate (QPE). Taranis data sets exist for CSAPR2 PPI, HSRHI, and sector RHI scans. Cartesian-gridded data sets were also produced as well as a near-surface rain rate retrieval. More details can be found in the README.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Advancing \textit{otsdaq}: Enhancements for Usability, Accuracy, and Robustness

High-energy physics (HEP) experiments require data acquisition (DAQ) systems that can orchestrate complex detector operations, high data throughput, and responsive, real-time feedback to operators. Traditional DAQ stacks, which are often bespoke, command-line driven and highly specific, impose large learning curves on users. The Off-The-Shelf Data Acquisition (\textit{otsdaq}) framework was created to address these issues by offering a highly customizable and scalable browser-based ’desktop’ environment, in which experiment-specific control and monitoring applications can be easily deployed and integrated. Although the initial development of the \textit{otsdaq} software was aimed at the Fermilab Test Beam Facility, \textit{otsdaq} is now being leveraged for broader deployment, including the upcoming Mu2e experiment, where real-time monitoring of field-programmable gate array (FPGA)-based Data Transfer Controllers (DTCs), Clock and Fanout (CFO) boards, and several other subsystems are critical. We contribute a set of targeted improvements to \textit{otsdaq}: bitmap visualization functionality for configured data, improved and corrected delta-based DTC throughput metrics, version control (VC)-backed source navigation for console messages, custom navigation hooks to eliminate disruptive user interface glitches, and copy-to-clipboard support for macro execution history. These changes improve usability, reduce debugging time, and increase accuracy in performance data as Mu2e moves toward commissioning.

Mohammed, A. [Unlisted, US]↗