Search NASA⌕ Search

SEARCH · Search NASA

Results for “test frameworks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

IBPSA Project 2 BOPTEST: An update on the test cases available in the framework for testing advanced control strategies in buildings

Project 2 develops software infrastructure, test cases, and extensions for the Building Optimization Testing Framework (BOPTEST) to address the expanding needs of building and urban energy system controls through open international collaboration. This paper provides an overview of the new test cases available as of BOPTEST version 0.7.1. Each test case is developed using open-source Modelica libraries and Spawn of EnergyPlus, enabling the creation of high-fidelity building models that incorporate envelope dynamics, Heating Ventilation and Air Conditioning (HVAC) systems, and explicit control representations. Currently, eight test cases are available, with five additional cases under development. These test cases cover a wide range of climates, building types, and HVAC systems. This paper compiles and summarizes test case descriptions, cites original manuscripts that developed them for a more detailed description, and reports baseline control performance metrics. Furthermore, two example applications are presented: one illustrating different levels of control, from supervisory to low-level, and another demonstrating how Model Predictive Control (MPC) solutions must be adapted from continuous to integer to control some building actuators.

Zanetti, Ettore↗
DOE OSTI2018

Test Harness Framework

The code facilitates running tests and then validating the results. The tests are normally attacks on a system and the verification is normally to validate protection software detected the attack, however the framework can be used in any area.

McCarty, MichaelV↗
DOE OSTI2019

Parallel Computing for Module-Based Computational Experiment

Large-scale scientific code plays an important role in scientific researches. In order to facilitate module and element evaluation in scientific applications, we introduce a unit testing framework and describe the demand for module-based experiment customization. We then develop a parallel version of the unit testing framework to handle long-term simulations with a large amount of data. Specifically, we apply message passing based parallelization and I/O behavior optimization to improve the performance of the unit testing framework and use profiling result to guide the parallel process implementation. Finally, we present a case study on litter decomposition experiment using a standalone module from a large-scale Earth System Model. This case study is also a good demonstration on the scalability, portability, and high-efficiency of the framework.

Yao, Zhuo↗
DOE OSTI2019

Comparative Assessment of Experimental Testing of Instrument with an Embedded Digital Device Using Model-Based and Conventional Methods

The use of digital equipment has not been widespread in U.S. nuclear facilities. A major contributing factor to this situation arises from regulatory concerns about common cause failures (CCFs) stemming from the use of identical software in redundant systems, questions as to how to quantify the reliability of software-based products, validity of tools that are used for software verification and validation (V&V), and the effectiveness of software QA procedures. Compounding the concerns are sporadic incidents of digital equipment malfunctions in nuclear power plants, including lockups resulting from software memory leaks, feedwater control malfunctions due to excessive traffic on Ethernet networks, software logic errors, and other problems leading to plant trips. Additionally, the limited amount of demonstrated research and a lack of industry consensus on quantitative software reliability methodologies has hindered the widespread adoption of digital I&C in the nuclear industry. In a recent Regulatory Issue Summary (RIS), the U.S. Nuclear Regulatory Commission (NRC) identified concerns about the impact of greater use of instrumentation and control (I&C) equipment with an embedded digital device (EDD) [1]. Specifically, the NRC staff states that increased use of such devices “may increase a facility’s vulnerability to a CCF” (i.e., common-cause failure). The prevailing NRC guidance regarding software CCF (SCCF) identifies two design attributes that are acceptable for eliminating CCF concerns: (1) diversity or (2) testability (specifically, 100% testability). Either solution can result in high costs and remaining licensing uncertainty (i.e., how much diversity is enough? how to ensure test coverage of every possible sequence of device states?). Consequently, many utilities and reactor designers have limited or avoided more extensive use of digital technology to minimize licensing, scheduling, and financial risk. Without development of cost effective qualification methods to satisfy regulatory requirements and address the potential for CCF vulnerability associated with EDDs, the nuclear power industry may not be able to realize the benefits of digital technology achieved by other industries. In order to address these challenges, the U.S. Department of Energy (DOE) is sponsoring this research to assess the prospect of CCF in digital I&C equipment. The purpose of the current research is to develop an effective approach employing science-based methods to resolve concerns about CCF vulnerability that serve to inhibit deployment of advanced instrumentation (e.g., sensors, actuators, microcontrollers) with EDDs in nuclear power applications. The research objectives address the challenge of establishing high levels of safety and reliability assurance needed for the qualification of EDDs (e.g., microprocessors, programmable logic devices) that are subject to software design faults, complex failure modes, and CCF vulnerability. Specific objectives are: (1) assess the regulatory context for treatment of CCF vulnerability a graded approach to their qualification, (3) develop and extend model-based testing methods to enable effective demonstration of whether devices are subject to CCF, which may arise from vulnerabilities introduced at any stage of the design lifecycle, (4) establish a cost-effective testing framework that incorporates automation and test scenario prioritization, and (5) demonstrate the qualification approach through selection and testing of candidate digital device(s). This report documents an assessment of the results from experimental testing of a representative instrument with an embedded digital device. The experiment serves as a proof-of-concept demonstration of a model-based testing (MBT) framework. The objective of MBT is to ensure that an EDD undergoes suitably comprehensive testing to establish that it is not subject to SCCF. The approach by which MBT accomplishes this objective is through development and application of an effective test suite that provides the appropriate coverage to detect the full range of postulated faults that may arise at the requirements, design, and code (implementation) level of the EDD software. The findings of the experiment involve the application of the MBT framework by Ohio State University (OSU) through testing of the selected demonstration instrument (i.e., the prototype smart pressure sensor developed by Virginia Commonwealth University, VCU). In addition, the experiment findings also involve application of a more conventional black-box testing approach that serves as a baseline against which the MBT methodology can be compared. The primary outcome documented in this report is the comparative analysis of the capabilities demonstrated for each method. The results of the analysis demonstrate clear benefits that arise from the MBT methodology. The comparison indicates greater effectiveness from the more comprehensive evidence available through MBT and enhanced efficiency achieved through automation within the MBT framework. The structure of the report addresses the approach and application of each test method as well as the comparative analysis of the experimental results. Chapter 2 gives a summary overview of the demonstration approach. Chapter 3 covers the baseline testing and Chapter 4 documents the model-based testing. Chapter 5 describes the findings of the comparative analysis and presents the conclusions that can be drawn about the effectiveness and efficiency of the MBT methodology.

47 OTHER INSTRUMENTATION↗
DOE OSTI2016

Framework for Testing the Effectiveness of Bat and Eagle Impact-Reduction Strategies at Wind Energy Projects

The objectives of this framework are to facilitate the study design and execution to test the effectiveness of bat and eagle impact-reduction strategies at wind energy sites. Through scientific field research, the wind industry and its partners can help determine if certain strategies are ready for operational deployment or require further development. This framework should be considered a living document to be improved upon as fatality-reduction technologies advance from the initial concepts to proven readiness (through project- and technology-specific testing) and as scientific field methods improve.

17 WIND ENERGY↗
DOE OSTI2024

An Entropy-Based Test and Development Framework for Uncertainty Modeling in Level-Set Visualizations

We present a simple comparative framework for testing and developing uncertainty modeling in uncertain marching cubes implementations. The selection of a model to represent the probability distribution of uncertain values directly influences the memory use, run time, and accuracy of an uncertainty visualization algorithm. We use an entropy calculation directly on ensemble data to establish an expected result and then compare the entropy from various probability models, including uniform, Gaussian, histogram, and quantile models. Our results verify that models matching the distribution of the ensemble indeed match the entropy. We further show that fewer bins in nonparametric histogram models are more effective whereas large numbers of bins in quantile models approach data accuracy.

Sisneros, Robert↗
DOE OSTI2021

Acquire and Test IDAES Framework Components

The cooperation under the Grid Modernization Initiative and the Applied Energy Tri-Laboratory umbrellas requires mutual understanding of capabilities and a common effort to identify gaps existing in the available computational modeling and analysis tools. In this work, we report acquisition and testing of the National Energy Technology Laboratory’s Institute for the Design of Advanced Energy Systems (IDAES) framework, particularly including both electric power systems production cost model simulator Prescient as well as system optimization IDAES itself. Further, we consider the methods in which developments in Prescient and IDAES for grid-energy simulation and optimization may be included in the Framework for Optimization of ResourCes and Economics (FORCE) ecosystem that has been under collaborative development by Idaho National Laboratory, Argonne National Laboratory, and Oak Ridge National Laboratory. We propose two integration methods to introduce Prescient and IDAES into existing FORCE automated workflows. One integration involves using Prescient as a market reduced-order model and IDAES models similar to how existing Modelica models are used. The other integration involves wrapping the Prescient-IDAES dispatch optimization process with the stochastic capacity optimization workflow in FORCE.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

A framework for testing soil carbon dynamics post land-use transition in a multisector dynamics model

Soil carbon plays a crucial role in the global carbon cycle. Changes in land use can determine whether carbon is stored or is emitted into the atmosphere as carbon dioxide, which has broad implications for the human and Earth systems. These feedbacks to the carbon cycle and their socio-economic drivers are modelled by many global multisector dynamics models to project future possibilities for the human-Earth system. One notable model of this class is the Global Change Analysis Model (GCAM), which uses a simplified process to model soil organic carbon (SOC) content after land-use transition across 384 land units. While the current GCAM soil carbon framework is based on scientific principles, it has not been tested against experimental data. This work examines rates of SOC change from GCAM input data. Specifically, first order rate constants derived from model inputs were compared to values from two syntheses to assess GCAM’s accuracy. Welch’s t-tests and linear models were used to determine if rate constants were consistent across all tested geographical areas and land-use transition types. While we found that there was general agreement on the direction and magnitude (i.e., rate) of SOC change, the rate constant derived from GCAM and empirical values differed strongly in a subset of specific instances. These results indicate that GCAM’s current SOC dynamics during land use transition successfully capture broad patterns of change in this critical carbon pool, but should be interpreted with caution at finer spatial scales. One potential cause of these discrepancies is our highly aggregated variable, soil timescale, which could be made more granular to improve accuracy. When using economically rooted multisector dynamics models, such as GCAM, it is critical to understand such model limitations for representing specific Earth system processes.

carbon↗
DOE OSTI2022

Towards Ultra-high-resolution E3SM Land Modeling on Exascale Computers

Here we present an ultra-high-resolution E3SM land model (uELM) for high-fidelity land simulations targeting new Exascale computers. After considering modeling infrastructure compatibility and ELM software features, we designed a parallel model for the uELM development targeting hybrid architectures of new US Exascale computers. We also described a function unit test framework to expedite the piece-wise code porting (with compiler directives), verification, and global variable management. Furthermore, in this study, we report an early uELM model development using OpenACC within a function unit test framework on a pre-Exascale computer, demonstrate the performance of a uLEM submodel with a 3.0-time speedup, and summarize the code porting experience regarding global variable handling, deepcopy, memory reduction, and parallel loop reconstruction.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Testing contamination source identification methods for water distribution networks

In the event of contamination in a water distribution network (WDN), source identification (SI) methods that analyze sensor data can be used to identify the source location(s). Knowledge of the source location and characteristics are important to inform contamination control and cleanup operations. Various SI strategies that have been developed by researchers differ in their underlying assumptions and solution techniques. The following manuscript presents a systematic procedure for testing and evaluating SI methods. The performance of these SI methods is affected by various factors including the size of WDN model, measurement error, modeling error, time and number of contaminant injections, and time and number of measurements. This paper includes test cases that vary these factors and evaluates three SI methods on the basis of accuracy and specificity. The tests are used to review and compare these different SI methods, highlighting their strengths in handling various identification scenarios. These SI methods and a testing framework that includes the test cases and analysis tools presented in this paper have been integrated into EPA’s Water Security Toolkit (WST), a suite of software tools to help researchers and others in the water industry evaluate and plan various response strategies in case of a contamination incident. Lastly, a set of recommendations are made for users to consider when working with different categories of SI methods.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Assessing Local Structure Motifs Using Order Parameters for Motif Recognition, Interstitial Identification, and Diffusion Path Characterization

Structure–property relationships form the basis of many design rules in materials science, including synthesizability and long-term stability of catalysts, control of electrical and optoelectronic behavior in semiconductors, as well as the capacity of and transport properties in cathode materials for rechargeable batteries. The immediate atomic environments (i.e., the first coordination shells) of a few atomic sites are often a key factor in achieving a desired property. Some of the most frequently encountered coordination patterns are tetrahedra, octahedra, body and face-centered cubic as well as hexagonal close packed-like environments. Here, we showcase the usefulness of local order parameters to identify these basic structural motifs in inorganic solid materials by developing classification criteria. We introduce a systematic testing framework, the Einstein crystal test rig, that probes the response of order parameters to distortions in perfect motifs to validate our approach. Subsequently, we highlight three important application cases. First, we map basic crystal structure information of a large materials database in an intuitive manner by screening the Materials Project (MP) database (61,422 compounds) for element-specific motif distributions. Second, we use the structure-motif recognition capabilities to automatically find interstitials in metals, semiconductor, and insulator materials. Our Interstitialcy Finding Tool (InFiT) facilitates high-throughput screenings of defect properties. Third, the order parameters are reliable and compact quantitative structure descriptors for characterizing diffusion hops of intercalants as our example of magnesium in MnO 2 -spinel indicates. Finally, the tools developed in our work are readily and freely available as software implementations in the pymatgen library, and we expect them to be further applied to machine-learning approaches for emerging applications in materials science.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Integrating CVMix into GOTM (v6.0): a consistent framework for testing, comparing, and applying ocean mixing schemes

Abstract. The General Ocean Turbulence Model (GOTM) is a one-dimensional water column model, including a set of state-of-the-art turbulence closure models, and has widely been used in various applications in the ocean modeling community. Here, we extend GOTM to include a set of newly developed ocean surface vertical mixing parameterizations of Langmuir turbulence via coupling with the Community Vertical Mixing Project (CVMix). A Stokes drift module is also implemented in GOTM to provide the necessary ocean surface waves information to the Langmuir turbulence parameterizations, as well as to facilitate future development and evaluation of new Langmuir turbulence parameterizations. In addition, a streamlined workflow with Python and Jupyter notebooks is also described, enabled by the newly developed and more flexible configuration capability of GOTM. The newly implemented Langmuir turbulence parameterizations are evaluated against theoretical scalings and available observations in four test cases, including an idealized wind-driven entrainment case and three realistic cases at Ocean Station Papa, the northern North Sea, and the central Baltic Sea, and compared with the existing general length scale scheme in GOTM. The results are consistent with previous studies. This development extends the capability of GOTM towards including the effects of ocean surface waves and provides useful toolsets for the ocean modeling community to further study the effects of Langmuir turbulence in a broader scope.

58 GEOSCIENCES↗