A common task framework for testing and evaluation at the Space Domain Awareness Tools, Applications, and Processing Lab
Explore the source record for details and available documents.
Astronomy and AstroPhysics↗
SEARCH · Search NASA
Results for “test frameworks”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Who watches the watchers?: preventing fault in a fault tolerance library
The Scalable Checkpoint/Restart library (SCR) was developed and is used by researchers at Lawrence Livermore National Laboratory to provide a fast and efficient method of saving and recovering large applications during runtime on high-performance computing (HPC) systems. Though SCR protects other programs, up until June 2017, nothing was actively protecting SCR. The goal of this project was to automate the building and testing of this library on the varying HPC architectures on which it is used. Our methods centered around the use of a continuous integration tool called Bamboo that allowed for automation agents to be installed on the HPC systems themselves. These agents provided a way for us to establish a new and unique way to automate and customize the allocation of resources and running of tests with CMake’s unit testing framework, CTest, as well as integration testing scripts though an HPC package manager called Spack. These methods provided a parallel environment in which to test the more complex features of SCR. As a result, SCR is now automatically built and tested on several HPC architectures any time changes are made by developers to the library’s source code. The results of these tests are then communicated back to the developers for immediate feedback, allowing them to fix functionality of SCR that may have broken. Hours of developers’ time are now being saved from the tedious process of manually testing and debugging, which saves money and allows the SCR project team to focus their efforts towards development. Thus, HPC system users can use SCR in conjunction with their own applications to efficiently and effectively checkpoint and restart as needed with the assurance that SCR itself is functioning properly.
97 MATHEMATICS AND COMPUTING↗
Modernization efforts for the R -Matrix code SAMMY [Abstract]
The R-Matrix code SAMMY is a widely used nuclear data evaluation code focused on the resolved range, which includes corrections for experimental effects. The code is still mostly written in Fortran 77, and uses a memory management system suitable for the time of its initial writing (1984). A modernization effort is under way to bring the code in-line with modern software development practices. A continuous-integration testing framework was added, automating the large existing set of test cases. It is run on every commit. The memory management was updated to current standard practices suitable for modern software analysis tools. The code can be obtained from https://code.ornl.gov/RNSD/SAMMY. The resonance parameters and covariance information are now stored in C++ objects shared by SAMMY and AMPX, the processing code that generates nuclear data libraries for SCALE. This allows for easier maintenance and access to the resonance parameters inside and outside of SAMMY. This feature is already used by accessing and changing parameters in memory in the Bayesian Monte Carlo Evaluation Framework for Cross Sections Nuclear Data and Integral Benchmark Experiments project, Further plans include the switch to the ENDF reading and writing routines in AMPX, as these routines are more robust, easier to maintain, and support more features. Of note here is support for the new GNDS format. Previously it wasn’t easy to share the full covariance matrix for evaluations containing more than one isotope due to limitations on the ENDF format; this is now supported in GNDS. The data are currently available in a binary SAMMY format and can be exported to GNDS to make them more widely available and sharable. The next step will be to use the same resonance processing code at 0K in AMPX and SAMMY as one of the available Reich-Moore R-Matrix formalism. The first step toward this goal is to isolate the reconstruction into a module that takes resonance parameters as its input and does not depend on SAMMY global parameters. This goal has been achieved and it should now be possible to more easily change the resonance formalism and add enhancements as the Phenomenological R-Matrix parameterization of direct, doorway, and compound nuclear reactions discussed elsewhere on this conference. This concerted modernization and enhancement effort provides multiple advantages to the nuclear data community. It will allow parameter optimization using enhanced formalisms, including experimental effects, that better match complex experimental data. Then those evaluated parameters can immediately be passed off to AMPX to be reconstructed with the exact same cross section model and be put into a data library for subsequent testing using SCALE and the Valid Benchmark suite or other suitable benchmark suites.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
A streamlined build system foundation for developing HPC software
BLT bundles custom CMake macros, unit testing frameworks for C++ and Fortran, and a set of smoke tests for common HPC dependencies. The combination of these three provides a foundation for quickly bootstrapping a CMale-based system for developing HPC softward.
White, Chris↗
Large scale maximum average power multiple inference on time-course count data with application to RNA-seq analysis
Not provided.
Life Sciences & Biomedicine - Other Topics↗
A Cybersecurity Event Simulation Tool and Platform
Digital Instrumentation and Control Systems (ICSs) have replaced analog control systems in nuclear power plants raising cybersecurity concerns. To study and understand the cybersecurity risks of nuclear power plants both high fidelity models of the plant physics and controllers must be created, and a framework to test and evaluate cyber security events must be established. A testing and evaluation framework of cybersecurity events consists of a method of interfering with control systems, a simulation of the plant network, and a network packet capture and recording tool. Sandia National Labs (SNL) in collaboration with the University of New Mexico’s Institute for Space and Nuclear Power Studies (UNM-ISNPS) is developing such a cybersecurity testing framework.
21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
Pyctos
Pyctos is a concolic testing framework for standard, dynamically-typed Python. Pyctos is able to generate exhaustive test inputs reaching 100% coverage for a subset of pure Python in the absence of type annotations, even where modern fuzzers would fail. Pyctos's underlying reasoning engine is the CVC5 SMT solver, though Z3 is also supported. Pyctos also supports a growing subset of the standard library and some third-party libraries, such as NumPy.
Washbourne, ErickN [Lawrence Livermore National La↗
Autonomous Closed-Loop Experimental Characterization and Dynamic Model Validation of a Scaled Underwater Kite
This paper presents the closed-loop experimental framework and dynamic model validation for a 1/12-scale underwater kite design. The pool-based tow testing framework described herein, which involves a fully actuated, closed-loop controlled kite and flexible tether, significantly expands upon the capabilities of any previously developed open-source framework for experimental underwater kite characterization. Specifically, the framework has allowed for the validation of three closed-loop flight control strategies, along with a critical comparison between dynamic model predictions and experimental results. In this paper, we provide a detailed presentation of the experimental tow system and kite setup, describe the control algorithms implemented and tested, and quantify the level of agreement between our multi-degree-of-freedom kite dynamic model and experimental data. Finally, we also present a sensitivity analysis that helps to identify the most influential parameters to kite performance and further explain the remaining mismatches between the model and data.
42 ENGINEERING↗
UnitTesterSG
UnitTesterSG is a unit testing framework that is designed for nested and/or scientific/engineering data structures. It is designed primarily for testing the outputs if a single function or simulation run by storing the expected results file such that comparisons to the stored output can be made with unit tests after the function or software has been edited. This module is designed to be able to unit test arrays, strings, as well as nested and deeply nested objects (including arrays with tuples inside, arrays with strings inside, etc.). The module is compatable with nested objects, and thus has a dependency on the nestedObjectsFunctions module within. The module takes *calculated* results and then compares them to *expected* results. If no expected results are available (or they do not match the calculated results) the module then offers to store the calculated results as expected results for next time.
Savara, Aditya [Oak Ridge National Lab. (ORNL), Oa↗
Development of an MC&A toolbox for liquid-fueled molten salt reactors with online reprocessing (Final Report)
A critical barrier to the deployment of MSRs is the absence of a well-defined nuclear material control and accounting (MC&A) approach, a vital prerequisite to meet NRC licensing requirements as well as facilitating future international exports. Liquid-fueled MSR variants (especially those incorporating online refueling or reprocessing) present a unique set of challenges to traditional nuclear material control and accountancy. Unlike solid-fueled light-water reactors, traditional item-counting methods cannot be applied as an accountancy strategy. Rather, MC&A approaches to MSR variants (including both uranium and thorium fueled designs) are more analogous to bulk material handling facilities (e.g., enrichment and reprocessing); yet further complicating matters is the fact that fuel medium is also highly radioactive. Meanwhile, the space of MSRs covers a broad range of design parameters, including thermal and fast spectra designs; operation in actinide breeder or burner modes, choice of actinide fuel used (e.g., 235 U, 232 Th / 233 U, denatured 233 U, etc.), pool or loop-type configuration, and even different salt chemistry. Each of these design choices introduce significant challenges to MC&A approaches within MSR facilities. We propose to bridge this gap for liquid-fueled MSRs by developing a modular, component-based test framework for evaluating viable process monitoring and MC&A techniques specifically suited to liquid-fueled MSR system variants employing online refueling or reprocessing. This test platform will consist of a toolbox of independent process modules representing discrete physical units (such as the reactor core, off-gas processing, decay tanks, and actinide separation units), each with its own self-contained physics responsive to the input mass flow, along with appropriate measurement models that can be coupled to key flow points. These dynamic physical signatures thus afford the ability to test the viability and efficacy of potential accountancy techniques under the full range of reactor operating conditions. As process modules are connected via mass flows, the result is a reconfigurable, generic MSR mass flow model capable of serving as an MC&A test platform for a broad spectrum of possible MSR configurations. The resulting MSR MC&A toolbox will enable robust assessment of accountancy strategies for this unique facility type, including analysis of physical feedbacks arising both from depletion of the fuel over time as well as from potential off-normal events, including those introduced by equipment failures (e.g., a pump failure) as well as by malicious action (i.e., attempts to divert material). The proposed toolbox both addresses a critical needs area for the MPACT analysis toolkit while leveraging existing MPACT-sponsored tools, especially with respect to simulation of measurement and accountancy techniques for advanced fuel cycle facilities. Beyond enabling new analysis capabilities for MSR systems, the design of this toolbox will be to such to maximize compatibility with existing MPACT tools, such to enhance existing facility MC&A analysis capabilities.
11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
Gene expression of functionally-related genes coevolves across fungal species: detecting coevolution of gene expression using phylogenetic comparative methods
Researchers often measure changes in gene expression across conditions to better understand the shared functional roles and regulatory mechanisms of different genes. Analogous to this is comparing gene expression across species, which can improve our understanding of the evolutionary processes shaping the evolution of both individual genes and functional pathways. One area of interest is determining genes showing signals of coevolution, which can also indicate potential functional similarity, analogous to co-expression analysis often performed across conditions for a single species. However, as with any trait, comparing gene expression across species can be confounded by the non-independence of species due to shared ancestry, making standard hypothesis testing inappropriate. We compared RNA-Seq data across 18 fungal species using a multivariate Brownian Motion phylogenetic comparative method (PCM), which allowed us to quantify coevolution between protein pairs while directly accounting for the shared ancestry of the species. Our work indicates proteins which physically-interact show stronger signals of coevolution than randomly-generated pairs. Interactions with stronger empirical and computational evidence also showing stronger signals of coevolution. We examined the effects of number of protein interactions and gene expression levels on coevolution, finding both factors are overall poor predictors of the strength of coevolution between a protein pair. Simulations further demonstrate the potential issues of analyzing gene expression coevolution without accounting for shared ancestry in a standard hypothesis testing framework. Furthermore, our simulations indicate the use of a randomly-generated null distribution as a means of determining statistical significance for detecting coevolving genes with phylogenetically-uncorrected correlations, as has previously been done, is less accurate than PCMs, although is a significant improvement over standard hypothesis testing. These methods are further improved by using a phylogenetically-corrected correlation metric. Our work highlights potential benefits of using PCMs to detect gene expression coevolution from high-throughput omics scale data. This framework can be built upon to investigate other evolutionary hypotheses, such as changes in transcription regulatory mechanisms across species.
59 BASIC BIOLOGICAL SCIENCES↗
Portable Software Environment for Ultrahigh-Resolution ELM Development on GPUs
This paper presents our endeavors in developing the large-scale, ultra-high-resolution E3SM Land Model (uELM), specifically designed for exascale computers furnished with accelerators such as Nvidia GPUs. The uELM is a sophisticated code that substantially relies on High-Performance Computing (HPC) environments, necessitating particular machine and software configurations. To facilitate community-based uELM developments employing GPUs, we have created a portable, standalone software environment preconfigured with uELM input datasets, simulation cases, and source code. This environment, utilizing Docker, encompasses all essential code, libraries, and system software for uELM development on GPUs. It also features a functional unit test framework and an offline model testbed for comprehensive numerical experiments. From a technical perspective, the paper discusses GPU-ready container generations, uELM code management, and input data distribution across computational platforms. Lastly, the paper demonstrates the use of environment for functional unit testing, end-to-end simulation on CPUs and GPUs, and collaborative code development.
E3SM Land Model↗
Dual‐Transformer Deep Learning Framework for Seasonal Forecasting of Great Lakes Water Levels
Abstract The Great Lakes of North America form one of the largest freshwater systems on Earth, and their lake‐wide average water levels (lake levels) can fluctuate by more than 0.5 m on a seasonal scale. These fluctuations pose substantial challenges for coastal resilience, flood risk management, and navigation planning. Accurate seasonal forecasting of lake levels using traditional mechanistic models is challenging due to the complex physical mechanisms and coupled hydroclimatic processes involved. Recently, deep learning has gained prominence in geoscience applications for its ability to recognize intricate patterns within multiphysical data sets. Here, we introduce a novel Dual‐Transformer deep learning framework, tested on the Great Lakes. This architecture integrates two modified Transformer models: the Prophet, which predicts underlying trends, and the Critic, which refines the Prophet's predictions. The final lake level prediction is derived by weighting the outputs of both models through a multi‐layer perceptron, jointly trained with the Prophet and Critic to enhance overall accuracy. Our results demonstrate that the innovative learning framework achieves the highest prediction accuracy compared to established deep learning models when using identical input features. It attains a root mean square error of 4–7 cm in predicting lake levels up to 6 months in advance across the lakes. Additionally, the Dual‐Transformer model runs six orders of magnitude faster than conventional mechanistic models, producing results in less than one second on a typical personal computer. These findings suggest that our deep learning framework has strong potential to advance lake level prediction and carries important implications for water management and disaster mitigation, thereby enhancing the quality of life in coastal regions.
Chen, Yi [Great Lakes Research Center Michigan Tec↗
Bioverse: A Simulation Framework to Assess the Statistical Power of Future Biosignature Surveys
Next-generation space observatories will conduct the first systematic surveys of terrestrial exoplanet atmospheres and search for evidence of life beyond Earth. While in-depth observations of the nearest habitable worlds may yield enticing results, there are fundamental questions about planetary habitability and evolution that can only be answered through population-level studies of dozens to hundreds of terrestrial planets. To determine the requirements for next-generation observatories to address these questions, we have developed Bioverse. Bioverse combines existing knowledge of exoplanet statistics with a survey simulation and hypothesis testing framework to determine whether proposed space-based direct imaging and transit-spectroscopy surveys will be capable of detecting various hypothetical statistical relationships between the properties of terrestrial exoplanets. Following a description of the code, we apply Bioverse to determine whether an ambitious direct imaging or transit survey would be able to determine the extent of the circumstellar habitable zone and study the evolution of Earth-like planets. Given recent evidence that Earth-sized habitable zone planets are likely much rarer than previously believed, we find that space missions with large search volumes will be necessary to study the population of terrestrial and habitable worlds. Moving forward, Bioverse provides a methodology for performing trade studies of future observatory concepts to maximize their ability to address population-level questions, including and beyond the specific examples explored here.
60 APPLIED LIFE SCIENCES↗
Julienne v1.0.0
Julienne is a compiler-portable unit-testing framework for Fortran software projects, including those that use the parallel/accelerator-programming features of Fortran 2023. Julienne achieves portability across compilers through minimalism and isolation. The minimal design ensures that Julienne uses only features supported by the majority of Fortran compilers. The isolation through zero dependencies ensures that no other projects block Julienne from building with a particular compiler. Julienne also contains with additional services that support its unit-testing code. These include functions for manipulating strings, command lines and input/output format strings; and a user-defined collective subroutine for verifying that all processes pass a test in parallel testing. Julienne's name derives from the term for vegetables sliced into thin strings: julienne vegetables. Julienne captures the authors' most frequently used thin slice of the Veggies and Sourcery software repositories while avoiding certain compiler limitations of the those two packages.
Rouson, Damian↗
Enhancing Autonomous Control of Microreactors Using Multi-Agent Reinforcement Learning
In order for microreactors to be economically competitive, operation costs will need to be minimized through some degree of autonomous control. Previous work has demonstrated the effectiveness of reinforcement learning (RL) for load-following control in a drum-controlled microreactor. This study extends that work by exploring the potential of RL to independently control each of the reactor’s drums. We compare a single-agent RL approach with a multi-agent RL (MARL) framework, testing them for generalization across different load-following power profiles and control timescales, and for robustness in cases of randomly disabled control drums. Since the point kinetics simulation environment used in this study cannot resolve spatial effects, we assume that in the absence of spatially localized disturbances, optimal drum movements should be symmetrical. We demonstrate that single-agent RL is able to achieve accurate performance only when symmetric actions are ignored; otherwise, it fails to train a useful controller. Meanwhile, the MARL framework performs symmetric actions by design and trains a robust, accurate agent, as evidenced by mean absolute errors in power matching of 0.41% for the training power profile, 0.68% for a profile with half the drums disabled, and 0.21% for a profile on a realistic load-following time horizon.
22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
Performance portable ice-sheet modeling with MALI
High-resolution simulations of polar ice sheets play a crucial role in the ongoing effort to develop more accurate and reliable Earth system models for probabilistic sea-level projections. These simulations often require a massive amount of memory and computation from large supercomputing clusters to provide sufficient accuracy and resolution; therefore, it has become essential to ensure performance on these platforms. Many of today’s supercomputers contain a diverse set of computing architectures and require specific programming interfaces in order to obtain optimal efficiency. In an effort to avoid architecture-specific programming and maintain productivity across platforms, the ice-sheet modeling code known as MPAS-Albany Land Ice (MALI) uses high-level abstractions to integrate Trilinos libraries and the Kokkos programming model for performance portable code across a variety of different architectures. In this article, we analyze the performance portable features of MALI via a performance analysis on current CPU-based and GPU-based supercomputers. The analysis highlights not only the performance portable improvements made in finite element assembly and multigrid preconditioning within MALI with speedups between 1.26 and 1.82x across CPU and GPU architectures but also identifies the need to further improve performance in software coupling and preconditioning on GPUs. We perform a weak scalability study and show that simulations on GPU-based machines perform 1.24–1.92x faster when utilizing the GPUs. The best performance is found in finite element assembly, which achieved a speedup of up to 8.65x and a weak scaling efficiency of 82.6% with GPUs. We additionally describe an automated performance testing framework developed for this code base using a changepoint detection method. The framework is used to make actionable decisions about performance within MALI. We provide several concrete examples of scenarios in which the framework has identified performance regressions, improvements, and algorithm differences over the course of 2 years of development.
54 ENVIRONMENTAL SCIENCES↗