Search NASA⌕ Search

SEARCH · Search NASA

Results for “scientific software testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Testing Scientific Software: Challenges and Remedies

Over the past two decades, the emergence of highly effective software testing frameworks has greatly simplified the development and use of unit tests and has led to new software development paradigms such as test driven development (TDD). However, scientific computing introduces a number of unique testing challenges, including numerical algorithms, distributed parallelism, and exascale environments. This presentation will begin with a brief introduction to unit testing, testing frameworks, and some simple examples using pFUnit, a unit testing framework for Fortran + MPI. I will then take a closer look at several of the obstacles one faces when testing technical software and suggest methodologies that can mitigate these difficulties.

Clune, Tom↗

Modeling input spaace for testing scientific computational software: a case study

An application of a method of test case generation for scientific computational software is presented. NEWTRNX, neutron transport software being developed at Oak Ridge National Laboratory, is treated as a case study. A model of dependencies between input parameters of NEWTRNX is created. Results of NEWTRNX model analysis and test case generation are evaluated.

97 MATHEMATICS AND COMPUTING↗

Methodology for Building Granular Testing in Multicomponent Scientific Software

Computational science and engineering communities develop complex application software with multiple mathematical models that need to interact with one another. Partly due to complexity of verifying scientific software, and partly because of the way incentives work in science, there has been insufficient testing of these codes. With a spotlight on the results produced with scientific software, and increasing awareness of software testing and verification as a critical contributor to the reliability of these results, testing is gaining more attention by the developing teams. However, many science teams struggle to find a good solution for themselves due either to lack of training or lack of resources within the team. In this experience paper, we describe test development methodologies utilized in two different scenarios: one explains a methodology for building granular tests where none existed before, while the second demonstrates a methodology for selecting test cases that build confidence in the software through a process similar to scaffolding. The common insight from both the experiences is that testing should be a part of software design from the beginning for better software and scientific productivity.

test building, granular testing, Earth system mode↗

Virtual Environment for Testing Software-Defined Networking Solutions for Scientific Workflows

Recent developments in software-defined infrastructures promise that scientific workflows utilizing supercomputers, instruments, and storage systems will be dynamically composed and orchestrated using software at unprecedented speed and scale in the near future. Testing of the underlying networking software, particularly during initial exploratory stages, remains a challenge due to potential disruptions, and resource allocation and coordination needed over the multi-domain physical infrastructure. To overcome these challenges, we develop the Virtual Science Network Environment (VSNE) that emulates the multi-site host, storage, and network infrastructure using Virtual Machines (VMs), wherein the production and nascent software can be tested. Within each VM, which represents a site, the hosts and local-area networks are emulated using Mininet, and the Software-Defined Network (SDN) controllers and service daemon codes are natively run to support dynamic provisioning of network connections. Additionally, Lustre filesystem support at the sites and an emulation of the long-haul network using Mininet, are provided using separate VMs. As case studies, we describe Lustre file transfers using XDD, Red5 streaming service demonstration, and an emulated experiment with remote monitoring and steering modules, all supported over dynamically configured connections using SDN controllers.

97 MATHEMATICS AND COMPUTING↗

Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics

Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code-test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.

Md mushfiqur rahman, Sheikh [University of Tenness↗

Computational Simulations and the Scientific Method

As scientific simulation software becomes more complicated, the scientific-software implementor's need for component tests from new model developers becomes more crucial. The community's ability to follow the basic premise of the Scientific Method requires independently repeatable experiments, and model innovators are in the best position to create these test fixtures. Scientific software developers also need to quickly judge the value of the new model, i.e., its cost-to-benefit ratio in terms of gains provided by the new model and implementation risks such as cost, time, and quality. This paper asks two questions. The first is whether other scientific software developers would find published component tests useful, and the second is whether model innovators think publishing test fixtures is a feasible approach.

Kleb, Bil↗

Towards Ultra-high-resolution E3SM Land Modeling on Exascale Computers

Here we present an ultra-high-resolution E3SM land model (uELM) for high-fidelity land simulations targeting new Exascale computers. After considering modeling infrastructure compatibility and ELM software features, we designed a parallel model for the uELM development targeting hybrid architectures of new US Exascale computers. We also described a function unit test framework to expedite the piece-wise code porting (with compiler directives), verification, and global variable management. Furthermore, in this study, we report an early uELM model development using OpenACC within a function unit test framework on a pre-Exascale computer, demonstrate the performance of a uLEM submodel with a 3.0-time speedup, and summarize the code porting experience regarding global variable handling, deepcopy, memory reduction, and parallel loop reconstruction.

97 MATHEMATICS AND COMPUTING↗

Increasing Software Testing Coverage and Portability with Spack

Adequate testing coverage is critical for successful scientific software development. For numerically intensive codes, the test matrix should cover a variety of machines, compilers, third-party library dependencies, as well as possible other features such as memory testing, parallelization paradigms, and application options that may be enabled or disabled, as some examples. Such coverage can be a major burden on developers without powerful methods for managing the matrix. Spack is a great tool developed by Lawrence Livermore National Lab for managing software on a Unix-based system and in addition to this, we are leveraging Spack to simplify application testing development and management. Here we demonstrate exploiting many of Spack's features to orchestrate the test coverage for the Nalu-Wind application we are developing under the Exascale Computing Project. By using Spack underneath our testing infrastructure, we have increased our testing coverage, portability, robustness, and software quality. While we use our Nalu-Wind application as an example, the benefits of exploiting Spack for application testing can be had for other applications as well.

97 MATHEMATICS AND COMPUTING↗

Challenges and Strategies for Testing Automation Practices at Sandia National Laboratories

Sandia National Laboratories is a premier United States national security laboratory which develops science-based technologies in areas such as nuclear deterrence, energy production, and climate change. Computing plays a key role in its diverse missions, and within that environment, Research Software Engineers (RSEs) and other scientific software developers utilize testing automation to ensure quality and maintainability of their work. We conducted a Participatory Action Research study to explore the challenges and strategies for testing automation through the lens of academic literature. Through the experiences collected and comparison with open literature, we identify these challenges in testing automation and then present strategies for mitigation grounded in evidence-based practice and experience reports that other, similar institutions can assess for their automation needs.

97 MATHEMATICS AND COMPUTING↗

Systems aspects of COBE science data compression

A general approach to compression of diverse data from large scientific projects has been developed and this paper addresses the appropriate system and scientific constraints together with the algorithm development and test strategy. This framework has been implemented for the COsmic Background Explorer spacecraft (COBE) by retrofitting the existing VAS-based data management system with high-performance compression software permitting random access to the data. Algorithms which incorporate scientific knowledge and consume relatively few system resources are preferred over ad hoc methods. COBE exceeded its planned storage by a large and growing factor and the retrieval of data significantly affects the processing, delaying the availability of data for scientific usage and software test. Embedded compression software is planned to make the project tractable by reducing the data storage volume to an acceptable level during normal processing.

Freedman, I.↗

In Situ Data Infrastructure for Scientific Unit Testing Platform

Testing is a significant software development process for the management of software systems and scientific code. However, as the complexity of scientific codes increases, extra checks are needed to monitor impacts to dependent models and to verify system constraints. The software complexity also impedes the efforts of module developers and software engineers to rapidly develop and extend their code. Recently, we have developed an automatic methodology and prototype platform to facilitate scientific verification of individual functions within complex scientific codes. With this system, the scientific module builders are able to track variables conveniently in one module or track variables’ changes among different modules. In this paper, we present a procedure for automatic unit testing generation. For the interest of a general audience of this conference, we are emphasizing the technical details of integrating the In Situ data infrastructure into our platform. At the end of this paper, we have included an implementation of unit testing for the ACME Land Model (ALM) to demonstrate the usefulness and correctness of the platform. We have also used single- and multipoint checks to demonstrate the efficient variable tracking capability of this platform.

97 MATHEMATICS AND COMPUTING↗

Software Testing and Verification in Climate Model Development

Over the past 30 years most climate models have grown from relatively simple representations of a few atmospheric processes to a complex multi-disciplinary system. Computer infrastructure over that period has gone from punch card mainframes to modem parallel clusters. Model implementations have become complex, brittle, and increasingly difficult to extend and maintain. Existing verification processes for model implementations rely almost exclusively upon some combination of detailed analysis of output from full climate simulations and system-level regression tests. In additional to being quite costly in terms of developer time and computing resources, these testing methodologies are limited in terms of the types of defects that can be detected, isolated and diagnosed. Mitigating these weaknesses of coarse-grained testing with finer-grained "unit" tests has been perceived as cumbersome and counter-productive. In the commercial software sector, recent advances in tools and methodology have led to a renaissance for systematic fine-grained testing. We discuss the availability of analogous tools for scientific software and examine benefits that similar testing methodologies could bring to climate modeling software. We describe the unique challenges faced when testing complex numerical algorithms and suggest techniques to minimize and/or eliminate the difficulties.

Clune, Thomas L.↗

"Who Has Plots?": Contextualizing Scientific Software, Practice, and Visualizations

We present that software is an integral element of the work of science yet it is not commonly an object of inquiry in studies of scientific infrastructures. This paper presents findings from an ethnographic study of a cosmology group's collaborative scientific software production. We demonstrate how these cosmologists use plots to simultaneously test their software and analyze data while interrogating multiple layers of infrastructural components. We broaden perspectives on scientific software development using a sociotechnical, software studies lens to examine this work of scientific discovery as a creative and embodied, yet exacting and methodical, activity that requires a 'human in the loop'. Here, we offer a new reading of scientific software practices to convey how creating scientific software is often really the act of doing science itself--an intervention we believe is necessary to more successfully support scientific software sharing and infrastructure production.

97 MATHEMATICS AND COMPUTING↗