Search NASA⌕ Search

SEARCH · Search NASA

Results for “scientific software testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

ENHANCING SEISMIC CALIBRATION RESEARCH THROUGH SOFTWARE AUTOMATION AND SCIENTIFIC INFORMATION MANAGEMENT

The National Nuclear Security Administration (NNSA) Ground-Based Nuclear Explosion Monitoring Research and Development (GNEMRD) Program at LLNL continues to make significant progress enhancing the process of deriving seismic calibrations and performing scientific integration, analysis, and information management with software automation tools. Our tool efforts address the problematic issues of very large datasets and varied formats encountered during seismic calibration research. New information management and analysis tools have resulted in demonstrated gains in efficiency of producing scientific data products and improved accuracy of derived seismic calibrations. The foundation of a robust, efficient data development and processing environment is comprised of many components built upon engineered versatile libraries. We incorporate proven industry 'best practices' throughout our code and apply source code and bug tracking management as well as automatic generation and execution of unit tests for our experimental, development and production lines. Significant software engineering and development efforts have produced an object-oriented framework that provides database centric coordination between scientific tools, users, and data. Over a half billion parameters, signals, measurements, and metadata entries are all stored in a relational database accessed by an extensive object-oriented multi-technology software framework that includes stored procedures, real-time transactional database triggers and constraints, as well as coupled Java and C++ software libraries to handle the information interchange and validation requirements. Significant resources were applied to schema design to enable management of processing methods and station parameters, responses and metadata. This allowed for the development of merged ground-truth (GT) data sets compiled by the NNSA labs and AFTAC that include hundreds of thousands of events and tens of millions of arrivals. The schema design groundwork facilitated extensive quality-control and revalidation steps. In support of the GT merge effort, a comprehensive site merge process was also accomplished this year that included station site information for tens of thousands of entries from NNSA labs, AFTAC, NEIC, ISC, and IRIS. A core capability is the ability to rapidly select and present subsets of related signals and measurements to the researchers for analysis and distillation both visually (JAVA GUI client applications) and in batch mode (instantiation of multi-threaded applications on clusters of processors). RBAP Version 2 is one such example. Over the past year RBAP was significantly improved in capability and performance. A new role-based security model now allows fine-grain access control over all aspects of the tool's functions enabling researchers to share their work with others without fear of unintended parameter alterations. A new, faster and more reliable GIS mapping framework was added, as well as expanded powerful interactive plotting graphics. In addition, we implemented parent-child type projects to enhance calibration data management. Our specific automation methodology and tools improve the researchers ability to assemble quality-controlled research products for delivery into the NNSA Knowledge Base (KB). The software and scientific automation tasks provide the robust foundation upon which synergistic and efficient development of GNEMRD Program seismic calibration research may be built.

58 GEOSCIENCES↗

SABATH: Surrogate AI Benchmarking Applications' Testing Harness

SABATH provides benchmarking infrastructure for evaluating scientific ML/AI models. It contains support for scientific machine learning surrogates from external repositories. The software dependences are explicitly exposed in the surrogate model definition, which allows the use of advanced optimization, communication, and hardware features.

Luszczek, Piotr [Univ. of Tennessee, Knoxville, TN↗

epicsuite(EAS)

Software, Documentation, Tutorials, Testing data, Example data for processing, analysis, filtering, querying, visualization and otherwise transforming genomic data for scientific analysis and discovery.

Rogers, David H.↗

ComPort: Rigorous Testing Methods to Safeguard Software Porting (Final UW Report)

This report summarizes the University of Washington’s contributions to the ComPort project, Rigorous Testing Methods to Safeguard Software Porting, funded under the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR), award number DE-SC0022081. UW’s work addressed numerics-driven correctness challenges arising in heterogeneous and rapidly evolving computing environments. Our efforts focused on developing interactive analysis tools, correctness-preserving rewriting systems, semantic-aware code-transformation infrastructure, and multi-language support for reproducible numerical experiments.

97 MATHEMATICS AND COMPUTING↗

A general purpose CCD controller

A new type of controller for scientific CCDs that has been designed and tested at NASA-Goddard is described. It features the flexibility of software waveform generation as well as the speed and accuracy of clocking by hardware. The basic CCD operation is described as well as the controller architecture; controller operation; experimental verification; and software, hardware, and data acquisition.

Chen, P. C.↗

Lessons Learned from the Clementine Mission

According to BMDO, the Clementine mission achieved many of its technology objectives during its flight to the Moon in early 1994 but, because of a software error, was unable to test the autonomous tracking of a cold target. The preliminary analyses of the returned lunar data suggest that valuable scientific measurements were made on several important topics but that COMPLEX's highest-priority objectives for lunar science were not achieved. This is not surprising given that the rationale for Clementine was technological rather than scientific. COMPLEX lists below a few of the lessons that may be learned from Clementine. Although the Clementine mission was not conceived as a NASA science mission exactly like those planned for the Discovery program, many operational aspects of the two are similar. It is therefore worthwhile to understand the strengths and faults of the Clementine approach. Some elements of the Clementine operation that led to the mission's success include the following: (1) The mission's achievements were the responsibility of a single organization and its manager, which made that organization and that individual accountable for the final outcome; (2) The sponsor adopted a hands-off approach and set a minimum number of reviews (three); (3) The sponsor accepted a reasonable amount of risk and allowed the project team to make the trade-offs necessary to minimize the mission's risks while still accomplishing all its primary objectives; and (4) The development schedule was brief and the agreed-on funding (and funding profile) was adhered to. Among the operational shortcomings of Clementine were the following: (1) An overly ambitious schedule and a slightly lean budget (meaning insufficient time for software development and testing, and leading ultimately to human exhaustion); and (2) No support for data calibration, reduction, and analysis. The principal lesson to be learned in this category is that any benefits from the constructive application of higher risk for lower cost and faster schedule will be lost if the schedule does not allow adequate time for the development of all essential systems or makes no allowance for human frailties. Another lesson to be drawn is that despite its limitations, if judged strictly as a science mission, Clementine attested that significant scientific information can be gathered during a technology-demonstration mission. In the current era of limited funds, when science missions will be infrequent, the opportunity to fly scientific instruments aboard missions whose objectives might be other than science must be seized and, indeed, encouraged. During such opportunities it would be inexcusable to do second-class science. Thus the scientific community must be actively involved in such projects from their initiation.

Source record↗

Simulating Mars: Enabling Testing of the Perseverance Rover Sampling and Caching Subsystem on Earth

The development of the Sampling and Caching Subsystem (SCS) on the JPL Perseverance Rover lies at the intersection of testing, robotics, and geology. The SCS team established three primary system test campaigns and venues to aid in the development of SCS through verification and validation testing – Qualification Model Dirty Testing (QMDT) to provide a venue for testing in a Martian environment, Vehicle System Testbed (VSTB) for testing while integrated with the mobility subsystem on Martian-like terrain, and the Flight Software Testbed (FSWTB) for conducting tests using the flight motor controllers and software system on a hexapod which had the ability to simulate rover tilt. Each venue contributed a vital piece to the SCS building blocks. However, the QMDT venue operating within a 10-ft diameter Thermal Vacuum chamber to simulate Martian environment provided a sui generis opportunity to fine tune the entire sampling and caching process while building the team’s knowledge base about rock drillability, system life, and target selection. On Earth, because Martian rocks are not readily available, the development team must utilize geoanalogs to the rocks and regolith on Mars. Geologists on the team helped establish a set of standard rock types to use for Mars missions, like Basalt, Sandstone, Mudstone, Gypsum, and other related geoanalogs. These geoanalogs are characterized with a standard suite of tests for density, compressibility, and other characteristics to categorize potential drillability. This concept of drillability is what links the geoanalogs on Earth to the samples we collect on Mars. With the simulant characteristics defined, these geoanalog rocks are ready to be drilled into as we do on the Martian surface. A key aspect of interacting with the surface on Mars is rock target identification and selection. The Perseverance robotic system uses the on-board cameras, instrumentation, and software to collect enough information to identify potential scientific targets. With the targets identified, SCS can place the Corer and abrade the surface or collect a sample. For a ground test activity like QMDT, the test team did not have all of the camera and instrumentation systems that the rover does, so the team developed ground test equivalents to process a rock, build a target map, and define the target. The team constructed a Rock Scanning Station to build a 3D point cloud of the rock. This point cloud was then processed and evaluated with predefined and programmed criteria in a Target Downselect Tool. A primary output of the Target Downselect Tool is a defined target that can be uploaded directly to the robotic software system to simulate and build the robotic sequences used in tests. With these insights and programmatic definition of targets, the QMDT test team was able to make the same decisions that the Perseverance surface operations team does. In addition, valuable lessons learned from developing the target selection ground tools and using them were implemented into the tools used for surface operations.

Kim, Junggon↗

A Python Object Oriented & Data Driven Testing Infrastructure for an Environmental Information Management System

Software systems such as TEIMS, the Taurus Environmental Information Management System evolve in size and complexity making manual testing no longer sufficient or feasible, becoming a performance and quality risk. TEIMS is a system of systems comprised of dynamic, data-driven web-based applications that support planning, operational monitoring, data collection, sample management, reporting and, scientific findings for LLNL’s Environmental Restoration Department (ERD). Selenium’s Integrated Development Environment (IDE) was previously used to support TEIMS testing. An approach with limited flexibility for future software development. As a student intern I was assigned the task to develop an object oriented testing infrastructure for TEIMS. A critical resource in this approach is the development of a catalogue of page elements. To accomplish this I created a comma separated value file, which is comprised of key identifiers based on page elements. Using the Python language and the Selenium Web driver I then developed the object mapping structure, which serves as a wrapper for exposure of each element allowing interaction with the tests and test suites. A master driver “Module” allowed the web driver to be shared among the objects and tests. I then developed a web-based graphical user interface (GUI) for test history, metrics and searching. This data driven approach provides flexibility and extensibility, allowing changes to the structure to be performed in a single place. The result is an object oriented/data driven infrastructure that allows tests to be simpler, flexible, and maintainable, with searchable results.

97 MATHEMATICS AND COMPUTING↗

Agile collaboration: Citizen science as a transdisciplinary approach to heliophysics

Citizen science connects scientists with the public to enable discovery, engaging broad audiences across the world. There are many attributes that make citizen science an asset to the field of heliophysics, including agile collaboration. Agility is the extent to which a person, group of people, technology, or project can work efficiently, pivot, and adapt to adversity. Citizen scientists are agile; they are adaptable and responsive. Citizen science projects and their underlying technology platforms are also agile in the software development sense, by utilizing beta testing and short timeframes to pivot in response to community needs. As they capture scientifically valuable data, citizen scientists can bring expertise from other fields to scientific teams. The impact of citizen science projects and communities means citizen scientists are a bridge between scientists and the public, facilitating the exchange of information. These attributes of citizen scientists form the framework of agile collaboration. In this paper, we contextualize agile collaboration primarily for aurora chasers, a group of citizen scientists actively engaged in projects and independent data gathering. Nevertheless, these insights scale across other domains and projects. Citizen science is an emerging yet proven way of enhancing the current research landscape. To tackle the next-generation’s biggest research problems, agile collaboration with citizen scientists will become necessary.

79 ASTRONOMY AND ASTROPHYSICS↗

RELAP-7 Software Verification and Validation Plan

This INL plan comprehensively describes the software for RELAP-7 and documents the software, interface, and software design requirements for the application. The plan also describes the testing-based software verification and validation (SV&V) process—a set of specially designed software models used to test RELAP-7. The RELAP-7 (Reactor Excursion and Leak Analysis Program) code is a nuclear reactor system safety analysis code being developed at Idaho National Laboratory (INL). The code is based on the INL’s modern scientific software development framework – MOOSE (Multi-Physics Object-Oriented Simulation Environment). The overall design goal of RELAP-7 is to take advantage of the previous thirty years of advancements in computer architecture, software design, numerical integration methods, and physical models. The end result will be a reactor systems analysis capability that retains and improves upon RELAP5’s capability and extends the analysis capability for all reactor system simulation scenarios.

97 MATHEMATICS AND COMPUTING↗

Report on Automated Semantic Analysis of Scientific and Engineering Codes

The loss of the Mars Climate Orbiter due to a software error reveals what insiders know: software development is difficult and risky because, in part, current practices do not readily handle the complex details of software. Yet, for scientific software development the MCO mishap represents the tip of the iceberg; few errors are so public, and many errors are avoided with a combination of expertise, care, and testing during development and modification. Further, this effort consumes valuable time and resources even when hardware costs and execution time continually decrease. Software development could use better tools! This lack of tools has motivated the semantic analysis work explained in this report. However, this work has a distinguishing emphasis; the tool focuses on automated recognition of the fundamental mathematical and physical meaning of scientific code. Further, its comprehension is measured by quantitatively evaluating overall recognition with practical codes. This emphasis is necessary if software errors-like the MCO error-are to be quickly and inexpensively avoided in the future. This report evaluates the progress made with this problem. It presents recommendations, describes the approach, the tool's status, the challenges, related research, and a development strategy.

Stewart. Maark E. M.↗

Towards Test Driven Development for Computational Science with pFUnit

Developers working in Computational Science & Engineering (CSE)/High Performance Computing (HPC) must contend with constant change due to advances in computing technology and science. Test Driven Development (TDD) is a methodology that mitigates software development risks due to change at the cost of adding comprehensive and continuous testing to the development process. Testing frameworks tailored for CSE/HPC, like pFUnit, can lower the barriers to such testing, yet CSE software faces unique constraints foreign to the broader software engineering community. Effective testing of numerical software requires a comprehensive suite of oracles, i.e., use cases with known answers, as well as robust estimates for the unavoidable numerical errors associated with implementation with finite-precision arithmetic. At first glance these concerns often seem exceedingly challenging or even insurmountable for real-world scientific applications. However, we argue that this common perception is incorrect and driven by (1) a conflation between model validation and software verification and (2) the general tendency in the scientific community to develop relatively coarse-grained, large procedures that compound numerous algorithmic steps.We believe TDD can be applied routinely to numerical software if developers pursue fine-grained implementations that permit testing, neatly side-stepping concerns about needing nontrivial oracles as well as the accumulation of errors. We present an example of a successful, complex legacy CSE/HPC code whose development process shares some aspects with TDD, which we contrast with current and potential capabilities. A mix of our proposed methodology and framework support should enable everyday use of TDD by CSE-expert developers.

pFUnit↗

SITCOMTN-154: Initial studies of photometric redshifts with LSSTComCam from DP1

This technote holds reports based on the first analyses of the Data Preview 1 (DP1) data by the Science Unit for photometric redshifts. Although photometric redshifts are not an official DP1 data product, the "Photo-z Science Unit" generated photo-z estimates for every galaxy in DP1 using the available multi-band imaging on a best-effort basis. This work included developing training and test datasets by matching DP1 data to high-quality reference redshifts obtained with spectroscopy, Grism data, and multi-band photometry. The Science Unit used the RAIL software package to make photometric redshift estimates using eight different algorithms, developed simple scientific performance metrics, used those metrics to explore how the performance of the algorithms varied with configuration changes, derived more optimized configurations of the algorithms and tested the performance of those configurations. This work, the resulting data products and expected data distribution mechanism are all described there.

79 ASTRONOMY AND ASTROPHYSICS↗

Intelligent Mobile Technologies

Testing involves commercial radio equipment approved for export and use in Canada. Testing was conducted in the Canadian High Arctic, where hilly terrain provided the worst-case testing. SFU and Canadian governmental agencies made significant technical contributions. The only technical data related to radio testing was exchanged with SFU. Test protocols are standard radio tests performed by communication technicians worldwide. The Joint Fields Operations objectives included the following: (1) to provide Internet communications services for field science work and mobile exploration systems; (2) to evaluate the range and throughput of three different medium-range radio link technologies for providing coverage of the crater area; and (3) to demonstrate collaborative software such as NetMeeting with multi-point video for exchange of scientific information between remote node and base-base camp and science centers as part of communications testing.

Alena, Rick↗

Orbit Determination and Navigation Software Testing for the Mars Reconnaissance Orbiter

During the extended science phase of the Mars Reconnaissance Orbiter's lifecycle, the operational duties pertaining to navigation primarily involve orbit determination. The orbit determination process utilizes radiometric tracking data and is used for the prediction and reconstruction of MRO's trajectories. Predictions are done twice per week for ephemeris updates on-board the spacecraft and for planning purposes. Orbit Trim Maneuvers (OTM-s) are also designed using the predicted trajectory. Reconstructions, which incorporate a batch estimator, provide precise information about the spacecraft state to be synchronized with scientific measurements. These tasks were conducted regularly to validate the results obtained by the MRO Navigation Team. Additionally, the team is in the process of converting to newer versions of the navigation software and operating system. The capability to model multiple densities in the Martian atmosphere is also being implemented. However, testing outputs among these different configurations was necessary to ensure compliance to a satisfactory degree.

Deep Space Network↗

ComPort: Rigorous Testing Methods to Safeguard Software Porting (Final Technical Report)

This is a technical report from the lead institution – University of Utah, Kahlert School of Computing – funded under the Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under award number DE-SC0022252. We summarize our work done over the three years of funding received. The relevant papers and software have already been uploaded at the DOE site.

97 MATHEMATICS AND COMPUTING↗

Perspectives on distributed computing : thirty people, four user types, and the distributed computing user experience.

This report summarizes the methodology and results of a user perspectives study conducted by the Community Driven Improvement of Globus Software (CDIGS) project. The purpose of the study was to document the work-related goals and challenges facing today's scientific technology users, to record their perspectives on Globus software and the distributed-computing ecosystem, and to provide recommendations to the Globus community based on the observations. Globus is a set of open source software components intended to provide a framework for collaborative computational science activities. Rather than attempting to characterize all users or potential users of Globus software, our strategy has been to speak in detail with a small group of individuals in the scientific community whose work appears to be the kind that could benefit from Globus software, learn as much as possible about their work goals and the challenges they face, and describe what we found. The result is a set of statements about specific individuals experiences. We do not claim that these are representative of a potential user community, but we do claim to have found commonalities and differences among the interviewees that may be reflected in the user community as a whole. We present these as a series of hypotheses that can be tested by subsequent studies, and we offer recommendations to Globus developers based on the assumption that these hypotheses are representative. Specifically, we conducted interviews with thirty technology users in the scientific community. We included both people who have used Globus software and those who have not. We made a point of including individuals who represent a variety of roles in scientific projects, for example, scientists, software developers, engineers, and infrastructure providers. The following material is included in this report: (1) A summary of the reported work-related goals, significant issues, and points of satisfaction with the use of Globus software; (2) A method for characterizing users according to their technology interactions, and identification of four user types among the interviewees using the method; (3) Four profiles that highlight points of commonality and diversity in each user type; (4) Recommendations for technology developers and future studies; (5) A description of the interview protocol and overall study methodology; (6) An anonymized list of the interviewees; and (7) Interview writeups and summary data. The interview summaries in Section 3 and transcripts in Appendix D illustrate the value of distributed computing software--and Globus in particular--to scientific enterprises. They also document opportunities to make these tools still more useful both to current users and to new communities. We aim our recommendations at developers who intend their software to be used and reused in many applications. (This kind of software is often referred to as 'middleware.') Our two core recommendations are as follows. First, it is essential for middleware developers to understand and explicitly manage the multiple user products in which their software components are used. We must avoid making assumptions about the commonality of these products and, instead, study and account for their diversity. Second, middleware developers should engage in different ways with different kinds of users. Having identified four general user types in Section 4, we provide specific ideas for how to engage them in Section 5.

97 MATHEMATICS AND COMPUTING↗