Search NASA⌕ Search

SEARCH · Search NASA

Results for “test frameworks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

cFS Test Framework (CTF)

NASA's Core Flight System (cFS) provides a generic flight software framework architecture for developing flight software. As the cFS framework has gained popularity over the years within the flight software community, supporting software tools have been developed to assist in the design, development, testing and verification of flight software. The cFS Test Framework (CTF) is a recently developed cFS tool with capabilities to develop and run automated test and verification scripts against flight software targets. The CTF tool parses and executes JSON-based test scripts containing test instructions, while logging and reporting the results. CTF utilizes a plugin-based architecture to allow developers to extend CTF with new test instructions, external interfaces, and custom functionality. To interface with flight software, CTF parses a set of CCSDS message definition files to create the necessary command and telemetry structures for use during the test run. Additionally, CTF also supports interfacing with multiple cFS instances, allowing a test script to verify requirements that involve multiple flight software targets. Lastly, CTF provides support for executing test scripts against FSW running on remote or embedded hardware. This allows CTF to execute the same test scripts across different target configurations throughout the development process. In this presentation, we will introduce the cFS Test Framework (CTF) architecture, discuss the history of cFS testing frameworks, and present the features and capabilities currently provided by CTF. Lastly, we will show a demo of the CTF tool being used to execute test scripts against flight software.

Aly I Shehata↗
NASA NTRSDate not supplied

CFS Test Framework

NASA's Core Flight System (cFS) provides a generic flight software framework architecture for developing flight software. As the cFS framework has gained popularity over the years within the flight software community, supporting software tools have been developed to assist in the design, development, testing and verification of flight software. The cFS Test Framework (CTF) is a recently developed cFS tool with capabilities to develop and run automated test and verification scripts against flight software targets. The CTF tool parses and executes JSON-based test scripts containing test instructions, while logging and reporting the results. CTF utilizes a plugin-based architecture to allow developers to extend CTF with new test instructions, external interfaces, and custom functionality. To interface with flight software, CTF parses a set of CCSDS message definition files to create the necessary command and telemetry structures for use during the test run. Additionally, CTF also supports interfacing with multiple cFS instances, allowing a test script to verify requirements that involve multiple flight software targets. Lastly, CTF provides support for executing test scripts against FSW running on remote or embedded hardware. This allows CTF to execute the same test scripts across different target configurations throughout the development process. In this presentation, we will introduce the cFS Test Framework (CTF) architecture, discuss the history of cFS testing frameworks, and present the features and capabilities currently provided by CTF. Lastly, we will show a demo of the CTF tool being used to execute test scripts against flight software.

cfs↗
NASA NTRS2022

Evaluating Network Performance of Containerized Test Framework for Distributed Space Systems

Distributed space systems are a mission architecture consisting of multiple spacecraft as a cohesive system which provide multipoint sampling, increased mission coverage, or improved sample resolution, while reducing mission risk through redundancy. To fully realize the potential of these systems, eventually scaling to hundreds or thousands of spacecraft, distributed space systems need to be operated as a single entity, which will enable a variety of novel scientific space missions. The Distributed Spacecraft Autonomy (DSA) project is a software project which aims to mature the technology needed for those systems, namely autonomous decision-making and swarm networking. The DSA project leverages a containerized swarm test framework to simulate spacecraft software, which can identify emergent behavior early in development. Container virtualization allows distributed spacecraft systems to be simulated entirely in software on a single computer, avoiding the overhead associated with conventional approaches like hardware facsimiles and virtual machines. For this approach to be effective, the simulated system behavior must not be artificially influenced by the swarm test framework itself. To address this, we present a series of benchmarks to quantify virtual network bandwidth available on a single-host computer and contextualize this against the network and application behavior of the DSA swarm test framework.

Distributed Spacecraft Autonomy↗
NASA NTRS2022

Rapid Flight Control Law Deployment and Testing Framework for Subscale VTOL Aircraft

A set of procedures was developed to enable rapid flight control law deployment and testing on subscale vertical takeoff and landing (VTOL) aircraft. Low-cost, subscale flight vehicles have become well-suited testbeds for rapid flight dynamics and controls research progression; however, integration of custom flight control laws onto flight hardware has historically been an arduous task. The toolchain described in this report leverages Simulink with the UAV Toolbox, a Pixhawk flight computer running PX4 firmware, and QGroundControl to efficiently design and flight test custom control algorithms. A subscale CL-84 VTOL aircraft was used as a testbed in this investigation to exercise the hardware integration process on a physical model. Implementation of custom attitude stabilization control laws and programmed test input excitations for aircraft system identification were demonstrated using the expeditious hardware integration process. The detailed procedures given in this report are expected to be used in future flight test efforts.

flight testing↗
NASA NTRS2018

Utilizing Testing Frameworks for Launch Control Systems Continuous Integration

Command and control software is an integral part of the launch procedure. The most important part of this type of software is its ability to communicate well with the user and relay information in a correctly formatted way such that the user can understand the data. There is a tool that aides the communication between the different parts of the system, and effectively, the user. This instrument is capable of taking several complex values and ensuring that they are correctly sorted into their distinctive message values and distributed properly among the different facets of the system. This tool will easily translate and publish the data inside of messages in the system to something that is readable and understandable. The tool also allows for transmission of the recorded data to the user, effectively ensuring the communication between different components of the system. As well as keeping track of messages and ensuring that the information contained within each of them reaches the correct location, this tool has the ability to keep track of its own statistics and determine how many messages passed in were erroneous and how many were successfully transmitted. It is able to check and see what the total message failure count is when an invalid message is given, as well as the number of different messages and their respective types passed into the tool. This tool is of great value to the new Space Launch System (SLS). As such, the tool must be thoroughly tested with test cases that, although improbable, are possible, where the tool may not function properly. Testing an interface this complex is necessary to ensure mission safety and create unlikely scenarios where the tool would work as intended, and stretch its limits to test that even under the most uncommon conditions it would still continue to function. This software will be an important part of the control system for the newest spacecraft which will fly deeper into space than humans have ever travelled. It will fly beyond the moon, into deep space to Mars and perhaps set the groundwork for a manned mission even further to create more opportunities for interplanetary and even interstellar travel by humans. This mission relies heavily on software and hardware to ensure the safety of the humans that will be on board and therefore must be checked, exhausting each and every different situation, such that there is not a doubt surrounding the well-being of the humans aboard the rocket. That is why testing is such an important part of the mission. It provides evidence that the systems aboard the rocket and on the launch pad are safe.

Unit Testing↗
DOE OSTI2024

MUPPET: An automated OpenMP mutation testing framework for performance optimization

MUPPET is a tool for OpenMP programs that identifies program modifications, called mutations, aimed at improving program performance. Existing performance optimization techniques, including profiling-based and auto-tuning techniques, fail to indicate program modifications at the source level thus preventing their portability across compilers. MUPPET aims to help HPC developers reason about performance defects and missed opportunities to improve performance at the source code level.

Parasyris, Konstantinos↗
DOE OSTI2025

IBPSA Project 2 BOPTEST: An update on the test cases available in the framework for testing advanced control strategies in buildings

Project 2 develops software infrastructure, test cases, and extensions for the Building Optimization Testing Framework (BOPTEST) to address the expanding needs of building and urban energy system controls through open international collaboration. This paper provides an overview of the new test cases available as of BOPTEST version 0.7.1. Each test case is developed using open-source Modelica libraries and Spawn of EnergyPlus, enabling the creation of high-fidelity building models that incorporate envelope dynamics, Heating Ventilation and Air Conditioning (HVAC) systems, and explicit control representations. Currently, eight test cases are available, with five additional cases under development. These test cases cover a wide range of climates, building types, and HVAC systems. This paper compiles and summarizes test case descriptions, cites original manuscripts that developed them for a more detailed description, and reports baseline control performance metrics. Furthermore, two example applications are presented: one illustrating different levels of control, from supervisory to low-level, and another demonstrating how Model Predictive Control (MPC) solutions must be adapted from continuous to integer to control some building actuators.

Zanetti, Ettore↗
NASA NTRS2013

Adoption of Test Driven Development and Continuous Integration for the Development of the Trick Simulation Toolkit

This paper describes the adoption of a Test Driven Development approach and a Continuous Integration System in the development of the Trick Simulation Toolkit, a generic simulation development environment for creating high fidelity training and engineering simulations at the NASA/Johnson Space Center and many other NASA facilities. It describes what was learned and the significant benefits seen, such as fast, thorough, and clear test feedback every time code is checked-in to the code repository. It also describes a system that encourages development of code that is much more flexible, maintainable, and reliable. The Trick Simulation Toolkit development environment provides a common architecture for user-defined simulations. Trick builds executable simulations using user-supplied simulation-definition files (S_define) and user supplied "model code". For each Trick-based simulation, Trick automatically provides job scheduling, checkpoint / restore, data-recording, interactive variable manipulation (variable server), and an input-processor. Also included are tools for plotting recorded data and various other supporting tools and libraries. Trick is written in C/C++ and Java and supports both Linux and MacOSX. Prior to adopting this new development approach, Trick testing consisted primarily of running a few large simulations, with the hope that their complexity and scale would exercise most of Trick's code and expose any recently introduced bugs. Unsurprising, this approach yielded inconsistent results. It was obvious that a more systematic, thorough approach was required. After seeing examples of some Java-based projects that used the JUnit test framework, similar test frameworks for C and C++ were sought. Several were found, all clearly inspired by JUnit. Googletest, a freely available Open source testing framework, was selected as the most appropriate and capable. The new approach was implemented while rewriting the Trick memory management component, to eliminate a fundamental design flaw. The benefits became obvious almost immediately, not just in the correctness of the individual functions and classes but also in the correctness and flexibility being added to the overall design. Creating code to be testable, and testing as it was created resulted not only in better working code, but also in better-organized, flexible, and readable (i.e., articulate) code. This was, in essence the Test-driven development (TDD) methodology created by Kent Beck. Seeing the benefits of Test Driven Development, other Trick components were refactored to make them more testable and tests were designed and implemented for them.

Penn, John M.↗
NASA NTRS2009

Exploring the Use of a Test Automation Framework

It is known that software testers, more often than not, lack the time needed to fully test the delivered software product within the time period allotted to them. When problems in the implementation phase of a development project occur, it normally causes the software delivery date to slide. As a result, testers either need to work longer hours, or supplementary resources need to be added to the test team in order to meet aggressive test deadlines. One solution to this problem is to provide testers with a test automation framework to facilitate the development of automated test solutions.

TAF STAF Test Automation↗
NASA NTRS2019

Testing Scientific Software: Challenges and Remedies

Over the past two decades, the emergence of highly effective software testing frameworks has greatly simplified the development and use of unit tests and has led to new software development paradigms such as test driven development (TDD). However, scientific computing introduces a number of unique testing challenges, including numerical algorithms, distributed parallelism, and exascale environments. This presentation will begin with a brief introduction to unit testing, testing frameworks, and some simple examples using pFUnit, a unit testing framework for Fortran + MPI. I will then take a closer look at several of the obstacles one faces when testing technical software and suggest methodologies that can mitigate these difficulties.

Clune, Tom↗
NASA NTRS2009

(Quickly) Testing the Tester via Path Coverage

The configuration complexity and code size of an automated testing framework may grow to a point that the tester itself becomes a significant software artifact, prone to poor configuration and implementation errors. Unfortunately, testing the tester by using old versions of the software under test (SUT) may be impractical or impossible: test framework changes may have been motivated by interface changes in the tested system, or fault detection may become too expensive in terms of computing time to justify running until errors are detected on older versions of the software. We propose the use of path coverage measures as a "quick and dirty" method for detecting many faults in complex test frameworks. We also note the possibility of using techniques developed to diversify state-space searches in model checking to diversify test focus, and an associated classification of tester changes into focus-changing and non-focus-changing modifications.

evaluation of test systems↗
NASA NTRS2019

Testing Fortran Software with pFunit

Over the past two decades, the emergence of highly effective software testing frameworks has greatly simplified the development and use of unit tests and has led to new software development paradigms such as test driven development (TDD). However, technical computing introduces a number of unique testing challenges, including distributed parallelism and numerical accuracy. This webinar will begin with a basic introduction to the use of pFUnit (parallel Fortran Unit testing framework) to develop tests for Message Passing Interface (MPI) plus Fortran (MPI+Fortran) software and then present some of the new capabilities in the latest release. We will also discuss some specialized methodologies for testing numerical algorithms and speculate about future framework capabilities that may improve our ability to test at exascale.

Clune, Tom↗
NASA NTRS2019

Writing a New Automated Testing Method for Custom Display Components

During the launch process of the Space Launch System (SLS), engineers in the Firing Room of the Launch Control Center (LCC) will be analyzing displays that show data and statistics about the vehicle and launch. These displays are subject to a battery of tests, but the custom components they use are not currently supported by any testing framework. My project is to modify a testing framework so that the custom components on these displays can be tested similarly to how a user would interact with them.

Automated Testing↗
NASA NTRS2017

Launch Control System Software Development System Automation Testing

The Spaceport Command and Control System (SCCS) is the National Aeronautics and Space Administration's (NASA) launch control system for the Orion capsule and Space Launch System, the next generation manned rocket currently in development. This system requires high quality testing that will measure and test the capabilities of the system. For the past two years, the Exploration and Operations Division at Kennedy Space Center (KSC) has assigned a group including interns and full-time engineers to develop automated tests to save the project time and money. The team worked on automating the testing process for the SCCS GUI that would use streamed simulated data from the testing servers to produce data, plots, statuses, etc. to the GUI. The software used to develop automated tests included an automated testing framework and an automation library. The automated testing framework has a tabular-style syntax, which means the functionality of a line of code must have the appropriate number of tabs for the line to function as intended. The header section contains either paths to custom resources or the names of libraries being used. The automation library contains functionality to automate anything that appears on a desired screen with the use of image recognition software to detect and control GUI components. The data section contains any data values strictly created for the current testing file. The body section holds the tests that are being run. The function section can include any number of functions that may be used by the current testing file or any other file that resources it. The resources and body section are required for all test files; the data and function sections can be left empty if the data values and functions being used are from a resourced library or another file. To help equip the automation team with better tools, the Project Lead of the Automated Testing Team, Jason Kapusta, assigned the task to install and train an optical character recognition (OCR) tool to Brandon Echols, a fellow intern, and I. The purpose of the OCR tool is to analyze an image and find the coordinates of any group of text. Some issues that arose while installing the OCR tool included the absence of certain libraries needed to train the tool and an outdated software version. We eventually resolved the issues and successfully installed the OCR tool. Training the tool required many images and different fonts and sizes, but in the end the tool learned to accurately decipher the text in the images and their coordinates. The OCR tool produced a file that contained significant metadata for each section of text, but only the text and coordinates of the text was required for our purpose. The team made a script to parse the information we wanted from the OCR file to a different file that would be used by automation functions within the automated framework. Since a majority of development and testing for the automated test cases for the GUI in question has been done using live simulated data on the workstations at the Launch Control Center (LCC), a large amount of progress has been made. As of this writing, about 60% of all of automated testing has been implemented. Additionally, the OCR tool will help make our automated tests more robust due to the tool's text recognition being highly scalable to different text fonts and text sizes. Soon we will have the whole test system automated, allowing for more full-time engineers working on development projects.

Automation↗