Search NASA⌕ Search

SEARCH · Search NASA

Results for “benchmark and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Expansion of Check-Cases for 6DOF Simulation

This effort expands upon a previous NASA activity that developed flight simulation benchmark check-cases to include new check-cases for the Cislunar domain, comparing multiple NASA simulation tools. The results of this effort describe the benefits of standardizing inputs, simulation comparisons and describe an interactive website that enables comparison of externally provided simulation data. Participating simulations improved their software and identified implementation errors. This activity elevated simulation credibility and provided a measure of validation for the simulations actively in use for NASA’s Human Landing Systems (HLS).

Modeling↗

Expansion of Check-Cases for 6DOF Simulation: Appendix A

This is the Appendix containing figures of simulation output data plots for comparison from the assessment, “Expansion of Check-Cases for 6DOF Simulation”. This effort expands upon a previous NASA activity that developed flight simulation benchmark check-cases to include new check-cases for the Cislunar domain, comparing multiple NASA simulation tools. The results of this effort describe the benefits of standardizing inputs, simulation comparisons and describe an interactive website that enables comparison of externally provided simulation data. Participating simulations improved their software and identified implementation errors. This activity elevated simulation credibility and provided a measure of validation for the simulations actively in use for NASA’s Human Landing Systems (HLS).

Modeling↗

Benchmark Data for Evaluation of NASA Impedance Eduction Methods

Results achieved with three acoustic liner configurations are used to evaluate the effects of recent modifications to the NASA Normal Incidence Tube and Grazing Flow Impedance Tube. These include a calibration liner, a wire mesh liner, and a perforate liner. The effects of source type, source level, and mean flow Mach number on the impedances educed with these liners are explored. Existing models are used to predict the impedance for each test condition and are compared with educed impedance spectra to determine their validity. The results suggest that educed impedances are similar for either a stepped sine or swept sine source. The predicted models are acceptable for the perforate liner and are quite good for the calibration and wire mesh liners.

acoustic↗

Benchmark Data for Evaluation of NASA Impedance Eduction Methods

Results achieved with three acoustic liner configurations are used to evaluate the effects of recent modifications to the NASA Normal Incidence Tube and Grazing Flow Impedance Tube. These include a calibration liner, a wire mesh liner, and a perforate liner. The effects of source type, source level, and mean flow Mach number on the impedances educed with these liners are explored. Existing models are used to predict the impedance for each test condition, and are compared with educed impedance spectra to determine their validity. The results suggest that educed impedances are similar for either a stepped sine or swept sine source. The predicted models are acceptable for the perforate liner, and are quite good for the calibration and wire mesh liners.

acoustic↗

A Vehicle Management End-to-End Testing and Analysis Platform for Validation of Mission and Fault Management Algorithms to Reduce Risk for NASAs Space Launch System

The engineering development of the National Aeronautics and Space Administration's (NASA) new Space Launch System (SLS) requires cross discipline teams with extensive knowledge of launch vehicle subsystems, information theory, and autonomous algorithms dealing with all operations from pre-launch through on orbit operations. The nominal and off-nominal characteristics of SLS's elements and subsystems must be understood and matched with the autonomous algorithm monitoring and mitigation capabilities for accurate control and response to abnormal conditions throughout all vehicle mission flight phases, including precipitating safing actions and crew aborts. This presents a large and complex systems engineering challenge, which is being addressed in part by focusing on the specific subsystems involved in the handling of off-nominal mission and fault tolerance with response management. Using traditional model-based system and software engineering design principles from the Unified Modeling Language (UML) and Systems Modeling Language (SysML), the Mission and Fault Management (M&FM) algorithms for the vehicle are crafted and vetted in Integrated Development Teams (IDTs) composed of multiple development disciplines such as Systems Engineering (SE), Flight Software (FSW), Safety and Mission Assurance (S&MA) and the major subsystems and vehicle elements such as Main Propulsion Systems (MPS), boosters, avionics, Guidance, Navigation, and Control (GNC), Thrust Vector Control (TVC), and liquid engines. These model-based algorithms and their development lifecycle from inception through FSW certification are an important focus of SLS's development effort to further ensure reliable detection and response to off-nominal vehicle states during all phases of vehicle operation from pre-launch through end of flight. To test and validate these M&FM algorithms a dedicated test-bed was developed for full Vehicle Management End-to-End Testing (VMET). For addressing fault management (FM) early in the development lifecycle for the SLS program, NASA formed the M&FM team as part of the Integrated Systems Health Management and Automation Branch under the Spacecraft Vehicle Systems Department at the Marshall Space Flight Center (MSFC). To support the development of the FM algorithms, the VMET developed by the M&FM team provides the ability to integrate the algorithms, perform test cases, and integrate vendor-supplied physics-based launch vehicle (LV) subsystem models. Additionally, the team has developed processes for implementing and validating the M&FM algorithms for concept validation and risk reduction. The flexibility of the VMET capabilities enables thorough testing of the M&FM algorithms by providing configurable suites of both nominal and off-nominal test cases to validate the developed algorithms utilizing actual subsystem models such as MPS, GNC, and others. One of the principal functions of VMET is to validate the M&FM algorithms and substantiate them with performance baselines for each of the target vehicle subsystems in an independent platform exterior to the flight software test and validation processes. In any software development process there is inherent risk in the interpretation and implementation of concepts from requirements and test cases into flight software compounded with potential human errors throughout the development and regression testing lifecycle. Risk reduction is addressed by the M&FM group but in particular by the Analysis Team working with other organizations such as S&MA, Structures and Environments, GNC, Orion, Crew Office, Flight Operations, and Ground Operations by assessing performance of the M&FM algorithms in terms of their ability to reduce Loss of Mission (LOM) and Loss of Crew (LOC) probabilities. In addition, through state machine and diagnostic modeling, analysis efforts investigate a broader suite of failure effects and associated detection and responses to be tested in VMET to ensure reliable failure detection, and confirm responses do not create additional risks or cause undesired states through interactive dynamic effects with other algorithms and systems. VMET further contributes to risk reduction by prototyping and exercising the M&FM algorithms early in their implementation and without any inherent hindrances such as meeting FSW processor scheduling constraints due to their target platform - the ARINC 6535-partitioned Operating System, resource limitations, and other factors related to integration with other subsystems not directly involved with M&FM such as telemetry packing and processing. The baseline plan for use of VMET encompasses testing the original M&FM algorithms coded in the same C++ language and state machine architectural concepts as that used by FSW. This enables the development of performance standards and test cases to characterize the M&FM algorithms and sets a benchmark from which to measure their effectiveness and performance in the exterior FSW development and test processes. This paper is outlined in a systematic fashion analogous to a lifecycle process flow for engineering development of algorithms into software and testing. Section I describes the NASA SLS M&FM context, presenting the current infrastructure, leading principles, methods, and participants. Section II defines the testing philosophy of the M&FM algorithms as related to VMET followed by section III, which presents the modeling methods of the algorithms to be tested and validated in VMET. Its details are then further presented in section IV followed by Section V presenting integration, test status, and state analysis. Finally, section VI addresses the summary and forward directions followed by the appendices presenting relevant information on terminology and documentation.

Trevino, Luis↗

A Vehicle Management End-to-End Testing and Analysis Platform for Validation of Mission and Fault Management Algorithms to Reduce Risk for NASA's Space Launch System

The development of the Space Launch System (SLS) launch vehicle requires cross discipline teams with extensive knowledge of launch vehicle subsystems, information theory, and autonomous algorithms dealing with all operations from pre-launch through on orbit operations. The characteristics of these systems must be matched with the autonomous algorithm monitoring and mitigation capabilities for accurate control and response to abnormal conditions throughout all vehicle mission flight phases, including precipitating safing actions and crew aborts. This presents a large complex systems engineering challenge being addressed in part by focusing on the specific subsystems handling of off-nominal mission and fault tolerance. Using traditional model based system and software engineering design principles from the Unified Modeling Language (UML), the Mission and Fault Management (M&FM) algorithms are crafted and vetted in specialized Integrated Development Teams composed of multiple development disciplines. NASA also has formed an M&FM team for addressing fault management early in the development lifecycle. This team has developed a dedicated Vehicle Management End-to-End Testbed (VMET) that integrates specific M&FM algorithms, specialized nominal and off-nominal test cases, and vendor-supplied physics-based launch vehicle subsystem models. The flexibility of VMET enables thorough testing of the M&FM algorithms by providing configurable suites of both nominal and off-nominal test cases to validate the algorithms utilizing actual subsystem models. The intent is to validate the algorithms and substantiate them with performance baselines for each of the vehicle subsystems in an independent platform exterior to flight software test processes. In any software development process there is inherent risk in the interpretation and implementation of concepts into software through requirements and test processes. Risk reduction is addressed by working with other organizations such as S&MA, Structures and Environments, GNC, Orion, the Crew Office, Flight Operations, and Ground Operations by assessing performance of the M&FM algorithms in terms of their ability to reduce Loss of Mission and Loss of Crew probabilities. In addition, through state machine and diagnostic modeling, analysis efforts investigate a broader suite of failure effects and detection and responses that can be tested in VMET and confirm that responses do not create additional risks or cause undesired states through interactive dynamic effects with other algorithms and systems. VMET further contributes to risk reduction by prototyping and exercising the M&FM algorithms early in their implementation and without any inherent hindrances such as meeting FSW processor scheduling constraints due to their target platform - ARINC 653 partitioned OS, resource limitations, and other factors related to integration with other subsystems not directly involved with M&FM. The plan for VMET encompasses testing the original M&FM algorithms coded in the same C++ language and state machine architectural concepts as that used by Flight Software. This enables the development of performance standards and test cases to characterize the M&FM algorithms and sets a benchmark from which to measure the effectiveness of M&FM algorithms performance in the FSW development and test processes. This paper is outlined in a systematic fashion analogous to a lifecycle process flow for engineering development of algorithms into software and testing. Section I describes the NASA SLS M&FM context, presenting the current infrastructure, leading principles, methods, and participants. Section II defines the testing philosophy of the M&FM algorithms as related to VMET followed by section III, which presents the modeling methods of the algorithms to be tested and validated in VMET. Its details are then further presented in section IV followed by Section V presenting integration, test status, and state analysis. Finally, section VI addresses the summary and forward directions followed by the appendices presenting relevant information on terminology and documentation.

Trevino, Luis↗

Bioreactor perfusion system for the long-term maintenance of tissue-engineered skeletal muscle organoids

Three-dimensional skeletal muscle organ-like structures (organoids) formed in tissue culture by fusion of proliferating myoblasts into parallel networks of long, unbranched myofibers provide an in vivo-like model for examining the effects of growth factors, tension, and space flight on muscle cell growth and metabolism. To determine the feasibility of maintaining either avian or mammalian muscle organoids in a commercial perfusion bioreactor system, we measured metabolism, protein turnover. and autocrine/paracrine growth factor release rates. Medium glucose was metabolized at a constant rate in both low-serum- and serum-free media for up to 30 d. Total organoid noncollagenous protein and DNA content decreased approximately 22-28% (P < 0.05) over a 13-d period. Total protein synthesis rates could be determined accurately in the bioreactors for up to 30 h and total protein degradation rates could be measured for up to 3 wk. Special fixation and storage conditions necessary for space flight studies were validated as part of the studies. For example, the anabolic autocrine/paracrine skeletal muscle growth factors prostaglandin F2alpha (PGF2alpha) and insulin-like growth factor-1 (IGF-1) could be measured accurately in collected media fractions, even after storage at 37 degrees C for up to 10 d. In contrast, creatine kinase activity (a marker of cell damage) in collected media fractions was unreliable. These results provide initial benchmarks for long-term ex vivo studies of tissue-engineered skeletal muscle.

NASA Discipline Musculoskeletal↗

Human Research Program Advanced Exercise Concepts (AEC) Overview

Exercise countermeasures provide benefits that are crucial for successful human spaceflight, to mitigate the spaceflight physiological deconditioning which occurs during exposure to microgravity. The NASA Human Research Program (HRP) within the Human Exploration and Operations Mission Directorate (HEOMD) is managing next generation Advanced Exercise Concepts (AEC) requirements development and candidate technology maturation to Technology Readiness Level (TRL) 7 (ground prototyping and flight demonstration) for all exploration mission profiles from Multi Purpose Crew Vehicle (MPCV) Exploration Missions (up to 21 day duration) to Mars Transit (up to 1000 day duration) missions. These validated and optimized exercise countermeasures systems will be provided to the ISS Program and MPCV Program for subsequent flight development and operations. The International Space Station (ISS) currently has three major pieces of operational exercise countermeasures hardware: the Advanced Resistive Exercise Device (ARED), the second-generation (T2) treadmill, and the cycle ergometer with vibration isolation system (CEVIS). This suite of exercise countermeasures hardware serves as a benchmark and is a vast improvement over previous generations of countermeasures hardware, providing both aerobic and resistive exercise for the crew. However, vehicle and resource constraints for future exploration missions beyond low Earth orbit will require that the exercise countermeasures hardware mass, volume, and power be minimized, while preserving the current ISS capabilities or even enhancing these exercise capabilities directed at mission specific physiological functional performance and medical standards requirements. Further, mission-specific considerations such as preservation of sensorimotor function, autonomous and adaptable operation, integration with medical data systems, rehabilitation, and in-flight monitoring and feedback are being developed for integration with the exercise countermeasures systems. Numerous technologies have been considered and evaluated against HRP-approved functional device requirements for these extreme mission profiles, and include wearable sensors, exoskeletons, flywheel, pneumatic, and closed-loop microprocessor controlled motor driven systems. Each technology has unique advantages and disadvantages. The Advanced Exercise Concepts project oversees development of candidate next generation exercise countermeasures hardware, performs trade studies of current and state of the art exercise technologies, manages and supports candidate systems physiological evaluations with human test subjects on the ground, in flight analogs and flight. The near term goal is evaluation of candidate systems in flight, culminating in an integrated candidate next generation exercise countermeasures suite on the ISS which coalesces research findings from HRP disciplines in the areas of exercise performance for muscle, bone, cardiovascular, sensorimotor, behavioral health, and nutrition for optimal benefit to the crew.

Countermeasures↗

A Performance Analysis on Soil Dielectric Models Over Organic Soils in Alaska for Passive Microwave Remote Sensing of Soil Moisture

Passive microwave remote sensing of soil moisture (SM) requires a physically based dielectric model that quantitatively converts the volumetric SM into the soil bulk dielectric constant. Mironov 2009 is the dielectric model used in the operational SM retrieval algorithms of the NASA Soil Moisture Active Passive (SMAP) and the ESA Soil Moisture and Ocean Salinity (SMOS) missions. However, Mironov 2009 suffers a challenge in deriving SM over organic soils, as it does not account for the impact of soil organic matter (SOM) on the soil bulk dielectric constant. To this end, we presented a comparative performance analysis of nine advanced soil dielectric models over organic soil in Alaska, four of which incorporate SOM. In the framework of the SMAP single-channel algorithm at vertical polarization (SCA-V), SM retrievals from different dielectric models were derived using an iterative optimization scheme. The skills of the different dielectric models over organic soils were reflected by the performance of their respective SM retrievals, which was measured by four conventional statistical metrics, calculated by comparing satellite-based SM time series with in-situ benchmarks. Overall, SM retrievals of organic-soil-based dielectric models tended to overestimate, while those from mineral-soil-based models displayed dry biases. All the models showed comparable values of unbiased root-mean-square error (ubRMSE) and Pearson Correlation (R), but Mironov 2019 exhibited a slight but consistent edge over the others. An integrated consideration of the model inputs, the physical basis, and the validated accuracy indicated that the separate use of Mironov 2009 and Mironov 2019 in the SMAP SCA-V for mineral soils (SOM < 15%) and organic soils (SOM ≥ 15%) would be the preferred option.

Soil Moisture↗

Improved Benchmarking of Cohesive Elements in Abaqus Standard for Predicting Disbond and Delamination in Composite Structures

Traditional approaches for aircraft certification require the assumption of an initial flaw condition, either represented as barely visible impact damage (BVID) or through inclusion of a Teflon insert to serve as surrogate damage. Based on the initial composite damage state, the structure must be shown to demonstrate structural durability and damage tolerance (DaDT) according to the following criteria: a. Damage displays no detrimental growth under cyclic loading. The structure is able to sustain design limit load (DLL). Currently, the only available manner for validating structural performance is through test. Since damage can occur over a wide variety of areas within a structure, this approach has proven to be increasingly expensive and time consuming for composite airframes and acreage structure within the design-test-certification building block. A further complicating factor is the requirement to accurately capture the most critical damage morphologies as a starting condition. To understand the severity of the damage, it is either required to experimentally determine the most critical areas at tremendous expense or rely on legacy data of similar structural testing, which limits design space expansion. A preferred solution is to use advanced analysis to provide improved understanding of load margins for critical locations based on a wide variety of potential starting damage conditions.

Eric M. Roberts↗

Expert system verification and validation study. ES V/V guidelines/workshop conference summary

The intent of the workshop was to start moving research on the verification and validation (V&V) of knowledge based systems (KBSs) in the direction of providing tangible 'products' that a KBS developer could use. In the near term research will focus on identifying the kinds of experiences encountered during KBS development of 'real' KBSs. These will be stored in a repository and will serve as the foundation for the rest of the activities described here. One specific approach to be pursued is 'benchmarking'. With this approach, a KBS developer can use either 'canned' KBSs with seeded errors or existing KBSs with known errors to evaluate a given tool's ability to satisfactorily identify errors.

French, Scott W.↗

AMD Radeon e9173 Low Power PCIE GPU Single Event Effects Test Report

The AMD Radeon Embedded e9170 Graphics Processing Unit (GPU), notably the e9173 Peripheral Component Interconnect Express (PCIE) variant, is of interest to Artemis generation programs with requirements for graphics rendering, compute, artificial intelligence (Ai) with a constraints-requiring piece-part procurement and power consumption of less than 50W. In addition to collecting heavy ion data on this device, a secondary purpose of this test campaign was to validate video capture hardware and software workflows used with GPU, microprocessor and system-on-chip device testing. Five (5) test patterns from the NEPP Processor Enclave (NPE) test suite were used with the e9173. The test patterns covered the operating system’s (OS) idle contribution towards the cross section, matrix math using tensorflow-rocm, two artificial intelligence models developed at NASA GSFC, and an industry standard GPU benchmarking application called Mesa GLXGears.

NASA Technical Memorandum (TM) test report for pos↗

High-Resolution Snowstorm Measurements and Retrievals Using Cross-Platform Multi-Frequency and Polarimetric Radars

Many studies have provided microphysical retrievals using radars of different frequencies, platforms, and methodologies. However, little is known about the consistency of retrievals derived from different radar platforms (i.e., airborne or spaceborne vs. groundbased) and their methodologies. This study is the first to directly compare snow mean volume diameter (Dm) retrievals from both nadir-pointing airborne multi-frequency radars and ground-based polarimetric range-height indicator (RHI) radar scans along a coincident airborne flight track. Compared to typical airborne/spaceborne/ground validation studies which at best provide O(1 km) resolution, these two platforms together provide directly collocated, high-resolution O(10m) retrievals. Dm retrieval values were over all both largest and smallest when using two separate dual-wavelength methods from aircraft measurements. A Triple-frequency analysis suggests the possibility that snow aggregates were generally composed of needles. The results shown here can be used as a benchmark for comparing retrieval methodologies.

Edwin L. Dunnavan↗

Assurance of COTS Boards for Space Flight

Space Flight hardware and software designers are increasingly turning to Commercial-Off-the-Shelf (COTS) products in hopes of meeting the demands imposed on them by projects with short development cycle times. The Technology Validation Assurance (TVA) team at NASA GSFC has embarked on applying a method for inserting COTS hardware into the Spartan 251 spacecraft. This method includes Procurement, Characterization, Ruggedization/Remediation and Verification Testing process steps which are intended to increase the uses confidence in the hardware's ability to function in the intended application for the required duration. As this method is refined with use, it has the potential for becoming a benchmark for industry-wide use of COTS in high reliability systems.

Plante, Jeannette↗

Towards the Prediction of the Mars 2020 Heatshield Material Response

Introduction: NASA’s next mission to Mars, the Mars 2020, will use the same heatshield of the Mars Science Laboratory (MSL) for thermal protection during entry, descent and landing. The heatshield is a tiled system made of Phenolic Impregnated Carbon Ablators (PICA) blocks [1]. PICA is a lightweight carbon fiber/polymeric resin material that offers excellent performances for protecting probes during planetary entry. The Mars Entry Descent and Landing Instrument (MEDLI) suite on MSL offers unique in-flight validation data for models of atmospheric entry and material response. MEDLI recorded, among others, time-resolved in-depth temperature data of PICA using thermocouple sensors assembled in the MEDLI Integrated Sensor Plugs (MISP). The objective of this work is to compare the thermal response of the MSL heatshield to the MISP flight data. In preparation to Mars 2020 post-flight analysis, the predictive material response capability is benchmarked against MEDLI flight data.

heatshield↗

Benchmarking Computational Tools for Calling SNPs and Indels in Complex Microbial Populations

The NASA BioNutrients missions seek to understand the suitability of microorganisms for bioproduction during space flight. One topic of interest is the stability of microbial genomes during long-term ambient storage and subsequent rehydration and growth. To address these questions, samples from 8 species were flown to ISS for 5 years of desiccated storage at ambient temperature (Stasis Packs) and 2 species were packaged along with powdered media inside a bioreactor system to allow hydration and growth in microgravity (Production Packs). For both systems, Whole Genome Sequencing (WGS) of the DNA extracted from the returned samples and paired ground controls will be conducted to identify changes in genome stability due to time, storage conditions and growth in space. Across the technical replicates, ground controls, 10 timepoints, and multiple experimental conditions, ~300 samples have been selected for initial analysis with WGS sequencing to 100x coverage. A flexible and resource efficient mutation calling pipeline is needed to process this large dataset and allow for comparisons between species. Many bioinformatics tools for calling Indels and Single Nucleotide Variants (SNVs) are designed for use with pure isolates, where true variations from the reference genome are expected to dominate the reads aligning to the location of mutation. In contrast, DNA from the Stasis Pack (SP) samples was collected directly after recovery from desiccated storage and the Production Pack (PP) samples were collected after fermentation. In this context, reads with mutations are expected to be less frequent than reads that align with the reference genome, as each sample will include multiple lines of cells. Thus, BioNutrients samples are expected to be similar to samples from cancer cell or “pooled” sequencing approaches. In preparation for the analysis of the BioNutrients samples, we have tested three mutation calling tools (GATK for Microbes, BreSeq and DiscoSNP) designed for complex samples. A challenge of validating mutation identification pipelines is a lack of “Ground Truth” datasets, especially for complex samples. To compare these three tools, we sought to identify mutations in pre-existing WGS data collected from populations of Chlamydomonas reinhardtii that were exposed to UV mutagenesis and growth in LEO as part of the Space Algae-1 mission. Here we present a summary of these tools against the analysis originally conducted using the CRISP tool. Critical metrics are compared such as runtime, the number of SNPs, the number and size of Indels, and patterns of transversion and transitions identified by each tool are reported. By sharing these benchmarking results collected in support of the BioNutrients mission, we aim to guide others seeking to identify SNVs in similarly complex microbial samples.

Biology↗

Assurance of COTS Boards for Space Flight

Space Flight hardware and software designers are increasingly turning to Commercial-Off-the-Shelf (COTS) products in hopes of meeting the demands imposed on them by projects with short development cycle times. The Technology Validation Assurance (TVA) team at NASA GSFC has embarked on applying a method for inserting COTS hardware into the Spartan 251 spacecraft. This method includes Procurement, Characterization, Ruggedization/Remediation and Verification Testing process steps which are intended to increase the user's confidence in the hardware's ability to function in the intended application for the required duration. As this method is refined with use, it has the potential for becoming a benchmark for industry-wide use of COTS in high reliability systems.

Plante Jeannette↗

Flight data acquisition methodology for validation of passive ranging algorithms for obstacle avoidance

An account is given of the methodology used to develop a data base containing low altitude imagery, augmented with the rotorcraft and sensor parameters required for passive-range estimation. The bata base is predicated on flight test data encompassing imagery, rotorcraft and sensor parameters, and ground truth range measurements. Data preparation was conducted by means of a novel technique for obtaining the sensor-calibration parameters. The data-base thus obtained will facilitate further development of computer vision-based obstacle-detection and passive range-estimation algorithms, as well as furnish a benchmark for verification of range estimates against ground-truth estimates.

Smith, Phillip N.↗