Search NASA⌕ Search

SEARCH · Search NASA

Results for “validations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS1989

NASA's CFD Validation Program

With computational fluid dynamics (CFD) becoming a productive research and design tool, the requirement to validate CFD codes has grown significantly. NASA had emphasized CFD validation activities since 1986 when a separate work element was formed to fund experimental activities related to validation. NASA's CFD and CFD validation programs are closely coordinated to ensure that experimental data bases are available as soon as possible for validating codes. In response to industry and academic requirements, four levels of experimental research have been defined as part of CFD validation with NASA's Aeronautics Advisory Committee (AAC) support although only the fourth level actually has the detailed information necessary for validating codes. Critical flow physics especially turbulence modeling are key to improved CFD codes. NASA has focused additional resources on transition and turbulence physics to meet these requirements. With improved turbulence models, CFD codes will be more accurate, robust, and efficient. However, with the level of detailed information available from CFD codes, highly accurate and detailed experiments are required to capture the critical information for validating codes. Advanced instrumentation especially non-intrusive instrumentation is required to acquire this information in validation experiments. The CFD validation program is being coordinated and managed to address these critical activities. A list of experiments which are currently being supported at least partially are included.

Satran, Dale R.↗
NASA NTRS1996

Validation of NASA Thermal Ice Protection Computer Codes: Program Overview - Part 1

The Icing Technology Branch at NASA Lewis has been involved in an effort to validate two thermal ice protection codes developed at the NASA Lewis Research Center. LEWICE/Thermal (electrothermal deicing & anti-icing), and ANTICE (hot-gas & electrothermal anti-icing). The Thermal Code Validation effort was designated as a priority during a 1994 'peer review' of the NASA Lewis Icing program, and was implemented as a cooperative effort with industry. During April 1996, the first of a series of experimental validation tests was conducted in the NASA Lewis Icing Research Tunnel(IRT). The purpose of the April 96 test was to validate the electrothermal predictive capabilities of both LEWICE/Thermal, and ANTICE. A heavily instrumented test article was designed and fabricated for this test, with the capability of simulating electrothermal de-icing and anti-icing modes of operation. Thermal measurements were then obtained over a range of test conditions, for comparison with analytical predictions. This paper will present an overview of the test, including a detailed description of: (1) the validation process; (2) test article design; (3) test matrix development; and (4) test procedures. Selected experimental results will be presented for de-icing and anti-icing modes of operation. Finally, the status of the validation effort at this point will be summarized. Detailed comparisons between analytical predictions and experimental results are contained in the following two papers: 'Validation of NASA Thermal Ice Protection Computer Codes: Part 2- The Validation of LEWICE/Thermal' and 'Validation of NASA Thermal Ice Protection Computer Codes: Part 3-The Validation of ANTICE'

Miller, Dean↗
NASA NTRSDate not supplied

TROPOMI Methane Validation in the GeoCarb Domain

Objective: GeoCarb validation •Purpose: •Do the four TCCON sites in the GeoCarb domain characterize satellite validation? •Method: •Assess the error estimates globally vs. in the GeoCarb domain •Assess the use of GML/DOE aircraft for CH4 validation •Systematic error estimate (from Kulawik, 2016): •+ regional bias (stdevof average bias versus validation at each station) •+ correlated error (stdev of daily average versus validation) •-co-location error (above quantities for model@satelliteminus model@valid) •-validation error (systematic error of validation estimates, e.g. model extension) Aircraft validation error •CH4 profile extension error estimate: 7 ppb •Extend aircraft observations to the top of the atmosphere with different model configurations •CAMS, scaling to stratosphere, then taking stratosphere •Extend by repeating top aircraft value to the troposphere, then taking CAMS •Extend with GEOS-Chem run (from 2010). •Difference CAMS-Scale vs. CAMS-Tropo: •3.6 +-5.5 ppb •Systematic error: 6.6 ppb •Extending aircraft to tropopause is blatantly wrong, but these two have the same stratosphere. •Difference CAMS-Scale vs. GEOS-Scale: •6.2 +-8.7 ppb •Do not consider bias since model is 10 years off. Systematic error ~8.7 ppb. •Estimate of systematic error from profile extension (possibly underestimated, resulting in possibly higher TROPOMI error estimate) •7 ppb •Would likely vary some by season, site, other conditions •Aircraft measurement error: 1 ppb •1 ppb from measurement error3

TROMPOMI↗
NASA NTRS2004

The Role of Structural Models in the Solar Sail Flight Validation Process

NASA is currently soliciting proposals via the New Millennium Program ST-9 opportunity for a potential Solar Sail Flight Validation (SSFV) experiment to develop and operate in space a deployable solar sail that can be steered and provides measurable acceleration. The approach planned for this experiment is to test and validate models and processes for solar sail design, fabrication, deployment, and flight. These models and processes would then be used to design, fabricate, and operate scaleable solar sails for future space science missions. There are six validation objectives planned for the ST9 SSFV experiment: 1) Validate solar sail design tools and fabrication methods; 2) Validate controlled deployment; 3) Validate in space structural characteristics (focus of poster); 4) Validate solar sail attitude control; 5) Validate solar sail thrust performance; 6) Characterize the sail's electromagnetic interaction with the space environment. This poster presents a top-level assessment of the role of structural models in the validation process for in-space structural characteristics.

Johnston, John D.↗
NASA NTRS2021

Validation Assessment for the Soil Moisture Active Passive (SMAP) Level 4 Carbon (L4_C) Data Product Version 5

The post-launch Cal/Val phase of the SMAP mission is guided by two primary objectives for each science product team: 1) to calibrate, verify, and improve the performance of the science algorithms, and 2) validate accuracies of the science data products as specified in the SMAP Level-1 mission science requirements. Algorithm science and product maintenance activities during the SMAP extended mission phase have also involved periodic algorithm calibration and product refinements to maintain or enhance product consistency and performance as well as science utility. This report provides an assessment of the latest (Version 5) SMAP Level 4 Carbon (L4_C) product. The L4_C Version 5 (v5) global record now spans more than six years (March 2015 – present) of SMAP operations and has benefited from five major reprocessing updates to the operational product. These reprocessing events and L4_C product release updates have incorporated various algorithm refinements and calibration adjustments to account for similar refinements to the upstream GEOS land model assimilation system, SMAP brightness temperatures, and MODIS vegetation inputs used for L4_C processing. The SMAP L4_C algorithms utilize a terrestrial carbon flux model informed by daily surface and root zone soil moisture information contributed from the SMAP Level 4 Soil Moisture (L4_SM) product along with optical remote sensing-based (e.g. MODIS-based) land cover and canopy fractional photosynthetic active radiation (fPAR), and other ancillary biophysical data. The carbon flux model estimates global daily net ecosystem CO2 exchange (NEE) and the component carbon fluxes, namely, vegetation gross primary production (GPP) and soil heterotrophic respiration (Rh). Other L4_C product elements include surface (~0-5 cm depth) soil organic carbon (SOC) stocks and associated environmental constraints to these processes, including soil moisture-related controls on GPP and ecosystem respiration (Kimball et al. 2014, Jones et al. 2017). The L4_C product addresses SMAP carbon cycle science objectives by: 1) providing a direct link between terrestrial carbon fluxes and underlying freeze/thaw and soil moisture-related constraints to these processes, 2) documenting primary connections between terrestrial water, energy and carbon cycles, and 3) improving understanding of terrestrial carbon sink activity. The SMAP L4_C algorithms and operational product are mature and at a CEOS Validation Stage 4 level (Jackson et al. 2012) based on extensive validation of the multi-year record against a diverse array of independent benchmarks, well characterized global performance, and systematic refinements gained from five major reprocessing events. There are no Level-1 mission science requirements for the L4_C product; however, self-imposed requirements have been established focusing on NEE as the primary product field for validation, and on demonstrating L4_C accuracy and success in meeting product science requirements (Jackson et al. 2012). The other L4_C product fields also have strong utility for carbon science applications (e.g., Liu et al. 2019, Endsley et al. 2020); however, analysis of these other fields is considered secondary relative to primary validation activities focusing on NEE. The L4_C targeted accuracy requirements are to meet or exceed a mean unbiased root-mean-square error (ubRMSE, or standard deviation of the error) for NEE of 1.6 g C m-2 d-1 and 30 g C m-2 yr-1, emphasizing northern (≥45°N) boreal and arctic ecosystems; this accuracy is similar to that of tower eddy covariance measurement-based observations (Baldocchi 2008). Methods used for the latest v5 L4_C product performance and validation assessment have been established from the SMAP Cal/Val plan and previous studies (Jackson et al. 2012, Jones et al. 2017) and include: 1) consistency evaluations of the product fields against earlier product releases (version 4 or earlier); 2) comparisons of daily carbon flux estimates with independent tower eddy 4covariance measurement-based daily carbon (CO2) flux observations from core tower validation sites (CVS); and 3) consistency checks against other global carbon products, including soil carbon inventory records, global GPP records derived from tower observation upscaling methods, and satellite-based observations of canopy solar induced chlorophyll fluorescence (SIF) as a surrogate for GPP. Metrics used to evaluate relative agreement between L4_C product fields and observational benchmarks include correlation (r-value), RMSE differences, bias and model sensitivity diagnostics. Following these validation criteria, the present report provides a validation assessment of the latest L4_C product release (v5). Detailed descriptions of the L4_C algorithm and additional global product accuracy and performance results are given elsewhere (Jones et al. 2017, Endsley et al. 2020). The v5 L4_C product replaces earlier product versions and continues to show: (i) accuracy and performance levels meeting or exceeding SMAP L4_C science requirements; (ii) improvement over the previous product version (version 4); and (iii) suitability for a diversity of science applications. Example L4_C applications from the recent literature include clarifying environmental trends and controls on the northern terrestrial carbon sink (Liu et al. 2019), diagnosing drought-related impacts on ecosystem productivity (Li et al. 2020), and regional monitoring of cropland conditions for projecting annual yields (Wurster et al. 2020). 2EXPECTED L4_C ALGORITHM AND PRODUCTPERFORMANCE The L4_C algorithm performance, including variance and uncertainty estimates of model outputs, was determined during the mission pre-launch phase through spatially explicit model sensitivity studies using available model inputs similar to those currently being used for operational production and evaluating the resulting model simulations over the observed range of northern (≥45 °N) and global conditions (Kimball et al. 2012, Entekhabi et al. 2014). The L4_C algorithm options were also evaluated during the mission prelaunch phase, including deriving canopy fPAR from lower order NDVI (Normalized Difference Vegetation Index) inputs in lieu of using MODIS (MOD15) fPAR; and including an explicit model representation of boreal fire disturbance recovery impacts. These results indicated that the L4_C accuracy requirements (i.e., NEE ubRMSE ≤ 30 g C m-2 yr-1or ≤ 1.6 g C m-2 d-1) could be met from the baseline algorithms over more than 82% and 89% of global and northern vegetated land areas, respectively (Yi et al. 2013, Kimball et al. 2014). The global L4_C algorithm error budget for NEE derived during the mission prelaunch phase indicated that the estimated NEE ubRMSE uncertainty is proportional to GPP and is therefore larger in higher biomass productivity areas, including forests and croplands (Kimball et al. 2014). Likewise, NEE ubRMSE uncertainty is expected to be lower in less-productive areas, including grasslands and shrublands. Expected model NEE ubRMSE levels were also generally within targeted accuracy levels for characteristically less-productive boreal and Arctic biomes, even though relative model error as a proportion of total productivity (NEE RMSE / GPP) may be large in these areas. The estimated NEE uncertainty was lower than expected in some warmer tropical high biomass productivity areas (e.g. Amazon rainforest) because of reduced low temperature and moisture constraints to the L4_C respiration calculations so that the bulk of model uncertainty is contributed by GPP in these areas. Model NEE uncertainty in the African Congo was estimated to

SMAP↗
NASA NTRSDate not supplied

Identification and Study of Validation Level Test Cases for Computational Modeling of Non-Charring Ablators

Computational modeling of Thermal Protection System (TPS) materials, used for aerospace applications, provides numerous advantages in preliminary selection and design of a heatshield material and shape for atmospheric entry vehicles. However, to serve as a reliable tool for prediction of material thermal and ablative behavior, the modeling approach needs to be validated against real experimental and flight data, preferably at a range of applied conditions. The validation study is typically very complex as it requires reliable measured data not only for the material thermal response and surface recession, but also well characterized environmental conditions. The validation problem becomes even more complex when the material thermal response is dictated by multi-physics effects such as solid conduction, in-depth thermal decomposition, pyrolysis gas flow and chemical reactions. The multi-physics effects complicate not only the modeling effort, but also the experimental measurement for validation of various aspects of the highly coupled problem. In this study, an attempt is made to identify suitable experimental data that could serve as a source for validation of material thermal response modeling tools. To reduce the computational complexity, this study focuses only on non-charring ablators, where the material thermal response could be modeled with a single governing equation for solid conduction and the ablation is limited only to the surface of the material. With a well characterized and publicly available experimental data being sparse, the study is limited in presenting test cases for only three materials: camphor, graphite and FiberForm® in the sequence of increased modeling complexity. Graphite is a commonly used TPS material for aerospace applications, both for leading edges of high-speed vehicles and internal insulation of solid rocket motors. FiberForm® is a porous carbon pre-form used in preparation of the well known PICA material Tran et al. [1996]. Inclusion of camphor into the list is conditioned with the relative simplicity in modeling the material thermal and chemical response and the low-enthalpy flow environment. In addition, camphor has been used as a simple test material for study of flow transition behavior by Stock and Ginoux [1973] and assessment of a heatshield shape change at flight relevant conditions by Rotondi et al. [2022]. In this work, the identified experimental data was extracted from the public literature and test cases that yet have been published. As it was found from the review, not a single test case contains an exhaustive set of data that would validate every aspect of the material physics. However, in the data collected, various aspects of the material behavior can be still validated, such as surface and in-depth temperature, amount of recession and a shape change. The identified experimental data for each case is accompanied with a characterized flow environment and simulated boundary conditions predicted by a Data-Parallel Line Relaxation (DPLR) code Wright et al. [1998]. In addition, material thermal response numerical simulations in each test case are performed with Kentucky Aerothermodynamics and Thermal Response System (KATS-MR) Zibitsker et al. [2022] providing a comparative study and a sanity check for the proposed validation data. Sample results from the performed numerical study are shown below. Figure 1 shows distribution of surface heat flux and pressure values on a hemi-cylinder model made of FiberForm® and tested in HyMETS arc-jet facility. The results are shown for the high pressure condition among the two tests. Flow simulation was performed with DPLR code on a quarter of original geometry. In the figure, the quarter shape was mirrored across zx and xy planes to show the complete distribution. Figure 2 shows the material response results for the high pressure case (7500 Pa), simulated with KATS-MR and a comparison to the experimental data for the surface temperature and shape shape. The simulation was performed on a 2-D slice, extracted in the xy plane at the middle of the sample. Figure 3 shows the material response simulation for the low pressure case (3500 Pa) and a comparison to the experimental data for surface temperature and shape change.

ablation↗
NASA NTRS2000

Earth Science Enterprise Scientific Data Purchase Project: Verification and Validation

This paper presents viewgraphs on the Earth Science Enterprise Scientific Data Purchase Project's verification,and validation process. The topics include: 1) What is Verification and Validation? 2) Why Verification and Validation? 3) Background; 4) ESE Data Purchas Validation Process; 5) Data Validation System and Ingest Queue; 6) Shipment Verification; 7) Tracking and Metrics; 8) Validation of Contract Specifications; 9) Earth Watch Data Validation; 10) Validation of Vertical Accuracy; and 11) Results of Vertical Accuracy Assessment.

Jenner, Jeff↗
NASA NTRS2004

Review of TRMM/GPM Rainfall Algorithm Validation

A review is presented concerning current progress on evaluation and validation of standard Tropical Rainfall Measuring Mission (TRMM) precipitation retrieval algorithms and the prospects for implementing an improved validation research program for the next generation Global Precipitation Measurement (GPM) Mission. All standard TRMM algorithms are physical in design, and are thus based on fundamental principles of microwave radiative transfer and its interaction with semi-detailed cloud microphysical constituents. They are evaluated for consistency and degree of equivalence with one another, as well as intercompared to radar-retrieved rainfall at TRMM's four main ground validation sites. Similarities and differences are interpreted in the context of the radiative and microphysical assumptions underpinning the algorithms. Results indicate that the current accuracies of the TRMM Version 6 algorithms are approximately 15% at zonal-averaged / monthly scales with precisions of approximately 25% for full resolution / instantaneous rain rate estimates (i.e., level 2 retrievals). Strengths and weaknesses of the TRMM validation approach are summarized. Because the dew of convergence of level 2 TRMM algorithms is being used as a guide for setting validation requirements for the GPM mission, it is important that the GPM algorithm validation program be improved to ensure concomitant improvement in the standard GPM retrieval algorithms. An overview of the GPM Mission's validation plan is provided including a description of a new type of physical validation model using an analytic 3-dimensional radiative transfer model.

Smith, Eric A.↗
NASA NTRSDate not supplied

Validation of Decompression Sickness Risk Mitigation Protocols for Planetary Spaceflight Missions

BACKGROUND: Apollo missions used a 100% O2 cabin atmosphere which effectively eliminated the risk of decompression sickness (DCS) during extravehicular activity (EVA) on the moon. NASA’s future missions to the moon and Mars are expected to use nitrox gas mixtures of up to 34% O2, 66% N2, which will reduce flammability risk compared with Apollo, but will necessitate Oxygen prebreathe prior to EVA to reduce DCS risk to acceptable levels. Prebreathe protocols used on the space shuttle and International Space Station are validated for microgravity EVAs, but the significantly increased risk of DCS during equivalent ambulatory EVAs make these protocols inapplicable to planetary EVA. An “exploration atmosphere” of 56.5 kPa (8.2 psia), 34% O2, 66% N2 has been recommended by NASA as a compromise that balances prebreathe duration, hypoxia, and flammability risk, assuming a 29.6 kPa (4.3 psi) spacesuit. However, this atmosphere may not be used for vehicles that do not support frequent EVA, and with commercial providers and international providers expected to provide landers, pressurized rovers, habitats, and spacesuits, different combinations of vehicle and spacesuit atmospheres are possible and will each require validated prebreathe protocols. OVERVIEW: Key components of a multi-year strategic roadmap include: 1) Establish hypobaric chamber facility capable of supporting 8-person EVA prebreathe validation tests at saturation atmospheres up to 36% O2; 2) validate an EVA physical workload simulation for use during prebreathe validation testing; 3) validate the recommended “exploration atmosphere” prebreathe protocol; 4) validate prebreathe protocols for additional atmospheric combinations that bound the most likely potential operating ranges of future vehicles and spacesuits; and 5) update DCS risk estimation models based on results of prebreathe validation studies. DISCUSSION: Details and data from completion of the first two steps of the strategic roadmap will be presented; the third step is currently underway, with pilot results provided in a companion presentation. Steps four and five will require a multi-year series of chamber tests; collaborations are being pursued.

Andrew F. J. Abercromby↗
NASA NTRS2016

Experience with Aero- and Fluid-Dynamic Testing for Engineering and CFD Validation

Ever since computations have been used to simulate aerodynamics the need to ensure that the computations adequately represent real life has followed. Many experiments have been performed specifically for validation and as computational methods have improved, so have the validation experiments. Validation is also a moving target because computational methods improve requiring validation for the new aspect of flow physics that the computations aim to capture. Concurrently, new measurement techniques are being developed that can help capture more detailed flow features pressure sensitive paint (PSP) and particle image velocimetry (PIV) come to mind. This paper will present various wind-tunnel tests the author has been involved with and how they were used for validation of various kinds of CFD. A particular focus is the application of advanced measurement techniques to flow fields (and geometries) that had proven to be difficult to predict computationally. Many of these difficult flow problems arose from engineering and development problems that needed to be solved for a particular vehicle or research program. In some cases the experiments required to solve the engineering problems were refined to provide valuable CFD validation data in addition to the primary engineering data. All of these experiments have provided physical insight and validation data for a wide range of aerodynamic and acoustic phenomena for vehicles ranging from tractor-trailers to crewed spacecraft.

experiment↗
NASA NTRSDate not supplied

AVT-297 Development of a Framework for Validation of Computational Tools

Design of aerospace vehicles relies on system models. These models rely on validation data to ensure accurate representation of the system. This paper provides an introduction to the problem and an overview of a NATO STO Activity, AVT-297, which developed a framework to determine validation experiments required to support system level models. The goals of the research group were to develop a process to identify a validation database for intended uses of a vehicle system, demonstrate the process on vehicles and document the process. Applying this process to multiple intended uses and collecting this set of validation databases identifies a set of validation experiments that could be applied to the design of multiple vehicles, further reducing the cost and time to develop air and naval vehicles. Four subgroup teams focused on different aspects of the validation database process: two focused on different aspects of a missile, and two focused on different aspects of a fixed-wing mobility transport class vehicle. This resulted in three different processes to decompose the multi-disciplinary problem to identify critical validation data. All four of the teams demonstrated their process on a sample problem demonstrating the benefit of the process.

Validation↗
NASA NTRS1999

EOS Terra Validation Program

The EOS Terra mission will be launched in July 1999. This mission has great relevance to the atmospheric radiation community and global change issues. Terra instruments include ASTER, CERES, MISR, MODIS and MOPITT. In addition to the fundamental radiance data sets, numerous global science data products will be generated, including various Earth radiation budget, cloud and aerosol parameters, as well as land surface, terrestrial ecology, ocean color, and atmospheric chemistry parameters. Significant investments have been made in on-board calibration to ensure the quality of the radiance observations. A key component of the Terra mission is the validation of the science data products. This is essential for a mission focused on global change issues and the underlying processes. The Terra algorithms have been subject to extensive pre-launch testing with field data whenever possible. Intensive efforts will be made to validate the Terra data products after launch. These include validation of instrument calibration (vicarious calibration) experiments, instrument and cross-platform comparisons, routine collection of high quality correlative data from ground-based networks, such as AERONET, and intensive sites, such as the SGP ARM site, as well as a variety field experiments, cruises, etc. Airborne simulator instruments have been developed for the field experiment and underflight activities including the MODIS Airborne Simulator (MAS), AirMISR, MASTER (MODIS-ASTER), and MOPITT-A. All are integrated on the NASA ER-2, though low altitude platforms are more typically used for MASTER. MATR is an additional sensor used for MOPITT algorithm development and validation. The intensive validation activities planned for the first year of the Terra mission will be described with emphasis on derived geophysical parameters of most relevance to the atmospheric radiation community. Detailed information about the EOS Terra validation Program can be found on the EOS Validation program homepage i/e.: http://ospso.gsfc.nasa.gov/validation/valpage.html).

Starr, David↗
NASA NTRS2003

Validation and Error Characterization for the Global Precipitation Measurement

The Global Precipitation Measurement (GPM) is an international effort to increase scientific knowledge on the global water cycle with specific goals of improving the understanding and the predictions of climate, weather, and hydrology. These goals will be achieved through several satellites specifically dedicated to GPM along with the integration of numerous meteorological satellite data streams from international and domestic partners. The GPM effort is led by the National Aeronautics and Space Administration (NASA) of the United States and the National Space Development Agency (NASDA) of Japan. In addition to the spaceborne assets, international and domestic partners will provide ground-based resources for validating the satellite observations and retrievals. This paper describes the validation effort of Global Precipitation Measurement to provide quantitative estimates on the errors of the GPM satellite retrievals. The GPM validation approach will build upon the research experience of the Tropical Rainfall Measuring Mission (TRMM) retrieval comparisons and its validation program. The GPM ground validation program will employ instrumentation, physical infrastructure, and research capabilities at Supersites located in important meteorological regimes of the globe. NASA will provide two Supersites, one in a tropical oceanic and the other in a mid-latitude continental regime. GPM international partners will provide Supersites for other important regimes. Those objectives or regimes not addressed by Supersites will be covered through focused field experiments. This paper describes the specific errors that GPM ground validation will address, quantify, and relate to the GPM satellite physical retrievals. GPM will attempt to identify the source of errors within retrievals including those of instrument calibration, retrieval physical assumptions, and algorithm applicability. With the identification of error sources, improvements will be made to the respective calibration, assumption, or algorithm. The instrumentation and techniques of the Supersites will be discussed. The GPM core satellite, with its dual-frequency radar and conically scanning radiometer, will provide insight into precipitation drop-size distributions and potentially increased measurement capabilities of light rain and snowfall. The ground validation program will include instrumentation and techniques commensurate with these new measurement capabilities.

Bidwell, Steven W.↗
NASA NTRS2014

Observations on CFD Verification and Validation from the AIAA Drag Prediction Workshops

The authors provide observations from the AIAA Drag Prediction Workshops that have spanned over a decade and from a recent validation experiment at NASA Langley. These workshops provide an assessment of the predictive capability of forces and moments, focused on drag, for transonic transports. It is very difficult to manage the consistency of results in a workshop setting to perform verification and validation at the scientific level, but it may be sufficient to assess it at the level of practice. Observations thus far: 1) due to simplifications in the workshop test cases, wind tunnel data are not necessarily the “correct” results that CFD should match, 2) an average of core CFD data are not necessarily a better estimate of the true solution as it is merely an average of other solutions and has many coupled sources of variation, 3) outlier solutions should be investigated and understood, and 4) the DPW series does not have the systematic build up and definition on both the computational and experimental side that is required for detailed verification and validation. Several observations regarding the importance of the grid, effects of physical modeling, benefits of open forums, and guidance for validation experiments are discussed. The increased variation in results when predicting regions of flow separation and increased variation due to interaction effects, e.g., fuselage and horizontal tail, point out the need for validation data sets for these important flow phenomena. Experiences with a recent validation experiment at NASA Langley are included to provide guidance on validation experiments.

Computational Fluid Dynamics↗
NASA NTRS2015

A Vehicle Management End-to-End Testing and Analysis Platform for Validation of Mission and Fault Management Algorithms to Reduce Risk for NASAs Space Launch System

The engineering development of the National Aeronautics and Space Administration's (NASA) new Space Launch System (SLS) requires cross discipline teams with extensive knowledge of launch vehicle subsystems, information theory, and autonomous algorithms dealing with all operations from pre-launch through on orbit operations. The nominal and off-nominal characteristics of SLS's elements and subsystems must be understood and matched with the autonomous algorithm monitoring and mitigation capabilities for accurate control and response to abnormal conditions throughout all vehicle mission flight phases, including precipitating safing actions and crew aborts. This presents a large and complex systems engineering challenge, which is being addressed in part by focusing on the specific subsystems involved in the handling of off-nominal mission and fault tolerance with response management. Using traditional model-based system and software engineering design principles from the Unified Modeling Language (UML) and Systems Modeling Language (SysML), the Mission and Fault Management (M&FM) algorithms for the vehicle are crafted and vetted in Integrated Development Teams (IDTs) composed of multiple development disciplines such as Systems Engineering (SE), Flight Software (FSW), Safety and Mission Assurance (S&MA) and the major subsystems and vehicle elements such as Main Propulsion Systems (MPS), boosters, avionics, Guidance, Navigation, and Control (GNC), Thrust Vector Control (TVC), and liquid engines. These model-based algorithms and their development lifecycle from inception through FSW certification are an important focus of SLS's development effort to further ensure reliable detection and response to off-nominal vehicle states during all phases of vehicle operation from pre-launch through end of flight. To test and validate these M&FM algorithms a dedicated test-bed was developed for full Vehicle Management End-to-End Testing (VMET). For addressing fault management (FM) early in the development lifecycle for the SLS program, NASA formed the M&FM team as part of the Integrated Systems Health Management and Automation Branch under the Spacecraft Vehicle Systems Department at the Marshall Space Flight Center (MSFC). To support the development of the FM algorithms, the VMET developed by the M&FM team provides the ability to integrate the algorithms, perform test cases, and integrate vendor-supplied physics-based launch vehicle (LV) subsystem models. Additionally, the team has developed processes for implementing and validating the M&FM algorithms for concept validation and risk reduction. The flexibility of the VMET capabilities enables thorough testing of the M&FM algorithms by providing configurable suites of both nominal and off-nominal test cases to validate the developed algorithms utilizing actual subsystem models such as MPS, GNC, and others. One of the principal functions of VMET is to validate the M&FM algorithms and substantiate them with performance baselines for each of the target vehicle subsystems in an independent platform exterior to the flight software test and validation processes. In any software development process there is inherent risk in the interpretation and implementation of concepts from requirements and test cases into flight software compounded with potential human errors throughout the development and regression testing lifecycle. Risk reduction is addressed by the M&FM group but in particular by the Analysis Team working with other organizations such as S&MA, Structures and Environments, GNC, Orion, Crew Office, Flight Operations, and Ground Operations by assessing performance of the M&FM algorithms in terms of their ability to reduce Loss of Mission (LOM) and Loss of Crew (LOC) probabilities. In addition, through state machine and diagnostic modeling, analysis efforts investigate a broader suite of failure effects and associated detection and responses to be tested in VMET to ensure reliable failure detection, and confirm responses do not create additional risks or cause undesired states through interactive dynamic effects with other algorithms and systems. VMET further contributes to risk reduction by prototyping and exercising the M&FM algorithms early in their implementation and without any inherent hindrances such as meeting FSW processor scheduling constraints due to their target platform - the ARINC 6535-partitioned Operating System, resource limitations, and other factors related to integration with other subsystems not directly involved with M&FM such as telemetry packing and processing. The baseline plan for use of VMET encompasses testing the original M&FM algorithms coded in the same C++ language and state machine architectural concepts as that used by FSW. This enables the development of performance standards and test cases to characterize the M&FM algorithms and sets a benchmark from which to measure their effectiveness and performance in the exterior FSW development and test processes. This paper is outlined in a systematic fashion analogous to a lifecycle process flow for engineering development of algorithms into software and testing. Section I describes the NASA SLS M&FM context, presenting the current infrastructure, leading principles, methods, and participants. Section II defines the testing philosophy of the M&FM algorithms as related to VMET followed by section III, which presents the modeling methods of the algorithms to be tested and validated in VMET. Its details are then further presented in section IV followed by Section V presenting integration, test status, and state analysis. Finally, section VI addresses the summary and forward directions followed by the appendices presenting relevant information on terminology and documentation.

Trevino, Luis↗
NASA NTRS2020

Hardware Verification and Validation for a Navigation Sensor Software Model in Support of Flight Vehicle Performance Analysis

… or, “It’s in the details, how to make complicated software perform like complicated hardware.” In attempts to minimize development time and quickly build an operational vehicle, NASA’s Space Launch System (SLS) has had to be intentional about integrated testing. Constraints on budget and schedule have required balance between testing needs and the desire for an integrated flight vehicle as soon as possible. To provide key insights early in design and analysis cycles, a large amount of effort has shifted into maturing and validating models at the component level with integrated testing as a means to validate their integration. In terms of SLS Navigation, this, and the model-based design approach have pushed explicit requirements for sensor models to be validated against flight hardware to high precision. This paper covers the approach taken to verify and validate the models for the two key navigation sensors on the SLS vehicle, the Redundant Inertial Navigation Sensor and the Rate Gyro Assembly. These models are used in performance evaluation, fault detection, and operations development extensively. Using a mix of data from hardware vendor documentation and testing reports, limited in-house testing, and integration activities, these models were able to be validated against flight hardware at multiple levels, from the internal software design to statistical behavior at the raw sensor and integrated box levels. The high level of insight into the hardware elements is instrumental to support flight certification activities and building confidence in SLS Navigation capability. Focused testing enabled additional insight and validation that proved invaluable and the resulting insights were used to focus and mature models. Additionally, of having validated performance-based hardware models enables a wide breadth of activities including detailed fault detection studies and integration into future vehicle frameworks, such as an upper stage and provide a valuable asset to continued SLS analysis and design.

Evan J Anzalone↗
NASA NTRS2020

Hardware Verification and Validation for a Navigation Sensor Software Model in Support of Flight Vehicle Performance Analysis

… or, “It’s in the details, how to make complicated software perform like complicated hardware.” In attempts to minimize development time and quickly build an operational vehicle, NASA’s Space Launch System (SLS) has had to be intentional about integrated testing. Constraints on budget and schedule have required balance between testing needs and the desire for an integrated flight vehicle as soon as possible. To provide key insights early in design and analysis cycles, a large amount of effort has shifted into maturing and validating models at the component level with integrated testing as a means to validate their integration. In terms of SLS Navigation, this, and the model-based design approach have pushed explicit requirements for sensor models to be validated against flight hardware to high precision. This paper covers the approach taken to verify and validate the models for the two key navigation sensors on the SLS vehicle, the Redundant Inertial Navigation Sensor and the Rate Gyro Assembly. These models are used in performance evaluation, fault detection, and operations development extensively. Using a mix of data from hardware vendor documentation and testing reports, limited in-house testing, and integration activities, these models were able to be validated against flight hardware at multiple levels, from the internal software design to statistical behavior at the raw sensor and integrated box levels. The high level of insight into the hardware elements is instrumental to support flight certification activities and building confidence in SLS Navigation capability. Focused testing enabled additional insight and validation that proved invaluable and the resulting insights were used to focus and mature models. Additionally, of having validated performance-based hardware models enables a wide breadth of activities including detailed fault detection studies and integration into future vehicle frameworks, such as an upper stage and provide a valuable asset to continued SLS analysis and design.

Thomas Park↗
NASA NTRS2017

Land Surface Temperature Product Validation Best Practice Protocol Version 1.0 - October, 2017

The Global Climate Observing System (GCOS) has specified the need to systematically generate andvalidate Land Surface Temperature (LST) products. This document provides recommendations on goodpractices for the validation of LST products. Internationally accepted definitions of LST, emissivity andassociated quantities are provided to ensure the compatibility across products and reference data sets. Asurvey of current validation capabilities indicates that progress is being made in terms of up-scaling and insitu measurement methods, but there is insufficient standardization with respect to performing andreporting statistically robust comparisons.Four LST validation approaches are identified: (1) Ground-based validation, which involvescomparisons with LST obtained from ground-based radiance measurements; (2) Scene-based intercomparisonof current satellite LST products with a heritage LST products; (3) Radiance-based validation,which is based on radiative transfer calculations for known atmospheric profiles and land surface emissivity;(4) Time series comparisons, which are particularly useful for detecting problems that can occur during aninstrument's life, e.g. calibration drift or unrealistic outliers due to undetected clouds. Finally, the need foran open access facility for performing LST product validation as well as accessing reference LST datasets isidentified.

best practice↗