Search NASA⌕ Search

SEARCH · Search NASA

Results for “Validation Methods”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRS2011

An Investigation Into HPLC Data Quality Problems

This report summarizes the analyses and results produced by a five-member investigative team of Government, university, and industry experts, established by NASA HQ. The team examined data quality problems associated with high performance liquid chromatography (HPLC) analyses of pigment concentrations in seawater samples produced by the San Diego State University (SDSU) Center for Hydro-Optics and Remote Sensing (CHORS). This report shows CHORS did not validate the methods used before placing them into service to analyze field samples for NASA principal investigators (PIs), even though the HPLC literature contained easily accessible method validation procedures, and the importance of implementing them, more than a decade ago. In addition, there were so many sources of significant variance in the CHORS methodologies, that the HPLC system rarely operated within performance criteria capable of producing the requisite data quality. It is the recommendation of the investigative team to a) not correct the data, b) make all the data that was temporarily sequestered available for scientific use, and c) label the affected data with an appropriate warning, e.g., "These data are not validated and should not be used as the sole basis for a scientific result, conclusion, or hypothesis--independent corroborating evidence is required."

Hooker, Stanford B.↗
NASA NTRS2020

Spatial validation reveals poor predictive performance of large-scale ecological mapping models

Mapping aboveground forest biomass is central for assessing the global carbon balance. However, current large-scale maps show strong disparities, despite good validation statistics of their underlying models. Here, we attribute this contradiction to a flaw in the validation methods, which ignore spatial autocorrelation (SAC) in data, leading to overoptimistic assessment of model predictive power. To illustrate this issue, we reproduce the approach of large-scale mapping studies using a massive forest inventory dataset of 11.8 million trees in central Africa to train and validate a random forest model based on multispectral and environmental variables. A standard nonspatial validation method suggests that the model predicts more than half of the forest biomass variation, while spatial validation methods accounting for SAC reveal quasi-null predictive power. This study underscores how a common practice in big data mapping studies shows an apparent high predictive power, even when predictors have poor relationships with the ecological variable of interest, thus possibly leading to erroneous maps and interpretations.

biomass↗
NASA NTRSDate not supplied

A Method for Validating Causal Diagrams of Human Health Risk in Space Flight

The complexity of cause-and-effect relationships between spaceflight hazards and resulting health conditions clouds understanding of the totality of human system risk in space. In response, NASA has introduced Directed Acyclic Graphs (causal diagrams) into the human systems risk management process. These diagrams allow for a common understanding of the mechanisms that lead from unique hazards of spaceflight to the health outcomes important to agencies and astronauts. However, the paucity of available biomedical data from spaceflight creates a need for methods of validating causal models that can accommodate data from spaceflight model analogs. Here we outline one approach utilizing open-access rodent bone datasets from the Ames Life Sciences Data Archive. The properties of directed acyclic graphs themselves can provide an epistemological and statistical framework for validation of a priori causal representations of human system risk in space flight. The assumed causal connections on the graph creates sets of logical implications: variables that – if the causal diagram is correct – should be correlated, as well as sets that should be conditionally independent. By testing these implied correlations and conditional independencies both statistically and heuristically, we can provide evidence for or against specific causal pathways on the causal diagram. In addition to validation of expert-generated causal diagrams, machine learning techniques can learn the most likely structure of a causal diagram from a given dataset. Comparison with and reconciliation between machine-learned causal diagrams and expert-generated diagrams is another technique for challenging assumptions and improving our understanding of causal mechanisms. Accurately representing complex causation is essential to systemic understanding of human health risks in space travel. Having a robust system of validating causal diagrams helps us arrive at more accurate representations of causal systems. This process will be integral to developing the countermeasures necessary for extended exploration of the moon and Mars.

Robert Reynolds↗
NASA NTRS2026

Methods for Validation of Random Uncertainty Estimates and Their Applications to Ozone Profiles From Limb-Viewing Satellite Instruments

For satellite measurements of atmospheric composition, the random uncertainty estimates provided by retrieval algorithms might be imperfect due to various approximations used in the retrievals or the presence of unknown error sources. This paper presents an overview of the methods used for the validation of random uncertainty estimates. All methods discussed in this study are categorized, and assumptions and limitations of each method are discussed. This overview evaluates these methods in application to ozone profile measurements from limb and occultation satellite instruments and provides practical illustrations of random uncertainty validation.

Viktoria F Sofieva↗
NASA NTRS1984

A validation methodology for fault-tolerant clock synchronization

A validation method for the synchronization subsystem of a fault-tolerant computer system is presented. The high reliability requirement of flight crucial systems precludes the use of most traditional validation methods. The method presented utilizes formal design proof to uncover design and coding errors and experimentation to validate the assumptions of the design proof. The experimental method is described and illustrated by validating an experimental implementation of the Software Implemented Fault Tolerance (SIFT) clock synchronization algorithm. The design proof of the algorithm defines the maximum skew between any two nonfaulty clocks in the system in terms of theoretical upper bounds on certain system parameters. The quantile to which each parameter must be estimated is determined by a combinatorial analysis of the system reliability. The parameters are measured by direct and indirect means, and upper bounds are estimated. A nonparametric method based on an asymptotic property of the tail of a distribution is used to estimate the upper bound of a critical system parameter. Although the proof process is very costly, it is extremely valuable when validating the crucial synchronization subsystem.

Johnson, S. C.↗
NASA NTRSDate not supplied

Validated Reduced Computational Methods for Realistic RDC Injection Modeling

This paper outlines a simplified computational method without chemistry for single and multi-injector performance for rotating detonation combustion (RDC) environment as well as relevant performance metrics. For non-reactive cases, an unsteady pressure profile from a Method of Characteristics (MOC) simulation was used to simulate an RDC combustion wave travelling around the annulus. In reactive cases, a pre-mixed stoichiometric mixture of hydrogen and air is injected into a 2D unwrapped combustion chamber with a specified injector area. The performance is evaluated based on several performance metrics including unsteady discharge coefficient, average mass flux coming into the combustor, and percentage time obstructed. Finally, the computational method was validated through an experimental RDC test setup with a single element injector in a hydrogen-air detonation driver environment.

Propulsion↗
NASA NTRS1987

Fault-tolerant clock synchronization validation methodology

A validation method for the synchronization subsystem of a fault-tolerant computer system is presented. The high reliability requirement of flight-crucial systems precludes the use of most traditional validation methods. The method presented utilizes formal design proof to uncover design and coding errors and experimentation to validate the assumptions of the design proof. The experimental method is described and illustrated by validating the clock synchronization system of the Software Implemented Fault Tolerance computer. The design proof of the algorithm includes a theorem that defines the maximum skew between any two nonfaulty clocks in the system in terms of specific system parameters. Most of these parameters are deterministic. One crucial parameter is the upper bound on the clock read error, which is stochastic. The probability that this upper bound is exceeded is calculated from data obtained by the measurement of system parameters. This probability is then included in a detailed reliability analysis of the system.

Computer systems↗
NASA NTRS2011

Validation of Methods to Predict Vibration of a Panel in the Near Field of a Hot Supersonic Rocket Plume

This paper describes the measurement and analysis of surface fluctuating pressure level (FPL) data and vibration data from a plume impingement aero-acoustic and vibration (PIAAV) test to validate NASA s physics-based modeling methods for prediction of panel vibration in the near field of a hot supersonic rocket plume. For this test - reported more fully in a companion paper by Osterholt & Knox at 26th Aerospace Testing Seminar, 2011 - the flexible panel was located 2.4 nozzle diameters from the plume centerline and 4.3 nozzle diameters downstream from the nozzle exit. The FPL loading is analyzed in terms of its auto spectrum, its cross spectrum, its spatial correlation parameters and its statistical properties. The panel vibration data is used to estimate the in-situ damping under plume FPL loading conditions and to validate both finite element analysis (FEA) and statistical energy analysis (SEA) methods for prediction of panel response. An assessment is also made of the effects of non-linearity in the panel elasticity.

Bremner, P. G.↗
NASA NTRS1991

Meteorological radar methods for validating space observations of precipitation

Meteorological approaches to verification of space measurements of rainfall are examined; validation of Tropical Rainfall Measuring Mission (TRMM) observations is expected to depend significantly on ground-based radars. Two methods of comparison are initially contemplated. TRMM rainfall data over time periods of a month for large areas (500 x 500 km) are averaged and compared with similarly averaged ground truth measurements. Both the rainfall and height distribution data from TRMM are compared with the instantaneous values observed at one or more 'ground truth' stations and from airborne radar and radiometers as available.

Thiele, Otto W.↗
NASA NTRS2019

Benchmarking CME Arrival Time and Impact: Progress on Metadata, Metrics, and Events

Accurate forecasting of the arrival time and subsequent geomagnetic impacts of coronal mass ejections (CMEs) at Earth is an important objective for space weather forecasting agencies. Recently, the CME Arrival and Impact working team has made significant progress toward defining communit yagreed metrics and validation methods to assess the current state of CME modeling capabilities. This will allow the community to quantify our current capabilities and track progress in models over time. First, it is crucial that the community focuses on the collection of the necessary metadata for transparency and reproducibility of results. Concerning CME arrival and impact we have identified six different metadata types: 3D CME measurement, model description, model input, CME (non)arrival observation, model output data, and metrics and validation methods. Second, the working team has also identified a validation time period, where all events within the following two periods will be considered: 1 January 2011 to 31 December 2012 and January 2015 to 31 December 2015. Those two periods amount to a total of about 100 hit events at Earth and a large amount of misses. Considering a time period will remove any bias in selecting events and the event set will represent a sample set that will not be biased by user selection. Lastly, we have defined the basic metrics and skill scores that the CME Arrival and Impact working team will focus on.

Verbeke, C.↗
NASA NTRS1989

Evidence flow graph methods for validation and verification of expert systems

The results of an investigation into the use of evidence flow graph techniques for performing validation and verification of expert systems are given. A translator to convert horn-clause rule bases into evidence flow graphs, a simulation program, and methods of analysis were developed. These tools were then applied to a simple rule base which contained errors. It was found that the method was capable of identifying a variety of problems, for example that the order of presentation of input data or small changes in critical parameters could affect the output from a set of rules.

Becker, Lee A.↗
NASA NTRS1988

Evidence flow graph methods for validation and verification of expert systems

This final report describes the results of an investigation into the use of evidence flow graph techniques for performing validation and verification of expert systems. This was approached by developing a translator to convert horn-clause rule bases into evidence flow graphs, a simulation program, and methods of analysis. These tools were then applied to a simple rule base which contained errors. It was found that the method was capable of identifying a variety of problems, for example that the order of presentation of input data or small changes in critical parameters could effect the output from a set of rules.

Becker, Lee A.↗
NASA NTRS1992

Evaluation of ultrasonic cavitation of metallic and non-metallic surfaces

1,1,2 trichloro-1,2,2 trifluoro ethane (CFC-113) commercially known as Freon-113 is the primary test solvent used for validating the cleaned hardware at the Kennedy Space Center (KSC). Due to the ozone depletion problem, the current United States policy calls for the phase out of Freons by 1995. NASAs chlorofluorocarbon (CFC) replacement group at KSC has opted to use water as a replacement fluid for the validation process since water is non-toxic, inexpensive, and is environmentally friendly. The replacement validation method calls for the ultrasonification of the small parts with water at 52 C for a cycle or two of 10 min duration wash using commercial ultrasonic baths. In this project, experimental data was obtained to assess the applicability of the proposed validation method for any damage of the metallic and non-metallic surfaces resulting from ultrasonic cavitation.

Mehta, Narinder K.↗
NASA NTRS1999

Lidar Altimeter Measurements of Canopy Structure: Methods and Validation for Closed Canopy, Broadleaf Forests

Lidar altimeter observations of vegetated landscapes provide a time-resolved measure of laser pulse backscatter energy from canopy surfaces and the underlying ground. Airborne lidar altimeter data was acquired using the Scanning Lidar Imager of Canopies by Echo Recovery (SLICER) for a successional sequence of four, closed-canopy, deciduous forest stands in eastern Maryland. The four stands were selected so as to include a range of canopy structures of importance to forest ecosystem function, including variation in the height and roughness of the outer-most canopy surface and the vertical organization of canopy stories and gaps. The character of the SLICER backscatter signal is described and a method is developed that accounts for occlusion of the laser energy by canopy surfaces, transforming the backscatter signal to a canopy height profile (CHP) that quantitatively represents the relative vertical distribution of canopy surface area. The transformation applies an increased weighting to the backscatter amplitude as a function of closure through the canopy and assumes a horizontally random distribution of the canopy components. SLICER CHPs, averaged over areas of overlap where lidar ground tracks intersect, are shown to be highly reproducible. CHP transects across the four stands reveal spatial variations in vegetation, at the scale of the individual 10 m diameter laser footprints, within and between stands. Averaged SLICER CHPs are compared to analogous height profile results derived from ground-based sightings to plant intercepts measured on plots within the four stands. Tbe plots were located on the segments of the lidar ground tracks from which averaged SLICER CHPs were derived, and the ground observations were acquired within two weeks of the SLICER data acquisition to minimize temporal change. The differences in canopy structure between the four stands is similarly described by the SLICER and ground-based CHP results, however a Chi-square test of similarity documents differences that are statistically significant. The differences are discussed in terms of measurement properties that define the smoothness of the resulting CHPs and Lidar Altimeter Measurements of Canopy Structure - Harding et al. canopy properties that may vertically bias the CHP representations of canopy structure. The statistical differences are most likely due to the more noisy character of the ground-based CHPs, especially high in the canopy where ground-based sightings are rare resulting in an underestimate of canopy surface area and height, and to departures from the assumption of horizontal randomness which bias the CHPs toward the observer (upward for SLICER and downward for ground-based CHPs). The results demonstrate that the SLICER observations reliably provide a measure of canopy structure that reveals ecologically interesting structural variations such as those characterizing a successional sequence of closed-canopy, broadleaf forest stands.

Harding, D. J.↗
NASA NTRS1987

Design and validation of fault-tolerant flight systems

Flight systems must be validated to show that they are consistent with the requirements of their intended applications. While high reliability is difficult to validate, the additional complexity of fault tolerance further compounds the validation problem. The objective of NASA’s research is to develop a methodology for designing validatable fault-tolerant systems. Under the design-for-validation philosophy, emphasis is placed on developing validation methods that can be incorporated into the design process right from the start and design methods and guidance which, while incorporating fault tolerance, can assure validatability. This paper examines the statistical issues of validating highly reliable, fault tolerant system. There are many problems associated with traditional methods of designing and validating these potentially complex hardware and software systems. Useful design-for-validation methods, which include structured specification and design methodologies, mathematical proof techniques, analytical modeling, simulation and emulation, and physical testing, are discussed. Important design issues associated with fault tolerance are presented along with the related validation concerns which must be addressed. Experience has shown that synchronization and Byzantine resilience must accompany fault tolerance. Other design attributes associated with fault tolerance may be used by a designer on the basis of cost, weight, performance, and validation considerations.

Computer systems↗
NASA NTRS1975

Development and validation of methods for man-made machine interface evaluation

The alternate methods of conducting a man-machine interface evaluation are classified as static and dynamic, and are evaluated. A dynamic evaluation tool is presented to provide for a determination of the effectiveness of the man-machine interface in terms of the sequence of operations (task and task sequences) and in terms of the physical characteristics of the interface. This dynamic checklist approach is recommended for shuttle and shuttle payload man-machine interface evaluations based on reduced preparation time, reduced data, and increased sensitivity of critical problems.

Malone, T. B.↗
NASA NTRS1990

Prediction of forces and moments for flight vehicle control effectors. Part 1: Validation of methods for predicting hypersonic vehicle controls forces and moments

Many types of hypersonic aircraft configurations are currently being studied for feasibility of future development. Since the control of the hypersonic configurations throughout the speed range has a major impact on acceptable designs, it must be considered in the conceptual design stage. The ability of the aerodynamic analysis methods contained in an industry standard conceptual design system, APAS II, to estimate the forces and moments generated through control surface deflections from low subsonic to high hypersonic speeds is considered. Predicted control forces and moments generated by various control effectors are compared with previously published wind tunnel and flight test data for three configurations: the North American X-15, the Space Shuttle Orbiter, and a hypersonic research airplane concept. Qualitative summaries of the results are given for each longitudinal force and moment and each control derivative in the various speed ranges. Results show that all predictions of longitudinal stability and control derivatives are acceptable for use at the conceptual design stage. Results for most lateral/directional control derivatives are acceptable for conceptual design purposes; however, predictions at supersonic Mach numbers for the change in yawing moment due to aileron deflection and the change in rolling moment due to rudder deflection are found to be unacceptable. Including shielding effects in the analysis is shown to have little effect on lift and pitching moment predictions while improving drag predictions.

Maughmer, Mark D.↗
NASA NTRS2004

Methods for Validation and Intercomparison of Remote Sensing and In situ Ice Water Measurements: Case Studies from CRYSTAL-FACE and Model Results

Remote sensing observations, such as those from AURA, are necessary to understand the role of cirrus in determining the radiative and humidity budgets of the upper troposphere. Using these measurements quantitatively requires comparisons with in situ measurements that have previously been validated. However, a direct comparison of remote and in situ measurements is difficult due to the requirement that the spatial and temporal overlap be sufficient in order to guarantee that both instruments are measuring the same air parcel. A difficult as this might be for gas phase intercomparisons, cloud inhomogeneities significantly exacerbate the problem for cloud ice water content measurements. The CRYSTAL-FACE mission provided an opportunity to assess how well such intercomparisons can be performed and to establish flight plans that will be necessary for validation of future satellite instruments. During CRYSTAL-FACE, remote and in situ instruments were placed on different aircraft (NASA's ER-2 and WB-59, and the two planes flew in tandem so that the in situ payload flew in the field of view of the remote instruments. We show here that, even with this type of careful flight planning, it is not always possible to guarantee that remote and in situ instruments are viewing the same air parcel. We use ice water data derived from the in situ Harvard Total Water (HV-TW) instrument, and the remote Goddard Cloud Radar System (CRS) and show that agreement between HV-TW and CRS is a strong function of the horizontal separation and the time delay between the aircraft transects. We also use a cloud model to simulate possible trajectories through a cloud and evaluate the use of statistical analysis in determining the agreement between the two instruments. This type of analysis should guide flight planning for future intercomparison efforts, whether for aircraft or satellite-borne instrumentation.

Sayres, D.S.↗