Permutation Testing and Bootstrapping: NonParametric Approaches to Statistical Testing and Estimation
No abstract available
Young, Millennia↗
SEARCH · Search NASA
Results for “Statistical Testing Approach”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A Statistical Approach to Establishing Subsystem Environmental Test Specifications
Results are presented of a research task to evaluate structural responses at various subsystem mounting locations during spacecraft level test exposures to the environments of mechanical shock, acoustic noise, and random vibration. This statistical evaluation is presented in the form of recommended subsystem test specifications for these three environments as normalized to a reference set of spacecraft test levels and are thus suitable for extrapolation to a set of different spacecraft test levels. The recommendations are dependent upon a subsystem's mounting location in a spacecraft, and information is presented on how to determine this mounting zone for a given subsystem.
Keegan, W. B.↗
Reducing V&V Cost of Flight Critical Systems: Myth or Reality?
This paper presents an overview of NASA research program on the V&V of flight critical systems. Five years ago, NASA started an effort to reduce the cost and possibly increase the effectiveness of V&V for flight critical systems. It is the right time to take a look back and realize what progress has been made. This paper describes our overall approach and the tools introduced to address different phases of the software lifecycle. For example, we have improved testing by developing a statistical learning approach tor defining test cases. The tool automatically identifies possible unsafe conditions by analyzing outliers in output data; using an iterative learning process, it can then generate more test cases that represent potentially unsafe regions of operation. At the code level, we have developed and made available as open source a static analyzer for C and C++ programs called IKOS. We have shown that IKOS is very precise in the analysis of embedded C programs (very few false positives) and a bit less for regular C and C++ code. At the design level, in collaboration with our NRA partners, we have developed a suite of analysis tools for Simulink models. The analysis is done in a compositional framework for scalability.
Brat, Guillaume P.↗
Cleanroom certification model
The Cleanroom software development methodology is designed to take the gamble out of product releases for both suppliers and receivers of the software. The ingredients of this procedure are a life cycle of executable product increments, representative statistical testing, and a standard estimate of the MTTF (Mean Time To Failure) of the product at the time of its release. A statistical approach to software product testing using randomly selected samples of test cases is considered. A statistical model is defined for the certification process which uses the timing data recorded during test. A reasonableness argument for this model is provided that uses previously published data on software product execution. Also included is a derivation of the certification model estimators and a comparison of the proposed least squares technique with the more commonly used maximum likelihood estimators.
Currit, P. A.↗
Design, development and manufacture of a breadboard radio frequency mass gauging system
The feasibility of the RF gauging mode, counting technique was demonstrated for gauging liquid hydrogen and liquid oxygen under all attitude conditions. With LH2, it was also demonstrated under dynamic fluid conditions, in which the fluid assumes ever changing positions within the tank, that the RF gauging technique on the average provides a very good indication of mass. It is significant that the distribution of the mode count data at each fill level during dynamic LH2 and LOX orientation testing does approach a statistical normal distribution. Multiple space-diversity probes provide better coupling to the resonant modes than utilization of a single probe element. The variable sweep rate generator technique provides a more uniform mode versus time distribution for processing.
Source record↗
Long-Term Stability of Radio Sources in VLBI Analysis
Positional stability of radio sources is an important requirement for modeling of only one source position for the complete length of VLBI data of presently more than 20 years. The stability of radio sources can be verified by analyzing time series of radio source coordinates. One approach is a statistical test for normal distribution of residuals to the weighted mean for each radio source component of the time series. Systematic phenomena in the time series can thus be detected. Nevertheless, an inspection of rate estimation and weighted root-mean-square (WRMS) variations about the mean is also necessary. On the basis of the time series computed by the BKG group in the frame of the ICRF2 working group, 226 stable radio sources with an axis stability of 10 as could be identified. They include 100 ICRF2 axes-defining sources which are determined independently of the method applied in the ICRF2 working group. 29 stable radio sources with a source structure index of less than 3.0 can also be used to increase the number of 295 ICRF2 defining sources.
Engelhardt, Gerald↗
The Role of Analysis and Testing in the Service Life Assessment of Ion Engines
Experience from tests and flights and engineering analysis represent the two sources of information on which to base conclusions on the reliability or failure risk of aerospace flight systems. It is rarely feasible to establish high reliability at high confidence by testing aerospace flight systems or components. The limitations of testing in evaluating failure risk are discussed and an alternate statistical approach which relies on both test experience and analysis to quantitatively assess reliability is outlined. The implementation of this methodology in the service life assessment of ion thrusters is discussed and examples of failure modes being addressed in the NASA Solar Electric Propulsion Technology Application Readiness (NSTAR) program are given.
Ion↗
Composite Overwrapped Pressure Vessels (COPV): Flight Rationale for the Space Shuttle Program
Each Orbiter Vehicle (Space Shuttle Program) contains up to 24 Kevlar49/Epoxy Composite Overwrapped Pressure Vessels (COPV) for storage of pressurized gases. In the wake of the Columbia accident and the ensuing Return To Flight (RTF) activities, Orbiter engineers reexamined COPV flight certification. The original COPV design calculations were updated to include recently declassified Kevlar COPV test data from Lawrence Livermore National Laboratory (LLNL) and to incorporate changes in how the Space Shuttle was operated as opposed to orinigially envisioned. 2005 estimates for the probability of a catastrophic failure over the life of the program (from STS-1 through STS-107) were one-in-five. To address this unacceptable risk, the Orbiter Project Office (OPO) initiated a comprehensive investigation to understand and mitigate this risk. First, the team considered and eventually deemed unfeasible procuring and replacing all existing flight COPVs. OPO replaced the two vessels with the highest risk with existing flight spare units. Second, OPO instituted operational improvements in ground procedures to signficiantly reduce risk, without adversely affecting Shuttle capability. Third, OPO developed a comprehensive model to quantify the likelihood of occurrance. A fully-instrumented burst test (recording a lower burst pressure than expected) on a flight-certified vessel provided critical understanding of the behavior of Orbiter COPVs. A more accurate model was based on a newly-compiled comprehensive database of Kevlar data from LLNL and elsewhere. Considering hardware changes, operational improvements and reliability model refinements, the mean reliability was determined to be 0.998 for the remainder of the Shuttle Program (from 2007, for STS- 118 thru STS-135). Since limited hardware resources precluded full model validation through multiple tests, additional model confidence was sought through the first-ever Accelerated Stress Rupture Test (ASRT) of a flown flight article. A Bayesian statistical approach was developed to interpret possible test results. Since the lifetime observed in the ASRT exceeded initial estimates by one to two orders of magnitude, the Space Shuttle Program deemed there was significant conservatism in the model and accepted continued operation with existing flight hardware. Given the variability in tank-to-tank original prooftest response, a non-destructive evaluation (NDE) technique utilizing Raman Spectroscopy was developed to directly measure COPV residual stress state. Preliminary results showed that patterns of low fiber elastic strains over the outside vessel surface, together with measured permanent volume growth during proof, could be directly correlated to increased fiber stress ratios on the inside fibers adjacent to the liner, and thus reduced reliability.
Kezirian, Michael T.↗
Adaptive identification for the dynamics of large space structures
A new approach for identifying the dynamics of large space structures using noisy measurements is presented. In this approach, a modal system description is avoided and, hence, the a priori specification of model order. Instead, the system's order is recursively determined on-line, along with its parameters, using recursive lattice filters which are widely used in adaptive signal processing. The order determination is carried out based on statistical hypothesis testing theory. The approach is illustrated for the problem of identifying the structural dynamic characteristics of a free-free beam. Results on the effectiveness and accuracy of the identification scheme with respect to noise levels, sampling rates, and data window sizes are presented
Sundararajan, N.↗
Failure risk assessment by analysis and testing
The sources of information on which to base an evaluation of reliability or failure risk of an aerospace flight system are (1) experience from tests and flights and (2) engineering analysis. It is rarely feasible to establish high reliability at high confidence by testing aerospace systems or components. Moreover, failure prediction by conventional, deterministic methods of engineering analysis can become arbitrary and subject to serious misinterpretation when uncertain or approximate information is used to establish analysis parameter values and to calibrate the accuracy of engineering models. The limitations of testing to evaluate failure risk are discussed, and a statistical approach which incorporates both engineering analysis and testing is presented.
Moore, N.↗
Contextual classification on the massively parallel processor
Classifiers are often used to produce land cover maps from multispectral Earth observation imagery. Conventionally, these classifiers have been designed to exploit the spectral information contained in the imagery. Very few classifiers exploit the spatial information content of the imagery, and the few that do rarely exploit spatial information content in conjunction with spectral and/or temporal information. A contextual classifier that exploits spatial and spectral information in combination through a general statistical approach was studied. Early test results obtained from an implementation of the classifier on a VAX-11/780 minicomputer were encouraging, but they are of limited meaning because they were produced from small data sets. An implementation of the contextual classifier is presented on the Massively Parallel Processor (MPP) at Goddard that for the first time makes feasible the testing of the classifier on large data sets.
Tilton, James C.↗
A Ground Flash Fraction Retrieval Algorithm for GLM
A Bayesian inversion method is introduced for retrieving the fraction of ground flashes in a set of N lightning observed by a satellite lightning imager (such as the Geostationary Lightning Mapper, GLM). An exponential model is applied as a physically reasonable constraint to describe the measured lightning optical parameter distributions. Population statistics (i.e., the mean and variance) are invoked to add additional constraints to the retrieval process. The Maximum A Posteriori (MAP) solution is employed. The approach is tested by performing simulated retrievals, and retrieval error statistics are provided. The approach is feasible for N greater than 2000, and retrieval errors decrease as N is increased.
Koshak, William J.↗
Structural Certification of Human-Rated Inflatable Space Structures
This paper details the results of an initial study to develop a certification plan for human-rated inflatable space structures, including guidelines for qualification testing. Habitable softgoods inflatables are multi-layered shell structures that use high-strength webbing, cordage and broadcloth fabric to carry the skin loads of a variety of volumetric shapes and structural architectures. The primary objectives of this study are to define the key parameters that affect these structures and propose a statistically robust approach to defining safety and knockdown factors based on test and analysis. Current NASA standards for habitable inflatable space structures use a factor of safety of 4, which was inherited from airship design criteria. An updated approach to defining a design factor, taking into account material strength variability, load variability in the article, number of test samples, and damage and degradation effects is specified. Accurate analytical modeling of these structures is hindered by the difficulty of obtaining accurate and consistent material data due to load-history- dependent, nonlinear load versus strain behavior. A building block approach to certification is detailed that uses stochastic modeling and statistical test design and analysis to address the unique challenges these high-strength softgoods structures present. Human-rated inflatable modules are a transformative capability for launching much larger habitable volumes into space than is possible with rigid shell structures. This research aims to provide the framework for certifying these structures for future human space exploration missions.
Jones, Thomas C.↗
Bayes Analysis and Reliability Implications of Stress-Rupture Testing a Kevlar/Epoxy COPV Using Temperature and Pressure Acceleration
Composite Overwrapped Pressure Vessels (COPVs) that have survived a long service time under pressure generally must be recertified before service is extended. Flight certification is dependent on the reliability analysis to quantify the risk of stress rupture failure in existing flight vessels. Full certification of this reliability model would require a statistically significant number of lifetime tests to be performed and is impractical given the cost and limited flight hardware for certification testing purposes. One approach to confirm the reliability model is to perform a stress rupture test on a flight COPV. Currently, testing of such a Kevlar49 (Dupont)/epoxy COPV is nearing completion. The present paper focuses on a Bayesian statistical approach to analyze the possible failure time results of this test and to assess the implications in choosing between possible model parameter values that in the past have had significant uncertainty. The key uncertain parameters in this case are the actual fiber stress ratio at operating pressure, and the Weibull shape parameter for lifetime; the former has been uncertain due to ambiguities in interpreting the original and a duplicate burst test. The latter has been uncertain due to major differences between COPVs in the database and the actual COPVs in service. Any information obtained that clarifies and eliminates uncertainty in these parameters will have a major effect on the predicted reliability of the service COPVs going forward. The key result is that the longer the vessel survives, the more likely the more optimistic stress ratio model is correct. At the time of writing, the resulting effect on predicted future reliability is dramatic, increasing it by about one "nine," that is, reducing the predicted probability of failure by an order of magnitude. However, testing one vessel does not change the uncertainty on the Weibull shape parameter for lifetime since testing several vessels would be necessary.
Phoenix, S. Leigh↗
Nasa Space Flight Instruments: Cost Time Trends
Are NASA’s space flight instruments becoming cheaper or more expensive as time marches forward? After analyzing the costs of hundreds of instruments launched over the last 30 years, the short answer to this question is no… and yes. This paper gives a visual analysis of the cost time trends for various NASA space flight instrument types, such as optical, particles detectors, fields detectors and microwave instruments. In addition to the statistical approaches utilized, such as significance tests, cluster analysis and principle components analysis (PCA), we will also discuss the intangibles which are likely at play, including technological progress, NASA policy and the luck of the draw associated with mission manifests. This analysis was performed as the main driver for the NASA Instrument Cost Model (NICM) recent cost estimating model redesign. Started in 2004, the first version of NICM was based off of instruments launched from 1985-2005, or 20 years’ worth of data. As NICM hit its 10-year anniversary, we wanted to know: should NICM continue to only use the most recent 20 years’ worth of data (1995-2015)? Are instruments becoming cheaper or more expensive as time marches forward? There is evidence in favor of a drop in the median dollar-per-kg value across some instrument types, but little in others. Whereas further research is needed to substantiate, Particles and Optical-Planetary instrument types show moderate to strong evidence of a downward trend in dollar-per-kg. Further research is required to study the nature of this trend (shift, taper, cyclic, etc.). Little evidence for a similar downward trend was detected for Fields or Microwave instruments, or Optical instruments on Earth Orbiting spacecraft. We presented evidence in favor of a drop in the median dollar-per-kg value for Particles and Optical-Planetary instrument types. While similar evidence was weak at best for Fields and Microwave instruments. We can speculate as to the causes for this effect, but we are also equipped to begin to rule out, or at least prioritize, some of the suspected drivers. We observed, for Particles and Optical-Planetary instruments, that perhaps a launch manifest effect was playing part of the role in the observed decrease in dollar-per-kg over the years, noting that the more flagship class missions, which have more money to spend on their instruments, were seen in the earlier years in our data, versus the later years which were dominated by less expensive class missions. However, if this were a dominating driver, would we not have seen the downward trend in the Fields and Microwave instruments as well, which were drawn from that same launch manifest? The fact that we did not observe this helps us rule out the launch manifest effect, and other drivers, such as advances in technology, that seem to be more likely suspects. In that case, however, why would technology advances be helping the Particles and Optical-Planetary instruments only? Why would it not be impacting Optical-Earth Orbiting instruments? Further suspects were looked at as well and ruled out, such as the “Faster, Better, Cheaper”era of NASA development which did not seem to actually impact trends by instrument type on a dollar-perkg scale. VI. Future Work A. Time Series Detailed Statistical Assessment The analysis discussed above sets the foundation for a more rigorous time series analysis of the data. Time series analysis will further explore evidence to-date of time trends for the instrument types which showed the strongest indicators for a decrease in dollar-per-kg: Optical (Planetary) and Particles instruments. More than providing evidence and top-level significance tests, time series analysis would help elucidate what kind of trend that exists in the data, their significance and allow statistically based forecasting (see Figure 10)
Mrozinksi, Joseph↗
Estimating the chlorophyll content in the waters of Guanabara Bay from the LANDSAT multispectral scanning digital data
Detection of water quality in Guanabara Bay using multispectral scanning digital data taken from LANDSAT satellites was examined. To test these processes, an empirical (statistical) approach was choosen to observe the degree of relationship between LANDSAT data and the in situ data taken simultaneously. The linear and nonlinear regression analyses were taken from among those developed by INPE in 1978. Results indicate that the major regression was in the number six MSS band, atmospheric effects, which indicated a correction coefficient of 0.99 and an average error of 6.59 micrograms liter. This error was similar to that obtained in the laboratory. The chlorophyll content was between 0 and 100 micrograms/liter, as taken from the MSS of LANDSAT.
Dejesusparada, N.↗
A statistical approach to deriving subsystem specifications
In order to produce cost effective environmental test programs, the test specifications must be realistic and to be useful, they must be available early in the life of a program. This paper describes a method for achieving such specifications for subsystems by utilizing the results of a statistical analysis of data acquired at subsystem mounting locations during system level environmental tests. The paper describes the details of this statistical analysis. The resultant recommended levels are a function of the subsystems' mounting location in the spacecraft. Methods of determining this mounting 'zone' are described. Recommendations are then made as to which of the various problem areas encountered should be pursued further.
Keegan, W. B.↗
Development of an Uncertainty Model for the National Transonic Facility
This paper introduces an uncertainty model being developed for the National Transonic Facility (NTF). The model uses a Monte Carlo technique to propagate standard uncertainties of measured values through the NTF data reduction equations to calculate the combined uncertainties of the key aerodynamic force and moment coefficients and freestream properties. The uncertainty propagation approach to assessing data variability is compared with ongoing data quality assessment activities at the NTF, notably check standard testing using statistical process control (SPC) techniques. It is shown that the two approaches are complementary and both are necessary tools for data quality assessment and improvement activities. The SPC approach is the final arbiter of variability in a facility. Its result encompasses variation due to people, processes, test equipment, and test article. The uncertainty propagation approach is limited mainly to the data reduction process. However, it is useful because it helps to assess the causes of variability seen in the data and consequently provides a basis for improvement. For example, it is shown that Mach number random uncertainty is dominated by static pressure variation over most of the dynamic pressure range tested. However, the random uncertainty in the drag coefficient is generally dominated by axial and normal force uncertainty with much less contribution from freestream conditions.
Walter, Joel A.↗