SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

NASA NTRS1993

An investigation of fighter aircraft agility

This report attempts to unify in a single document the results of a series of studies on fighter aircraft agility funded by the NASA Ames Research Center, Dryden Flight Research Facility and conducted at the University of Kansas Flight Research Laboratory during the period January 1989 through December 1993. New metrics proposed by pilots and the research community to assess fighter aircraft agility are collected and analyzed. The report develops a framework for understanding the context into which the various proposed fighter agility metrics fit in terms of application and testing. Since new metrics continue to be proposed, this report does not claim to contain every proposed fighter agility metric. Flight test procedures, test constraints, and related criteria are developed. Instrumentation required to quantify agility via flight test is considered, as is the sensitivity of the candidate metrics to deviations from nominal pilot command inputs, which is studied in detail. Instead of supplying specific, detailed conclusions about the relevance or utility of one candidate metric versus another, the authors have attempted to provide sufficient data and analyses for readers to formulate their own conclusions. Readers are therefore ultimately responsible for judging exactly which metrics are 'best' for their particular needs. Additionally, it is not the intent of the authors to suggest combat tactics or other actual operational uses of the results and data in this report. This has been left up to the user community. Twenty of the candidate agility metrics were selected for evaluation with high fidelity, nonlinear, non real-time flight simulation computer programs of the F-5A Freedom Fighter, F-16A Fighting Falcon, F-18A Hornet, and X-29A. The information and data presented on the 20 candidate metrics which were evaluated will assist interested readers in conducting their own extensive investigations. The report provides a definition and analysis of each metric; details of how to test and measure the metric, including any special data reduction requirements; typical values for the metric obtained using one or more aircraft types; and a sensitivity analysis if applicable. The report is organized as follows. The first chapter in the report presents a historical review of air combat trends which demonstrate the need for agility metrics in assessing the combat performance of fighter aircraft in a modern, all-aspect missile environment. The second chapter presents a framework for classifying each candidate metric according to time scale (transient, functional, instantaneous), further subdivided by axis (pitch, lateral, axial). The report is then broadly divided into two parts, with the transient agility metrics (pitch lateral, axial) covered in chapters three, four, and five, and the functional agility metrics covered in chapter six. Conclusions, recommendations, and an extensive reference list and biography are also included. Five appendices contain a comprehensive list of the definitions of all the candidate metrics; a description of the aircraft models and flight simulation programs used for testing the metrics; several relations and concepts which are fundamental to the study of lateral agility; an in-depth analysis of the axial agility metrics; and a derivation of the relations for the instantaneous agility and their approximations.

Valasek, John↗

DOE OSTI2022

Scalability Testing Approach for Internet of Things for Manufacturing SQL and NoSQL Database Latency and Throughput

The proliferation of low-cost sensors and industrial data solutions has continued to push the frontier of manufacturing technology. Machine learning and other advanced statistical techniques stand to provide tremendous advantages in production capabilities, optimization, monitoring, and efficiency. The tremendous volume of data gathered continues to grow, and the methods for storing the data are critical underpinnings for advancing manufacturing technology. This work aims to investigate the ramifications and design tradeoffs within a decoupled architecture of two prominent database management systems (DBMS): sql and NoSQL. A representative comparison is carried out with Amazon Web Services (AWS) DynamoDB and AWS Aurora MySQL. The technologies and accompanying design constraints are investigated, and a side-by-side comparison is carried out through high-fidelity industrial data simulated load tests using metrics from a major US manufacturer. The results support the use of simulated client load testing for comparing the latency of database management systems as a system scales up from the prototype stage into production. As a result of complex query support, MySQL is favored for higher-order insights, while NoSQL can reduce system latency for known access patterns at the expense of integrated query flexibility. Here, by reviewing this work, a manufacturer can observe that the use of high-fidelity load testing can reveal tradeoffs in IoTfM write/ingestion performance in terms of latency that are not observable through prototype-scale testing of commercially available cloud DB solutions.

AWS↗

NASA NTRS2012

Evaluation of Two Crew Module Boilerplate Tests Using Newly Developed Calibration Metrics

The paper discusses a application of multi-dimensional calibration metrics to evaluate pressure data from water drop tests of the Max Launch Abort System (MLAS) crew module boilerplate. Specifically, three metrics are discussed: 1) a metric to assess the probability of enveloping the measured data with the model, 2) a multi-dimensional orthogonality metric to assess model adequacy between test and analysis, and 3) a prediction error metric to conduct sensor placement to minimize pressure prediction errors. Data from similar (nearly repeated) capsule drop tests shows significant variability in the measured pressure responses. When compared to expected variability using model predictions, it is demonstrated that the measured variability cannot be explained by the model under the current uncertainty assumptions.

Horta, Lucas G.↗

NASA NTRS2010

Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness

In black-box testing, the tester creates a set of tests to exercise a system under test without regard to the internal structure of the system. Generally, no objective metric is used to measure the adequacy of black-box tests. In recent work, we have proposed three requirements coverage metrics, allowing testers to objectively measure the adequacy of a black-box test suite with respect to a set of requirements formalized as Linear Temporal Logic (LTL) properties. In this report, we evaluate the effectiveness of these coverage metrics with respect to fault finding. Specifically, we conduct an empirical study to investigate two questions: (1) do test suites satisfying a requirements coverage metric provide better fault finding than randomly generated test suites of approximately the same size?, and (2) do test suites satisfying a more rigorous requirements coverage metric provide better fault finding than test suites satisfying a less rigorous requirements coverage metric? Our results indicate (1) only one coverage metric proposed -- Unique First Cause (UFC) coverage -- is sufficiently rigorous to ensure test suites satisfying the metric outperform randomly generated test suites of similar size and (2) that test suites satisfying more rigorous coverage metrics provide better fault finding than test suites satisfying less rigorous coverage metrics.

Staats, Matt↗

DOE OSTI2022

Improving energy performance metrics to maximize the benefits of disruptive technologies

Advances in appliance and equipment systems' technologies also cause changes in their technical characteristics that can affect energy consumption and associated greenhouse gas emissions. Energy performance evaluation methods that use testing and metrics can evolve to account for energy consumption contributed from new technological characteristics. However, there is no systematic effort to enable the advancement of test procedures and performance metrics to maximize the energy, climate, and financial benefits of disruptive technologies. Therefore, it is important for countries to regularly update their energy-efficiency policy programs, such as standards and labels, by improving test procedures and metrics to reflect the innovation in emerging technologies as well as to mitigate the risk of deploying obsolete technology. This paper shows how energy performance metrics have been improving in selected appliances and equipment and provides insights to design a systematic effort to improve metrics and test procedures at the speed of ongoing changes in technologies and markets.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NASA NTRS2016

Crew Exploration Vehicle (CEV) (Orion) Occupant Protection

The purpose of this study was to determine the similarity between the response of the THUMS model and the Hybrid III Anthropometric Test Device (ATD) given existing Wright-Patterson (WP) sled tests. There were four tests selected for this comparison with frontal, spinal, rear, and lateral loading. The THUMS was placed in a sled configuration that replicated the WP configuration and the recorded seat acceleration for each test was applied to model seat. Once the modeling simulations were complete, they were compared to the WP results using two methods. The first was a visual inspection of the sled test videos compared to the THUMS d3plot files. This comparison resulted in an assessment of the overall kinematics of the two results. The other comparison was a comparison of the plotted data recorded for both tests. The metrics selected for comparison were seat acceleration, belt forces, head acceleration and chest acceleration. These metrics were recorded in all WP tests and were outputs of the THUMS model. Once the comparison of the THUMS to the WP tests was complete, the THUMS model output was also examined for possible injuries in these scenarios. These outputs included metrics for injury risk to the head, neck, thorax, lumbar spine and lower extremities. The metrics to evaluate head response were peak head acceleration, HIC15, and HIC36. For the neck, N (sub ij) was calculated. The thorax response was evaluated with peak chest acceleration, the Combined Thoracic Index (CTI), sternal deflection, chest deflection, and chest acceleration- 3 ms clip. The lumbar spine response was evaluated with lumbar spine force. Finally the lower extremity response was evaluated by femur and tibia force. The results of the simulation comparisons indicate the THUMS model had a similar response to the Hybrid III dummy given the same input. The primary difference seen between the two was a more flexible response of the THUMS compared to the Hybrid III. This flexibility was most pronounced in the neck flexion, shoulder deflection and chest deflection. Due to the flexibility of the THUMS, the resulting head and chest accelerations tended to lag the Hybrid III acceleration trace and have a lower peak value. The results of the injury metric comparison identified possible injury trends between simulations. Risk of head injury was highest for the lateral simulations. The risk of chest injury was highest for the rear impact. However, neck injury risk was approximately the same for all simulations. The injury metric value for lumbar spine force was highest for the spinal impact. The leg forces were highest for the rear and lateral impacts. The results of this comparison indicate the THUMS model performs in a similar manner as the Hybrid III ATD. The differences in the responses of model and the ATD are primarily due to the flexibility of the THUMS. This flexibility of the THUMS would be a more human like response. Based on the similarity between the two models, the THUMS should be used in further testing to assess risk of injury to the occupant.

Currie-Gregg, Nancy J.↗

NASA NTRS2022

Using Trajectory Smoothness Metrics to Identify Drones in Radar Track Data

The identification of unmanned aircraft systems (UAS) using trajectory data is considered. Specifically, a number of smoothness metrics are proposed, which can be used to distinguish UAS from other aerial objects even when they are engaged in accelerative maneuvers (non-constant-velocity flight). The metrics are evaluated on a data set from a UAS sense-and-avoid field test, which contains track data of aerial objects recorded by a vehicle-board radar system during a flight test. The metrics are found to effectively differentiate UAS from other objects such as birds for this data set. In addition, an initial statistical performance analysis of one of the smoothness metrics is undertaken, using 15 data sets deriving from multiple flight tests. The smoothness metric is shown to identify the target UAS with 95% accuracy (95% true positive rate), while achieving a false positive rate of less than 9%.

Sandip Roy↗

NASA NTRS1990

Dynamics and control of multipayload platforms - The Middeck Active Control Experiment (MACE)

A flight experiment entitled the Middeck Active Control Experiment (MACE) proposed by the Space Engineering Research Center (SERC) at the Massachusetts Institute of Technology is described. The objective of this program is to investigate and validate the modeling of the dynamics of an actively controlled flexible, articulating, multibody platform free floating in zero gravity. A rationale and experimental approach for the program are presented. The rationale shows that on-orbit testing, coupled with ground testing and a strong analytical program, is necessary in order to fully understand both how flexibility of the platform affects the pointing problem, as well as how gravity perturbs this structural flexibility causing deviations between 1-and 0-gravity behavior. The experimental approach captures the essential physics of multibody platforms, by identifying the appropriate attributes, tests, and performance metrics of the test article, and defines the tests required to successfully validate the analytical framework.

Miller, David W.↗

NASA NTRS1990

The MODE family of on-orbit experiments: The Middeck Active Control Experiment (MACE)

A flight experiment entitled the Middeck Active Control Experiment (MACE), proposed by the Space Engineering Research Center (SERC) at the Massachusetts Institute of Technology, is described. This is the second in a family of flight experiments being developed at MIT. The first is the Middeck 0-Gravity Dynamics Experiment (MODE) which investigates the nonlinear behavior of contained fluids and truss structures in zero gravity. The objective of the MACE program is to investigate and validate the modeling of the dynamics of an actively controlled flexible, articulating, multibody platform free floating in zero gravity. A rationale and experimental approach for the program are presented. The rationale shows that on-orbit testing, coupled with ground testing and a strong analytical program, is necessary in order to fully understand both how flexibility of the platform affects the pointing problem, as well as how gravity perturbs this structural flexibility causing deviations between 1- and 0-gravity behavior. The experimental approach captures the essential physics of multibody platforms, by identifying the appropriate attributes, tests, and performance metrics of the test article and defines the tests required to successfully validate the analytical framework.

Crawley, Edward F.↗

NASA NTRS1996

JPL/NASA/IEEE Test Effectiveness Workshop

(none given)From OBJECTIVES: Specific objectives of the working group are to support the innovation, development, evaluation and implementation of test methods, metrics and tools based on failure engineering/physics and/or root cause evaluations. Data sources systems and tools shall be developed and implemented that: 1) provide improved preventions, controls, analyses and tests (PACT) & field failure data collection, 2) facilities data analysis, archiving, retrieval, failure physics and/or root cause evaluations and 3) enable new and existing technology suitability evaluations to be performed.

effectiveness concurrent engineering metrics test ↗

NASA NTRS2015

Improving Climate Projections Using "Intelligent" Ensembles

Recent changes in the climate system have led to growing concern, especially in communities which are highly vulnerable to resource shortages and weather extremes. There is an urgent need for better climate information to develop solutions and strategies for adapting to a changing climate. Climate models provide excellent tools for studying the current state of climate and making future projections. However, these models are subject to biases created by structural uncertainties. Performance metrics-or the systematic determination of model biases-succinctly quantify aspects of climate model behavior. Efforts to standardize climate model experiments and collect simulation data-such as the Coupled Model Intercomparison Project (CMIP)-provide the means to directly compare and assess model performance. Performance metrics have been used to show that some models reproduce present-day climate better than others. Simulation data from multiple models are often used to add value to projections by creating a consensus projection from the model ensemble, in which each model is given an equal weight. It has been shown that the ensemble mean generally outperforms any single model. It is possible to use unequal weights to produce ensemble means, in which models are weighted based on performance (called "intelligent" ensembles). Can performance metrics be used to improve climate projections? Previous work introduced a framework for comparing the utility of model performance metrics, showing that the best metrics are related to the variance of top-of-atmosphere outgoing longwave radiation. These metrics improve present-day climate simulations of Earth's energy budget using the "intelligent" ensemble method. The current project identifies several approaches for testing whether performance metrics can be applied to future simulations to create "intelligent" ensemble-mean climate projections. It is shown that certain performance metrics test key climate processes in the models, and that these metrics can be used to evaluate model quality in both current and future climate states. This information will be used to produce new consensus projections and provide communities with improved climate projections for urgent decision-making.

Baker, Noel C.↗

NASA NTRS2015

Improving Climate Projections Using "Intelligent" Ensembles

Baker, Noel C.↗

DOE OSTI2022

First Sagittarius A* Event Horizon Telescope Results. VI. Testing the Black Hole Metric

Astrophysical black holes are expected to be described by the Kerr metric. This is the only stationary, vacuum, axisymmetric metric, without electromagnetic charge, that satisfies Einstein’s equations and does not have pathologies outside of the event horizon. We present new constraints on potential deviations from the Kerr prediction based on 2017 EHT observations of Sagittarius A* (Sgr A*). We calibrate the relationship between the geometrically defined black hole shadow and the observed size of the ring-like images using a library that includes both Kerr and non-Kerr simulations. We use the exquisite prior constraints on the mass-to-distance ratio for Sgr A* to show that the observed image size is within ∼10% of the Kerr predictions. We use these bounds to constrain metrics that are parametrically different from Kerr, as well as the charges of several known spacetimes. To consider alternatives to the presence of an event horizon, we explore the possibility that Sgr A* is a compact object with a surface that either absorbs and thermally reemits incident radiation or partially reflects it. Using the observed image size and the broadband spectrum of Sgr A*, we conclude that a thermal surface can be ruled out and a fully reflective one is unlikely. We compare our results to the broader landscape of gravitational tests. Together with the bounds found for stellar-mass black holes and the M87 black hole, our observations provide further support that the external spacetimes of all black holes are described by the Kerr metric, independent of their mass.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2024

Short-Term Load Forecasting Considering EV Charging Loads with Prediction Interval Evaluation

Short-term load forecasting plays a critical role in power system planning and operation. Along with the electrification of various loads, electricity demands are becoming increasingly hard to predict. Notably, the recent rise in electric vehicles (EVs) has further contributed to this unpredictability. To address this issue, this paper proposes a probabilistic load forecasting strategy utilizing Gaussian process regression, structured in a day-ahead manner. While many works focus on deterministic prediction, probabilistic forecasting offers additional insights into variability and uncertainty, enabling more flexible and reliable operation for power systems. To enhance the accuracy of the load forecasting model, the inputs include features related to EV charging habits as well as commonly used weather information. The load forecasting results are evaluated using various metrics, including conventional ones that assess the accuracy of point forecasts, as well as additional metrics that test the reliability of prediction intervals. The proposed load forecasting method is finally tested on real residential power consumption data and EV charging data sampled from real-world sources. The results prove that the new features can greatly improve the performance of the load forecasting method.

electrical vehicle↗

DOE OSTI2021

Assessing the Energy Resilience of Office Buildings: Development and Testing of a Simplified Metric for Real Estate Stakeholders

Increasing concern over higher frequency extreme weather events is driving a push towards a more resilient built environment. In recent years there has been growing interest in understanding how to evaluate, measure, and improve building energy resilience, i.e., the ability of a building to provide energy-related services in the event of a local or regional power outage. In addition to human health and safety, many stakeholders are keenly interested in the ability of a building to allow continuity of operations and minimize business disruption. Office buildings are subject to significant economic losses when building operations are disrupted due to a power outage. We propose “occupant hours lost” (OHL) as a means to measure the business productivity lost as the result of a power outage in office buildings. OHL is determined based on indoor conditions in each space for each hour during a power outage, and then aggregated spatially and temporally to determine the whole building OHL. We used quasi-Monte Carlo parametric energy simulations to demonstrate how the OHL metric varies due to different building characteristics across different climate zones and seasons. The simulation dataset was then used to develop simple regression models for assessing the impact of ten key building characteristics on OHL. The most impactful were window-to-wall ratio and window characteristics. The regression models show promise as a simple means to assess and screen for resilience using basic building characteristics, especially for non-critical facilities where it may not be viable to conduct detailed engineering analysis.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

NASA NTRS1980

A report on the gravitational redshift test for non-metric theories of gravitation

The frequencies of two atomic hydrogen masers and of three superconducting cavity stabilized oscillators were compared as the ensemble of oscillators was moved in the Sun's gravitational field by the rotation and orbital motion of the Earth. Metric gravitation theories predict that the gravitational redshifts of the two types of oscillators are identical, and that there should be no relative frequency shift between the oscillators; nonmetric theories, in contrast, predict a frequency shift between masers and SCSOs that is proportional to the change in solar gravitational potential experienced by the oscillators. The results are consistent with metric theories of gravitation at a level of 2%.

Source record↗

NASA NTRS1989

Microcomputer-based tests for repeated-measures: Metric properties and predictive validities

A menu of psychomotor and mental acuity tests were refined. Field applications of such a battery are, for example, a study of the effects of toxic agents or exotic environments on performance readiness, or the determination of fitness for duty. The key requirement of these tasks is that they be suitable for repeated-measures applications, and so questions of stability and reliability are a continuing, central focus of this work. After the initial (practice) session, seven replications of 14 microcomputer-based performance tests (32 measures) were completed by 37 subjects. Each test in the battery had previously been shown to stabilize in less than five 90-second administrations and to possess retest reliabilities greater than r = 0.707 for three minutes of testing. However, all the tests had never been administered together as a battery and they had never been self-administered. In order to provide predictive validity for intelligence measurement, the Wechsler Adult Intelligence Scale-Revised and the Wonderlic Personnel Test were obtained on the same subjects.

Kennedy, Robert S.↗

DOE OSTI2023

Can Simple Metrics Identify the Process(es) Driving Extreme Precipitation?

This work seeks an automatic algorithm to determine the primary meteorological cause(s) of individual extreme precipitation events. Such determinations have been made before, but required a by-hand analysis of each separate event. This is very time-consuming and the field would benefit from an automatic process. This is especially relevant when comparing different datasets to determine which ones most closely hew towards reality. This paper tests three simple metrics over the continental United States using the European Center for Medium-Range Weather Forecasting’s (ECMWF) atmospheric reanalysis (ERA5). The metrics tested measure and compare the strength of three meteorological processes associated with extreme precipitation: fronts, convection, and cyclones. A multivariate statistical technique as well as individual case studies show evidence that the three meteorological processes of interest cannot be isolated from one another using these simple physical metrics. This shows the difficulty in finding “pure” cases of these precipitation-generating processes and suggests approaching these processes with an eye toward mixed-type events.

Swenson, Leif M. (ORCID:0000000199708735)↗