Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2009

Real-Time Assessment of Robot Performance During Remote Exploration Operations

To ensure that robots are used effectively for exploration missions, it is important to assess their performance during operations. We are investigating the definition and computation of performance metrics for assessing remote robotic operations in real-time. Our approach is to monitor data streams from robots, compute performance metrics, and provide Web-based displays of these metrics for assessing robot performance during operations. We evaluated our approach for measuring robot performance with the K10 rovers from NASA Ames Research Center during a field test at Moses Lake Sand Dunes (WA) in June 2008. In this paper we present the results of evaluating our software for robot performance and discuss our conclusions from this evaluation for future robot operations.

SBIR TOPIC X7.02 PHASE 1↗
NASA NTRS2012

Evaluating Core Quality for a Mars Sample Return Mission

Sample return missions, including the proposed Mars Sample Return (MSR) mission, propose to collect core samples from scientifically valuable sites on Mars. These core samples would undergo extreme forces during the drilling process, and during the reentry process if the EEV (Earth Entry Vehicle) performed a hard landing on Earth. Because of the foreseen damage to the stratigraphy of the cores, it is important to evaluate each core for rock quality. However, because no core sample return mission has yet been conducted to another planetary body, it remains unclear as to how to assess the cores for rock quality. In this report, we describe the development of a metric designed to quantitatively assess the mechanical quality of any rock cores returned from Mars (or other planetary bodies). We report on the process by which we tested the metric on core samples of Mars analogue materials, and the effectiveness of the core assessment metric (CAM) in assessing rock core quality before and after the cores were subjected to shocking (g forces representative of an EEV landing).

Mars samples↗
NASA NTRS2011

Ensuring Success of Adaptive Control Research Through Project Lifecycle Risk Mitigation

Lessons Learne: 1. Design-out unnecessary risk to prevent excessive mitigation management during flight. 2. Consider iterative checkouts to confirm or improve human factor characteristics. 3. Consider the total flight test profile to uncover unanticipated human-algorithm interactions. 4. Consider test card cadence as a metric to assess test readiness. 5. Full-scale flight test is critical to development, maturation, and acceptance of adaptive control laws for operational use.

Pavlock, Kate M.↗
NASA NTRS2001

Coverage Metrics for Model Checking

When using model checking to verify programs in practice, it is not usually possible to achieve complete coverage of the system. In this position paper we describe ongoing research within the Automated Software Engineering group at NASA Ames on the use of test coverage metrics to measure partial coverage and provide heuristic guidance for program model checking. We are specifically interested in applying and developing coverage metrics for concurrent programs that might be used to support certification of next generation avionics software.

Penix, John↗
NASA NTRS2025

Provider of Services for Urban Air Mobility (PSU) Prototype Simulation (X5) Final Report

Urban Air Mobility (UAM) is a new air transportation service concept to carry passengers or cargo in metropolitan areas, leveraged by innovative aircraft and air traffic automation technologies. NASA has conducted a series of simulations, called the X-series simulation, to evaluate the UAM concept of operations and support the development of airspace procedures and services for UAM operations. The simulation called “X5” was conducted in 2023 to test a Provider of Services for UAM (PSU) prototype developed by NASA for UAM flight planning, strategic conflict management support, and data exchange between UAM operators. In this simulation, two strategic conflict management capabilities, Demand-Capacity Balancing and Sequencing and Scheduling, were further investigated. This document describes the UAM system architecture modeled, the X5 simulation environment to be executed (e.g., traffic scenario and UAM airspace construct), and the strategic conflict management processes developed and evaluated in this study. Then, the simulation results are provided using several system performance metrics, such as the number of operations planned and activated, demand-capacity imbalances detected and resolved, and pre-departure delays. Based on these metrics, the test findings and lessons learned from this simulation are discussed. NASA developed a PSU prototype as part of a reference implementation of UAM system architecture and evolved strategic conflict management capabilities for UAM operations from the previous collaborative simulations with industry partners. Below is the summary of the achievements: - Aligned NASA’s UAM reference architecture with the FAA’s UAM ConOps notional architecture - Extended UAM airspace management capabilities to include 1) Demand-Capacity Balancing (DCB) to ensure operators coordinate planned usage of shared vertiports, and 2) Sequencing and Scheduling (S&S) at UAM corridor entry and exit points to help facilitate an orderly flow of traffic - Defined the PSU information exchange APIs and requirements towards informing industry standards - Developed and tested a NASA PSU prototype as reference implementation to validate the requirements and APIs - Developed a prototype service connecting NASA’s PSU and the FAA system for testing future PSU-ATM interface requirements - Tested NASA-developed assumptions for UAM operations such as airspace design, procedures, vehicle performance, and strategic conflict management methods to inform future Cooperative Operating Practices (COPs) development with industry - Evaluated system performance metrics such as number of simultaneous operations and ground delays that can help define system-level requirements. The simulation results showed that the UAM traffic demand could be managed to minimize the needs of tactical separation provision with ground delays assigned by DCB and S&S. These accomplishments and the lessons learned from the PSU Prototype X5 simulation activities will be valuable inputs for the Air Mobility Pathfinders (AMP) project, which is NASA’s new project to create and evaluate a reference architecture for safe, secure, and scalable UAM operations.

Simulation↗
NASA NTRS2023

Comment on “Advanced Testing of Low, Medium, and High ECS CMIP6 GCM Simulations Versus ERA5-T2m” by N. Scafetta (2022)

Scafetta (2022, https://doi.org/10.1029/2022gl097716) purports to test Coupled Model Intercomparison Project Phase 6 (CMIP6) climate models through a comparison of temperature changes over three decades. Unfortunately, the paper contains numerous conceptual and statistical errors that undermine all of the conclusions. First, no uncertainty is given for the observational temperature difference, making it impossible to assess compatibility with any model result. Second, the CMIP6 data are the ensemble means for each model, but the metric being tested is sensitive to the internal variability and so the full ensemble for each model must be used. When this is corrected, the conclusion that “all models with ECS > 3.0°C overestimate the observed global surface warming” is not sustained. Third, the statistical test in Section 2 would reject all models even in a perfect model setup given sufficient ensemble members, thus the second conclusion “that spatial t-statistics rejects the data-model agreement” is also not sustainable.

CMIP6↗
NASA NTRS1977

General relativity and satellite orbits - The motion of a test particle in the Schwarzschild metric

The motion of a satellite of negligible mass about a massive spherically symmetric object in a space of Schwarzschild geometry is calculated by using ordinary Newtonian methods for an inverse-square gravitational field acted upon by an inverse-cube disturbing function. The disturbing function is expressed in terms of the Keplerian elements of the orbit and substituted in the Lagrange planetary equations. The equations can be integrated to calculate the displacement in position of the satellite due to the relativistic potential. For example, the Beacon Explorer C satellite is found to be displaced by about 17.4 cm after one revolution. For Mercury the advance of the perihelion is about 85 km after one orbit, and the maximum periodic displacement is about 13 km.

Rubincam, D. P.↗
NASA NTRS1987

Methodology requirements for intelligent systems architecture

The methodology required for the development of the 'intelligent system architecture' of distributed computer systems which integrate standard data processing capabilities with symbolic processing to provide powerful and highly autonomous adaptive processing capabilities must encompass three elements: (1) a design knowledge capture system, (2) computer-aided engineering, and (3) verification and validation metrics and tests. Emphasis must be put on the earliest possible definition of system requirements and the realistic definition of allowable system uncertainties. Methodologies must also address human factor issues.

Grant, Terry↗
NASA NTRS1991

Performance analysis of LAN bridges and routers

Bridges and routers are used to interconnect Local Area Networks (LANs). The performance of these devices is important since they can become bottlenecks in large multi-segment networks. Performance metrics and test methodology for bridges and routers were not standardized. Performance data reported by vendors is not applicable to the actual scenarios encountered in an operational network. However, vendor-provided data can be used to calibrate models of bridges and routers that, along with other models, yield performance data for a network. Several tools are available for modeling bridges and routers - Network II.5 was used. The results of the analysis of some bridges and routers are presented.

Hajare, Ankur R.↗
NASA NTRS1993

The impact of flying qualities on helicopter operational agility

Flying qualities standards are formally set to ensure safe flight and therefore reflect minimum, rather than optimum, requirements. Agility is a flying quality but relates to operations at high, if not maximum, performance. While the quality metrics and test procedures for flying, as covered for example in ADS33C, may provide an adequate structure to encompass agility, they do not currently address flight at high performance. This is also true in the fixed-wing world and a current concern in both communities is the absence of substantiated agility criteria and possible conflicts between flying qualities and high performance. AGARD is sponsoring a working group (WG19) title 'Operational Agility' that deals with these and a range of related issues. This paper is condensed from contributions by the three authors to WG19, relating to flying qualities. Novel perspectives on the subject are presented including the agility factor, that quantifies performance margins in flying qualities terms; a new parameter, based on maneuver acceleration is introduced as a potential candidate for defining upper limits to flying qualities. Finally, a probabilistic analysis of pilot handling qualities ratings is presented that suggests a powerful relationship between inherent airframe flying qualities and operational agility.

Padfield, Gareth D.↗
NASA NTRS1994

Intelligent vehicle control: Opportunities for terrestrial-space system integration

For 11 years the Department of Defense has cooperated with a diverse array of other Federal agencies including the National Institute of Standards and Technology, the Jet Propulsion Laboratory, and the Department of Energy, to develop robotics technology for unmanned ground systems. These activities have addressed control system architectures supporting sharing of tasks between the system operator and various automated subsystems, man-machine interfaces to intelligent vehicles systems, video compression supporting vehicle driving in low data rate digital communication environments, multiple simultaneous vehicle control by a single operator, path planning and retrace, and automated obstacle detection and avoidance subsystem. Performance metrics and test facilities for robotic vehicles were developed permitting objective performance assessment of a variety of operator-automated vehicle control regimes. Progress in these areas will be described in the context of robotic vehicle testbeds specifically developed for automated vehicle research. These initiatives, particularly as regards the data compression, task sharing, and automated mobility topics, also have relevance in the space environment. The intersection of technology development interests between these two communities will be discussed in this paper.

Shoemaker, Charles↗
NASA NTRS1995

Automated spectral classification and the GAIA project

Two dimensional spectral types for each of the stars observed in the global astrometric interferometer for astrophysics (GAIA) mission would provide additional information for the galactic structure and stellar evolution studies, as well as helping in the identification of unusual objects and populations. The classification of the large quantity generated spectra requires that automated techniques are implemented. Approaches for the automatic classification are reviewed, and a metric-distance method is discussed. In tests, the metric-distance method produced spectral types with mean errors comparable to those of human classifiers working at similar resolution. Data and equipment requirements for an automated classification survey, are discussed. A program of auxiliary observations is proposed to yield spectral types and radial velocities for the GAIA-observed stars.

Lasala, Jerry↗
NASA NTRS1995

NASA/WVU Software Research Laboratory, 1995

In our second year, the NASA/WVU Software Research Lab has made significant strides toward analysis and solution of major software problems related to V&V activities. We have established working relationships with many ongoing efforts within NASA and continue to provide valuable input into policy and decision-making processes. Through our publications, technical reports, lecture series, newsletters, and resources on the World-Wide-Web, we provide information to many NASA and external parties daily. This report is a summary and overview of some of our activities for the past year. This report is divided into 6 chapters: Introduction, People, Support Activities, Process, Metrics, and Testing. The Introduction chapter (this chapter) gives an overview of our project beginnings and targets. The People chapter focuses on new people who have joined the Lab this year. The Support chapter briefly lists activities like our WWW pages, Technical Report Series, Technical Lecture Series, and Research Quarterly newsletter. Finally, the remaining four chapters discuss the major research areas that we have made significant progress towards producing meaningful task reports. These chapters can be regarded as portions of drafts of our task reports.

Sabolish, George J.↗
NASA NTRS2005

Lightweight Nonmetallic Thermal Protection Materials Technology (LNTPMT) Project

Contents include the following: Exploration systems research and technology program structure. Project objective. Overview of project. Candidate thermal protection system (PS) materials. Definition of reference missions and space environments. Technical performance metrics (TPMs).Testing (types of tests). Conclusion.

Flynn, Kevin↗
NASA NTRS2013

Aircraft Engine Gas Path Diagnostic Methods: Public Benchmarking Results

Recent technology reviews have identified the need for objective assessments of aircraft engine health management (EHM) technologies. To help address this issue, a gas path diagnostic benchmark problem has been created and made publicly available. This software tool, referred to as the Propulsion Diagnostic Method Evaluation Strategy (ProDiMES), has been constructed based on feedback provided by the aircraft EHM community. It provides a standard benchmark problem enabling users to develop, evaluate and compare diagnostic methods. This paper will present an overview of ProDiMES along with a description of four gas path diagnostic methods developed and applied to the problem. These methods, which include analytical and empirical diagnostic techniques, will be described and associated blind-test-case metric results will be presented and compared. Lessons learned along with recommendations for improving the public benchmarking processes will also be presented and discussed.

Simon, Donald L.↗
NASA NTRSDate not supplied

Error Localization Examples: Looking for a Needle in a Hay-stack

Finite element models (FEM) are routinely developed and used during fabrication of high dollar-value hardware. NASA as part of the pre-flight certification of launch vehicles routinely conducts vibration and static tests to calibrate models used for flight-risk assessments. As part of the calibration process, certain areas in the model are modified, using engineering judgment and sensitivity analysis, to match the test results. Unfortunately, tools to identify problem areas in the FEM using test data directly are scarce. Over the years, Error Localization Algorithms (ELA) have been proposed with very limited success. Recently, the Analytical Dynamics Model Improvement (ADMI) algorithm, which computes closed-form mass and stiffness corrections to match the test data exactly, have been shown to be effective for error localization. The paper will present several FEM example problems where ELA is used with simulated test data to determine FEM problem areas. For each example, the correct answer is shown along with ELA results. It is shown that the ELA process is able to identify general problem areas in the FEM, which are consistent with known model perturbations. However, in most cases the ELA identified area of improvement is larger than the true answer. Nonetheless, with proper optimization tools, calibration results using the ELA identified areas provide excellent results.

error localization↗