Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Automated Testcase Generation for Numerical Support Functions in Embedded Systems

We present a tool for the automatic generation of test stimuli for small numerical support functions, e.g., code for trigonometric functions, quaternions, filters, or table lookup. Our tool is based on KLEE to produce a set of test stimuli for full path coverage. We use a method of iterative deepening over abstractions to deal with floating-point values. During actual testing the stimuli exercise the code against a reference implementation. We illustrate our approach with results of experiments with low-level trigonometric functions, interpolation routines, and mathematical support functions from an open source UAS autopilot.

Metrics↗

Activities of the NASA Urban Air Mobility Noise Working Group (UNWG)

The NASA-led Urban Air Mobility Noise Working Group was established in 2018 following an exploratory meeting of stakeholders from traditional and emerging aviation sectors across industry, government agencies, academia, and community groups. Its mission is to create and support a diverse community of acoustics experts to identify, discuss, and address noise issues associated with UAM vehicles and their operations. The UNWG is comprised of four subgroups including Tools and Technologies, Ground and Flight Testing, Human Response and Metrics, and Regulation and Policy. In 2020, NASA published the white paper entitled “Urban Air Mobility Noise: Current Practice, Gaps, and Recommendations,” that presents a set of high-level goals and makes recommendations to address gaps in the current practice. This presentation is about recent activities of each of the subgroups to address those recommendations, including work towards establishment of an aeroacoustic test database for model validation, development of a set of measurement guidelines to quantify vehicle acoustic emission, a remotely administered human response study for quantifying annoyance to UAM operations, and status on noise standard, regulation, policy, and community engagement developments.

Urban air mobility↗

Validation of an expert system intended for research in distributed artificial intelligence

The expert system discussed in this paper is designed to function as a testbed for research on cooperating expert systems. Cooperating expert systems are members of an organization which dictates the manner in which the expert systems will interact when solving a problem. The Blackbox Expert described in this paper has been constructed using the C Language Integrated Production System (CLIPS), C++, and X windowing environment. CLIPS is embedded in a C++ program which provides objects that are used to maintain the state of the Blackbox puzzle. These objects are accessed by CLIPS rules through user-defined functions calls. The performance of the Blackbox Expert is validated by experimentation. A group of people are asked to solve a set of test cases for the Blackbox puzzle. A metric has been devised which evaluates the 'correctness' of a solution proposed for a test case of Blackbox. Using this metric and the solutions proposed by the humans, each person receives a rating for their ability to solve the Blackbox puzzle. The Blackbox Expert solves the same set of test cases and is assigned a rating for its ability. Then the rating obtained by the Blackbox Expert is compared with the ratings of the people, thus establishing the skill level of our expert system.

Grossner, C.↗

Development of a standardized battery of performance tests for the assessment of noise stress effects

A set of 36 relatively independent categories of human performance were identified. These categories encompass human performance in the cognitive, perceptual, and psychomotor areas, and include diagnostic measures and sensitive performance metrics. Then a prototype standardized test battery was constructed, and research was conducted to obtain information on the sensitivity of the tests to stress, the sensitivity of selected categories of performance degradation, the time course of stress effects on each of the selected tests, and the learning curves associated with each test. A research project utilizing a three factor partially repeated analysis of covariance design was conducted in which 60 male subjects were exposed to variations in noise level and quality during performance testing. Effects of randomly intermittent noise on performance of the reaction time tests were observed, but most of the other performance tests showed consistent stability. The results of 14 analyses of covariance of the data taken from the performance of the 60 subjects on the prototype standardized test battery provided information which will enable the final development and test of a standardized test battery and the associated development of differential sensitivity metrics and diagnostic classificatory system.

Theologus, G. C.↗

2011 Ground Testing Highlights Article

Two tests supporting development of the launch abort system for the Orion MultiPurpose Crew Vehicle were run in the NASA Ames Unitary Plan wind tunnel last year. The first test used a fully metric model to examine the stability and controllability of the Launch Abort Vehicle during potential abort scenarios for Mach numbers ranging from 0.3 to 2.5. The aerodynamic effects of the Abort Motor and Attitude Control Motor plumes were simulated using high-pressure air flowing through independent paths. The aerodynamic effects of the proximity to the launch vehicle during the early moments of an abort were simulated with a remotely actuated Service Module that allowed the position relative to the Crew Module to be varied appropriately. The second test simulated the acoustic environment around the Launch Abort Vehicle caused by the plumes from the 400,000-pound thrust, solid-fueled Abort Motor. To obtain the proper acoustic characteristics of the hot rocket plumes for the flight vehicle, heated Helium was used. A custom Helium supply system was developed for the test consisting of 2 jumbo high-pressure Helium trailers, a twelve-tube accumulator, and a 13MW gas-fired heater borrowed from the Propulsion Simulation Laboratory at NASA Glenn Research Center. The test provided fluctuating surface pressure measurements at over 200 points on the vehicle surface that have now been used to define the ground-testing requirements for the Orion Launch Abort Vehicle.

Ross, James C.↗

Lunar Proving Grounds Definition

The Lunar Surface Innovation Consortium (LSIC) is hosting a hybrid Lunar Proving Grounds Definition Workshop, July 12-13,2023, at the Johns Hopkins University Applied Physics Laboratory in Laurel, Maryland, and on Zoom. The topic of facilities needed for testing hardware destined for the Moon and the need for Earth-based ‘Lunar Proving Grounds’ for testing systems has come up across all six Focus Areas of LSIC. While facilities exist for component- and instrument- level technology maturation (e.g., up to system/subsystem demo in relevant environments), and there are potential flight opportunities for component maturation to flight-qualified and even flight-proved systems, the Artemis Program vision for a sustained presence and transition to industry (e.g., the Moon to Mars Objectives and the LSIC “Path to an Enduring Lunar Presence” white paper) suggests an architecture of integrated systems and systems of systems more complex than Apollo or the International Space Station. Some questions we aim to address through this workshop include: (1) How will validation and verification of these systems and interactions, including human-robotic operations, be accomplished? (2) What metrics need to be tested, and thus what capabilities will such a facility or facilities need? (3) Which functionalities can be tested separately and which need to synergize? What can be the role of digital engineering?

Proving Grounds↗

Comparison of Cognitive Performance Tests for Promethazine Pharmacodynamics in Human Subjects

The objective of this study is to compare cognitive function tests, Automated Neurological Assessment Metrics (ANAM) based Readiness Evaluation System (ARES(Registered TradeMark)) on a Palm Pilot and Windows based Spaceflight Cognitive Assessment Tool (WinSCAT(Registered TradeMark)) on a personal computer (PC) to assess performance effects of promethazine (PMZ) after administration to human subjects. In a randomized placebo-controlled cross-over design, subjects received 12.5, 25, and 50 mg intramuscular (IM) PMZ or a placebo and completed 14 sessions with WinSCAT(Registered TradeMark) (v. 1.26) and ARES(Registered TradeMark) (v. 1.27) consecutively for 72 h post dose. Maximum plasma concentrations (4.25, 6.25 and 13.33 ng/ml) were linear with dose and were achieved by 0.75, 8, and 24 h after dosing for the three doses, respectively. No significant differences in cognitive function after PMZ dosing were detected using WinSCAT(Registered TradeMark), however, tests from ARES(Registered TradeMark) demonstrated concentration dependent decrements in reaction time associated with PMZ dose.

Vaksman, Z.↗

Software metrics: The key to quality software on the NCC project

Network Control Center (NCC) Project metrics are captured during the implementation and testing phases of the NCCDS software development lifecycle. The metrics data collection and reporting function has interfaces with all elements of the NCC project. Close collaboration with all project elements has resulted in the development of a defined and repeatable set of metrics processes. The resulting data are used to plan and monitor release activities on a weekly basis. The use of graphical outputs facilitates the interpretation of progress and status. The successful application of metrics throughout the NCC project has been instrumental in the delivery of quality software. The use of metrics on the NCC Project supports the needs of the technical and managerial staff. This paper describes the project, the functions supported by metrics, the data that are collected and reported, how the data are used, and the improvements in the quality of deliverable software since the metrics processes and products have been in use.

Burns, Patricia J.↗

Development of a Ground Test and Analysis Protocol for NASA's NextSTEP Phase 2 Habitation Concepts

The NASA Next Space Technologies for Exploration Partnerships (NextSTEP) program is a public-private partnership model that seeks commercial development of deep space exploration capabilities to support human spaceflight missions around and beyond cislunar space. NASA first issued the Phase 1 NextSTEP Broad Agency Announcement to U.S. industries in 2014, which called for innovative cislunar habitation concepts that leveraged commercialization plans for low-Earth orbit. These habitats will be part of the Deep Space Gateway (DSG), the cislunar space station planned by NASA for construction in the 2020s. In 2016, Phase 2 of the NextSTEP program selected five commercial partners to develop ground prototypes. A team of NASA research engineers and subject matter experts (SMEs) have been tasked with developing the ground-test protocol that will serve as the primary means by which these Phase 2 prototypes will be evaluated. Since 2008, this core test team has successfully conducted multiple spaceflight analog mission evaluations utilizing a consistent set of operational tools, methods, and metrics to enable the iterative development, testing, analysis, and validation of evolving exploration architectures, operations concepts, and vehicle designs. The purpose of implementing a similar evaluation process for the Phase 2 Habitation Concepts is to consistently evaluate different commercial partner ground prototypes to provide data-driven, actionable recommendations for Phase 3. This paper describes the process by which the ground test protocol was developed and the objectives, methods, and metrics by which the NextSTEP Phase 2 Habitation Concepts will be rigorously and systematically evaluated. The protocol has been developed using both a top-down and bottom-up approach. Top-down development began with the Human Exploration and Operations Mission Directorate (HEOMD) exploration objectives and ISS Exploration Capability Study Team (IECST) candidate flight objectives. Strategic questions and associated rationales, derived from these candidate architectural objectives, provide the framework by which the ground-test protocol will address the DSG stack elements and configurations, systems and subsystems, and habitation, science, and EVA functions. From these strategic questions, high-level functional requirements for the DSG were drafted and associated ground-test objectives and analysis protocols were established. Bottom-up development incorporated objectives from NASA SMEs in autonomy, avionics and software, communication, environmental control and life support systems, exercise, extravehicular activity, exploration medical operations, guidance navigation and control, human factors and behavioral performance, human factors and habitability, logistics, Mission Control Center operations, power, radiation, robotics, safety and mission assurance, science, simulation, structures, thermal, trash management, and vehicle health. Top-down and bottom-up objectives were integrated to form overall functional requirements - ground-test objectives and analysis mapping. From this mapping, ground-test objectives were organized into those that will be evaluated through inspection, demonstration, analysis, subsystem standalone testing, and human-in-the-loop (HITL) testing. For the HITL tests, mission-like timelines, procedures, and flight rules have been developed to directly meet ground test objectives and evaluate specific functional requirements. Data collected from these assessments will be analyzed to determine the acceptability of habitation element configurations and the combinations of capabilities that will result in the best habitation platform to be recommended by the test team for Phase 3.

Gernhardt, Michael L.↗

Program Monitoring with LTL in EAGLE

We briefly present a rule-based framework called EAGLE, shown to be capable of defining and implementing finite trace monitoring logics, including future and past time temporal logic, extended regular expressions, real-time and metric temporal logics (MTL), interval logics, forms of quantified temporal logics, and so on. In this paper we focus on a linear temporal logic (LTL) specialization of EAGLE. For an initial formula of size m, we establish upper bounds of O(m(sup 2)2(sup m)log m) and O(m(sup 4)2(sup 2m)log(sup 2) m) for the space and time complexity, respectively, of single step evaluation over an input trace. This bound is close to the lower bound O(2(sup square root m) for future-time LTL presented. EAGLE has been successfully used, in both LTL and metric LTL forms, to test a real-time controller of an experimental NASA planetary rover.

Barringer, Howard↗

Integrated Resilient Aircraft Control Project Full Scale Flight Validation

Objective: Provide validation of adaptive control law concepts through full scale flight evaluation. Technical Approach: a) Engage failure mode - destabilizing or frozen surface. b) Perform formation flight and air-to-air tracking tasks. Evaluate adaptive algorithm: a) Stability metrics. b) Model following metrics. Full scale flight testing provides an ability to validate different adaptive flight control approaches. Full scale flight testing adds credence to NASA's research efforts. A sustained research effort is required to remove the road blocks and provide adaptive control as a viable design solution for increased aircraft resilience.

Bosworth, John T.↗

Automated Storm Tracking and the Lightning Jump Algorithm Using GOES-R Geostationary Lightning Mapper (GLM) Proxy Data

This study develops a fully automated lightning jump system encompassing objective storm tracking, Geostationary Lightning Mapper proxy data, and the lightning jump algorithm (LJA), which are important elements in the transition of the LJA concept from a research to an operational based algorithm. Storm cluster tracking is based on a product created from the combination of a radar parameter (vertically integrated liquid, VIL), and lightning information (flash rate density). Evaluations showed that the spatial scale of tracked features or storm clusters had a large impact on the lightning jump system performance, where increasing spatial scale size resulted in decreased dynamic range of the system's performance. This framework will also serve as a means to refine the LJA itself to enhance its operational applicability. Parameters within the system are isolated and the system's performance is evaluated with adjustments to parameter sensitivity. The system's performance is evaluated using the probability of detection (POD) and false alarm ratio (FAR) statistics. Of the algorithm parameters tested, sigma-level (metric of lightning jump strength) and flash rate threshold influenced the system's performance the most. Finally, verification methodologies are investigated. It is discovered that minor changes in verification methodology can dramatically impact the evaluation of the lightning jump system.

lightning jump↗

Breaking the Cost Curve: Applying Lessons Learned from the James Webb Space Telescope Development to Build More Cost-Effective Large Space Telescopes in the Future

This paper looks at the key programmatic and technical drivers of the James Webb Space Telescope and assesses ways to building more cost-effective telescopes in the future. The paper evaluates the top level programmatics for JWST along with the key technical drivers from design through integration and testing. Actual data and metrics from JWST are studied to identify what ultimately drove cost on JWST. Finally, the paper assesses areas where applying lessons learned can reduce costs on future observatories and will provide better insights into critical areas to optimize for cost.

ATLAST↗

Breaking the Cost Curve: Applying Lessons Learned from the James Webb Space Telescope Development

This paper looks at the key programmatic and technical drivers of the James Webb Space Telescope and assesses ways to building more cost-effective telescopes in the future. The paper evaluates the top level programmatics for JWST along with the key technical drivers from design through integration and testing. Actual data and metrics from JWST are studied to identify what ultimately drove cost on JWST. Finally, the paper assesses areas where applying lessons learned can reduce costs on future observatories and will provide better insights into critical areas to optimize for cost.

Feinberg, Lee↗

Experimental studies of loudness and annoyance response to sonic booms

The purpose of this paper is to summarize the most recent sonic boom laboratory studies performed at NASA-LaRC using the Sonic Boom Simulator. The first used synthesized idealized outdoor boom shapes which were filtered to represent booms heard inside a house. The test explored the efficacy of various metrics in assessing both loudness and annoyance responses to these booms. The second test investigated the effects of adding single reflections to idealized boom signatures, and the third compared booms recorded from real aircraft with idealized boom signatures to determine if subjects rated the real booms differently. In these studies, as in previous studies performed at NASA-LaRC, there was a continuing effort to evaluate metrics for predicting the subjective effects of sonic booms.

Sullivan, Brenda M.↗

Manufacture of Hybrid Standard-Ply/Thin-Ply Carbon/Epoxy Panels for Notched Test Specimens

Weight is a critical metric for aerospace structures, and one possible way to reduce weight is to improve performance using hybrid standard-ply/thin-ply laminates from a single material system. Previous studies showed benefits for stiffness, strength, reduced damage after impact, and increased compression-after-impact strength for these hybrid laminates. Another study suggested that performance of compression-loaded notched hybrid laminate specimens performed better than standard-ply laminates under compression, but the standard-ply laminates performed better under tension loading. However, analysis predicted the failure load and mode better for the hybrid laminates. An analytical study determined several hybrid laminate designs to use in a follow-on testing regiment that is planned to further study the notched hybrid composite performance under compression and tension load. Described herein is a manufacturing method to efficiently manufacture hybrid laminates using automated fiber placement using a 16-tow head where 8 tows are standard-ply and 8 tows are thin-ply. The method saves time in that it eliminates the need to change spools of material when switching from standard-ply to thin-ply lamina, and reduces the waste associated with changing of spools. The method is ideal for small manufacturing operations or research facilities that have only a single head, and the method was successfully applied at the NASA Langley Research Center (LaRC) Integrated Structural Assembly of Advanced Composites (ISAAC) facility to manufacture nine panels that will be used for the follow-on notched specimen testing of hybrid and standard laminates.

composite↗