Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25

Automatic documentation system extension to multi-manufacturers' computers and to measure, improve, and predict software reliability

The DOMONIC system has been modified to run on the Univac 1108 and the CDC 6600 as well as the IBM 370 computer system. The DOMONIC monitor system has been implemented to gather data which can be used to optimize the DOMONIC system and to predict the reliability of software developed using DOMONIC. The areas of quality metrics, error characterization, program complexity, program testing, validation and verification are analyzed. A software reliability model for estimating program completion levels and one on which to base system acceptance have been developed. The DAVE system which performs flow analysis and error detection has been converted from the University of Colorado CDC 6400/6600 computer to the IBM 360/370 computer system for use with the DOMONIC system.

Simmons, D. B.↗

Time Domain Identification of an Optimal Control Pilot Model with Emphasis on the Objective Function

A method for the identification of the pilot's control compensation using time domain techniques is proposed. From this information we hope to infer a quadratic cost function, supported by the data, that represents a reasonable expression for the pilot's control objective in the task being performed, or an inferred piloting strategy. The objectives for this method are: (1) obtain a better understanding of the fundamental piloting techniques in complex tasks, such as landing approach; (2) the development of a metric measurable in simulations and flight test that correlate with subjective pilot opinion; and (3) to further validate pilot models and pilot vehicle analysis methods.

Schmidt, D. K.↗

Four applications of a software data collection and analysis methodology

The evaluation of software technologies suffers because of the lack of quantitative assessment of their effect on software development and modification. A seven-step data collection and analysis methodology couples software technology evaluation with software measurement. Four in-depth applications of the methodology are presented. The four studies represent each of the general categories of analyses on the software product and development process: blocked subject-project studies, replicated project studies, multi-project variation studies, and single project strategies. The four applications are in the areas of, respectively, software testing, cleanroom software development, characteristic software metric sets, and software error analysis.

Basili, Victor R.↗

Implementation and Evaluation of Multiple Adaptive Control Technologies for a Generic Transport Aircraft Simulation

Presented here is the evaluation of multiple adaptive control technologies for a generic transport aircraft simulation. For this study, seven model reference adaptive control (MRAC) based technologies were considered. Each technology was integrated into an identical dynamic-inversion control architecture and tuned using a methodology based on metrics and specific design requirements. Simulation tests were then performed to evaluate each technology s sensitivity to time-delay, flight condition, model uncertainty, and artificially induced cross-coupling. The resulting robustness and performance characteristics were used to identify potential strengths, weaknesses, and integration challenges of the individual adaptive control technologies

Campbell, Stefan F.↗

Unmanned Aircraft Systems (UAS) Traffic Management (UTM) National Campaign II

The Unmanned Aircraft System (UAS) Traffic Management (UTM) effort at NASA aims to enable access to low-altitude airspace for small UAS. This goal is being pursued partly through partnerships that NASA has developed with the UAS stakeholder community, the FAA, other government agencies, and the designated FAA UAS Test Sites. By partnering with the FAA UAS Test Sites, NASA's UTM project has performed a geographically diverse, simultaneous set of UAS operations at locations in six states. The demonstrations used an architecture that was developed by NASA in partnership with the FAA to safely coordinate such operations. These demonstrations-the second or 'Technical Capability Level (TCL 2)' National Campaign of UTM testing-was performed from May 15 through June 9, 2017. Multiple UAS operations occurred during the testing at sites located in Alaska, Nevada, Texas, North Dakota, Virginia, and New York with multiple organizations serving as UAS Service Suppliers and/or UAS Operators per the specifications provided by NASA. By engaging various members of the UAS community in development and operational roles, this campaign provided initial validation of different aspects of the UTM concept including: UAS Service Supplier technologies and procedures; geofencing technologies/conformance monitoring; ground-based surveillance/sense and avoid; airborne sense and avoid; communication, navigation, surveillance; and human factors related to UTM data creation and display. Additionally, measures of performance were defined and calculated from the flight data to establish quantitative bases for comparing flight test activities and to provide potential metrics that might be routinely monitored in future operational UTM systems.

Aweiss, Arwa S.↗

Unmanned Aircraft Systems (UAS) Integration in the National Airspace System (NAS) Project: Terminal Operations HITL 1B Primary Results

This presentation provides an overview of the primary results from the Unmanned Aircraft Systems (UAS) Integration in the National Airspace System (NAS) Project's second Terminal Operations human-in-the-loop simulation. This talk covers the background of this follow-on experiment, which includes an overview of the first Terminal Operations HITL performed by the project. The primary results include a look at the number and durations of detect and avoid (DAA) alerts issued by the two DAA systems under test. It also includes response time metrics and metrics on the ability of the pilot-in-command (PIC) to maintain sufficient separation. Additional interoperability metrics are included to illustrate how pilots interact with the tower controller. Implications and conclusions are covered at the end.

unmanned aircraft systems↗

Unmanned Aircraft Systems (UAS) Traffic Management (UTM) National Campaign II

The Unmanned Aircraft System (UAS) Traffic Management (UTM) effort at NASA aims to enable access to low-altitude airspace for small UAS. This goal is being pursued partly through partnerships that NASA has developed with the UAS stakeholder community, the FAA, other government agencies, and the designated FAA UAS Test Sites. By partnering with the FAA UAS Test Sites, NASA's UTM project has performed a geographically diverse, simultaneous set of UAS operations at locations in six states. The demonstrations used an architecture that was developed by NASA in partnership with the FAA to safely coordinate such operations. These demonstrations—the second or "Technical Capability Level (TCL 2)" National Campaign of UTM testing—was performed from May 15 through June 9, 2017. Multiple UAS operations occurred during the testing at sites located in Alaska, Nevada, Texas, North Dakota, Virginia, and New York with multiple organizations serving as UAS Service Suppliers and/or UAS Operators per the specifications provided by NASA. By engaging various members of the UAS community in development and operational roles, this campaign provided initial validation of different aspects of the UTM concept including: UAS Service Supplier technologies and procedures; geofencing technologies/conformance monitoring; ground-based surveillance/sense and avoid; airborne sense and avoid; communication, navigation, surveillance; and human factors related to UTM data creation and display. Additionally, measures of performance were defined and calculated from the flight data to establish quantitative bases for comparing flight test activities and to provide potential metrics that might be routinely monitored in future operational UTM systems.

Aweiss, Arwa↗

Statistical Classification of Biosignature Information using Multiple Instrument Observations

The accurate identification of biosignatures (indications of life) from data taken from remote or in situ planetary exploration is one of the most important challenges in astrobiology, the interdisciplinary field examining habitability and the potential for extraterrestrial life. This study employs machine learning algorithms to optimize the identification of biosignatures, with an emphasis on those which are agnostic to a specific biochemical basis. We exploit the wealth of terrestrial data available from biogenic and abiogenic systems to enhance efficient feature prioritization. Our dataset, pulled from public databases and laboratory recorded measurements, includes elemental abundance, isotopic fractionation, and VNIR/Raman spectra The data curation process included standardization for detection limits and ranges. Subsequent feature extraction yielded detailed inputs for machine learning, including combinations of elemental content, isotopic ratios, and parameters of spectral peaks and troughs. Feature significance was evaluated across diverse machine learning methodologies, such as k-nearest neighbors, logistic regression, Random Forest, support vector machines, and Gaussian Naïve Bayes, along with a combined voting classifier. We utilized Receiver Operating Characteristic Area Under the Curve (ROC AUC) across 2,000 50% test-train splits as a robust metric of model performance. Results revealed a promising ROC AUC of 0.853 for the combined voting classifier. Removing elemental abundance data notably reduced model accuracy (13% decrease in AUC), highlighting its critical role in biosignature detection. Several other individual data features exhibited significance within their respective data types, offering additional granularity. This research fortifies the relevance of machine learning to astrobiology, potentially enhancing life detection missions by allowing algorithmic prioritization of high-interest samples for further investigation. Future work will refine data standardization, expand the dataset to include more terrestrial systems, and incorporate convolutional neural networks for spectral feature extraction. The potential for public data sharing is also under exploration, reinforcing our commitment to collective scientific advancement.

Statistical↗

Validation of Pronghorn for Natural-Circulation Molten Salt Loops

This paper presents the development and validation of a high-fidelity thermal-hydraulic model of a molten salt natural circulation flow loop, designed for integration within a digital twin framework. The study evaluates the performance of Idaho National Laboratory’s Pronghorn against experimental data from Texas A&M University Molten Salt Flow Loop (MSFL) four Hitec-salt test benchmark data. Natural circulation of high-Prandtl-number fluids exhibits complex, counter-intuitive flow patterns that make pointwise thermocouple readings unreliable. Experimental work at TAMU’s MSFL provides benchmark data, including flow visualization at a test-section and centerline steady-state temperature measurements along the loop. Validation includes four single-phase natural circulation test cases with Hitec salt. Key metrics include flow profile agreement and steady-state temperature accuracy. Pronghorn results for two-dimensional single-phase agree qualitatively with the experimental flow profile. This paper illustrates the importance of Computational Fluid Dynamics (CFD) in elucidating the behavior of high-Prandtl-number thermal-hydraulics, along with how misleading centerline temperature measurements can be. Pronghorn reproduces the axial and radial stratification that makes single thermocouple readings unreliable. Future research will focus on reduced-order modeling techniques to enable rapid simulation suitable for real-time digital twin applications. The validated cases provide a basis for developing reduced-order surrogates aimed at real-time digital-twin applications.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Utilization of the Critic Subnetwork of a Generative Adversarial Network as Detector of Morphological Material Change in Image Data

The resolution of computed tomography (CT) has become high enough to monitor morphological changes due to aging in materials in long-term applications. For this work, we explored the utility of the critic of a generative adversarial network (GAN) to automatically detect such changes. The GAN was trained with images of pristine Pharmatose, which is used as a surrogate energetic material. It is important to note that images of the material with altered morphology were only used during the test phase. The GAN-generated images reproduced the microstructure of Pharmatose well, although some unrealistic particle fusion was seen. Calculated morphological metrics (volume fraction, interfacial line length, and local thickness) for the synthetic images also showed good agreement with the training data, albeit with signs of mode collapse in the interfacial line length. While the critic exposed changes in particle size, it showed limited ability to distinguish images by particle shape. The detection of shape differences was also a more challenging task for the selected morphological metrics that related to energetic material performance. We further tested the critic with images of aged Pharmatose. Subtle changes due to aging are difficult for the human analyst to detect; but both critic and morphological metrics analysis showed image differentiation.

36 MATERIALS SCIENCE↗

Modal Test and Model Correlation of NASA Plum Brook Station Mechanical Vibration Facility Head Expander –Lessons Learned from the Perspective of an Early-Career Engineer

In preparation for the Sierra Nevada Corporation’s (SNC) Dream Chaser spacecraft vibration test campaign at the Mechanical Vibration Facility (MVF) at NASA Plum Brook Station (PBS) in Sandusky, Ohio, a test-verified model of MVF is needed in order to be able to perform accurate pretest analysis used for determining response limits and abort levels. MVF was designed to vibration test MPCV Orion and was used to perform the system level vibration test of the European Service Module Structural Test Article (E-STA) in 2016. MVF is comprised of an 18 ft diameter annulus table that is driven with sixteen hydraulic vertical actuator assemblies and four hydraulic horizontal actuator assemblies, which allow it to perform single axis vibration testing in the vertical axis and in each of the two orthogonal horizontal axes without the need for reconfiguring the test article. A head expander for the MVF Table has been designed and built that fills in the center opening providing a continuous flat mounting surface with a maximum diameter of 16.25feet that expands the vibration testing capabilities of MVF. The MVF Table with this head expander will be used during the SNC Dream Chaser spacecraft vibration test campaign. Therefore, a critical element in a test-verified model of the MVF will be a test correlated finite element model (FEM) of the head expander. To obtain this, engineers from the Structural Dynamics Lab (SDL) at NASA Glenn Research Center (GRC) in Cleveland, Ohio performed a modal pretest analysis, conducted a modal test in July 2019, and most recently correlated the head expander finite element model to the modal test data up to 300 Hz. From the initial test preparations to the final delivery of a correlated finite element model, all efforts mentioned were led by the same early-career engineers at NASA GRC. From the viewpoint of an early-career engineer, lessons learned about modal pretest analysis, modal testing, and finite element model correlation of the MVF Table expander head will be presented and discussed. This will include the importance of understanding the limitations of using uncorrelated finite element models in the modal pretest analysis and planning, the importance of orthogonality metrics in judging adequacy and accuracy of test mode shapes, and the importance of having the FEM match the as built hardware in the model correlation effort.

Emma L Pierson↗

Quantification of morphological change in materials based on image data utilizing machine learning techniques

Computed tomography (CT) resolution has become high enough to monitor morphological changes due to aging in materials in long-term applications. We explored the utility of the critic of a generative adversarial network (GAN) to automatically detect such changes. The GAN was trained with images of pristine Pharmatose, which is used as a surrogate energetic material. It is important to note that images of the material with altered morphology were only used during the test phase. The GAN-generated images visually reproduced the microstructure of Pharmatose well, although some unrealistic particle fusion was seen. Calculated morphological metrics (volume fraction, interfacial line length, and local thickness) for the synthetic images also showed good agreement with the training data, albeit with signs of mode collapse in the interfacial line length. While the critic exposed changes in particle size, it showed limited ability to distinguish images by particle shape. The detection of shape differences was also a more challenging task for the selected morphological metrics that related to energetic material performance. We further tested the critic with images of aged Pharmatose. Subtle changes due to aging are difficult for the human analyst to detect. Both critic and morphological metrics analysis showed image differentiation.

36 MATERIALS SCIENCE↗

One network metric datastore to track them all: the OSG network metric service

The Open Science Grid (OSG) relies upon the network as a critical part of the distributed infrastructures it enables. In 2012, OSG added a new focus area in networking with a goal of becoming the primary source of network information for its members and collaborators. This includes gathering, organizing, and providing network metrics to guarantee effective network usage and prompt detection and resolution of any network issues, including connection failures, congestion, and traffic routing. In September of 2015, this service was deployed into the OSG production environment. We will report on the creation, implementation, testing, and deployment of the OSG Networking Service. Starting from organizing the deployment of perfSONAR toolkits within OSG and its partners, to the challenges of orchestrating regular testing between sites, to reliably gathering the resulting network metrics and making them available for users, virtual organizations, and higher level services, all aspects of implementation will be reviewed. In particular, several higher-level services were developed to bring the OSG network service to its full potential. These include a web-based mesh configuration system, which allows central scheduling and management of all the network tests performed by the instances; a set of probes to continually gather metrics from the remote instances and publish it to different sources; a central network datastore (esmond), which provides interfaces to access the network monitoring information in close to real time and historically (up to a year) giving the state of the tests; and a perfSONAR infrastructure monitor system, ensuring the current perfSONAR instances are correctly configured and operating as intended. We will also describe the challenges we encountered in ongoing operations of the network service and how we have evolved our procedures to address those challenges. Finally we will describe our plans for future extensions and improvements to the service.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Are light curve classification metrics good proxies for SN Ia cosmological constraining power?

Context. When selecting a light curve classifier for use as part of a photometric supernova Ia (SN Ia) cosmological analysis, it is common to make decisions based on metrics of classification performance, such as the contamination within the photometrically classified SN Ia sample, rather than a measure of cosmological constraining power. If the former is an appropriate proxy for the latter, this practice would eliminate the computational expense of a full cosmology forecast in the analysis pipeline design process. Aims. This study tests the assumption that light curve classification metrics are an appropriate proxy for cosmology metrics. Methods. We emulated photometric SN Ia cosmology light curve samples with controlled contamination rates of individual contaminant classes and evaluated each of them under a set of classification metrics. We then derived cosmological parameter constraints from all samples under two common analysis approaches and quantified the impact of contamination by each contaminant class on the resulting cosmological parameter estimates. Results. We observe that cosmology metrics are sensitive to both the contamination rate and the class of the contaminating population, whereas the classification metrics are shown to be insensitive to the latter. Conclusions. Based on these findings, we discourage any exclusive reliance on light curve classification-based metrics for analysis design decisions, which (counterintuitively) include but are not limited to the classifier choice. Instead, we recommend optimising science analysis pipeline design choices using a metric of the information gained about the physical parameters of interest.

79 ASTRONOMY AND ASTROPHYSICS↗

Safe, Advanced, Adaptable Isolation System Eliminates the Need for Critical Lifts

The Starr Soft Support isolation system incorporates an automatically reconfigurable aircraft jack into NASA's existing 1-Hertz isolators. This enables an aircraft to float in mid-air without the need for a critical lift during ground vibration testing (GVT), significantly reducing testing risk, time, and costs. Currently incorporating the most advanced technology available, the 60,000-poundcapacity (27-metric-ton) isolation system is used for weight and measurement tests, control-surface free-play tests, and structural mode interaction tests without the need for any major reconfiguration, often saving days of time and significantly reducing labor costs. The Starr Soft Support isolation system consists of an aircraft-jacking device with three jacking points, each of which has an individual motor and accommodates up to 20,000 pounds (9 metric tons) for a total 60,000-pound (27-metric-ton) capacity. The system can be transported to the aircraft by forklift and placed at its jacking points using a pallet jack. The motors power the electric actuators, raising the aircraft above the ground until the landing gear can retract. Inflatable isolators then deploy, enabling the aircraft to float in mid-air, simulating a 1-Hertz free-free boundary condition. Inflatable isolators have been in use at NASA for years, enabling aircraft to literally float unsupported for highly accurate GVT. These isolators must be placed underneath the aircraft for this to occur. Traditionally, this is achieved by a critical lift a high-risk procedure in which a crane and flexible cord system are used to lift the aircraft. In contrast, the Starr Soft Support isolation system eliminates the need for critical lift by integrating the inflatable isolators into an aircraft jacking system. The system maintains vertical and horizontal isolating capabilities. The aircraft can be rolled onto the system, jacked up, and then the isolators can be inflated and positioned without any personnel needing to work underneath the aircraft. Also, the system accommodates changes in aircraft configuration, automatically adapting to changes in mass, and it can adjust the height of the isolators in one basic setup. Dryden personnel used the Starr Soft Support system to successfully perform a GVT on an F-15 being structurally modified by Gulfstream, Dryden's Gulfstream III used for science research and the crew exploration module and adaptor cone assembly.

Ginn, Starr↗

A Physical Protection Systems Test Bed for International Counter-Trafficking System Development

Physical protection systems have a widespread impact on the nuclear industry in areas such as nuclear safeguards, arms control, and trafficking of illicit goods (e.g., nuclear materials) across international borders around the world. Many challenges must be overcome in design and deployment of foreign border security systems such as lack of infrastructure, extreme environmental conditions, limited knowledge of terrain, insider threats, and occasional cultural resistance. Successful security systems, whether it be a system designed to secure a single facility or a whole border security system, rely on the entire integrated system composed of multiple subsystems. This test bed is composed of many unique sensors and subsystems, including wireless unattended ground sensors, a buried fiber-optic acoustic sensor, a lossy coaxial distributed sensor, wireless links, pan-tilt-zoom cameras, mobile power generation systems, unmanned aerial vehicles, and fiber-optic-fence intrusion detection systems. A Common Operating Picture software architecture is utilized to integrate a number of these subsystems. We are currently performance testing each system for border security and perimeter security applications by examining metrics such as probability of sense and a qualitative understanding of the sensor s vulnerability of defeat. The testing process includes different soil conditions for buried sensors (e.g., dry, wet, and frozen) and an array of different tests including walking, running, stealth detection, and vehicle detection. Also, long term sustainability of systems is tested including performance differences due to seasonal variations (e.g. summer versus winter, while raining, in foggy conditions). The capabilities of the test bed are discussed. Performance testing results, both at the individual component level and integrated into a larger system for a specific deployment (in situ), help illustrate the usefulness and need for integrated testing facilities to carry out this mission. The test bed provides access to grassy fields, wooded areas, and a large waterway three distinct testing environments. The infrastructure supporting deployment of systems at the test bed includes grid power, renewable power systems, climate controlled enclosures, high bandwidth wireless links, and a fiber optic communications backbone. With over 10 acres of dedicated area and direct waterway access, the test bed is well suited for long term test and evaluation of physical protection and security systems targeting a wide range of applications.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Anisotropic Solution Adaptive Unstructured Grid Generation Using AFLR

An existing volume grid generation procedure, AFLR3, was successfully modified to generate anisotropic tetrahedral elements using a directional metric transformation defined at source nodes. The procedure can be coupled with a solver and an error estimator as part of an overall anisotropic solution adaptation methodology. It is suitable for use with an error estimator based on an adjoint, optimization, sensitivity derivative, or related approach. This offers many advantages, including more efficient point placement along with robust and efficient error estimation. It also serves as a framework for true grid optimization wherein error estimation and computational resources can be used as cost functions to determine the optimal point distribution. Within AFLR3 the metric transformation is implemented using a set of transformation vectors and associated aspect ratios. The modified overall procedure is presented along with details of the anisotropic transformation implementation. Multiple two-and three-dimensional examples are also presented that demonstrate the capability of the modified AFLR procedure to generate anisotropic elements using a set of source nodes with anisotropic transformation metrics. The example cases presented use moderate levels of anisotropy and result in usable element quality. Future testing with various flow solvers and methods for obtaining transformation metric information is needed to determine practical limits and evaluate the efficacy of the overall approach.

Marcum, David L.↗

Hoop/column and tetrahedral truss electromagnetic tests

The distortion of antennas was measured with a metric camera system at discrete target locations on the surface. Given are surface distortion for hoop column reflector antennas, for tetrahedral truss reflector antennas, and distortion contours for the tetrahedral truss reflector. Radiation patterns at 2.27-GHz, 4.26-GHz, 7.73-GHz and 11.6-GHz are given for the hoop column antenna. Also given are radiation patterns at 4.26-GHz and 7.73-GHz for the tetrahedral truss antenna.

Bailey, M. C.↗