Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

An Extensible Software and Communication Platform for Distributed Energy Resource Management

This paper introduces a novel Distributed Extensible Grid Control (DEGC) software and communication platform to facilitate the control of distributed energy resources on electric grids. The DEGC software platform leverages state-of-the-art advances in secure, distributed communication and decentralized authorization and authentication. We discuss how these advances enable the kind of robust and secure communication required for a distributed grid control platform, and show how DEGC applies these technologies to the agile development and deployment of grid software through an extensible and flexible API. Here, we describe how DEGC can implement both Volt-VAR voltage magnitude control and Phasor-Based Control as sample applications and demonstrate the DEGC platform in hardware with the demanding Phasor-Based Control test case, and provide performance metrics.

24 POWER TRANSMISSION AND DISTRIBUTION↗

SU-E-T-618: Plan Robustness Study of Volumetric-Modulated Arc Therapy Vs. Intensity-Modulated Radiation Therapy for Head and Neck Cancer

Purpose: Lack of plan robustness may contribute to local failure in volumetric-modulated arc therapy (VMAT) to treat head and neck (H&N) cancer. Thus we compared plan robustness of VMAT with intensity-modulated radiation therapy (IMRT). Methods: VMAT and IMRT plans were created for 9 H&N cancer patients. For each plan, six new perturbed dose distributions were computed — one each for ± 3mm setup deviations along the S-I, A-P and L-R directions. We used three robustness quantification tools: (1) worst-case analysis (WCA); (2) dose-volume histograms (DVHs) band (DVHB); and (3) root-mean-square-dose deviation (RMSD) volume histogram (DDVH). DDVH represents the relative volume (y) on the vertical axis and the RMSD (x) on the horizontal axis. Similar to DVH, this means that y% of the volume of the indicated structure has the RMSD at least x Gy[RBE].The width from the first two methods at different target DVH indices (such as D95 and D5) and the area under the DDVH curves (AUC) for the target were used to indicate plan robustness. In these robustness quantification tools, the smaller the value, the more robust the plan is. Plan robustness evaluation metrics were compared using Wilcoxon test. Results: DVHB showed the width at D95 from IMRT to be larger than from VMAT (unit Gy) [1.59 vs 1.18 (p=0.49)], while the width at D5 from IMRT was found to be slightly larger than from VMAT [0.59 vs 0.54 (p=0.84)]. WCA showed similar results [D95: 3.28 vs 3.00 (p=0.56); D5: 1.68 vs 1.95 (p=0.23)]. DDVH showed the AUC from IMRT to be slightly smaller than from VMAT [1.13 vs 1.15 (p=0.43)]. Conclusion: VMAT plan robustness is comparable to IMRT plan robustness. The plan robustness conclusions from WCA and DVHB are DVH parameter dependent. On the other hand DDVH captures the overall effect of uncertainties on the dose to a volume of interest. NIH/NCI K25CA168984; Eagles Cancer Research Career Development; The Lawrence W. and Marilyn W. Matteson Fund for Cancer Research Mayo ASU Seed Grant; The Kemper Marley Foundation.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

A Functional Response Metric for the Temperature Sensitivity of Tropical Ecosystems

Abstract Earth system models (ESMs) simulate a large spread in carbon cycle feedbacks to climate change, particularly in their prediction of cumulative changes in terrestrial carbon storage. Evaluating the performance of ESMs against observations and assessing the likelihood of long-term climate predictions are crucial for model development. Here, we assessed the use of atmospheric growth rate variations to evaluate the sensitivity of tropical ecosystem carbon fluxes to interannual temperature variations. We found that the temperature sensitivity of the observed growth rate depended on the time scales over which atmospheric observations were averaged. The temperature sensitivity of the growth rate during Northern Hemisphere winter is most directly related to the tropical carbon flux sensitivity since winter variations in Northern Hemisphere carbon fluxes are relatively small. This metric can be used to test the fidelity of interactions between the physical climate system and terrestrial ecosystems within ESMs, which is especially important since the short-term relationship between ecosystem fluxes and temperature stress may be related to the long-term feedbacks between ecosystems and climate. If the interannual temperature sensitivity is used to constrain long-term temperature responses, the inferred sensitivity may be biased by 20%, unless the seasonality of the relationship between the observed growth rate and tropical fluxes is taken into account. These results suggest that atmospheric data can be used directly to evaluate regional land fluxes from ESMs, but underscore that the interaction between the time scales for land surface processes and those for atmospheric processes must be considered.

54 ENVIRONMENTAL SCIENCES↗

ADRIANO2 Calorimeter Performance from 2022 Test Beams

A novel high-granularity dual-readout calorimetric technique was developed as part of the T1604 collaboration. The ADRIANO2 Calorimeter Prototype consists of a pair of optically isolated, small sized tiles made of scintillating plastic and lead glass. Čerenkov light from the lead glass are exploited to for high resolution timing measurements, while high granularity from scintillating plastic can be used to probe the spatial component of the particle shower. This setup works for excellent energy resolution and particle detection for REDTOP as it is crucial for a calorimeter to detect the decay products of eta/eta-prime mesons. Measurements were collected on ADRIANO2 between February to December 2022 to evaluate the detector performance at Fermilab’s Test Beam Facility. The key metrics extracted from my analysis are the detector’s efficiency for various tile configuration and light-yield which will then be used as parameters for an upgraded REDTOP monte-carlo simulation campaign. An in-depth analysis of ADRIANO2 performance are detailed in this presentation.

43 PARTICLE ACCELERATORS↗

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗

Test of parametrized post-newtonian gravity with galaxy-scale strong lensing systems

Based on a mass-selected sample of galaxy-scale strong gravitational lenses from the SLACS, BELLS, LSD, and SL2S surveys and using a well-motivated fiducial set of lens-galaxy parameters, we tested the weak-field metric on kiloparsec scales and found a constraint on the post-Newtonian parameter γ=0.995{sub −0.047}{sup +0.037} under the assumption of a flat ΛCDM universe with parameters taken from Planck observations. General relativity (GR) predicts exactly γ = 1. Uncertainties concerning the total mass density profile, anisotropy of the velocity dispersion, and the shape of the light profile combine to systematic uncertainties of ∼25%. By applying a cosmological model-independent method to the simulated future LSST data, we found a significant degeneracy between the PPN γ parameter and the spatial curvature of the universe. Setting a prior on the cosmic curvature parameter −0.007 < Ω{sub k} < 0.006, we obtained the constraint on the PPN parameter that γ=1.000{sub −0.0025}{sup +0.0023}. We conclude that strong lensing systems with measured stellar velocity dispersions may serve as another important probe to investigate validity of the GR, if the mass-dynamical structure of the lensing galaxies is accurately constrained in future lens surveys.

79 ASTRONOMY AND ASTROPHYSICS↗

Testing the Kerr Black Hole Hypothesis Using X-Ray Reflection Spectroscopy

We present the first X-ray reflection model for testing the assumption that the metric of astrophysical black holes is described by the Kerr solution. We employ the formalism of the transfer function proposed by Cunningham. The calculations of the reflection spectrum of a thin accretion disk are split into two parts: the calculation of the transfer function and the calculation of the local spectrum at any emission point in the disk. The transfer function only depends on the background metric and takes into account all the relativistic effects (gravitational redshift, Doppler boosting, and light bending). Our code computes the transfer function for a spacetime described by the Johannsen metric and can easily be extended to any stationary, axisymmetric, and asymptotically flat spacetime. Transfer functions and single line shapes in the Kerr metric are compared to those calculated from existing codes to check that we reach the necessary accuracy. We also simulate some observations with NuSTAR and LAD/eXTP and fit the data with our new model to show the potential capabilities of current and future observations to constrain possible deviations from the Kerr metric.

79 ASTRONOMY AND ASTROPHYSICS↗

Capturing the Impact of Fuel Octane Number on Modern Gasoline Vehicles with Octane Indices

The need for high efficiency automotive engines has led to more complex air handling and fuel injection systems, higher compression ratios, more advanced combustion and aftertreatment systems, and the use of fuels with higher octane ratings. Higher octane number fuels have a lower propensity to knock. This work studies the influence of changing fuel octane rating on two modern production gasoline vehicles, one with a naturally aspirated, port injected engine and the other with a turbocharged, direct injected engine, using fuels with four different octane number grades (with 85, 87, 91, and 93 anti-knock indices) and operated over a variety of driving cycles and temperature conditions. Unlike previous studies, this effort develops and demonstrates a methodology that isolates fuel effects on fuel consumption and provides a clear view of the octane impact on existing vehicles. While fuel octane rating can also impact factors such as the allowable compression ratio and gear shifting strategies, this study examines fuel consumption changes that are solely attributable to octane rating on production vehicles. The developed approach uses results from drive cycle as well as steady-state tests along with response surface modeling to predict the engine combustion phasing and changes in efficiency for each fuel relative to the baseline 87 anti-knock index (AKI) fuel. The methodology is then used to capture the role of fuel octane rating on powertrain efficiency over Environmental Protection Agency (EPA) certification drive cycles and demonstrate the variation of that impact as a function of powertrain output power. Based on the vehicles and fuels tested in this study, the metrics indicate that the effects of octane on efficiency are limited to a portion of the operating modes represented in the driving cycles used for vehicle certification. In addition, an octane potential benefit index (OPBI) suggests that the vehicle with the turbocharged, direct injected engine has a greater potential to benefit from higher octane number fuel compared to the naturally aspirated, port injected engine and proposes a way to quantify this impact for production vehicles.

Jehlik, Forrest↗

Software interface verifier

A Telos study of 40 recent subsystem deliveries into the DSN at JPL found software interface testing to be the single most expensive and error-prone activity, and the study team suggested creating an automated software interface test tool. The resulting Software Interface Verifier (SIV), which was funded by NASA/JPL and created by Telos, employed 92 percent software reuse to quickly create an initial version which incorporated early user feedback. SIV is now successfully used by developers for interface prototyping and unit testing, by test engineers for formal testing, and by end users for non-intrusive data flow tests in the operational environment. Metrics, including cost, are included. Lessons learned include the need for early user training. SIV is ported to many platforms and can be successfully used or tailored by other NASA groups.

Soderstrom, Tomas J.↗

Parameter Sensitivity Study of the Wall Interference Correction System (WICS)

An off-line version of the Wall Interference Correction System (WICS) has been implemented for the "NASA Langley National Transonic Facility. The correction capability is currently restricted to corrections for solid wall interference in the model pitch plane for Mach numbers, less than 0.45 due to a limitation in tunnel calibration data. A study to assess output sensitivity to the aerodynamic parameters of Reynolds number and Mach number was conducted on this code to further ensure quality during the correction process. In addition, this paper includes all investigation into possible correction due to a semispan test technique using a non metric standoff and all improvement to the standard data rejection algorithm.

Walker, Eric L.↗

Robust Multivariable Optimization and Performance Simulation for ASIC Design

Application-specific-integrated-circuit (ASIC) design for space applications involves multiple challenges of maximizing performance, minimizing power, and ensuring reliable operation in extreme environments. This is a complex multidimensional optimization problem, which must be solved early in the development cycle of a system due to the time required for testing and qualification severely limiting opportunities to modify and iterate. Manual design techniques, which generally involve simulation at one or a small number of corners with a very limited set of simultaneously variable parameters in order to make the problem tractable, are inefficient and not guaranteed to achieve the best possible results within the performance envelope defined by the process and environmental requirements. What is required is a means to automate design parameter variation, allow the designer to specify operational constraints and performance goals, and to analyze the results in a way that facilitates identifying the tradeoffs defining the performance envelope over the full set of process and environmental corner cases. The system developed by the Mixed Signal ASIC Group (MSAG) at the Goddard Space Flight Center is implemented as a framework of software modules, templates, and function libraries. It integrates CAD tools and a mathematical computing environment, and can be customized for new circuit designs with only a modest amount of effort as most common tasks are already encapsulated. Customization is required for simulation test benches to determine performance metrics and for cost function computation.

DuMonthier, Jeffrey↗

Acoustics Technical Working Group and UAM Noise Working Group Proceedings

The NASA Acoustics Technical Working Group Meeting originally started as a program planning meeting in 1992 and has grown through the years. The purpose of these biannual meetings is to foster communication and collaboration among NASA researchers and their university, industry, and government colleagues on activities and work of current and future mutual interest. The NASA-led Urban Air Mobility (UAM) Noise Working Group (UNWG) Meeting began in 2018 as a one-day meeting immediately following the Acoustics Technical Working Group Meeting. The UNWG utilizes four subgroups that conduct regular meetings throughout the year to focus on challenges facing UAM vehicles for community noise and acceptance. The four subgroups coordinate research for development of noise prediction tools/noise reduction technologies, ground and flight test methods, human response and metrics, and regulations and policy (led by the FAA). This work is a compilation of the presentations given at ATWG/UNWG Fall 2022 held in person and via Teams at NASA Glenn Research Center. All NASA content was previously submitted through STRIVES for public presentation. This is merely a compilation. Non-NASA content has permission to distribute. Slide 208 video is included in the Available Downloads as Medial.MOV.

Advanced Air Mobility↗

The Development and Application of Cognitive Performance Assessment for Exploration Class Mission EVAs

The objectives of our research are to: 1) Define cognitive performance requirements for future Lunar and Mars mission EVAs, 2) Characterize cognitive performance in subjects conducting simulated EVAs in analog studies, and 3) Develop and test unobtrusive EVA cognitive performance metrics. Our long-term goal is to develop exploration class cognitive performance monitoring capabilities for EVA and to provide data that can inform EVA planning for Lunar and Mars missions.

S R Anderson↗

Suitable low Global Warming Potential (GWP) refrigerants for two-speed Heat Pumps for residential applications based on speed Heat Pumps for residential applications based on simulated performance simulated performance

Cold climate Heat Pumps (CCHPs) using single-speed compressors in tandem have been successfully built and tested to reach its performance metrics with outdoor temperatures as low as -25°C. CCHPs must be designed to alleviate the problems of too high discharge temperatures, low suction pressure and high-pressure ratio at low ambient conditions and insufficient heating capacity relative to the heating capacity at 8.3°C. In this paper we recognize those concerns. Low-GWP refrigerants are screened based on the shape of their T-S saturation boundary. Simulations of the performance of low-GWP refrigerants (R32, R454B, R466A, and R452B) relative to R410A is done using the DOE/ORNL Heat Pump Design Model at the component level. Systemic inefficiency, the root cause of lost work, is addressed by component-level exergy analysis to refine design options. CCHPs using low-GWP refrigerants addresses issues of reducing energy consumption, lowering carbon footprint, and enabling environmentally sustainability. The CCHP was designed to meet a HSPF > 11.0 and SEER > 15.0 having adequate heat pump heating capacity down to -25°C. It is estimated that at least 2.6 million U.S homes using electric furnaces and heat pumps in cold or very old climates with an annual energy consumption of 0.17EJ would benefit by CCHPs.

Ally, Moonis↗

JPSS-1 VIIRS Radiometric Characterization and Calibration Based on Pre-Launch Testing

The Visible Infrared Imaging Radiometer Suite (VIIRS) on-board the first Joint Polar Satellite System (JPSS) completed its sensor level testing on December 2014. The JPSS-1 (J1) mission is scheduled to launch in December 2016, and will be very similar to the Suomi-National Polar-orbiting Partnership (SNPP) mission. VIIRS instrument has 22 spectral bands covering the spectrum between 0.4 and 12.6 m. It is a cross-track scanning radiometer capable of providing global measurements twice daily, through observations at two spatial resolutions, 375 m and 750 m at nadir for the imaging and moderate bands, respectively. This paper will briefly describe J1 VIIRS characterization and calibration performance and methodologies executed during the pre-launch testing phases by the government independent team to generate the at-launch baseline radiometric performance and the metrics needed to populate the sensor data record (SDR) Look-Up-Tables (LUTs). This paper will also provide an assessment of the sensor pre-launch radiometric performance, such as the sensor signal to noise ratios (SNRs), radiance dynamic range, reflective and emissive bands calibration performance, polarization sensitivity, spectral performance, response-vs-scan (RVS), and scattered light response. A set of performance metrics generated during the pre-launch testing program will be compared to both the VIIRS sensor specification and the SNPP VIIRS pre-launch performance.

VIIRS↗

Up Periscope! Designing a New Perceptual Metric for Imaging System Performance

Modern electronic imaging systems include optics, sensors, sampling, noise, processing, compression, transmission and display elements, and are viewed by the human eye. Many of these elements cannot be assessed by traditional imaging system metrics such as the MTF. More complex metrics such as NVTherm do address these elements, but do so largely through parametric adjustment of an MTF-like metric. The parameters are adjusted through subjective testing of human observers identifying specific targets in a set of standard images. We have designed a new metric that is based on a model of human visual pattern classification. In contrast to previous metrics, ours simulates the human observer identifying the standard targets. One application of this metric is to quantify performance of modern electronic periscope systems on submarines.

imaging↗

JPSS-1 VIIRS Pre-Launch Radiometric Performance

The Visible Infrared Imaging Radiometer Suite (VIIRS) on-board the first Joint Polar Satellite System (JPSS) completed its sensor level testing on December 2014. The JPSS-1 (J1) mission is scheduled to launch in December 2016, and will be very similar to the Suomi-National Polar-orbiting Partnership (SNPP) mission. VIIRS instrument was designed to provide measurements of the globe twice daily. It is a wide-swath (3,040 kilometers) cross-track scanning radiometer with spatial resolutions of 370 and 740 meters at nadir for imaging and moderate bands, respectively. It covers the wavelength spectrum from reflective to long-wave infrared through 22 spectral bands [0.412 microns to 12.01 microns]. VIIRS observations are used to generate 22 environmental data products (EDRs). This paper will briefly describe J1 VIIRS characterization and calibration performance and methodologies executed during the pre-launch testing phases by the independent government team, to generate the at-launch baseline radiometric performance, and the metrics needed to populate the sensor data record (SDR) Look-Up-Tables (LUTs). This paper will also provide an assessment of the sensor pre-launch radiometric performance, such as the sensor signal to noise ratios (SNRs), dynamic range, reflective and emissive bands calibration performance, polarization sensitivity, bands spectral performance, response-vs-scan (RVS), near field and stray light responses. A set of performance metrics generated during the pre-launch testing program will be compared to the SNPP VIIRS pre-launch performance.

RSB↗

In-Depth Analysis of Subject Functional Performance and Subjective Feedback Data for Evaluating Argos Simulation Quality

Mechanical offloading systems such as the Active Response Gravity Offload System (ARGOS) at NASA Johnson Space Center (JSC) are used to simulate various partial gravity environments. ARGOS offloads pressurized suited subjects through a gimbaling pivot point system attached to a volumetric Portable Life Support System (PLSS). The pivot point can be configured to multiple locations with respect to the human-suit system center of gravity (CG). Previous work has tested and documented the interaction between different pivot point locations and measured CG. Observations indicated that small changes in pivot point location substantially affect the stability and functional performance (simulation quality) of a subject. This led to the development of a standardized gimbal assessment protocol to assess the functional performance of a pressurized suited subject as a function of the pivot point location, using the data obtained from a variety of planetary suit mobility tasks. As a result, an expansive repository of CG-related task metrics was generated from numerous ARGOS spacesuit test events, for different gimbal configurations and different test subjects. Based on the evaluation metrics and observed trends, several pivot point locations were iteratively identified to determine an “appropriate/optimal” configuration. This study thus describes the in-depth analysis of the CG evaluation metrics to better quantify trends and determine if specific factors are strongly associated with appropriate/optimal configurations. Quantitative and qualitative variables relating to subject task performance, subjective feedback, and anthropometry will be evaluated via probabilistic methods. The results from this study are expected to improve our understanding of optimal ARGOS gimbal settings, which will better inform the gimbal configuration identification process and improve simulation quality for extravehicular activity (EVA) training.

Joseph Yao↗