Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 685 records · Page 38

Challenges in the detection of gases in exoplanet atmospheres

Claims of detections of gases in exoplanet atmospheres often rely on comparisons between models including and excluding specific chemical species. However, the space of molecular combinations available for model construction is vast and highly degenerate. Only a limited subset of these combinations is typically explored for any given detection. As a result, apparent detections of trace gases risk being artefacts of incomplete modelling rather than robust identification of atmospheric constituents, especially in the low-signal-to-noise regime. Here, using the sub-Neptune K2-18 b as a case study, we show that recent biosignature claims vanish when the model space is expanded, with numerous alternatives providing equally good or better fits. We demonstrate that the significance of a claimed detection relies on the choice of models being compared, and that model preference does not in itself imply the presence of a specific gas. In conclusion, we recommend treating model comparisons instead as relative adequacy tests, which should be supported by theoretical predictions and complementary metrics of statistical significance to attribute a signal to a particular gas.

Astronomy and AstroPhysics↗

DefectTrack: a deep learning-based multi-object tracking algorithm for quantitative defect analysis of in-situ TEM videos in real-time

Abstract In-situ irradiation transmission electron microscopy (TEM) offers unique insights into the millisecond-timescale post-cascade process, such as the lifetime and thermal stability of defect clusters, vital to the mechanistic understanding of irradiation damage in nuclear materials. Converting in-situ irradiation TEM video data into meaningful information on defect cluster dynamic properties (e.g., lifetime) has become the major technical bottleneck. Here, we present a solution called the DefectTrack , the first dedicated deep learning-based one-shot multi-object tracking (MOT) model capable of tracking cascade-induced defect clusters in in-situ TEM videos in real-time. DefectTrack has achieved a Multi-Object Tracking Accuracy (MOTA) of 66.43% and a Mostly Tracked (MT) of 67.81% on the test set, which are comparable to state-of-the-art MOT algorithms. We discuss the MOT framework, model selection, training, and evaluation strategies for in-situ TEM applications. Further, we compare the DefectTrack with four human experts in quantifying defect cluster lifetime distributions using statistical tests and discuss the relationship between the material science domain metrics and MOT metrics. Our statistical evaluations on the defect lifetime distribution suggest that the DefectTrack outperforms human experts in accuracy and speed.

42 ENGINEERING↗

Constraint-preserving boundary conditions in the 3+1 first-order approach

A set of energy-momentum constraint-preserving boundary conditions is proposed for the first-order Z4 case. The stability of a simple numerical implementation is tested in the linear regime (robust stability test), both with the standard corner and vertex treatment and with a modified finite-differences stencil for boundary points which avoids corners and vertices even in Cartesian-like grids. Moreover, the proposed boundary conditions are tested in a strong-field scenario, the Gowdy waves metric, showing the expected rate of convergence. The accumulated amount of energy-momentum constraint violations is similar or even smaller than the one generated by either periodic or reflection conditions, which are exact in the Gowdy waves case. As a side theoretical result, a new symmetrizer is explicitly given, which extends the parametric domain of symmetric hyperbolicity for the Z4 formalism. The application of these results to first-order Baumgarte-Shapiro-Shibata-Nakamura-like formalisms is also considered.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Additional Capacity Value From Synergy of Variable Renewable Energy and Energy Storage

Current capacity markets often consider capacity credits from each resource independently, irrespective of the portfolio of resources, potentially overvaluing or undervaluing the capacity contribution of variable renewable energy (VRE) and energy storage (ES) in the grid. We propose a method for calculating the standalone and integrated capacity value of an added VRE resource with existing ES resources. The difference between the integrated and standalone value is the portfolio effect. This is the additional capacity value gained by the synergy of VRE and the existing fleet. Using chronological dispatch simulations and two different reliability metrics to estimate firm capacity, we demonstrate on a small test system that the portfolio effect can be substantial.

capacity credit↗

TU-C-17A-06: Evaluating IMRT Plan Deliverability Via PTV Shape and MLC Motion

Purpose: For step-and-shoot intensity-modulated radiation therapy (IMRT) plans, the dosimetry and deliverability can be affected by the number and shape of the segments used. Thus, plan deliverability is likely related to target volume and shape. We investigated whether the sphericity of target volumes and the previously proposed Modulation Complexity Score (MCS) could be used together to improve the detection of IMRT fields that failed quality assurance (QA). Methods: 526 and 353 IMRT fields from 32 prostate and 28 head-and-neck (H'N) patients, respectively, were analyzed. MCS was used to quantify the complexity of multi-leaf collimator shapes and motion patterns for each field. Sphericity was calculated using the surface area and volume of each patient’s planning target volume (PTV). Logistic regression models with MCS-alone or MCS and sphericity terms were fit to PlanUNC IMRT pass/fail results (5% dose difference, 4mm distance-to-agreement criteria) using SAS 9.3 (Cary, NC). Model concordance, discordance and area under the curve (AUC) were used to quantify model accuracy. Results: Mean (±1 standard deviation) MCS for prostate and H'N were 0.58(±0.15) and 0.40 (±0.14), respectively. Mean sphericity scores were 0.75(±0.05) for prostate and 0.63 (±0.12) for H'N. Both metrics were significantly different between treatment locations (p<0.01, Wilcoxon Rank Sum Test) indicating greater complexity in shape and variations for H'N PTVs. For prostate, concordance, discordance and AUC using MCS alone were 80.8%, 18.7% and 0.811. Including sphericity in the model improved these to 81.7%, 17.7% and 0.820. For H'N, the original concordance, discordance and AUC were of 72.9%, 26.9% and 0.729. Including sphericity into the model improved these metrics to 76.5%, 23.2% and 0.729. Conclusion: Sphericity provides a quantitative measure of PTV shape. While improvement in IMRT QA failure detection was modest for both prostate and H'N plans, including sphericity in the model does improve IMRT field failure prediction.

60 APPLIED LIFE SCIENCES↗

MO-FG-202-09: Virtual IMRT QA Using Machine Learning: A Multi-Institutional Validation

Purpose: To validate a machine learning approach to Virtual IMRT QA for accurately predicting gamma passing rates using different QA devices at different institutions. Methods: A Virtual IMRT QA was constructed using a machine learning algorithm based on 416 IMRT plans, in which QA measurements were performed using diode-array detectors and a 3%local/3mm with 10% threshold. An independent set of 139 IMRT measurements from a different institution, with QA data based on portal dosimetry using the same gamma index and 10% threshold, was used to further test the algorithm. Plans were characterized by 90 different complexity metrics. A weighted poison regression with Lasso regularization was trained to predict passing rates using the complexity metrics as input. Results: In addition to predicting passing rates with 3% accuracy for all composite plans using diode-array detectors, passing rates for portal dosimetry on per-beam basis were predicted with an error <3.5% for 120 IMRT measurements. The remaining measurements (19) had large areas of low CU, where portal dosimetry has larger disagreement with the calculated dose and, as such, large errors were expected. These beams need to be further modeled to correct the under-response in low dose regions. Important features selected by Lasso to predict gamma passing rates were: complete irradiated area outline (CIAO) area, jaw position, fraction of MLC leafs with gaps smaller than 20 mm or 5mm, fraction of area receiving less than 50% of the total CU, fraction of the area receiving dose from penumbra, weighted Average Irregularity Factor, duty cycle among others. Conclusion: We have demonstrated that the Virtual IMRT QA can predict passing rates using different QA devices and across multiple institutions. Prediction of QA passing rates could have profound implications on the current IMRT process.

60 APPLIED LIFE SCIENCES↗

On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems. In: EASC '16 Proceedings of the Exascale Applications and Software Conference 2016, Article No. 5

The present work is targeted at performing a strong scaling study of the high-order spectral element fluid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent flow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000-10,000 degrees of freedom per core on Mira, 30,000 - 50,0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10,000 and 220,000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.

Offermans, Nicolas↗

Citing a Data Repository: A Case Study of the Protein Data Bank

The Protein Data Bank (PDB) is the worldwide repository of 3D structures of proteins, nucleic acids and complex assemblies. The PDB’s large corpus of data (> 100,000 structures) and related citations provide a well-organized and extensive test set for developing and understanding data citation and access metrics. In this paper, we present a systematic investigation of how authors cite PDB as a data repository. We describe a novel metric based on information cascade constructed by exploring the citation network to measure influence between competing works and apply that to analyze different data citation practices to PDB. Based on this new metric, we found that the original publication of RCSB PDB in the year 2000 continues to attract most citations though many follow-up updates were published. None of these follow-up publications by members of the wwPDB organization can compete with the original publication in terms of citations and influence. Meanwhile, authors increasingly choose to use URLs of PDB in the text instead of citing PDB papers, leading to disruption of the growth of the literature citations. A comparison of data usage statistics and paper citations shows that PDB Web access is highly correlated with URL mentions in the text. The results reveal the trend of how authors cite a biomedical data repository and may provide useful insight of how to measure the impact of a data repository.

59 BASIC BIOLOGICAL SCIENCES↗

Cell-level reliability testing procedures for CIGS photovoltaics

The reliability of photovoltaics is commonly studied at the module level. Many reliability problems originate from module attributes, such as metal interconnections to cells, junction boxes. However, significant work in reliability can also be done prior to module design. Testing for reliability earlier in the research cycle increases the probability of avoiding common module reliability problems before cell changes are implemented on a large scale. Cell-level reliability studies can thus lower the rates of module failures in the field and provide confidence to investors that new technologies will perform as advertised. This report summarizes how we investigated three reliability concerns in Cu(In,Ga)Se 2 (CIGS) photovoltaics at the cell level: metastability, shading-induced damage, and potential-induced degradation (PID). We find that examining these concerns required developing robust measurement protocols including the fabrication of novel testing structures. This information will allow readers to incorporate sound metrics for investigating reliability phenomena and aid their studies of cell and module reliability improvements.

14 SOLAR ENERGY↗

DISTINCT: Diversity in Solar Talent Through INnovative Curriculum and Training: An Integrated Research and Education Approach towards Creating Diversity and Advancing Utility-Scale Solar Technology

The DISTINCT project research objective is to develop an innovative N-port power converter for a utility-scale PV system that is modular, compact and cost-effective and that will enable the integration of a high-frequency, high-voltage solid-state transformer. The novelty of the proposed research is the electrical power conversion architecture using an N-port converter system that replaces the output 60Hz transformer with an integrated high-frequency low-weight solid-state transformer reducing power electronics and BOS costs to meet SunShot goals through modularity and direct high-voltage interconnection. A challenge in direct integration with a 13.8kV line is the high voltage handling capacity of the converters combined with high efficiency operation. The front-end converter for each port is a Neutral-Point Clamped (NPC) Multi-Level dc-dc Dual-Active Bridge (ML-DAB) which allows Maximum Power Point Tracking (MPPT). The integrated high frequency transformer provides the galvanic isolation between the PV and grid side and also steps up the low dc voltage from PV source. Following the ML-DAB stage, in each port, is an inverter with H-bridge configuration or NPC configuration. N number of NPC inverters’ outputs are cascaded to attain the per-phase line-to-neutral voltage to connect directly to the distribution grid (i.e. 13.8 kV). The cascaded inverters have the inherent advantage of using lower rated devices, smaller filters and low Total Harmonic Distortion (THD) required for PV grid interconnection. Our analysis and simulation results show improved performance on cost, efficiency, service life with zero downtime and THD. A comprehensive control scheme is presented to ensure the maximum power from each port and each phase are sent to the grid. A functional prototype of a 2-port converter with ML-DAB and cascaded H-bridges has been designed, built, and tested in a laboratory setup to verify the target technical metrics. The N-port converter system due to its modular structure with individual control per port can be easily adapted to integrate functionalities that go well beyond the conventional grid support functions and mitigates impacts of forecasted fast ramp downs or ramp ups and single-fault conditions by automatic reconfiguration of the output.

14 SOLAR ENERGY↗

Remote Sensing Approach for Monitoring Tree Health Adjacent to Transmission Corridors

This study presents an initial proof-of-concept for a satellite-based remote sensing approach to identify and monitor potential areas of poor tree health across the entire BPA service territory on an annual basis. We tested three variants of “delta peak NDVI” ( ΔPN ) change detection metrics that express interannual variation in primary productivity relative to a baseline by comparing ΔPN values for known insect/disease disturbances and nearby reference locations. All three metrics showed promise for detecting poor tree health in the year during disturbance, but the metric based on the difference from the long-term (2016-2024) median ( Δ Med PN ) was preferred due to its responsiveness to change in the years during and after disturbance, resilience to interannual variation, and ease of interpretation as being above or below normal. Comparison of Δ Med PN grouped by relative severity of disturbance indicated it was not sensitive enough to detect “low” severity disturbances, as mapped by USGS’s LANDFIRE program, but could distinguish “moderate” and “high” severity disturbances from reference locations. These findings informed selection of a threshold for Δ Med PN , which was combined with areas exhibiting negative NDVI to map potential areas of concern. Visual inspection of before/after high-resolution imagery and NDVI time series showed that many areas of concern aligned with visible signs of defoliation and die-off as well as other types of disturbance (e.g., landslides, logging, road grading, flooding). Some areas of concern are thought to be false detections caused by persistent shadow, and some could not be explained with visual inspection due to spatiotemporal limitations of before/after imagery. In summary, our approach shows promise for large-scale monitoring of tree health adjacent to BPA transmission lines, but additional work is recommended to improve model sophistication and remove noise.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Intrusive Uncertainty Quantification and Optimal Experiment Design in the Open-Source Pyomo Ecosystem

This contribution describes ParmEst and Pyomo.DoE, two pillars of the open-source Python-based Pyomo ecosystem for computational optimization with (partial differential) algebraic equation mathematical models. Specifically, ParmEst facilitates intrusive frequentist parameter estimation (PE) and uncertainty quantification (UQ) through built-in features, such as covariance matrix estimation, bootstrapping, and likelihood ratio tests. Complementary, Pyomo.DoE enables optimal experiment design by maximizing various metrics of the Fisher information matrix, such as A-optimality (trace), D-optimality (determinant), E-optimality (minimum eigenvalue), and ME-optimality (condition number). ParmEst and Pyomo.DoE can solve high-dimensional optimization problems by leveraging the model structure and exact derivative information. Finally, we will discuss future opportunities to integrate PE and UQ capabilities with optimization under uncertainty, including robust optimization with non-convex models via PyROS.

97 MATHEMATICS AND COMPUTING↗

Preliminary geologic map of the region around the candidate Proclus Apollo landing site, part J

The Proclus Crater region was mapped to test the value, for photogeologic mapping purposes, of Apollo 15 metric photographs and to estimate the scientific value of the area as a potential landing site. Adjacent frames of the metric photography were overlapped with the base frame to provide stereographic images. Because of the high quality of the photographs, it was found that the geologic units could be more definitely interpreted and dated than those of earlier maps. The photographs tend to confirm the earlier interpretation of the rugged area as composed of bedrock uplifted when the Crisium basin formed. They also suggest that an earlier predominantly volcanic interpretation of the terra in this area might be replaced by an interpretation in which mass wasting and fracturing play the major roles in producing different terrain types. It was concluded that because of apparent lithologic homogeneity, the terra of the Proclus region is an undesirable objective for an extensive manned lunar landing mission.

Wilhelms, D. E.↗

Numerical Estimation of the Curvature of Biological Surfaces

Many biological systems may profitably be studied as surface phenomena. A model consisting of isotropic growth of a curved surface from a flat sheet is assumed. With such a model, the Gaussian curvature of the final surface determines whether growth rate of the surface is subharmonic or superharmonic. These properties correspond to notions of convexity and concavity, and thus to local excess growth and local deficiency of growth. In biological models where the major factors controlling surface growth are intrinsic to the surface, researchers thus gained from geometrical study information on the differential growth undergone by the surface. These ideas were applied to an analysis of the folding of the cerebral cortex, a geometrically rather complex surface growth. A numerical surface curvature technique based on an approximation to the Dupin indicatrix of the surface was developed. A metric for comparing curvature estimates is introduced, and considerable numerical testing indicated the reliability of this technique.

Todd, P. H.↗

An exact solution for a thick domain wall in general relativity

An exact solution of the Einstein equations for a static, planar domain wall with finite thickness is presented. At infinity, density and pressure vanish and the space-time tends to the Minkowski vacuum on one side of the wall and to the Taub vacuum on the other side. A surprising feature of this solution is that the density and pressure distribution are symmetric about the central plane of the wall whereas the space-time metric and therefore also the gravitational field experienced by a test particle is asymmetric.

Goetz, Guenter↗

Comparison of DAC and MONACO DSMC Codes with Flat Plate Simulation

Various implementations of the direct simulation Monte Carlo (DSMC) method exist in academia, government and industry. By comparing implementations, deficiencies and merits of each can be discovered. This document reports comparisons between DSMC Analysis Code (DAC) and MONACO. DAC is NASA's standard DSMC production code and MONACO is a research DSMC code developed in academia. These codes have various differences; in particular, they employ distinct computational grid definitions. In this study, DAC and MONACO are compared by having each simulate a blunted flat plate wind tunnel test, using an identical volume mesh. Simulation expense and DSMC metrics are compared. In addition, flow results are compared with available laboratory data. Overall, this study revealed that both codes, excluding grid adaptation, performed similarly. For parallel processing, DAC was generally more efficient. As expected, code accuracy was mainly dependent on physical models employed.

Padilla, Jose F.↗

EVA Health and Human Performance Benchmarking Study

Multiple HRP Risks and Gaps require detailed characterization of human health and performance during exploration extravehicular activity (EVA) tasks; however, a rigorous and comprehensive methodology for characterizing and comparing the health and human performance implications of current and future EVA spacesuit designs does not exist. This study will identify and implement functional tasks and metrics, both objective and subjective, that are relevant to health and human performance, such as metabolic expenditure, suit fit, discomfort, suited postural stability, cognitive performance, and potentially biochemical responses for humans working inside different EVA suits doing functional tasks under the appropriate simulated reduced gravity environments. This study will provide health and human performance benchmark data for humans working in current EVA suits (EMU, Mark III, and Z2) as well as shirtsleeves using a standard set of tasks and metrics with quantified reliability. Results and methodologies developed during this test will provide benchmark data against which future EVA suits, and different suit configurations (eg, varied pressure, mass, CG) may be reliably compared in subsequent tests. Results will also inform fitness for duty standards as well as design requirements and operations concepts for future EVA suits and other exploration systems.

Abercromby, A. F.↗

Effective L/D: A Theoretical Approach to the Measurement of Aero-Structural Efficiency in Aircraft Design

There are many trade-offs in aircraft design that ultimately impact the overall performance and characteristics of the final design. One well recognized and well understood trade-off is that of wing weight and aerodynamic efficiency. Higher aerodynamic efficiency can be obtained by increasing wing span, usually at the expense of higher wing weight. The proper balance of these two competing factors depends on the objectives of the design. For example, aerodynamic efficiency is preeminent for sailplanes and long slender wings result. Although the wing weight-drag trade is universally recognized, aerodynamic efficiency and structural efficiency are not usually considered in combination. This paper discusses the concept of "aero-structural efficiency," which combines weight and drag characteristics. A metric to quantify aero-structural efficiency, termed effective L/D, is then derived and tested with various scenarios. Effective L/D is found to be a practical and robust means to simultaneously characterize aerodynamic and structural efficiency in the context of aircraft design. The primary value of the effective L/D metric is as a means to better communicate the combined system level impacts of drag and structural weight.

Guynn, Mark D.↗