Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24

Simplified Readability Metric Drives Improvement of Radiology Reports: an Experiment on Ultrasound Reports at a Pediatric Hospital

Highly complex medical documents, including ultrasound reports, are greatly mismatched with patient literacy levels. While improving radiology reports for readability is a longstanding concern, few articles objectively measure the effectiveness of physician training for readability improvement. We hypothesized that writing styles may be evaluated using an objective two-dimensional measure and writing training could improve the writing styles of radiologists. To test it, a simplified “grade vs. length” readability metric is developed based on results from factor analysis of ten readability metrics applied to more than 500,000 radiology reports. To test the short-term effectiveness of a writing workshop, we measured the writing style improvement before and after the training. Statistically significant writing style improvement occurred as a result of the training. Although the degree of improvement varied for different measures, it is evident that targeted training could provide potential benefits to improve readability due to our statistically significant results. The simplified grade vs. length metric enables future clinical decision support systems to quantitatively guide physicians to improve writing styles through writing workshops.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models

Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models (LLMs) have been incorporated into the training dataset of the models. This gives a false sense of performance that can overestimate how these models will function on truly unseen data. This problem becomes worse with commercial LLMs with larger and non-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box (text access only) and grey box (access to model loss/probabilities required) techniques and apply them to both commercial and non-commercial models. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness (or ineffectiveness) of these contamination detection techniques. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination (broadly) in fine-tuned models when both a baseline and fine-tuned model is present. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar (e.g., drawn from the same set of code projects).

97 MATHEMATICS AND COMPUTING↗

Risk Reduction Testing of Superconducting Coils for the High Efficiency Megawatt Motor

Second generation high temperature superconducting coils provide a very attractive level of performance for use in the field winding of electric machines for aircraft. Aside from maintaining these coils at cryogenic temperatures, much care must be taken to prevent damaging these relatively fragile materials due to excessive stress. Thermal stresses due to temperature cycling and mechanical stresses due to centrifugal forces are the most critical for the High Efficiency Megawatt Motor. This paper summarizes two test campaigns that were undertaken to reduce the risks posed by these loads. In one campaign, several superconducting coils were fabricated, thermally cycled up to 50~times, and electrically tested in liquid nitrogen to assess the change in superconductivity metrics throughout thermal cycling. In the other campaign, a full-scale superconducting coil was fabricated, a test article was designed to produce a representative stress environment in the coil, the test article and coil were rotated at room temperature up to a speed of 11,800~rpm, and the superconductivity metrics were measured in liquid nitrogen before and after each rotation. The experimental results demonstrate that the superconducting coils manufactured in house can survive both thermal cycling and high speed rotation with no appreciable degradation in superconducting performance.

partially superconducting electric machine↗

Quantitative Analysis Guidance in Safety Software Validation

Quantitative analysis in software development has been a challenging topic over the years in safety software. While qualitative processes for vigorous safety software verification and validation (V and V) have been provided through standard such as IEEE 1012, quantitative analysis for V and V tasks such as correctness, completeness or accuracy evaluation have not been formally defined. In practice, there are quantitative analyses in terms of using software metrics. Many of these software metrics are useful in analyzing the test results. Some of them are used for providing guidance in code evaluation activities e.g. numbers of line of codes not exceeding certain limits, recursive calls are not allowed, etc. However, few can be used for providing quantitative analysis guidance for the V and V tasks such as completeness evaluation. AREVA Federal Services (AFS) has provided services for many nuclear projects in US. Through compliance with IEEE 1012, AFS V and V processes have provided vigorous qualitative processes ensuring quality level of the software products used in the project meeting the required nuclear software level. In addition, quantitative analysis is developed to provide structured validation framework ensuring 'zero defect' in the software products. This paper describes some aspects of the quantitative approaches using a sample safety application. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Oculometric Analysis of Saccadic Compensation for Visual Motion Processing Impairment due to Alcohol and Sleep Disruption

The Visuomotor Control Laboratory at Ames Research Center has developed a 5-minute ocular tracking test that computes 21 largely independent metrics of visuomotor performance, reflecting neural signal processing along a number of distinct pathways through cortex, brainstem, and cerebellum. Human sensorimotor performance is resilient to the challenges and stressors of many operational environments, in part, because overall performance is achieved through multiple parallel systems. Our multidimensional oculometrics allow us to examine impacts on these sub-components separately. To illustrate this, we contrasted the effects of two mild neural stressors, acute sleep-deprivation and low-dose alcohol. We have previously shown that, in both cases, oculometric analysis is a highly sensitive indicator of impairment. Here we quantified not only the observed impact on the performance of one sub-system, smooth pursuit, which uses high-level cortical processing of visual motion to track a moving object, but also the observed (partial) compensation by an evolutionarily older mid-brain and brainstem subsystem, saccades, which generates jumps in eye position to catch up with the target when smooth pursuit is inadequate. Specifically, we examined the dose-response (effect size vs. dose size) of the ground lost (pursuit deficit) and the ground recouped (saccadic compensation) across three separate studies – acute low-dose alcohol administration (16 subjects), acute sleep loss (12 subjects), and acute sleep loss with caffeine intervention (9 subjects). We computed the dose-response slopes using linear regression. The figure below shows that, in the case of acute sleep deprivation, the resulting slopes for ground lost and ground recouped (mean ± SE across subjects) were significantly different (paired t-test, t(11) = 5.17, p < 0.001), indicating poor saccadic compensation. However, when sleep loss was coupled with caffeine ingestion, ground lost was decreased and ground recouped increased such that the slopes were no longer different (t(8) = -0.05, p = 0.965). With alcohol, the two slopes were large albeit not significantly different (t(15) = 0.96, p = 0.351), indicating significant pursuit impairment but effective saccadic compensation. Our findings show that sleep deprivation and alcohol affect oculomotor performance differently. Low-dose alcohol effects appear predominantly cortical, with effective brainstem compensation. Sleep loss and circadian disruption however appears to affect both cortical and brainstem pathways with caffeine providing an effective countermeasure to both effects. Beyond the mere detection of impairment, our oculometric assessment allows us to characterize the nature of the deficit, to provide insight into the neural substrate, and to assess the effectiveness of countermeasures.

pursuit↗

Assessing ecological risks to the fish community from residual coal fly ash in Watts Bar Reservoir, Tennessee

For this research, extensive site-specific biological and environmental data were collected to support an evaluation of risks to the fish community in Watts Bar Reservoir from residual ash from the December 2008 Tennessee Valley Authority (TVA) Kingston ash release. This paper describes the approach used and results of the risk assessment for the fish community, which consists of multiple measurement endpoints (measures of exposure and effects) for fish. The lines of evidence included 1) comparing postspill annual fish community assessments with nearby prespill data and data from other TVA reservoirs, 2) evaluating possible effects of exposures of fish eggs and larval fish to ash in controlled laboratory toxicity tests, 3) evaluating reproductive competence of field-exposed fish, 4) assessing individual fish health through physical examination, histopathology, and blood chemistry, 5) comparing fish tissue concentrations with literature-based critical body residues, and 6) comparing concentrations of ash-related contaminants in surface waters with US Environmental Protection Agency's (USEPA) Ambient Water Quality Standards for Fish and Aquatic Life. These measurement endpoints were treated as independent lines of evidence that were integrated into an overall weight-of-evidence estimate of risk to the fish community. Collectively, the data and analysis presented here indicate that ash and ash-related constituents pose negligible risks to the fish communities in Watts Bar Reservoir. This conclusion contradicts the predictions by some researchers immediately following the ash release of devastating effects on the aquatic ecology of Watts Bar Reservoir. The information presented in this article reaffirms the wisdom of carefully evaluating the evidence before predicting probable ecological effects of a major event such as the TVA Kingston ash release. Lastly, this study demonstrates that a thorough and detailed investigation using multiple measurement endpoints is needed to properly evaluate ecological effects.

54 ENVIRONMENTAL SCIENCES↗

Crew Scheduling Performance

As NASA considers long-duration exploration missions (LDEMs), it is envisioned that crew will behave more autonomously as compared to low-Earth orbit missions. The necessary shift of Ops Planners’ complete management of scheduling and planning to provide flexibility for crew to manage their own schedule in real-time requires significant research and investigation in evaluation of concepts of operations, of software tools to support tasks, and of crew performance to complete scheduling tasks. Our research objective is to characterize the human performance envelope for the task of planning and scheduling (crew self-scheduling), develop countermeasures to mitigate adverse performance effects due to plan complexity, and inform performance standards and guidelines based on research results. In our efforts to understand and characterize scheduling performance of crew members, we have conducted subject matter expert interviews to obtain feedback on scheduling plan complexity drivers and plan goodness. Following the pilot study conducted last year, we designed a study and have begun remote human subject testing to evaluate non-expert human performance, workload, and situational awareness for the task of planning and scheduling. This study investigates the effects of the number and type of constraints on human performance, and the differences in scheduling metrics between scheduling and rescheduling tasks. Subject testing is currently in progress, and we aim to collect data from approximately thirty subjects. We will summarize the methods used and impacts of the number and type of scheduling constraints. We will also present lessons learned as well as relevant results of the study.

crew scheduling↗

Physiological Imaging-Defined, Response-Driven Subvolumes of a Tumor

Purpose: To develop an image analysis framework to delineate the physiological imaging-defined subvolumes of a tumor in relating to treatment response and outcome. Methods and Materials: Our proposed approach delineates the subvolumes of a tumor based on its heterogeneous distributions of physiological imaging parameters. The method assigns each voxel a probabilistic membership function belonging to the physiological parameter classes defined in a sample of tumors, and then calculates the related subvolumes in each tumor. We applied our approach to regional cerebral blood volume (rCBV) and Gd-DTPA transfer constant (K{sup trans}) images of patients who had brain metastases and were treated by whole-brain radiation therapy (WBRT). A total of 45 lesions were included in the analysis. Changes in the rCBV (or K{sup trans})–defined subvolumes of the tumors from pre-RT to 2 weeks after the start of WBRT (2W) were evaluated for differentiation of responsive, stable, and progressive tumors using the Mann-Whitney U test. Performance of the newly developed metrics for predicting tumor response to WBRT was evaluated by receiver operating characteristic (ROC) curve analysis. Results: The percentage decrease in the high-CBV-defined subvolumes of the tumors from pre-RT to 2W was significantly greater in the group of responsive tumors than in the group of stable and progressive tumors (P<.007). The change in the high-CBV-defined subvolumes of the tumors from pre-RT to 2W was a predictor for post-RT response significantly better than change in the gross tumor volume observed during the same time interval (P=.012), suggesting that the physiological change occurs before the volumetric change. Also, K{sup trans} did not add significant discriminatory information for assessing response with respect to rCBV. Conclusion: The physiological imaging-defined subvolumes of the tumors delineated by our method could be candidates for boost target, for which further development and evaluation is warranted.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Assessing tension metrics with dark energy survey and Planck data

ABSTRACT Quantifying tensions – inconsistencies amongst measurements of cosmological parameters by different experiments – has emerged as a crucial part of modern cosmological data analysis. Statistically significant tensions between two experiments or cosmological probes may indicate new physics extending beyond the standard cosmological model and need to be promptly identified. We apply several tension estimators proposed in the literature to the dark energy survey (DES) large-scale structure measurement and Planck cosmic microwave background data. We first evaluate the responsiveness of these metrics to an input tension artificially introduced between the two, using synthetic DES data. We then apply the metrics to the comparison of Planck and actual DES Year 1 data. We find that the parameter differences, Eigentension, and Suspiciousness metrics all yield similar results on both simulated and real data, while the Bayes ratio is inconsistent with the rest due to its dependence on the prior volume. Using these metrics, we calculate the tension between DES Year 1 3 × 2pt and Planck, finding the surveys to be in ∼2.3σ tension under the ΛCDM paradigm. This suite of metrics provides a toolset for robustly testing tensions in the DES Year 3 data and beyond.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

High power conditioning and benchmarking of planar nitrogen-incorporated ultrananocrystalline diamond field emission electron source

Planar nitrogen-incorporated ultrananocrystalline diamond [(N)UNCD] has emerged as a unique field emission source attractive for accelerator applications because of its capability to generate a high charge beam and handle moderate vacuum conditions. Most importantly, (N)UNCD sources are simple to produce: Conventional high aspect ratio isolated emitters are not required to be formed on the surface, and the actual emitter surface roughness is on the order of only 10–100 nm. A careful reliability assessment of (N)UNCD is required before it may find routine application in accelerator systems. In the present study using an L-band normal conducting single-cell rf gun, a (N)UNCD cathode has been conditioned to ~ 42 MV / m in a well-controlled manner. It reached a maximum output charge of 15 nC corresponding to an average current of 6 mA during an emission period of 2.5 $μ$s . Imaging of the emission current revealed a large number of isolated emitters (density over 100/cm 2 ) distributed on the cathode, which is consistent with previous tests in dc environments. The performance metrics, the emission imaging, and the systematic study of emission properties during rf conditioning in a wide gradient range assert (N)UNCD as an enabling electron source for rf injector designs serving industrial and scientific applications. These studies also improve the fundamental knowledge of the practical conditioning procedure via a better understanding of the emission mechanisms.

43 PARTICLE ACCELERATORS↗

Classical dynamics of brane-world extended objects

We make use of the universally valid stress-energy conservation law to study the motion of various branelike extended objects in a generic brane-world. Without specifying any particular action, we are able to derive the world-sheet equations that govern the dynamics of brane-world test branes. In particular, the brane-world test particles are shown to follow geodesics with respect to the brane-world induced metric. At the same time, the presence of extended objects is shown to influence the brane-world geometry. It is demonstrated that codimension-1 branes necessarily violate the brane-world smooth structure, while lower-dimensional branes violate the very continuity. In particular, the truly zero-size massive particles are shown not to exist in a continuous brane-world. As an example, static, axially symmetric membrane-world in 4d Minkowski background is analyzed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Stochastic Multi-Timescale Power System Operations With Variable Wind Generation

This paper describes a novel set of stochastic unit commitment and economic dispatch models that consider stochastic loads and variable generation at multiple operational timescales. The stochastic model includes four distinct stages: stochastic day-ahead security-constrained unit commitment (SCUC), stochastic real-time SCUC, stochastic real-time security-constrained economic dispatch (SCED), and deterministic automatic generation control (AGC). These sub-models are integrated together such that they are continually updated with decisions passed from one to another. The progressive hedging algorithm (PHA) is applied to solve the stochastic models to maintain the computational tractability of the proposed models. Comparative case studies with deterministic approaches are conducted in low wind and high wind penetration scenarios to highlight the advantages of the proposed methodology, one with perfect forecasts and the other with current state-of-the-art but imperfect deterministic forecasts. The effectiveness of the proposed method is evaluated with sensitivity tests using both economic and reliability metrics to provide a broader view of its impact.

17 WIND ENERGY↗

SU-E-T-450: How Important Is a Reproducible Breath Hold for DIBH Breast Radiotherapy?

Purpose: Deep inspiration breath hold (DIBH) for left-sided breast cancer has been shown to reduce heart dose. Surface imaging helps to ensure accurate breast positioning, but does not guarantee a reproducible breath hold (BH) at DIBH treatments. We examine the effects of variable BH positions for DIBH treatments. Methods: Twenty-Five patients with free breathing (FB) and DIBH scans were reviewed. Four plans were created for each patient: 1) FB, 2) DIBH, 3) FB-DIBH – the DIBH plans were copied to the FB images and recalculated (image registration was based on breast tissue), and 4) P-DIBH – a partial BH with the heart shifted midway between the FB and DIBH positions. The FB-DIBH plans give “worst case” scenarios for surface imaging DIBH, where the breast is aligned by surface imaging but the patient is not holding their breath. Students t-tests were used to compare dose metrics. Results: The DIBH plans gave lower heart dose and comparable breast coverage versus FB in all cases. The FB-DIBH plans showed no significant difference versus FB plans for breast coverage, mean heart dose, or maximum heart dose (p >= 0.10). The mean heart dose differed between FB-DIBH and FB by < 2 Gy for all cases, the maximum heart dose differed by < 2 Gy for 21 cases. The P-DIBH plans showed significantly lower mean heart dose than FB (p = 0.01). The mean heart doses for the P-DIBH plans were < FB for 22 cases, the maximum dose < FB for 18 cases. Conclusions: A DIBH plan delivered to a FB patient set-up with surface imaging will yield similar dosimetry to a plan created and delivered FB. A DIBH plan delivered with even a partial BH can give reduced heart dose compared to FB techniques when the breast tissue is well aligned.

60 APPLIED LIFE SCIENCES↗

Carbon Mineralization by Aqueous Precipitation for Beneficial Use of CO2 from Flue Gas

Calera's innovative Mineralization via Aqueous Precipitation (MAP) technology for the capture and conversion of CO{sub 2} to useful materials for use in the built environment was further developed and proven in the Phase 1 Department of Energy Grant. The process was scaled to 300 gallon batch reactors and subsequently to Pilot Plant scale for the continuous production of product with the production of reactive calcium carbonate material that was evaluated as a supplementary cementitious material (SCM). The Calera SCM{trademark} was evaluated as a 20% replacement for ordinary portland cement and demonstrated to meet the industry specification ASTM 1157 which is a standard performance specification for hydraulic cement. The performance of the 20% replacement material was comparable to the 100% ordinary portland cement control in terms of compressive strength and workability as measured by a variety of ASTM standard tests. In addition to the performance metrics, detailed characterization of the Calera SCM was performed using advanced analytical techniques to better understand the material interaction with the phases of ordinary portland cement. X-ray synchrotron diffraction studies at the Advanced Photon Source in Argonne National Lab confirmed the presence of an amorphous phase(s) in addition to the crystalline calcium carbonate phases in the reactive carbonate material. The presence of carboaluminate phases as a result of the interaction of the reactive carbonate materials with ordinary portland cement was also confirmed. A Life Cycle Assessment was completed for several cases based on different Calera process configurations and compared against the life cycle of ordinary portland cement. In addition to the materials development efforts, the Calera technology for the production of product using an innovative building materials demonstration plant was developed beyond conceptual engineering to a detailed design with a construction schedule and cost estimate.

43 PARTICLE ACCELERATORS↗

Detecting and Testing for Structural Error in Computer Models with Application to the Community Atmospheric Model

Bayesian calibration/inversion methods for uncertain input parameters to simulation codes have generated an increasing amount of interest in the climate community in the recent years. This approach uses multiple disparate observational datasets related to the output of the simulator of interest to yield a probabilistic constraint on the uncertain inputs (i.e. probability distribution). A major challenge in applying Bayesian calibration methods to simulation codes is related to the possible presence of a large structural error in the output fields of interest; that is, the presence of discrepancy between the observed and the simulated output that is neither explained by the uncertainty in the inputs nor the observations. We present here methods to test for and visualize structural error in an ensemble of input-perturbed simulations. The methods are based on bootstrap (resmpling) test statistics and well-known metrics to quantify the difference between two probability distributions. The proposed methods are demonstrated using selected precipitation output fields of the Community Atmospheric Model (CAM), which are compared to observations provided by the Global Precipitation Climatology Project (GPCP). The methods are found to be useful in identifying regions and seasons with a large structural error. The proposed structural error detection methodology can be used to screen multiple observational datasets of interest for structural error, making it possible to retain only the datasets with a small structural error for use in Bayesian calibration. In addition, information about the amount of structural error can be valuable to model developers.

58 GEOSCIENCES↗

Scalable Metadata Management for a Large Multi-Source Seismic Data Repository

In this work, we implemented the key metadata management components of a scalable seismic data ingestion framework to address limitations in our existing system, and to position it for anticipated growth in volume and complexity. We began the effort with an assessment of open source data flow tools from the Hadoop ecosystem. We then began the construction of a layered architecture that is specifically designed to address many of the scalability and data quality issues we experience with our current pipeline. This included implementing basic functionality in each of the layers, such as establishing a data lake, designing a unified metadata schema, tracking provenance, and calculating data quality metrics. Our original intent was to test and validate the new ingestion framework with data from a large-scale field deployment in a temporary network. This delivered somewhat unsatisfying results, since the new system immediately identified fatal flaws in the data relatively early in the pipeline. Although this is a correct result it did not allow us to sufficiently exercise the whole framework. We then widened our scope to process all available metadata from over a dozen online seismic data sources to further test the implementation and validate the design. This experiment also uncovered a higher than expected frequency of certain types of metadata issues that challenged us to further tune our data management strategy to handle them. Our result from this project is a greatly improved understanding of real world data issues, a validated design, and prototype implementations of major components of an eventual production framework. This successfully forms the basis of future development for the Geophysical Monitoring Program data pipeline, which is a critical asset supporting multiple programs. It also positions us very well to deliver valuable metadata management expertise to our sponsors, and has already resulted in an NNSA Office of Defense Nuclear Nonproliferation commitment to a multi-year project for follow-on work.

58 GEOSCIENCES↗

Evaluation of the Need to Consider Colloid-Facilitated Plutonium Transport in Contaminant Boundary Calculations for Pahute Mesa at the Nevada Nuclear Security Site

The testing of nuclear weapons at the Nevada National Security Site (NNSS), formerly the Nevada Test Site, has led to the deposition of substantial quantities of plutonium (Pu) into the environment. While tritium ( 3 H) is the most abundant anthropogenic radionuclide deposited in the NNSS subsurface from an activity standpoint (4.7×10 6 TBq decay corrected to September 23, 1992), Pu is the most abundant anthropogenic element by mass. Between 1951 and 1992, 828 underground nuclear tests were performed and approximately 2.8 metric tons (3.1×10 4 TBq) of Pu remains in the NNSS subsurface (Smith et al., 2003). The vast majority of this Pu is sequestered in nuclear melt glass that pools at the bottom of cavities formed by nuclear explosions.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗