Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Deep-Learning-Derived Evaluation Metrics Enable Effective Benchmarking of Computational Tools for Phosphopeptide Identification

Tandem mass spectrometry (MS/MS)-based phosphoproteomics is a powerful technology for global phosphorylation analysis. However, applying four computational pipelines to a typical mass spectrometry (MS)-based phosphoproteomic dataset from a human cancer study, we observed a large discrepancy among the reported phosphopeptide identification and phosphosite localization results, underscoring a critical need for benchmarking. While efforts have been made to compare performance of computational pipelines using data from synthetic phosphopeptides, evaluations involving real application data have been largely limited to comparing the numbers of phosphopeptide identifications due to the lack of appropriate evaluation metrics. We investigated three deep learning-derived features as potential evaluation metrics: phosphosite probability, Delta RT and spectral similarity. Predicted phosphosite probability is computed by MusiteDeep, which provides high accuracy as previously reported; Delta RT is defined as the absolute retention time (RT) difference between RTs observed and predicted by AutoRT; and spectral similarity is defined as the Pearson’s correlation coefficient between spectra observed and predicted by pDeep2. Using a synthetic peptide dataset, we found that both Delta RT and spectral similarity provided excellent discrimination between correct and incorrect peptide-spectrum matches (PSMs) both when incorrect PSMs involved wrong peptide sequences and even when incorrect PSMs were caused by only incorrect phosphosite localization. Based on these results, we used all the three deep learning-derived features as evaluation metrics to compare different computational pipelines on diverse set of phosphoproteomic datasets and showed their utility in benchmarking performance of the pipelines. The benchmark metrics demonstrated in this study will enable users to select computational pipelines and parameters for routine analysis of phosphoproteomics data and will offer guidance for developers to improve computational methods.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

How to select distracted driving countermeasures evaluation metrics: A systematic review

While there are numerous performance metrics that have been developed for the evaluation of distracted driving prevention programs, there is little information on how to select them depending on the requirements and/or objectives of the study. Here, this paper describes a systematic literature review that was conducted on the metrics for evaluating distracted driving countermeasures in order to bridge this research gap. A summary of the evaluation metrics used for the existing distracted driving countermeasures was provided. Guidance for choosing an evaluation measure was provided by analyzing the metrics from the perspectives of functionality, spatial-temporal dimension, and equity. Three examples of distracted driving countermeasure evaluation processes were thoroughly reviewed in order to offer insight into metric selection and measurement. This paper contributes to the body of knowledge by discussing the implications for policy on how to enhance the thoroughness and accuracy of the evaluation of distracted driving countermeasures. Analysis from multiple angles, the development of data collection tools and direct behavior indicators, taking into account temporal dimensions, and equity considerations, are proposed.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2022

A New Evaluation Metric for Demand Response-Driven Real-Time Price Prediction Towards Sustainable Manufacturing

Abstract The increasing industry energy demand highlights the urgency of demand response management, while the emerging smart manufacturing technologies pave the way for the implementation of real-time price (RTP)-based demand response management towards sustainable manufacturing. The demand response management requires scheduling of manufacturing systems based on RTP predictions, and thus the prediction quality can directly alter the effectiveness of demand response. However, since the general price prediction algorithms and prediction evaluation metrics are not specifically designed for RTP in demand response problems, a good RTP prediction obtained and evaluated by these algorithms and metrics may not be suitable for demand response scheduling. Therefore, in this study, the relationships between the effectiveness of demand response for manufacturing systems and evaluation results from six commonly used metrics are investigated. Meanwhile, a new metric called k-peak distance (KPD), considering the characteristics of the demand response problem, is proposed and compared with the other six metrics. Furthermore, an encoder-decoder long short-term memory recurrent neural network with KPD is proposed to provide better RTP prediction for manufacturing demand response problems. The case studies indicate that the proposed KPD metric shows a 1.8–3.6 times higher correlation with the demand response effectiveness compared to the other metrics. In addition, the production schedule based on the RTP prediction obtained from the proposed algorithm can improve the effectiveness of demand response by 23.4% on average.

Engineering↗
DOE OSTI2022

Review of Wave Energy Converter Power Take-Off Systems, Testing Practices, and Evaluation Metrics: Preprint

While the field of wave energy has been the subject of numerical simulation, scale model testing, and precommercial project testing for decades, wave energy technologies remain in the early stages of development and must continuing proving themselves as a promising modern renewable energy field. One of the difficulties that wave energy systems have been struggling to overcome is the design of highly efficient energy conversion systems that can convert the mechanical power, derived from the oscillation of wave activated bodies, into another useful product. Often the power take-off (PTO) is defined as the single unit responsible for converting mechanical power into another usable form such as electricity, pressurized fluid, compressed air, and others. The PTO, and the entire power conversion chain (PCC), is of great importance as it affects not only how efficient wave power is converted into electricity, but also contributes to the mass, size, structural dynamics, and levelized cost of energy (LCOE) of the wave energy converter (WEC). Unlike wind and solar, there is no industrial standard device, or devices, for wave energy conversion and this diversity is transferred to the PTO system. The majority of current WEC PTO systems incorporate a mechanical or hydraulic drive train, power generator, and an electrical control system. The challenge of WEC PTO designs is designing a mechanical-to-electrical component that can efficiently convert irregular, bi-directional, low frequency and low alternating velocity wave motions. While gross average power levels can be predicted in advance, the variable wave elevation input has to be converted into smooth electrical output and hence usually necessitates some type of energy storage system, such as battery storage, accumulator super capacitors, etc., or other means of compensation such as an array of devices. One of the primary challenges for wave energy converter systems is the fluctuating nature of wave resources, which require WEC components to be designed to handle loads (i.e. torques, forces, and powers) that are many times greater than the average load. This approach requires a much greater PTO capacity than the average power output and indicates a higher cost. In addition, supporting mechanical coupling and or gearing can be added to the PCC to help alleviate the difficulties with transmission and control of fluctuating large loads with low frequencies (indicative of wave forcing) into smaller loads at higher frequencies (optimum for conventional electrical machine design) can quickly increase the complexity of the PCC which could result in a greater number of failure modes and increased maintenance costs. All of the previous points demonstrate how the PTO influences WEC dynamics, reliability, performance and cost which are critical design factors. This paper further explores these topics by providing a review of the state-of-the-art PTO systems currently under development, how these novel PTO systems are tested and derisked prior to precommercial deployment, and the evaluation metrics historically used to differentiate between PTO designs and how they can be improved to support control co-design focused development of wave energy systems.

laboratory testing↗
DOE OSTI2025

Power-Capping Metric Evaluation for Improving Energy Efficiency in HPC Applications

With high-performance computing systems now running at exascale, optimizing power-scaling management and resource utilization has become more critical than ever. This paper explores runtime power-capping optimizations that leverage integrated CPU-GPU power management on architectures like the NVIDIA GH200 superchip. We evaluate energy-performance metrics that account for simultaneous CPU and GPU power-capping effects by using two complementary approaches: speedup-energy-delay and a Euclidean distance-based multi-objective optimization method. By targeting a mostly compute-bound exascale science application, the Locally Self-Consistent Multiple Scattering (LSMS), we explore challenging scenarios to identify potential opportunities for energy savings in exascale applications, and we recognize that even modest reductions in energy consumption can have significant overall impacts. Our results highlight how GPU task-specific dynamic power-cap adjustments combined with integrated CPU-GPU power steering can improve the energy utilization of certain GPU tasks, thereby laying the groundwork for future adaptive optimization strategies.

Patrou, Maria [ORNL] (ORCID:0000000339754638)↗
DOE OSTI2020

FY2020 Fourth Quarter Performance Metric: Evaluate Improvement in Simulations of Mesoscale Convective Systems from New Parameterization Developments in E3SM

Mesoscale convective systems (MCSs) consist of an assembly of cumulonimbus clouds on scales of 100 km or more and produce mesoscale circulations (Houze, 2004, 2018). As the largest form of deep convective storms, MCSs contribute to 30% – 70% of annual and warm season rainfall in the U.S. and in the global tropics (Houze 2018; Stevenson & Schumacher, 2014; Feng et al., 2019; Haberlie & Ashley, 2019). Since MCSs contribute importantly to mean and extreme precipitation in the U.S. and many other regions around the world, understanding how well they are simulated by E3SM may guide future development towards more skillful modeling of convective storms and associated hydrologic impacts. The FY2020 Second Quarter Performance Metric Report documented comparisons of MCSs in the central and eastern U.S. in a high-resolution simulation produced by E3SM v1 at 25 km resolution (Caldwell et al. 2019) with observations. MCSs in the simulation occur less frequently and produce less intense precipitation, resulting in large underestimation of MCS volumetric rain-rate compared to observations. The first and third quarter performance metric report indicated that these model biases in simulating MCSs can be attributed to model limitations in parameterizing convection, clouds, and other related processes, as well as model biases in simulating the MCS large-scale environment. In the current FY2020 Fourth Quarter Performance Metric Report, we evaluate MCSs simulated in E3SM with several new developments in convection parameterizations that are being developed for its next generation. The goal is to summarize what have been improved with the new developments and highlight what need more work in the future.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Developing and Evaluating Metrics for Demand Flexibility in Buildings: Comparing Simulations and Field Data

Building demand flexibility (DF) has attracted significant attention in recent years among researchers, technology developers and control companies, aggregators, utilities, and many others. There are numerous challenges with today’s electricity systems such as managing peak demand capacity and integrating variable renewables into the grid. Flexible building loads can provide various grid services to help reduce electricity costs, smooth out renewables intermittency and balance supply and demand. Recognizing this, the US DOE is leading the Grid-interactive Efficient Buildings (GEB) initiative which includes research to evaluate the potential, availability and timing of flexible loads. In this paper we present load shed metrics for three building types – medium office, large office and retail store – and compare prototype simulation results with measured data from 12 actual buildings that participated in hot summertime utility demand response (DR) events. The DR strategies included varying zone temperature and reducing light levels. The magnitude of a key DF metric, “demand decrease intensity” (or “shed intensity”) (W/ft2), between the simulation results and field data are similar (14-32% differences) for both mean and median values, though the field data show much larger variation among DR events. The coefficient p-values from linear regression model tests showed that outside air temperature is a significant variable for the whole building shed intensity when the resetting zone temperature strategy is deployed. These findings support the concept of using prototype building simulation to estimate building DF and expanding future simulation research to additional building types and climate zones.

Liu, Jingjing↗
DOE OSTI2023

Quantitative Power System Resilience Metrics and Evaluation Approach

Power system resilience is an emerging topic and plays an essential role in helping the power industry understand and respond to the increasing threats of extreme weather events. The first step of power system resilience analysis is to introduce metrics to quantify the resilience reasonably. Existing resilience metrics are typically restrained by the limited data for extreme event modeling and fall short in terms of physical interpretation and comparability. This paper develops novel quantitative metrics to evaluate power system resilience in pre- and post-event contexts. The developed metrics illustrate clear physical meanings and can be effectively used to compare resilience across different systems under different extreme events. Moreover, the developed metrics can be applied to both transmission and distribution systems. Simulation on a distribution system is employed to validate the effectiveness of the proposed resilience metrics and resilience evaluation approach.

power system resilience↗
DOE OSTI2023

Identifying strengths and weaknesses of methods for computational network inference from single-cell RNA-seq data

Single-cell RNA-sequencing (scRNA-seq) offers unparalleled insight into the transcriptional programs of different cellular states by measuring the transcriptome of thousands of individual cells. An emerging problem in the analysis of scRNA-seq is the inference of transcriptional gene regulatory networks and a number of methods with different learning frameworks have been developed to address this problem. Here, we present an expanded benchmarking study of eleven recent network inference methods on seven published scRNA-seq datasets in human, mouse, and yeast considering different types of gold standard networks and evaluation metrics. We evaluate methods based on their computing requirements as well as on their ability to recover the network structure. We find that, while most methods have a modest recovery of experimentally derived interactions based on global metrics such as Area Under the Precision Recall curve, methods are able to capture targets of regulators that are relevant to the system under study. Among the top performing methods that use only expression were SCENIC, PIDC, MERLIN or Correlation. Addition of prior biological knowledge and the estimation of transcription factor activities resulted in the best overall performance with the Inferelator and MERLIN methods that use prior knowledge outperforming methods that use expression alone. We found that imputation for network inference did not improve network inference accuracy and could be detrimental. Comparisons of inferred networks for comparable bulk conditions showed that the networks inferred from scRNA-seq datasets are often better or at par with the networks inferred from bulk datasets. Our analysis should be beneficial in selecting methods for network inference. At the same time, this highlights the need for improved methods and better gold standards for regulatory network inference from scRNAseq datasets.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Quantitative metrics for evaluating thermonuclear design codes and physics models applied to the National Ignition Campaign

In this study, physics models and design codes for hot dense plasmas undergoing thermonuclear burn are evaluated objectively using statistical metrics that compare the difference between calculations and data relative to the experimental uncertainties. The analysis is applied to the National Ignition Campaign (NIC) because it is relevant, comprehensive, and well documented. The statistics confirm that a key process afflicting NIC performance is mix driven by hydrodynamic instabilities as approximated here using the KL model [G. Dimonte and R. Tipton, Phys. Fluids 18, 085101 (2006)]. New physics models are also presented for instability-driven magnetic fields [B. Srinivasan et al., Phys. Rev. Lett. 108, 165002 (2012)] and the Coulomb logarithm for electron–ion thermal relaxation [G. Dimonte and J. Daligault, Phys. Rev. Lett. 101, 135001 (2008)]. The plasma-generated magnetic fields improve the agreement between code and data in a statistically significant manner by reducing the electron thermal conduction in the hot-spot via the Hall term. The Coulomb logarithm is presented in a numerically practical form that incorporates recent theoretical advances. However, it does not improve the agreement between the code and data because the thermal relaxation is so fast in non-ignited NIC plasmas that the typical 20% error does not change the tight coupling between electrons and ions. Even with these improved physics models, the one-dimensional simulations presented here are not able to describe the high-convergence NIC implosions in a statistically acceptable manner. Of the heroic multi-dimensional simulations of Clark et al. [Phys. Plasmas 23, 056302 (2016)], only those in three-dimensions satisfy the statistical acceptance criterion.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

Quantitative Power System Resilience Metrics and Evaluation Approach: Preprint

Power system resilience is an emerging topic and plays an essential role in helping power industry understand and respond to the increasing threats of extreme weather events. The first step of power system resilience analysis is to introduce metrics to quantify the resilience reasonably. Existing resilience metrics are typically restrained by the limited data for extreme event modeling and fall short in terms of physical interpretation and comparability. This paper develops novel quantitative metrics to evaluate power system resilience in pre- and post-event contexts. The developed metrics illustrate clear physical meanings and can be effectively used to compare resilience across different systems under different extreme events. Moreover, the developed metrics can be applied to both transmission and distribution systems. Simulation on a distribution system is employed to validate the effectiveness of the proposed resilience metrics and resilience evaluation approach.

power system resilience↗
DOE OSTI2023

Metrics for Evaluating Grid Service Provision from Communities of Grid-interactive and Efficient Buildings and other DER

Tapping into the flexibility of aggregations of Grid-interactive and Efficiency Buildings (GEBs) represent a large opportunity to cost-effectively improve operations in future low-carbon power grids. The US Department of Energy's Connected Communities program seeks to demonstrate collections of these GEBs at 10 different sites across the country. However, consistent metrics for evaluation across all projects are needed to build confidence in the approach. This paper presents the metrics relating to Grid Service Provision that will be computed at the 10 demonstrations. The metrics proposed quantity the magnitude of service offers, the consistency and quality of services provided by the community, and the individual DER contributions to the community-level service. The goal of this work is to present these metrics and describe some of their intended insights. The hope is that these insights will be useful and build confidence for grid operators, regulators, and aggregators and practitioners as they look to deploy these resources in grids of the future.

MacDonald, Jason S↗
DOE OSTI2019

A metric for evaluating conformance robustness during geologic CO 2 sequestration operations

A metric for quantifying the robustness of a designation of conformance of a geologic CO 2 sequestration (GCS) project during its operational phase is developed and demonstrated. Conformance in this context is a measure of the degree to which the sequestration system is understood and can be accurately modeled along with the degree to which the storage system is performing as designed. The robustness of conformance quantifies the degree to which parameter values can deviate from their current nominal estimates and still produce model forecasts that meet the performance criteria for the GCS operation. We develop and demonstrate the approach on a simplified scenario to illustrate the concept using a single uncertain parameter (homogeneous reservoir permeability) and a single performance criterion (critical pressure at a monitoring well in the reservoir; i.e., one that may displace brine from the reservoir to an overlying drinking water aquifer for example). Increased confidence in conformance assessment as more monitoring data are obtained is incorporated through the standard error of the coefficient (reservoir permeability in the case presented here), which we designate as the concordance metric. As more monitoring data become available during the course of the GCS operation, the standard error of the coefficient decreases (in general), thereby leading to increased conformance robustness as a larger deviation from nominal is required to fail to meet performance criteria. Increasing conformance robustness over time builds confidence that a GCS project will continue to meet performance criteria during the life-span of the project, thereby supporting designations of conformance. Furthermore a lack of conformance robustness provides a critical warning that the performance criteria of the GCS operation are not robust against probabilistic and non-probabilistic uncertainty in model conceptualization and/or model parameters.

58 GEOSCIENCES↗
DOE OSTI2022

An Operational Resilience Metric to Evaluate Inertia and Inverter-based Generation on the Grid

In an effort to reduce carbon emissions and curtail the effects of climate change there has been considerable effort to increase the penetration of inverter-based renewable energy sources. The adoption of renewable generation over conventional inertia-based generation sources is forming considerable challenges for the operation and stability of the power system. The power system has been designed around generation units characterized by high inertia and primary frequency response (PFR), allowing them to respond to disturbances such as faults or generators tripping off line. In contrast, the modern inverter-based assets are characterized by no contribution to inertia and they typically provide stochastic generation at their maximum output, thus having no contribution to PFR during low-frequency disturbance events. Because of this, inertia in power systems is reducing, resulting in a faster rate of frequency change after a disturbance occurs. As more inverter-based generation units are added to the grid it is important to understand the stability of the system and the size of disturbance a system is capable of withstanding. This paper presents a resilience metric that evaluates the maximum size of disturbance a systems can withstand based on the system inertia and the primary frequency control of inverter and inertia-based generation. The results are shown visually and are based on the real-time operation of generation units and their characteristics such as latency, ramp rates, and energy constraints. It is demonstrated that the real-time positioning or bias of the generating units has an effect of the size of disturbance that a system can withstand, i.e. its resilience. It is expected that this type of analysis can help operators increase the resilience of power systems in the future.

13 HYDRO ENERGY↗
DOE OSTI2018

Life-cycle analysis of integrated biorefineries with co-production of biofuels and bio-based chemicals: co-product handling methods and implications: Co-product handling methods and implications of life-cycle analysis of biofuel and bio-chemical co-producing biorefineries

New integrated biorefinery (IBR) concepts are being investigated to co-produce hydrocarbon fuels and high-value bio-based chemicals to improve the economic viability of IBRs, to enhance biomass resource utilization efficiencies, and to maximize potential greenhouse gas (GHG) emission reductions. Unlike fuel-only biorefineries, IBRs may co-produce a significant amount of bio-based chemicals, whose emission implications for specific biorefinery products and the biorefinery as a whole need to be evaluated. We discuss this in principle and apply three sets of co-product handling methods to conduct life-cycle analysis (LCA) of modeled IBRs with co-production of two bioproduct examples - succinic acid and adipic acid - alongside a renewable diesel blendstock fuel product. The LCA results for the specific co-product handling methods that were examined shed light on potential artifacts of product-specific LCA with selected co-product methods. We discuss the advantages and limitations of each method and conclude that (i) a system-level or 'black-box' LCA allocation method is too simplistic to reflect appropriately the GHG burdens of distinctly different processing trains for fuels and chemicals in the IBR context, and (ii) the displacement method is the only co-product handling method that accounts fully for the emission effects of both the fuel product and the non-fuel bio-based co-products in the IBRs within the context of the existing fuel-focused GHG regulatory framework. Alternatively, biorefinery system-level LCA combines benefits of individual products to offer a complete picture. This system-level LCA approach offers a holistic LCA without somewhat arbitrary decisions either on an allocation basis or by the selection of an evaluation metric based on specific products.

09 BIOMASS FUELS↗