Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,027 records · Page 57
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Time-Based CAN Intrusion Detection Benchmark

Modern vehicles are complex cyber-physical systems made of hundreds of electronic control units (ECUs) that communicate over controller area networks (CANs). This inherited complexity has expanded the CAN attack surface by the injection of malicious messages that vary their time-based characteristics. To detect these malicious messages, time-based intrusion detection systems (IDS) have been proposed. However, time-based IDS are usually trained and tested on low-fidelity datasets with unrealistic labeled attacks. This makes difficult the task of evaluating, comparing, and validating IDS. Here we detail and benchmark four time-based IDS in a dataset with real and advanced attacks. We found that methods with strong assumptions regarding the distribution of inter-arrival times have lower performance than distribution agnostic based methods. In particular, distribution agnostic based methods outperform distribution based methods at least on $55\%$ in area under the precision-recall (AUC-PR) curve. Our results expand the body of knowledge of CAN time-based IDS by providing details of these methods and reporting their results when tested on datasets with real and advanced attacks. We describe limitations, open challenges, and how lessons learnt from this research can inform the design of deployable time-based IDS in modern vehicles.

Blevins, Deborah↗
DOE OSTI2024

DAISY Benchmark Performance Data

This repository contains the underlying data from benchmark experiments for Drifting Acoustic Instrumentation SYstems (DAISYs) in waves and currents described in "Performance of a Drifting Acoustic Instrumentation SYstem (DAISY) for Characterizing Radiated Noise from Marine Energy Converters" (https://link.springer.com/article/10.1007/s40722-024-00358-6). DAISYs consist of a surface expression connected to a hydrophone recording package by a tether. Both elements are instrumented to provide metadata (e.g., position, orientation, and depth). Information about how to build DAISYs is available at https://www.pmec.us/research-projects/daisy. The repository's primary content is three compressed archives (.zip format), each containing multiple MATLAB binary data files (.mat format). A table relating individual data files to figures in the paper, as well as the structure of each file, is included in the repository as a Word document (Data Description MHK-DR.docx). Most of the files contain time series information for a single DAISY deployment (file naming convention: [site]_DAISY_[Drift #].mat) consisting of processed hydrophone data and associated metadata. For a limited number of DAISY deployments, the hydrophone package was replaced with an acoustic Doppler velocimeter (file naming convention: [site]_DAISY_[Drift #]_ADV.mat). Data were collected over several years at three locations: (1) Sequim Bay at Pacific Northwest National Laboratory's Marine & Coastal Research Laboratory (MCRL) in Sequim, WA, the energetic tidal channel in Admiralty Inlet, WA (Admiralty Inlet), and the U.S. Navy's Wave Energy Test Site (WETS) in Kaneohe, HI. Brief descriptions of data files at each location follow. - MCRL - (1) Drift #4 and #16 contrast the performance of a DAISY and a reference hydrophone (icListen HF Reson), respectively, in the quiescent interior of Sequim Bay (September 2020). (2) Drift #152 and #153 are velocity measurements for a drifting acoustic Doppler velocimeter in in the tidally-energetic entrance channel inside a flow shield and exposed to the flow, respectively (January 2018). (3) Two non-standard files are also included: DAISY_data.mat corresponds to a subset of a DAISY drift over an Adaptable Monitoring Package (AMP) and AMP_data.mat corresponds to approximately co-temporal data for a stationary hydrophone on the AMP (February 2019). - Admiralty Inlet - (1) Drift #1-12 correspond to tests with flow shielded DAISYs, unshielded DAISYs, a reference hydrophone, and drifting acoustic Doppler velocimeter with 5, 10, and 15 m tether lengths between surface expression and hydrophone recording package (July 2022). (2) Drift #13-20 correspond to tests of flow shielded DAISYs with three different tether materials (rubber cord, nylon line, and faired nylon line) in lengths of 5, 10, and 15 m (July 2022). - WETS - (1) Drift #30-32 correspond to tests with a heave plate incorporated into the tether (standard configuration for wave sites), rubber cord only, and rubber cord, but with a flow shielded hydrophone (November 2022). (2) Drift #49-58 and Drift #65-68 correspond to measurements around mooring infrastructure at the 60 m berth where time-delay-of-arrival localization was demonstrated for different DAISY arrangements and hydrophone depths (November 2022).

16 TIDAL AND WAVE POWER↗
DOE OSTI2020

Alaskan carbon-climate feedbacks will be weaker than inferred from short-term manipulations: Alaskan Benchmark Data and Model runs

This submission aimed to assess differences in short-term step warming manipulations and long-term chronic response to climate change in Alaskan ecosystems. Briefly, climate warming is occurring fastest at high latitudes. Based on short-term field experiments, this warming is projected to stimulate soil organic matter decomposition, and promote a positive feedback to climate change. We show here that the tightly coupled, nonlinear nature of high-latitude ecosystems implies that short-term (< 10 year) warming experiments produce emergent ecosystem carbon stock temperature sensitivities inconsistent with emergent multi-decadal responses. We first demonstrate that a well-tested mechanistic ecosystem model accurately represents observed carbon cycle and active layer depth responses to short-term summer warming in four diverse Alaskan sites. We then show that short-term warming manipulations do not capture the non-linear, long-term dynamics of vegetation, and thereby soil organic matter, that occur in response to thermal, hydrological, and nutrient transformations belowground. Our results demonstrate significant spatial heterogeneity in multi-decadal Arctic carbon cycle trajectories and argue for more mechanistic models to improve predictive capabilities.The model used in the current study is available publicly (https://github.com/jinyun1tang/ECOSYS), and the current submission contains the python/ matlab codes for analyzing output from the model (includng a readme file to explain the codes). The benchmark data, also enclosed, was collected from a range of published and publicly available sources (extracted using GRABIT: https://www.mathworks.com/matlabcentral/fileexchange/7173-grabit). These sources describe warming induced changes in tundra/ boreal ecosystems.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Model-ready benchmarks for NPP, ANPP, litter fluxes, and recruitment into the 1 cm dbh size class

The intended use of this dataset is to serve as an observational benchmark to evaluate model predictions of NPP, ANPP, litter fluxes, and recruitment at Barro Colorado Island, Panama. This dataset contains four CSV files and one text file. “Benchmarks-NPP-ANPP-R-L.csv” provides estimates of annual ecosystem-level reproductive litter flux (R), leaf litter flux (L), aboveground net primary productivity (ANPP), and net primary productivity (NPP) for 61 field plots throughout tropical, temperate, and boreal forest biomes. An additional 499 plots (n = 550) include estimates of just R, L, and R/L. Each row reports a distinct set of estimates for one sampling interval at one plot. “Metadata-Benchmarks-NPP-ANPP-R-L.csv” contains field descriptions for all data fields in “Benchmarks-NPP-ANPP-R-L.csv”. “References-Benchmarks-NPP-ANPP-R-L.txt” contains full references to the original studies used to produce the observations at each plot included in the data. “Benchmarks-Recruitment.csv” provides estimates of species-level recruitment rates into the 1 cm dbh size class at four CTFS-ForestGeo sites using methods that account for unobserved mortality of new recruits between census intervals (Kohyama et al., 2018). “Metadata-Benchmarks-Recruitment.csv” contains field descriptions for all data fields in “Benchmarks-Recruitment.csv”.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

2024 Water Splitting Technologies Benchmarking and Protocols Workshop

The sixth annual Advanced Water Splitting Pathways Benchmarking meeting was held on June 11-12, 2024 at the Arizona State University California Center- Los Angeles, CA. A total of 117 people participated (102 in person and 15 via Zoom). Attendance at most breakout sessions ranged from 10 - 25 attendees. The focus of many of the sessions was on developing plans to validate protocols written to date, defining future protocols to be written and aligning with international efforts. The plenary session provided perspectives on international activities in each technology area, as well as an overview of the ARCHES Hydrogen Hub.

08 HYDROGEN↗
DOE OSTI2019

Building Benchmark Data Platform

This project is a three-year, four-laboratory collaboration to collect and curate a handful of high-resolution building systems datasets that have broad applicability to address highest-impact use cases. We will collect and curate high-resolution, well-calibrated time series of building operational and indoor/outdoor environmental data, which are crucial to understanding and optimizing building energy efficiency performance and demand flexibility capabilities as well as benchmarking energy algorithms.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2025

Benchmarking the Use of BPM Quadrupole Moments to Measure Emittance

For the PIP-II program, transverse emittance in the Fermilab Booster must remain well controlled at higher bunch intensities. 4-plate beam position monitors (BPMs) have a small but measurable quadrupole moment, making it possible to infer transverse emittance. By compositing many BPMs together, it becomes possible to improve the quality of the quadrupole signal. The Fermilab Booster BPM system has been used to measure these quadrupole moments in the past year and derive emittances from them. Recent benchmarks show that the derived BPM emittances show similar emittance evolution and value to IPM and Multiwire data. This approach can both supplement and complement existing non-intercepting emittance monitors in accelerators.

Balcewicz, M. A. [Fermilab]↗
DOE OSTI2020

Curation of Ground-Truth Validated Benchmarking Datasets for Fault Detection & Diagnostics Tools

Fault detection and diagnostics (FDD) analytical tools for heating, ventilation and air conditioning (HVAC) systems represent one of the most active areas of smart building technology development. A diversity of techniques is used for FDD analytics, spanning physical models, black box, and rule-based approaches, and researchers continuously strive to develop improved algorithms. With FDD algorithm numbers now in the hundreds, there is a need for performance evaluation of these algorithms in order to assess improvements, improve costeffectiveness, and to prioritize investment in the further development of these technologies. A persistent challenge of FDD advance has been the lack of common datasets to benchmark the performance accuracy of FDD algorithms. This paper summarizes the successful curation of HVAC operational data, paired with validated ground-truth information regarding the presence and absence of faults. The current dataset, consisting of both simulation and experimental data, will evolve to include a larger set of HVAC systems with the objective of creating the largest publicly available dataset to be used by FDD developers, users, and researchers to compare and contrast performance accuracy across FDD algorithms, helping to drive improvements that will spur greater market adoption of FDD tools. Furthermore, in order to avoid previously observed issues with contributed datasets and ensure high quality and consistency of future submissions, the development of data validation and ground-truth assessment protocol is detailed in this study.

Casillas, Armando↗
DOE OSTI2025

DeepBench: A simulation package for physical benchmarking data

We introduce **DeepBench**, a python library that generates simple simulated image data from first principles, such as basic geometric shapes and astronomical objects. These data are highly valuable for developing (calibration, testing, and benchmarking) statistical and machine learning models because they make it possible to connect the final data product to physically interpretable inputs. This software includes tools to curate and store the datasets to maximize reproducibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

High-Resolution Simulations of Geological CO 2 Injection: Application to the SPE11 Benchmark

Geological carbon sequestration (GCS) will play a critical role in decarbonization and in facilitating the transition to clean energy systems. Because CO 2 is highly mobile, ensuring its safe and permanent injection into subsurface geological formations involves monitoring over larger spatial domains and longer time periods than is typical for hydrocarbon reservoirs. This can benefit from simulation tools capable of modeling key CO 2 trapping mechanisms, particularly those optimized for speed and scalability on high-performance computing systems. Using isothermal versions of the SPE11B and SPE11C benchmark cases, we conduct a mesh refinement study simulating CO 2 injection into kilometer-scale rock formations at centimeter resolution with the GEOS open-source simulation framework. We focus on how mesh refinement improves the accuracy of convective mixing in both 2D and 3D simulations. The computational costs associated with achieving a converged solution highlight the need for predictive upscaling techniques. A systematic performance scaling analysis—including both central processing unit (CPU) and graphics processing unit (GPU) architectures—complements the “Results” section.

Geosciences↗
DOE OSTI2015

U.S. Photovoltaic Prices and Cost Breakdowns. Q1 2015 Benchmarks for Residential, Commercial, and Utility-Scale Systems

The price of photovoltaic (PV) systems in the United States (i.e., the cost to the system owner) has continued to decline across all major market sectors. This report provides a Q1 2015 update regarding the prices of residential, commercial, and utility scale PV systems, based on an objective methodology that closely approximates the book value of a PV system. Several cases are benchmarked to represent common variations in business models, labor rates, and system architecture choice. We estimate a weighted-average cash purchase price of $3.09/W for residential scale rooftop systems, $2.15/W for commercial scale rooftop systems, $1.77/W for utility scale systems with fixed mounting structures, and $1.91/W for utility scale systems using single-axis trackers. All systems are modeled assuming standard-efficiency, polycrystalline-silicon PV modules, and further assume installation within the United States.

14 SOLAR ENERGY↗
DOE OSTI2016

East Bay Area Benchmark Study

This report is a benchmarking studying for the Sandia Create Project provided by Cornerstone Project Management, Inc.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2016

The Benchmark Extensible Tractable Testbed Engineering Resource (BETTER)

The Benchmark Extensible Tractable Testbed Engineering Resource (BETTER) is proposed as a family of modular test bodies that are intended to support engineering capability development by helping to identify weaknesses and needs. Weapon systems, subassemblies, and components are often complex and difficult to test and analyze, resulting in low confidence and high uncertainties in experimental and simulated results. The complexities make it difficult to distinguish between inherent uncertainties and errors due to insufficient capabilities. BETTER test bodies will first use simplified geometries and materials such that testing, data collection, modeling and simulation can be accomplished with high confidence and low uncertainty. Modifications and combinations of simple and well-characterized BETTER test bodies can then be used to increase complexity in order to reproduce relevant mechanics and identify weaknesses. BETTER can provide both immediate and long-term improvements in testing and simulation capabilities. This document presents the motivation, concept, benefits and examples for BETTER.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2016

U.S. Solar Photovoltaic System Cost Benchmark: Q1 2016

NREL has been modeling U.S. photovoltaic (PV) system costs since 2009. This report benchmarks costs of U.S. solar PV for residential, commercial, and utility-scale systems built in the first quarter of 2016 (Q1 2016). Our methodology includes bottom-up accounting for all system and project-development costs incurred when installing residential, commercial, and utility-scale systems, and it models the capital costs for such systems.

14 SOLAR ENERGY↗
DOE OSTI2016

Engine Benchmarking - Final CRADA Report

Detailed benchmarking of the powertrains of three light-duty vehicles was performed. Results were presented and provided to CRADA partners. The vehicles included a MY2011 Audi A4, a MY2012 Mini Cooper and a MY2014 Nissan Versa.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2017

Benchmarking transition energies and emission strengths for X-ray astrophysics with measurements at the Livermore EBITs

K-shell transitions in astrophysically abundant metals and L-shell transitions in Fe group elements show characteristic signatures in the soft X-ray spectrum in the energy range 0.1–10 keV. These signatures have great diagnostic value for plasma parameters such as electron and ion temperatures and densities, and can thus help understand the physics controlling the energetic processes in astrophysical sources. This diagnostic power increases with advances in spectral resolution and effective area of the employed X-ray observatories. However, to make optimal use of the diagnostic potential – whether through global spectral modeling or through diagnostics from local modeling of individual lines – the underlying atomic physics has to be complete and well known. With the next generation of soft X-ray observatories featuring micro-calorimeters such as the SXS on Astro- H/Hitomi and the X-IFU on Athena, broadband high-resolution spectroscopy with large effective area will become more commonly available in the next decade. With these spectrometers, the accuracy of the plasma parameters derived from spectral modeling will be limited by the uncertainty of the reference atomic data rather than by instrumental factors, as is sometimes already the case for the high-resolution grating observations with Chandra-HETG and XMM-Newton-RGS. To take full advantage of the measured spectra, assessment of the accuracy of and improvements to the available atomic reference data are therefore important. Dedicated measurements in the laboratory are essential to benchmark the theoretical calculations providing the bulk of the reference data used in astrophysics. Experiments at the Lawrence Livermore National Laboratory electron beam ion traps (EBIT-I and SuperEBIT) have a long history of providing this service. In this work, I present new measurements of transition energies and absolute electron impact excitation cross sections geared towards currently open atomic physics data needs.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2017

Benchmarking of DFLAW Solid Secondary Wastes and Processes with UK/Europe Counterparts

This report provides information and background on UK solid wastes and waste processes that are similar to those which will be generated by the Direct-Feed Low Activity Waste (DFLAW) facilities at Hanford. The aim is to further improve the design case for stabilizing and immobilizing of solid secondary wastes, establish international benchmarking and review possibilities for innovation.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗