Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Machine Learning meets Algebraic Combinatorics: A Suite of Benchmark Datasets to Accelerate AI for Mathematics Research

The use of benchmark datasets has become an important engine of progress in machine learning (ML) over the past 15 years. Recently there has been growing interest in utilizing machine learning to drive advances in research-level mathematics. However, off-the-shelf solutions often fail to deliver the types of insights required by mathematicians. This suggests the need for new ML methods specifically designed with mathematics in mind. The question then is: what benchmarks should the community use to evaluate these? On the one hand, toy problems such as learning the multiplicative structure of small finite groups have become popular in the mechanistic interpretability community whose perspective on explainability aligns well with the needs of mathematicians. While toy datasets are a useful benchmark for initial work, they lack the scale, complexity, and sophistication of many of the principal objects of study in modern mathematics. To address this, we introduce a new collection of benchmark datasets, Algebraic Combinatorics Benchmarks (ACBench), representing either classic or open problems in algebraic combinatorics, a subfield of mathematics that studies discrete structures arising from abstract algebra. After describing the datasets, we discuss the challenges involved in constructing “good” mathematics benchmarks, describe baseline model performance, and discuss some of the insights these datasets can provide that may be of interest even to those who are not interested in mathematics research itself.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Curation of Ground-Truth Validated Benchmarking Datasets for Fault Detection & Diagnostics Tools

Fault detection and diagnostics (FDD) analytical tools for heating, ventilation and air conditioning (HVAC) systems represent one of the most active areas of smart building technology development. A diversity of techniques is used for FDD analytics, spanning physical models, black box, and rule-based approaches, and researchers continuously strive to develop improved algorithms. With FDD algorithm numbers now in the hundreds, there is a need for performance evaluation of these algorithms in order to assess improvements, improve costeffectiveness, and to prioritize investment in the further development of these technologies. A persistent challenge of FDD advance has been the lack of common datasets to benchmark the performance accuracy of FDD algorithms. This paper summarizes the successful curation of HVAC operational data, paired with validated ground-truth information regarding the presence and absence of faults. The current dataset, consisting of both simulation and experimental data, will evolve to include a larger set of HVAC systems with the objective of creating the largest publicly available dataset to be used by FDD developers, users, and researchers to compare and contrast performance accuracy across FDD algorithms, helping to drive improvements that will spur greater market adoption of FDD tools. Furthermore, in order to avoid previously observed issues with contributed datasets and ensure high quality and consistency of future submissions, the development of data validation and ground-truth assessment protocol is detailed in this study.

Casillas, Armando↗
DOE OSTI2021

Highly Accurate Experimental Heave Decay Tests with a Floating Sphere: A Public Benchmark Dataset for Model Validation of Fluid–Structure Interaction

Highly accurate and precise heave decay tests on a sphere with a diameter of 300 mm were completed in a meticulously designed test setup in the wave basin in the Ocean and Coastal Engineering Laboratory at Aalborg University, Denmark. The tests were dedicated to providing a rigorous benchmark dataset for numerical model validation. The sphere was ballasted to half submergence, thereby floating with the waterline at the equator when at rest in calm water. Heave decay tests were conducted, wherein the sphere was held stationary and dropped from three drop heights: a small drop height, which can be considered a linear case, a moderately nonlinear case, and a highly nonlinear case with a drop height from a position where the whole sphere was initially above the water. The precision of the heave decay time series was calculated from random and systematic standard uncertainties. At a 95% confidence level, uncertainties were found to be very low—on average only about 0.3% of the respective drop heights. Physical parameters of the test setup and associated uncertainties were quantified. A test case was formulated that closely represents the physical tests, enabling the reader to do his/her own numerical tests. The paper includes a comparison of the physical test results to the results from several independent numerical models based on linear potential flow, fully nonlinear potential flow, and the Reynolds-averaged Navier–Stokes (RANS) equations. A high correlation between physical and numerical test results is shown. The physical test results are very suitable for numerical model validation and are public as a benchmark dataset.

42 ENGINEERING↗
DOE OSTI2022

Case Study: NREL Campus Chilled Water Storage Potential: Benchmark Datasets Development and Applications, Task 4 - Use Case Demonstration

The Benchmark Datasets Development and Applications project is a three-year collaboration between the National Renewable Energy Laboratory (NREL), Oak Ridge National Laboratory, Pacific Northwest National Laboratory, and Lawrence Berkeley National Laboratory. The project seeks to collect and curate high-resolution, well-calibrated time series of building operational and indoor/outdoor environmental data, which are crucial to understanding and optimizing building energy efficiency performance and demand flexibility capabilities as well as benchmarking energy algorithms. Project outcomes include approximately twelve high-fidelity building datasets, enhanced data representation tools, and four case studies to illustrate example applications. The goal of these case studies is to define and execute analyses that demonstrate how one or more datasets collected through this project can address a data gap or challenge historically faced by building stakeholders. This technical paper summarizes the findings of one of these case studies, in which we studied the operational efficiencies of the central cooling system at NREL. We looked at three years of data from the three chillers in the Field Test Laboratory Building (FTLB), from 2019 to 2021, to compare equipment operation and demand throughout the time period. Our analysis indicates that all three chillers are operating at or below the optimal loading conditions for most of the operation time, and thus there was no efficiency drop due to loading of the chillers at full capacity. Our recommendation is that no chiller capacity increase is needed; instead, the central plant could benefit from adopting advanced control logics for optimal sequencing of chillers during part load operations. Analysis of adding chilled water thermal storage to the central plant indicated 34% savings in demand cost and 24.5% savings in total cost (energy consumption and demand charge cost). The payback period is estimated to be 11-22 years with an assumed TES cost of $\$$100-$200 per ton. This case study shows how a selected dataset is used to solve a practical building problem - learning the operational status of its components, analyzing the effectiveness of a proposed new technique, and aiding decision-making for the building operations and maintenance team.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Panta Rhei benchmark dataset: socio-hydrological data of paired events of floods and droughts

As the adverse impacts of hydrological extremes increase in many regions of the world, a better understanding of the drivers of changes in risk and impacts is essential for effective flood and drought risk management and climate adaptation. However, there is currently a lack of comprehensive, empirical data about the processes, interactions, and feedbacks in complex human–water systems leading to flood and drought impacts. Here we present a benchmark dataset containing socio-hydrological data of paired events, i.e. two floods or two droughts that occurred in the same area. The 45 paired events occurred in 42 different study areas and cover a wide range of socio-economic and hydro-climatic conditions. The dataset is unique in covering both floods and droughts, in the number of cases assessed and in the quantity of socio-hydrological data. The benchmark dataset comprises (1) detailed review-style reports about the events and key processes between the two events of a pair; (2) the key data table containing variables that assess the indicators which characterize management shortcomings, hazard, exposure, vulnerability, and impacts of all events; and (3) a table of the indicators of change that indicate the differences between the first and second event of a pair. The advantages of the dataset are that it enables comparative analyses across all the paired events based on the indicators of change and allows for detailed context- and location-specific assessments based on the extensive data and reports of the individual study areas. The dataset can be used by the scientific community for exploratory data analyses, e.g. focused on causal links between risk management; changes in hazard, exposure and vulnerability; and flood or drought impacts. The data can also be used for the development, calibration, and validation of socio-hydrological models. The dataset is available to the public through the GFZ Data Services (Kreibich et al., 2023, https://doi.org/10.5880/GFZ.4.4.2023.001).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Li1−xNiO2 Many-body DMC Benchmark Dataset

The dataset contains all numerical data generated in support of the manuscript “Many‑body Benchmark of Electronic Charge and Spin Densities for Li1–xNiO2​” (Journal of Chemical Theory and Computation, DOI: 10.1021/acs.jctc.5c02097, URL: https://pubs.acs.org/doi/10.1021/acs.jctc.5c02097). The materials included in this repository are: 1. Data files used to produce all figures and tables in the main manuscript and supporting information. 2. Benchmark density‑functional theory (DFT) datasets used for the charge‑ and spin‑density analyses. 3. Reference many‑body diffusion Monte Carlo (DMC) calculations and associated input/output files.

36 MATERIALS SCIENCE↗
NASA NTRS2012

Integrating CFD, CAA, and Experiments Towards Benchmark Datasets for Airframe Noise Problems

Airframe noise corresponds to the acoustic radiation due to turbulent flow in the vicinity of airframe components such as high-lift devices and landing gears. The combination of geometric complexity, high Reynolds number turbulence, multiple regions of separation, and a strong coupling with adjacent physical components makes the problem of airframe noise highly challenging. Since 2010, the American Institute of Aeronautics and Astronautics has organized an ongoing series of workshops devoted to Benchmark Problems for Airframe Noise Computations (BANC). The BANC workshops are aimed at enabling a systematic progress in the understanding and high-fidelity predictions of airframe noise via collaborative investigations that integrate state of the art computational fluid dynamics, computational aeroacoustics, and in depth, holistic, and multifacility measurements targeting a selected set of canonical yet realistic configurations. This paper provides a brief summary of the BANC effort, including its technical objectives, strategy, and selective outcomes thus far.

Choudhari, Meelan M.↗
DOE OSTI2022

Benchmark datasets for SARS-CoV-2 surveillance bioinformatics

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the cause of coronavirus disease 2019 (COVID-19), has spread globally and is being surveilled with an international genome sequencing effort. Surveillance consists of sample acquisition, library preparation, and whole genome sequencing. This has necessitated a classification scheme detailing Variants of Concern (VOC) and Variants of Interest (VOI), and the rapid expansion of bioinformatics tools for sequence analysis. These bioinformatic tools are means for major actionable results: maintaining quality assurance and checks, defining population structure, performing genomic epidemiology, and inferring lineage to allow reliable and actionable identification and classification. Additionally, the pandemic has required public health laboratories to reach high throughput proficiency in sequencing library preparation and downstream data analysis rapidly. However, both processes can be limited by a lack of a standardized sequence dataset. We identified six SARS-CoV-2 sequence datasets from recent publications, public databases and internal resources. In addition, we created a method to mine public databases to identify representative genomes for these datasets. Using this novel method, we identified several genomes as either VOI/VOC representatives or non-VOI/VOC representatives. To describe each dataset, we utilized a previously published datasets format, which describes accession information and whole dataset information. Additionally, a script from the same publication has been enhanced to download and verify all data from this study.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

ProvSec: Open Cybersecurity System Provenance Analysis Benchmark Dataset with Labels

System provenance forensic analysis has been studied by a large body of research work. This area needs fine granularity data such as system calls along with event fields to track the dependencies of events. While prior work on security datasets has been proposed, we found a useful dataset of realistic attacks and details that are needed for high-quality provenance tracking is lacking. We created a new dataset of eleven vulnerable cases for system forensic analysis. It includes the full details of system calls including syscall parameters. Realistic attack scenarios with real software vulnerabilities and exploits are used. For each case, we created two sets of benign and adversary scenarios which are manually labeled for supervised machine-learning analysis. In addition, we present an algorithm to improve the data quality in the system provenance forensic analysis. We demonstrate the details of the dataset events and dependency analysis of our dataset cases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

A benchmark dataset for Hydrogen Combustion

The generation of reference data for deep learning models is challenging for reactive systems, and more so for combustion reactions due to the extreme conditions that create radical species and alternative spin states during the combustion process. Here, we extend intrinsic reaction coordinate (IRC) calculations with ab initio MD simulations and normal mode displacement calculations to more extensively cover the potential energy surface for 19 reaction channels for hydrogen combustion. A total of ~290,000 potential energies and ~1,270,000 nuclear force vectors are evaluated with a high quality range-separated hybrid density functional, ωB97X-V, to construct the reference data set, including transition state ensembles, for the deep learning models to study hydrogen combustion reaction.

08 HYDROGEN↗
DOE OSTI2021

A High-Fidelity Cyber-Physical Testbed-Based Benchmarking Dataset For Testing Operational Technology Specific Intrusion Detection Systems

Quality datasets serve a critical purpose in cyber security research. Data is needed to understand system behavior and develop security controls to protect critical systems. However, for critical infrastructure operational environments there is a lack of available datasets to study because of the high cost and specialized capabilities necessary to generate them. This paper documents the development of a dataset of high fidelity hardware in the loop laboratory simulated models of electric and natural gas distribution systems with real cyber attack test cases. A deep dive discussion for the experimental setup and controls for generating the data is provided along with observations from using the data in evaluating intrusion detection approaches.

Ashok, Aditya↗
DOE OSTI2024

Advancing our understanding of instabilities in high-energy-density systems through the study of benchmark datasets with advanced modeling tools

Numerous types of pulsed power driven inertial confinement fusion (ICF) and high energy density (HED) systems rely on implosion stability to achieve desired temperatures, pressures, and densities. Sandia National Laboratories Pulsed Power Sciences Center’s main ICF platform, Magnetized Liner Inertial Fusion (MagLIF), suffers from implosion instabilities which limit attainable fuel conditions and can compromise fuel confinement. This Truman Fellowship research primarily focused on computationally exploring (a) methods for improving our understanding of hydrodynamic and magnetohydrodynamic instabilities that form during cylindrical liner implosions, (b) methods for mitigating implosion instabilities, particularly those that degrade performance of MagLIF targets, and (c) novel MagLIF target designs intended to improve target performance primarily via enhanced implosion stability. Several multi-dimensional computational tools were used, including the magnetohydrodynamics code ALEGRA, the radiation-magnetohydrodynamics code HYDRA, and the magnetohydrodynamics code KRAKEN. This research succeeded in executing and analyzing simulations of automagnetizing liner implosions, shockless MagLIF implosions, dynamic screw pinch driven cylindrical liner implosions, and cylindrically convergent HED instability studies. The methods and tools explored and developed in this Truman Fellowship research have been published in several peer-reviewed journal articles and will serve as useful contributions to the fields of pulsed power science and engineering, particularly pertaining to pulsed power ICF and HED science.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗