Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data manipulation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Patch-level CO2 and CH4 fluxes and porewater concentrations in experimental wetlands, 5 and 10 PPT saltwater intrusion simulations, Louisiana 2023-2024

This dataset containes carbon dioxide (CO2) and methane (CH4) flux measurements collected from wetland vegetation patches dominated by Typha domingensis and Panicum hemitomon to assess greenhouse gas flux responses to experimental saltwater intrusion (SWI) pulses. Measurements were conducted before, during, and after simulated SWI events at target salinities of approximately 5 parts per thousand (ppt) with durations of 6, 10, and 17 days and 10 ppt with a duration of 48 days, alongside a control wetland (with no salinity added, flood manipulation only). These data were generated to evaluate how the magnitude and duration of SWI alter wetland carbon exchange and related biogeochemical and plant responses. This data package includes flux measurements from the wetland surface (i.e, soil/water surface and enclosed vegetation) and from the soil/water surface only; porewater and surface water concentrations of CO2 and CH4; salinity, pH, electrical conductivity collected in porewater (at 5, 10, and 20 cm soil depths) and in surface water; soil redox potential; leaf spectral indices, leaf vapor pressure deficit, stomatal conductance; water level, salinity, and photosynthetically active radiation; and aboveground biomass.

EARTH SCIENCE > AGRICULTURE > SOILS > SOIL RESPIRA↗
DOE OSTI2022

EVSE Cybersecurity and Resilience

Consequence-driven Cybersecurity Analysis for Extreme Fast Charging Electric Vehicle Infrastructure Electric vehicle (EV) development and associated charging infrastructure are expected to advance rapidly. Thirty percent of all global vehicle sales may be EVs and hybrid EVs by 2025, and they will rely on increasingly sophisticated strategies for grid integration. Next-generation EV charging infrastructure is expected to include interconnected renewable resources, such as photovoltaic (PV) arrays and battery storage systems, along with grid-edge devices. Although distributed energy resources (DERs) are useful in several ways, such as peak shaving at high demand times and backup supply for added resilience, the integration of vehicle charging and DERs could create more avenues for cyberattack. Physical and/or remote access to EV charging station components, including charge ports, power electronics, controllers, and local generation (e.g., PV and energy storage) could be paths to cause power fluctuations, leading to altered operations at the charging station, escalated privileges to administrative systems, exfiltration of financial information (including personally identifiable information), and reduced grid stability. One compromised EV supply equipment component can open the door to a variety of exploitable vulnerabilities. Cloud computing and mobile application control have the potential to expand the threat surface to non-repudiation and firmware integrity challenges. Vendor clouds have access to hundreds of chargers, and if compromised, can scale the attack surface exponentially. The high power and voltage levels of xFC infrastructure (e.g., 400 kW at 1000- V DC) increase the hazards and ability to impact the grid and vehicles more than lower-power charging systems. Legacy communications systems and protocols could also put EV infrastructure at risk of cyberattacks requiring a robust patch management process. Communications networks link EVs and chargers to several stakeholders - including charging station operators, grid operators, vendors/manufacturers, and aggregators - who have both physical and network access to share information for control, monitoring, and analytics. Information in these networks that is vulnerable to compromise includes the state of charge, charging duration, payment information, electricity price, and load control. Analyzing and prioritizing these interconnections risks could help address cybersecurity related to data leakage and manipulation.

charging↗
DOE OSTI2024

Geospatial Data Platform for All

Spatiotemporal data has evolved in scale due to augmented use in cross-domain applications. Simultaneously, there is substantial growth in the availability of Geographic Information Systems (GIS) data provided by the United States Geological Survey (USGS) along with other federal, state, county, or local agencies through open-data portals and public access APIs. However, data availability does not equate with accessibility. Large-scale analyses and applications require robust, performant data management with co-location of data storage and computing. The insufficiency of data management infrastructure compels researchers to adopt ad hoc project- specific GIS data storage solutions (e.g., copying data to High-Performance computer file systems). As an ad hoc storage strategy does not scale, it hampers cross-domain analyses causing difficulty in data reuse and utilizing existing code bases. Furthermore, GIS data is complex and requires expertise to analyze and manipulate due to its intricate data structures and data-specific projection transformations. Despite the challenges, we recognize that derived GIS data products, e.g., satellite or LIDAR-based images, can be used in downstream applications such as AI by domain, but non-GIS experts. To address the data needs and overcome the challenges, we are working towards a GIS Data Platform focused on efficient data storage, data discovery and access, and an API to enable common workflows. We propose a knowledge-graph (KG) approach for data discovery, whereby datasets are semantically linked to higher- level constructs such as projects and research areas. The semantic data links enable researchers to explore datasets in a top-down approach by specifying relevant and meaningful terms (assists in finding hidden data). An advantage is that the nodes and edges in a knowledge graph create built-in semantic documentation. Deeper spatiotemporal connections between data sources can be encoded via Graph Neural Networks (GNN) (Zhang et al., 2021). The KG approach can be extended to integrate the data itself in a Virtual KG (VKG). Our work will derive inspiration from large-scale VKG efforts that have been undertaken or are currently underway as part of the OpenStreetMap project (Ding et al., 2021). For DOE Data Days, we share the proposed geospatial data platform hybrid (cloud/on-prem) architecture, our work-to-date on storing, retrieving, and transforming LiDAR and raster data relevant to two important NREL use-cases, including the Renewable Energy Potential (reV) Model, and present our proposal for a KG based data discovery engine.

data platform↗
DOE OSTI2020

SPND Sensitivity Calculations Using MCNP and Experimental Data from ACRR

The use of the Monte Carlo N-Particle Transport Code (MCNP) to calculate detector sensitivity for Self-Powered Neutron Detectors (SPNDs) in the Annular Core Research Reactor (ACRR) could be a vital tool in the effort to optimize the design of next-generation SPNDs. Next-generation SPND designs, which consider specific materials and geometry, may provide experimenters with capabilities for advanced mixed field dosimetry. These detectors will need to be optimized for configuration, materials, and geometries and the ability to model and iterate must be available in order to decide on the ideal. SPND design. SPNDs were modeled in MCNP which closely resembled the dimensions and location of actual detectors used in the ACRR. Tallies were used to calculate detector sensitivity. Using metrics from a previous report, oscilloscope data from pulses were manipulated in a Matrix Laboratory computing environment (MATLAB) script to calculate experimental detector sensitivity. This report outlines the process in which experimental data from ACRR pulses verified results from tallies in an MCNP ACRR model. The sensitivity values from experiments and MCNP calculations agreed within one standard deviation. Parametric studies were also performed with MCNP to investigate the effects of materials and dimensions of different SPNDs.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

HDF5-Version 1.12.0

The HDF5 library and file format implement the HDF5 data model for storing and managing data. This implementation supports an unlimited variety of datatypes and is designed for flexible and efficient I/O and for high volume and complex data. It is highly portable and extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format and other implementations.

The HDF Group, None↗
DOE OSTI2024

pyEGAF: Modernization of the EGAF database

One of the most comprehensive resources for thermal neutron-capture data is the Evaluated Gamma-ray Activation File (EGAF), containing data from prompt gamma activation analysis measurements carried out at the Budapest Research Reactor for 245 isotopes. Although these valuable datasets have been freely available for many years, the outdated and cryptic adopted format makes it difficult to utilize the data and it is not generally suitable for modern computational technologies. Furthermore, to help overcome these challenges, we have converted the datasets into an open standard JSON format. Additionally, we have developed a Python implementation of an open-source software package, pyEGAF, designed to interact with the JSON data structures for general purpose access, manipulation, and rapid assessment of the capture-gamma data in EGAF.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 1 Sensor Data v1-2

This is the version 1-2 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments.L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds and out-of-service flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project**This dataset includes:- An overall dataset README file that describes the current version, gives citation and contact information, etc.- Site- and year-specific folders, each holding up to 12 CSV (comma separated value) data files for each site and plot in that year.- Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site.- Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes.Please see v1-2 TEMPEST L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning.The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods.* Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021* TEMPEST 1: June 22, 2022* TEMPEST 2: June 6-7, 2023* TEMPEST 3: June 11-13, 2024

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 1 Sensor Data v2-1

This is the version 2-1 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments. L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds, out-of-service, and outlier flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project** This dataset includes: - An overall dataset README file that describes the current version, gives citation and contact information, etc. - Site- and year-specific folders, each holding variable-specific CSV (comma separated value) data files for each site and plot in that year. - Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site. - Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes. Please see v2-1 TEMPEST L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning. The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods. * Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021 * TEMPEST 1: June 22, 2022 * TEMPEST 2: June 6-7, 2023 * TEMPEST 3: June 11-13, 2024 This dataset was updated 2026-03-12: (i) data now go through 2025-12-31 (previous end was 2025-06-30) and (ii) dataset and file names updated to “…v2-1” (previously was “v2-0”).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 2 Sensor Data v2-1

This is the version v2-1 Level 2 (L2) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments. Level 2 (L2) data consist of sensor observations from the COMPASS-FME synoptic sites, TEMPEST, and DELUGE. Compared to the L1 data, these are more consistent (always 15-minute timestamps for the entire year); better QA/QC’d (out of bounds, out of service, and extreme outlier values are removed); and more complete, with a gap-filled time series available alongside the main observations, and additional derived (calculated) variables. L2 data are intended to be rapidly and easily usable in analyses and simulations. However, algorithmic outlier identification always carries the risk of removing valid data, and Level 1 data may be more suitable for analyses that focus on variability or extreme events. This dataset includes: - An overall dataset README file that describes the current version, gives citation and contact information, etc. - Site- and year-specific folders, each holding variable-specific Parquet (a high performance, space efficient format; see https://parquet.apache.org) data files for each site and plot in that year. - Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site. - Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are reported every 15 minutes. Please see v2-1 TEMPEST L2 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning. Data files are in Apache Parquet, a high performance, space efficient format for tabular data. These files can be read using R's `arrow` package (https://arrow.apache.org/docs/r/), with similar tools available in other languages. The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods. * Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021 * TEMPEST 1: June 22, 2022 * TEMPEST 2: June 6-7, 2023 * TEMPEST 3: June 11-13, 2024

EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATU↗
DOE OSTI2023

Analysis of X-ray images and spectra (aXis2000): A toolkit for the analysis of X-ray spectromicroscopy data

Spectromicroscopy refers to analytical methods that combine imaging and spectroscopy to provide detailed, spatially resolved analytical information about a sample, such as the type and quantitative spatial distributions of chemical components, geometric or magnetic alignment information, crystal structure, etc. The analysis of X-ray images and spectra (aXis2000) software described in this work provides a set of routines within a single, integrated, graphical-oriented package to read, display, manipulate and analyze spectromicroscopy data, with particular focus on soft X-ray spectromicroscopy methods such as scanning transmission X-ray microscopy (STXM), X-ray photoemission electron microscopy (XPEEM), scanning photoelectron X-ray microscopy (SPEM) and transmission X-ray microscopy (TXM). Here, this free software is described and compared to other software that can provide similar or complementary capabilities. Examples of spectromicroscopic analyses using advanced features of aXis2000 are provided.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Tree Inventory

This is the tree inventory (diameter, species, and live/dead status) data from the Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment is part of the COMPASS-FME (Coastal Observations, Mechanisms, and Predictions Across Systems and Scales: Field Measurements and Experiments; see https://compass.pnnl.gov/FME/COMPASSFME) project. It addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in eastern Maryland, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments.This dataset includes:- An overall dataset README file.- The tree inventory data in both "wide" and "long" forms. These contain the same information but are structured differently, with the former more useful for human viewers and the latter more amenable for programmatic analyses.- A key to the species/genus codes used, which follow the U.S. Department of Agriculture's PLANTS schema (https://plants.usda.gov/).- A copy of the R code used to generate the wide- and long-form data files.All files are comma-separated value (CSV) and no special software is required to read them.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

pyEGAF: An open-source Python library for the Evaluated Gamma-ray Activation File

The Evaluated Gamma-ray Activation File (EGAF) is one of the most comprehensive resources for thermal neutron-capture data. This database contains data from prompt gamma activation analysis measurements carried out in a consistent manner using the same experimental configuration at the Budapest Research Reactor for 245 isotopes. Although these valuable datasets have been freely available for many years, one of the drawbacks is the outdated and cryptic Evaluated Nuclear Structure Data File (ENSDF) format that is currently adopted for dissemination, making it difficult for users unfamiliar with the format to access and utilize the data contained therein. Furthermore, the ENSDF format does not readily lend itself to modern computational technologies and a parser is required to interpret the complicated mixed-record format. To help overcome these challenges, we have developed a translator to convert the ENSDF-formatted datasets into an open standard JavaScript Object Notation (JSON) format enabling accessibility to applications using different programming languages running in different environments. To compliment this effort, we have also developed an open-source software package implemented in Python, pyEGAF, that is designed to interact with the JSON data structures for general purpose access, manipulation, and analysis of the neutron-capture $\gamma$-ray data in EGAF. The new format, together with the pyEGAF library, greatly enhances access to the wider applications community where EGAF data may be useful or is required.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

A Graph-Net with Node Embeddings to Detect False Data Injection Attacks in Photovoltaic Systems

Distributed energy resources (DER) contribute to the operational stability of the larger power grid both at utility-scale as well as commercial and residential scales in aggregated forms. These DER in-turn are susceptible to increasing cyber threats. An adversary can plug into the same local network that a field photovoltaic (PV) system uses to interconnect its data loggers and inverters and manipulate certain measurements collected from the network or trick existing irradiance and inverter readings through false data injection attacks (FDIA). Control routines that rely on these measurements can propagate the false data, impacting critical decisions that result in a suboptimal operation or even cause intentional harm leading to inverter-tripping or unscheduled loads that need to be shed. To detect FDIA in PV systems, the paper introduces an attention-based graph neural network with node embeddings and applied it to a simple prototypical DC-coupled microgrid with PV, energy storage, and load. The algorithm shows a detection accuracy of up to 98.95%. The proposed FDIA detection technique will provide micro-grid operators with an effective method to safeguard their systems, guaranteeing the secure and reliable operation.

Parvez, Imtiaz [Utah Valley University]↗
DOE OSTI2020

Dynamic compressed sensing for real-time tomographic reconstruction

Electron tomography has achieved higher resolution and quality at reduced doses with recent advances in compressed sensing. Compressed sensing (CS) exploits the inherent sparse signal structure to efficiently reconstruct three-dimensional (3D) volumes at the nanoscale from undersampled measurements. However, the process bottlenecks 3D reconstruction with computation times that run from hours to days. Here we demonstrate a framework for dynamic compressed sensing that produces a 3D specimen structure that updates in real-time as new specimen projections are collected. Researchers can begin interpreting 3D specimens as data is collected to facilitate high-throughput and interactive analysis. Using scanning transmission electron microscopy (STEM), we show that dynamic compressed sensing accelerates the convergence speed by ~3-fold while also reducing its error by 27% for a Au/SrTiO 3 nanoparticle specimen. Before a tomography experiment is completed, the 3D tomogram has interpretable structure within ~33% of completion and fine details are visible as early as ~66%. Upon completion of an experiment, a high-fidelity 3D visualization is produced without further delay. Additionally, we report reconstruction parameters that tune data fidelity can be manipulated throughout the computation without re-running the entire process.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

Perturbation-Based Diagnosis of False Data Injection Attack Using Distributed Energy Resources

Modern smart grid relies on various sensor measurements for its operational control. In a successful false data injection attack, the attacker manipulates the measurements from the grid sensors such that undetected errors are introduced into the estimates of the system parameters leading to catastrophic situations. This paper proposes a novel perturbation based false data injection attack detection mechanism that utilizes inverter based distributed energy resources (DERs) to create low magnitude perturbation signal in the distribution system voltage that is inconsequential to the normal grid operation. Two voltage sensitivity analysis based algorithms are designed to identify the optimal set of DERs that can create the voltage perturbation signal of desired magnitude. An analytical method of voltage sensitivity analysis is used to compute the magnitude of voltage perturbation signal at each node in a computationally efficient manner. Then, a detection mechanism is developed that checks for the presence of the perturbation sequence in each sensor measurement. A sensor measurement is deemed authentic if the voltage perturbation signal is present in the data. In case of sensor malfunction or cyber-attack, the perturbation signal will not be present in the measurement data. Performance of the proposed attack detection mechanism is validated via simulation of the IEEE 69 bus test system.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Laboratory time series moisture manipulative experiment from sediment across San Antonio, Texas: time series aerobic respiration and geochemistry

This dataset supports a broader study examining the effects of wetting and drying on hyporheic zone respiration. The dataset provides data generated from a laboratory moisture manipulation experiment. The contents include time series aerobic respiration and moisture; dissolved oxygen; sediment geochemistry data; and field metadata (including qualitative information on instream and river corridor characteristics). Samples were collected as part of the WHONDRS Allison Veach collaboration (AV1). The data package associated with the AV1 study is available at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/2529428. AV1 sampling occurred across 7 perennial and 7 intermittent streams in San Antonio, Texas. Each stream/site was visited both in summer during base flow (July-September 2023) and winter during peak flow (January-February 2024). This study uses subsamples from a subset of AV1 samples. The original field samples were labeled as AV1_###. Subsequent subsamples for this study were labeled as EV_###. The labels from the field samples and the EV subsamples can be mapped directly based on the digits following the prefix and underscore (i.e., EV_001 is a subsample from AV1_001). See the critical details section below for more details on sample naming. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. This dataset is comprised of one main data folder containing (1) file-level metadata; (2) data dictionary; (3) field metadata; (4) readme; (5) field protocol; and a (6) a subfolder with sediment sample data from the incubation experiment. The sample data subfolder contains (1) effect size; (2) iron (II); (3) gravimetric moisture; (4) respiration rates; (5) raw dissolved oxygen values and plots; (6) specific conductance; (7) pH; (8) temperature; (9) a summary containing mean, median, and standard deviation values of each data type for each treatment (wet and dry); and (10) methods codes. All files are .csv or.pdf.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation

The ever-increasing volume of data produced by HPC simulations necessitates scalable methods for data exploration and knowledge extraction. Scientific data analysis often involves complex queries across distributed datasets, requiring manipulation of multiple primary variables and generating derived data that needs to be handled efficiently, creating challenges for applications that need to parse many large datasets. Relying on individual applications to handle all intermediate data generally leads to redundant computations across studies and unnecessary data transfers. In this paper, we investigate the performance of different approaches where applications define derived variables as quantities of interest (QoIs) and offload the computation and transfer of these QoIs to the I/O library. This significantly reduces redundancy and optimizes data movement across the distributed storage and processing infrastructure by allowing control over when and where derived variables are computed. We present a detailed analysis of the performance-storage trade-offs associated with different solutions and showcase results for our study on two large-scale datasets created from climate and combustion simulations.

Gainaru, Ana↗