SEARCH · Search NASA

Results for “large-scale data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Adaptive Spatially Aware I/O for Multiresolution Particle Data Layouts

Large-scale simulations on nonuniform particle distributions that evolve over time are widely used in cosmology, molecular dynamics, and engineering. Such data are often saved in an unstructured format that neither preserves spatial locality nor provides metadata for accelerating spatial or attribute subset queries, leading to poor performance of visualization tasks. Furthermore, the parallel I/O strategy used typically writes a file per process or a single shared file, neither of which is portable or scalable across different HPC systems. We present a portable technique for scalable, spatially aware adaptive aggregation that preserves spatial locality in the output. We evaluate our approach on two supercomputers, Stampede2 and Summit, and demonstrate that it outperforms prior approaches at scale, achieving up to 2.5× faster writes and reads for nonuniform distributions. Furthermore, the layout written by our method is directly suitable for visual analytics, supporting low-latency reads and attribute-based filtering with little overhead.

Usher, Will↗

DOE OSTI2026

Dataset of Generative AI Workload Power Profiles

This dataset provides a collection of high-resolution (5/10 Hz or every 0.2/0.1 seconds) power consumption profiles for generative artificial intelligence (GenAI) workloads executed on NLR's High Performance Computing (HPC) platform Kestrel. The dataset also includes examples of representative whole-facility power profiles generated using a bottom-up, event-driven, data center energy model . This dataset is designed to support research in energy modeling, infrastructure planning, energy system integration, and sustainability analysis for AI-driven computing systems. The dataset captures time-resolved electrical power measurements across a diverse set of configurations, including variations in job type (inference vs. training), workload (LLM vs. image generation), datasets, and number of compute nodes. Power traces are provided in a standardized format and include both raw/instantaneous and aggregated files. Each profile is accompanied by metadata describing workload parameters, enabling reproducibility and cross-study comparison. The dataset is intended for use in applications such as data center infrastructure planning, energy modeling, demand response and grid impact studies, and development and validation of system-level simulation tools. By making these workload-specific power profiles publicly available, this dataset aims to address the current lack of open, empirical energy data for generative AI systems and to facilitate transparent, reproducible research on the energy and environmental impacts of large-scale AI deployment. If you use this dataset, please cite the associated publication: Vercellino et al., “Measurement of Generative AI Workload Power Profiles for Whole-Facility Data Center Infrastructure Planning,” arXiv:2604.07345 (2026).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Distributed quantum approximate optimization algorithm on a quantum-centric supercomputing architecture

Quantum approximate optimization algorithm (QAOA) has shown promise in solving combinatorial optimization problems by providing quantum speedup on near-term gate-based quantum computing systems. However, QAOA faces challenges for high-dimensional problems due to the large number of qubits required and the complexity of deep circuits, limiting its scalability for real-world applications. In this study, we present a distributed QAOA (DQAOA), which leverages distributed computing strategies to decompose a large computational workload into smaller tasks that require fewer qubits and shallower circuits than are necessary to solve the original problem. These sub-problems are processed using a combination of high-performance and quantum computing resources. The global solution is iteratively updated by aggregating sub-solutions, allowing convergence toward the optimal solution. We demonstrate that DQAOA can handle considerably large-scale optimization problems (e.g., 1000-bit problem), achieving a high solution quality and short time-to-solution, outperforming existing strategies. Furthermore, we realize DQAOA on a quantum-centric supercomputing architecture, paving the way for practical applications of gate-based quantum computers in real-world optimization tasks. To extend DQAOA’s applicability to materials science, we further develop an active learning algorithm integrated with our DQAOA (AL-DQAOA), which involves machine learning, DQAOA, and active data production in an iterative loop. We successfully optimize photonic structures using AL-DQAOA, indicating that solving real-world optimization problems using gate-based quantum computing is feasible. We expect the proposed DQAOA to be applicable to a wide range of optimization problems and AL-DQAOA to find broader applications in material design.

Kim, Seongmin [ORNL] (ORCID:0000000159063004)↗

DOE OSTI2023

Institutional Framework of Variable Renewable Energy Forecasting in India

The share of variable renewable energy (VRE) in India is growing rapidly, with a national goal of reaching 50% capacity from non-fossil fuel generation by 2030. One implication of this growth is the need for improved VRE forecasting methods. For this reason, the Ministry of New and Renewable Energy (MNRE) in India commissioned this study with support from the United States Agency for International Development (USAID), the National Renewable Energy Laboratory (NREL) in the United States, and the National Institute of Wind Energy (NIWE) in India. The objective of this study was to review the existing institutional framework and suggest changes needed to support the plans for large-scale VRE integration in the country. To achieve that objective, the authors consulted local stakeholders about the status of VRE forecasting in India, reviewed existing studies, and examined VRE forecasting methods around the world to identify best practices. Based on those best practices, the study presents six potential approaches to improve the VRE forecasting framework in India. Approaches include incentivizing VRE forecast improvement and use of the most accurate VRE forecast, creating an institution that will optimize VRE forecasts while maintaining and ensuring access to necessary data for forecasting, implementing a review and certification process for VRE forecast providers, forecasting closer to dispatch time and allowing for more frequent forecast revisions, increasing the frequency of weather forecasts, and aggregating VRE forecasts at the point of interconnection.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2023

FPGA Acceleration of GCN in Light of the Symmetry of Graph Adjacency Matrix

Graph Convolutional Neural Networks (GCNs) are widely used to process large-scale graph data. Different from deep neural networks (DNNs), GCNs are sparse, irregular, and unstructured, posing unique challenges to hardware acceleration with regular processing elements (PEs). In particular, the adjacency matrix of a GCN is extremely sparse, leading to frequent but irregular memory access, low spatial/temporal data locality and poor data reuse. Furthermore, a realistic graph usually consists of unstructured data (e.g., unbalanced distributions), creating significantly different processing times and imbalanced workload for each node in GCN acceleration. To overcome these challenges, we propose an end-to-end hardware-software co-design to accelerate GCNs on resource-constrained FPGAs with the features including: (1) A custom dataflow that leverages symmetry along the diagonal of the adjacency matrix to accelerate feature aggregation for undirected graphs. We utilize either the upper or the lower triangular matrix of the adjacency matrix to perform aggregation in GCN to improve data reuse. (2) Unified compute cores for both aggregation and transform phases, with full support to the symmetry-based dataflow. These cores can be dynamically reconfigured to the systolic mode for transformation or as individual accumulators for aggregation in GCN processing. (3) Preprocessing of the graph in software to rearrange the edges and features to match the custom dataflow. This step improves the regularity in memory access and data reuse in the aggregation phase. Moreover, we quantize the GCN precision from FP32 to INT8 to reduce the memory footprint without losing the inference accuracy. We implement our accelerator design in Intel Stratix10 MX FPGA board with HBM2, and demonstrate 1.3x-110.5x improvement in end-to-end GCN latency as compared to the state-of the-art FPGA implementations, on the graph datasets of Cora, Pubmed, Citeseer and Reddit.

Nair, Gopikrishnan R.↗

DOE OSTI2023

Alkali-silica reaction expansion model for confined concrete with stress-dependency and casting direction anisotropy

Alkali-silica reaction (ASR) is a deleterious chemical reaction between alkali hydroxyl ions and types of silica found in some aggregates of concrete. Owners and regulators of nuclear power plants aim to ensure the safety of the concrete structures with optimal maintenance strategies. A new model was developed for predicting the expansion of concrete structures affected by alkali-silica reaction. The model includes a novel combination of existing models as an alkali-silica reaction advancement model, a casting direction anisotropic expansion model, a stress-dependent anisotropic expansion model, and a material property evolution model dependent on the degree of ASR expansion. The model parameters were calibrated based on existing literature data and data generated by previous efforts of this study. The calibrated model was then validated with the experiments carried out in previous efforts of this study. The model was shown to accurately predict the ASR-expansion of large-scale reinforced concrete specimens with confinement.

36 MATERIALS SCIENCE↗

DOE OSTI2024

PyPop: a mature open-source software pipeline for population genomics

Python for Population Genomics (PyPop) is a software package that processes genotype and allele data and performs large-scale population genetic analyses on highly polymorphic multi-locus genotype data. In particular, PyPop tests data conformity to Hardy-Weinberg equilibrium expectations, performs Ewens-Watterson tests for selection, estimates haplotype frequencies, measures linkage disequilibrium, and tests significance. Standardized means of performing these tests is key for contemporary studies of evolutionary biology and population genetics, and these tests are central to genetic studies of disease association as well. Here, we present PyPop 1.0.0, a new major release of the package, which implements new features using the more robust infrastructure of GitHub, and is distributed via the industry-standard Python Package Index. New features include implementation of the asymmetric linkage disequilibrium measures and, of particular interest to the immunogenetics research communities, support for modern nomenclature, including colon-delimited allele names, and improvements to meta-analysis features for aggregating outputs for multiple populations.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Conservation management decreases surface runoff and soil erosion

Conservation management practices – including agroforestry, cover cropping, no-till, reduced tillage, and residue return – have been applied for decades to control surface runoff and soil erosion, yet results have not been integrated and evaluated across cropping systems. In this study we collected data comparing agricultural production with and without conservation management strategies. We used a bootstrap resampling analysis to explore interactions between practice type, soil texture, surface runoff, and soil erosion. We then used a correlation analysis to relate changes in surface runoff and soil erosion to 13 other soil health and agronomic indicators, including soil organic carbon, soil aggregation, infiltration, porosity, subsurface leaching, and cash crop yield. Across all conservation management practices, surface runoff and erosion had respective mean decreases of 67% and 80% compared with controls. Use of cover cropping provided the largest decreases in erosion and surface runoff, thus emphasizing the importance of maintaining continuous vegetative cover on soils. Coarse- and medium-textured soils had greater decreases in both erosion and runoff than fine-textured soils. Changes in surface runoff and soil erosion under conservation management were highly correlated with soil organic carbon, aggregation, porosity, infiltration, leaching, and yield, showing that conservation practices help drive important interactions between these different facets of soil health. This study offers the first large-scale comparison of how different conservation agriculture practices reduce surface runoff and soil erosion, and at the same time provides new insight into how these interactions influence the improvement or loss of soil health.

58 GEOSCIENCES↗

DOE OSTI2025

Evaluation of cloud height, optical thickness, and phase retrievals from the CHROMA algorithm applied to Sentinel-3 OLCI data

We previously developed the Cloud Height Retrieval from O 2 Molecular Absorption (CHROMA) algorithm for the Ocean Color Instrument (OCI) on the new NASA Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) mission. Here, we apply CHROMA to observations from the Ocean Land Colour Instrument (OLCI) to guide expectations for PACE, as it will take some time to obtain large-scale validation data for OCI. We use cloud top height (CTH), phase, and (for liquid clouds) cloud optical thickness (COT) data from the ground-based Atmospheric Radiation Measurement (ARM) network to evaluate the OLCI retrievals. We found that OLCI and Moderate Resolution Imaging Spectroradiometer (MODIS) CTH compare similarly well to the ARM reference. OLCI has a tendency to underestimate CTH as CTH increases, and algorithm assumptions about cloud geometric thickness may contribute to this. ARM COT from multifilter shadowband radiometers (MFRSR) and Sun photometers are well-correlated with one another, albeit with a roughly 30 % offset on average; OLCI and MODIS COT agree more closely with the MFRSR data. OLCI retrieval uncertainty estimates show skill at telling low-uncertainty cases from high-uncertainty ones, although CTH uncertainties are underestimated. Additionally, we compare the OLCI data to satellite retrievals based on thermal infrared measurements from MODIS and Sea and Land Surface Temperature Radiometer (SLSTR) data. Differences are broadly consistent with physical expectations based on the A-band vs. thermal techniques, although one key challenge in such aggregated comparisons is different cloud masking sensitivities and algorithm failure rates meaning additional sampling differences are introduced. We conclude by discussing the transition to and possible enhancements for PACE OCI.

Sayer, Andrew M. [Univ. of Maryland Baltimore Coun↗

DOE OSTI2025

HydraGNN v4.0

The new version of HydraGNN v4.0 provides additional core capabilities, such as: Inclusion of multi-body atomistic cluster expansion MACE, polarizable atom interaction neural network PAINN, and equivariant principal neighborhood aggregation (PNAEq) among the message passing layers supported -Inclusion of graph transformers to directly model long-range interactions between nodes that are distant in the graph topology Integration of graph transformers with message passing layers by combining the graph embedding generated by the two mechanisms, which allows for an improved expressivity of the HydraGNN architecture Improved re-implementation of multi-task learning (MTL) to allow its use for stabilized training across imbalanced, multi-source, multi-fidelity data Introduction of multi-task parallelism, a newly proposed type of model parallelism specifically for MTL architectures, which allows to dispatch different output decoding heads to different GPU devices Integration of multi-task parallelism with pre-existing distributed data parallelism to enable a 2D parallelization for distributed training Improved portability of the distributed training across Intel GPUs, which has been testes on ALCF exascale supercomputer Aurora Inclusion of 2-level fine-grained energy profilers portable across NVIDIA, AMD, and Intel GPUs to monitor the power and energy consumption associated with different functions executed by the HydraGNN code during data pre-load and training Restructuring of previous examples and inclusion of new sets of examples to illustrate the download, preprocess, and training of HydraGNN models on new large-scale open-source datasets for atomistic materials modeling (e.g., Alexandria, Transition1x, OMat24, OMol25)

Lupo Pasini, Massimiliano [Oak Ridge National Labo↗

DOE OSTI2022

AbacusSummit Cleaned Halo Catalogs

This is a data release from the AbacusSummit suite of cosmological N-body simulations. It contains dark matter halo catalogs and particle samples from those halos. The halo catalogs have been augmented (cleaned) using halo merger tree information. Descriptions of AbacusSummit, the cleaned halo catalogs, and instructions for manipulating these catalogs can be found at the following URL: abacussummit.readthedocs.io/ The halo catalog cleaning is structured as a set of auxiliary files that are ingested by the reader code at runtime. But this release contains both the original catalogs and the auxiliary files, thus forming a self-contained dataset. The files in this release are aggregated by redshift. We define sets of simulations (like all c000 sims or all hugebase sims), and then store one redshift from all simulations per tarball. This is designed to support the use-case where a user wants to analyze multiple simulations at a certain redshift.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

What you get is not always what you see—pitfalls in solar array assessment using overhead imagery

Effective integration planning for small, distributed solar photovoltaic (PV) arrays into electric power grids requires access to high quality data: the location and power capacity of individual solar PV arrays. Unfortunately, national databases of small-scale solar PV do not exist; those that do are limited in their spatial resolution, typically aggregated up to state or national levels. While several promising approaches for solar PV detection have been published, strategies for evaluating the performance of these models are often highly heterogeneous from study to study. The resulting comparison of these methods for practical applications for energy assessments becomes challenging and may imply that the reported performance evaluations overly optimistic. The heterogeneity comes in many forms, each of which we explore in this work: the degree of diversity of the locations and sensors (e.g. different satellites, aerial photography) from which the training and validation data originate, the validation of ground truth (manual annotation of imagery vs known solar PV locations), the level of spatial aggregation (e.g. array-level vs regional estimates), and inconsistencies in the training and validation datasets (e.g. different datasets are used for each study and those data are not always made accessible). For each, we discuss emerging practices from the literature to address them or suggest directions of future research. As part of our investigation, we evaluate solar PV identification performance in two large regions: the entire state of Connecticut and the city of San Diego, CA. In Connecticut, we also use 33,114 known parcel-level solar PV installations from Berkeley Lab’s Tracking the Sun dataset to evaluate parcel-level performance and evaluate capacity estimates using 169 municipalities. We also make our code (which we call SolarMapper), pre-trained models, training data, and predictions publicly available and provide a web portal for interactively inspecting each prediction that was made. Here our findings suggest that traditional performance evaluation of the automated identification of solar PV from satellite imagery may be optimistic due to common limitations in the validation process. The takeaways from this work are intended to inform and catalyze the large-scale practical application of automated solar PV assessment techniques by energy researchers and professionals.

14 SOLAR ENERGY↗

DOE OSTI2021

Influence of alkali-silica reaction on the shear capacity of reinforced concrete beams with minimum transverse reinforcement

A growing number of critical concrete infrastructure are affected by alkali-silica reaction (ASR) damage such as the Seabrook Nuclear Power Plant in New Hampshire, Parker arch-gravity dam in Arizona, and several highway bridges in California and Texas. ASR causes expansion and cracking and degrades the concrete mechanical properties. Despite a wealth of material level studies there is still limited large-scale experimental data regarding the effects of ASR on reinforced concrete (RC) members. Due to the brittle nature of the shear failure in RC structures, this study focuses on the shear response of full-scale ASR damaged RC beams with minimum shear reinforcement. Six RC beams were built with different levels of ASR susceptibility and conditioned in different environments during which continuous expansion monitoring was performed. The beams all contained reactive fine aggregate (sand) and two of them had additional alkali, 1.25% by weight of cement, to accelerate ASR. The highest expansion rate of the beams happened during the first 150 days and the expansions stayed constant after 240 days and 330 days for beams conditioned in outdoor and laboratory conditions, respectively, until the last measurement at 575 days. The highest expansion, 0.4%, was seen in the beams with additional alkali and conditioned outside with regular water spray. Out of the six beams, three were selected at different levels of ASR damage and two shear tests were performed on the minimally reinforced spans close to the ends of each beam. Results indicated that beams gain shear strength from ongoing cement hydration in the presence of moisture for ASR expansions less than 0.2%. Compared to one of the beams with 0.2% ASR expansion, one of the other samples with 0.4% expansion lost 6% of its shear strength, 25% of its shear stiffness, and showed about two times larger shear cracks and shear deformations at peak load. Finally, the shear reinforcement yielded at 20% less load in the beam with 0.4% expansion compared to the beam with 0.2% expansion.

42 ENGINEERING↗

DOE OSTI2021

A case study on parallel HDF5 dataset concatenation for high energy physics data analysis

In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. Here, the lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

Exploring Climate-Disease Connections in Geopolitical Versus Ecological Regions: The Case of West Nile Virus in the United States

Many infectious disease forecasting models in the United States (US) are built with data partitioned into geopolitical regions centered on human activity as opposed to regions defined by natural ecosystems; although useful for data collection and intervention, this has the potential to mask biological relationships between the environment and disease. We explored this concept by analyzing the correlations between climate and West Nile virus (WNV) case data aggregated to geopolitical and ecological regions. We compared correlations between minimum, maximum, and mean annual temperature; precipitation; and annual WNV neuroinvasive disease (WNND) case data from 2005 to 2019 when partitioned into (a) climate regions defined by the National Oceanic and Atmospheric Administration (NOAA) and (b) Level I ecoregions defined by the Environmental Protection Agency (EPA). We found that correlations between climate and WNND in NOAA climate regions and EPA ecoregions were often contradictory in both direction and magnitude, with EPA ecoregions more often supporting previously established biological hypotheses and environmental dynamics underlying vector-borne disease transmission. Using ecological regions to examine the relationships between climate and disease cases can enhance the predictive power of forecasts at various scales, motivating a conceptual shift in large-scale analyses from geopolitical frameworks to more ecologically meaningful regions.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2024

The Collaborative Seismic Earth Model: Generation 2

Geological interpretations, earthquake source inversions and ground motion modeling, among other applications, require models that jointly resolve crustal and mantle structure. With the second generation of the Collaborative Seismic Earth Model (CSEM2), we present a global multi-resolution tomographic Earth model that serves this purpose. The model evolves through successive regional- and global-scale refinements. While the first generation aggregated regional models, with this study, we ensure consistency between all individual submodels, resulting in a model that accurately explains wave propagation across scales. Recent regional tomographic models were incorporated, comprising continental-scale inversions for Asia and Africa, as well as regional inversions for the Western US, Central Andes, Iran, and Southeast Asia. Across all regional refinements, over 793,000 source-receiver pairs contributed. Moreover, the long-wavelength Earth model (LOWE) introduces large-scale structures outside of pre-existing local refinements. A full-waveform inversion for global anisotropic P-and S-wave speed structure over a total of 194 iterations with a minimum period of 50 s on a large data set of 1 hr of waveform data from 2,423 earthquakes and over 6 million source-receiver pairs ensures that regional updates in the crust and uppermost mantle translate into updates of deeper, global-scale structure. To test the performance of CSEM2, we evaluate waveform fits between observed and synthetic seismograms at 50 s for an independent data set on the global scale, and on the regional scale for lower periods. We accurately simulate waveforms within and across regional refinements, maintaining the original resolution of the submodels embedded in the global framework.

58 GEOSCIENCES↗

DOE OSTI2023

Building the I (Interoperability) of FAIR for performance reproducibility of large-scale composable workflows in RECUP

Abstract-Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

BrickQA: Bridging the Semantic Gap in Building Operations with Dynamic Graph Exploration

While standardized ontologies like the Brick schema address data heterogeneity in Building Automation Systems (BAS), accessing this semantic data remains a challenge as domain experts often lack the expertise to formulate complex SPARQL queries. To bridge this gap, we present BrickQA, a Large Language Model (LLM)-based framework that translates natural language into executable SPARQL queries through structured query decomposition, dynamic schema exploration, and inline validation. BrickQA utilizes an iterative reasoning agent to actively navigate graph topology through dynamic exploration actions without requiring exhaustive context injection or model fine-tuning. This approach effectively mitigates hallucinations, particularly in large-scale building knowledge graphs. Empirical evaluation on BuildingQA, a standardized benchmark, demonstrates that BrickQA significantly outperforms ReAct baselines, delivering a 0.291–0.355 absolute F1 improvement while achieving 3 × –12.7 × higher token cost-efficiency. Beyond these metrics, the framework maintains structural fidelity across heterogeneous buildings and remains resilient to ambiguous queries without requiring site-specific fine-tuning. Furthermore, a case study on operational analytics validates the framework’s capability to handle temporal and aggregation constraints, effectively transforming abstract semantic models into actionable facility management insights.1

Ko, Yun-Dam↗