Search NASA⌕ Search

SEARCH · Search NASA

Results for “large-scale data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

45 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

A case study on parallel HDF5 dataset concatenation for high energy physics data analysis

In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. Here, the lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Exploring Climate-Disease Connections in Geopolitical Versus Ecological Regions: The Case of West Nile Virus in the United States

Many infectious disease forecasting models in the United States (US) are built with data partitioned into geopolitical regions centered on human activity as opposed to regions defined by natural ecosystems; although useful for data collection and intervention, this has the potential to mask biological relationships between the environment and disease. We explored this concept by analyzing the correlations between climate and West Nile virus (WNV) case data aggregated to geopolitical and ecological regions. We compared correlations between minimum, maximum, and mean annual temperature; precipitation; and annual WNV neuroinvasive disease (WNND) case data from 2005 to 2019 when partitioned into (a) climate regions defined by the National Oceanic and Atmospheric Administration (NOAA) and (b) Level I ecoregions defined by the Environmental Protection Agency (EPA). We found that correlations between climate and WNND in NOAA climate regions and EPA ecoregions were often contradictory in both direction and magnitude, with EPA ecoregions more often supporting previously established biological hypotheses and environmental dynamics underlying vector-borne disease transmission. Using ecological regions to examine the relationships between climate and disease cases can enhance the predictive power of forecasts at various scales, motivating a conceptual shift in large-scale analyses from geopolitical frameworks to more ecologically meaningful regions.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

The Collaborative Seismic Earth Model: Generation 2

Geological interpretations, earthquake source inversions and ground motion modeling, among other applications, require models that jointly resolve crustal and mantle structure. With the second generation of the Collaborative Seismic Earth Model (CSEM2), we present a global multi-resolution tomographic Earth model that serves this purpose. The model evolves through successive regional- and global-scale refinements. While the first generation aggregated regional models, with this study, we ensure consistency between all individual submodels, resulting in a model that accurately explains wave propagation across scales. Recent regional tomographic models were incorporated, comprising continental-scale inversions for Asia and Africa, as well as regional inversions for the Western US, Central Andes, Iran, and Southeast Asia. Across all regional refinements, over 793,000 source-receiver pairs contributed. Moreover, the long-wavelength Earth model (LOWE) introduces large-scale structures outside of pre-existing local refinements. A full-waveform inversion for global anisotropic P-and S-wave speed structure over a total of 194 iterations with a minimum period of 50 s on a large data set of 1 hr of waveform data from 2,423 earthquakes and over 6 million source-receiver pairs ensures that regional updates in the crust and uppermost mantle translate into updates of deeper, global-scale structure. To test the performance of CSEM2, we evaluate waveform fits between observed and synthetic seismograms at 50 s for an independent data set on the global scale, and on the regional scale for lower periods. We accurately simulate waveforms within and across regional refinements, maintaining the original resolution of the submodels embedded in the global framework.

58 GEOSCIENCES↗
DOE OSTI2020

Planet Microbe: a platform for marine microbiology to discover and analyze interconnected ‘omics and environmental data

In recent years, large-scale oceanic sequencing efforts have provided a deeper understanding of marine microbial communities and their dynamics. These research endeavors require the acquisition of complex and varied datasets through large, interdisciplinary and collaborative efforts. However, no unifying framework currently exists for the marine science community to integrate sequencing data with physical, geological, and geochemical datasets. Planet Microbe is a web-based platform that enables data discovery from curated historical and on-going oceanographic sequencing efforts. In Planet Microbe, each ‘omics sample is linked with other biological and physiochemical measurements collected for the same water samples or during the same sample collection event, to provide a broader environmental context. This work highlights the need for curated aggregation efforts that can enable new insights into high-quality metagenomic datasets.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Building the I (Interoperability) of FAIR for performance reproducibility of large-scale composable workflows in RECUP

Abstract-Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism

Here, we present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up to 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

BrickQA: Bridging the Semantic Gap in Building Operations with Dynamic Graph Exploration

While standardized ontologies like the Brick schema address data heterogeneity in Building Automation Systems (BAS), accessing this semantic data remains a challenge as domain experts often lack the expertise to formulate complex SPARQL queries. To bridge this gap, we present BrickQA, a Large Language Model (LLM)-based framework that translates natural language into executable SPARQL queries through structured query decomposition, dynamic schema exploration, and inline validation. BrickQA utilizes an iterative reasoning agent to actively navigate graph topology through dynamic exploration actions without requiring exhaustive context injection or model fine-tuning. This approach effectively mitigates hallucinations, particularly in large-scale building knowledge graphs. Empirical evaluation on BuildingQA, a standardized benchmark, demonstrates that BrickQA significantly outperforms ReAct baselines, delivering a 0.291–0.355 absolute F1 improvement while achieving 3 × –12.7 × higher token cost-efficiency. Beyond these metrics, the framework maintains structural fidelity across heterogeneous buildings and remains resilient to ambiguous queries without requiring site-specific fine-tuning. Furthermore, a case study on operational analytics validates the framework’s capability to handle temporal and aggregation constraints, effectively transforming abstract semantic models into actionable facility management insights.1

Ko, Yun-Dam↗
DOE OSTI2024

Scalability of Real-time Distribution Models

This work will focus on developing the capabilities and validating the models for a sub transmission network with multiple feeders and microgrids. To achieve this scale of Hardware-in-the-loop (HitL) simulation, it is necessary to federate and collaborate. The work aims to design the large-scale feeder models to allow federation with complementary testbeds in the future. The feeder would be designed to be reconfigurable to put the system into a variety of modes. Aggregators models will be included in each distribution network’s federate to take control actions and interact with the management systems. Lastly, the feeder model will support large scale resilience studies involving complex Distributed Energy Resources (DER) controls, microgrid studies and emulation of complex data flows in future grid architectures.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

A Modularized Urban Scale Building Energy Modeling Framework Designed with An Open Mind

In recent years, physics-based building energy modeling (BEM) has started being used to evaluate the performance of buildings in the context of connected communities and on an urban scale to study their aggregated energy use, interactions, and impacts on the energy supply infrastructure and environment. The development of urban-scale BEM solutions needs extensive effort. Existing attempts tend to focus on different aspects of BEM on an urban scale, such as collecting as-built building data from different information sources, integrating geometry modeling with geographic information systems (GISs), representing operational and occupancy profiles, automating workflow, processing and visualizing the results, and conducting large-scale simulations. Urban-scale BEM development would benefit from multi-disciplinary research areas and from an open platform to adopt advancements on data sources and tools. For these purposes, this research proposes a modularized bottom-up model creation and simulation framework that is built on the state-of-the-art BEM tools and can accommodate different building stock data. This framework uses a standardized schema to describe building design and operational characteristics, and it can be instantiated from different building survey datasets with heterogeneous structures. The paper demonstrates how thousands of surveyed buildings from the 2012 U.S. Energy Information Administration’s Commercial Buildings Energy Consumption Survey (CBECS) were one-to-one converted to EnergyPlus models through the schema and the model generation process, then simulated with distributed computing, and their results are summarized.

Lei, Xuechen↗