Search NASA⌕ Search

SEARCH · Search NASA

Results for “Massive Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

MLClouds [SWR-24-24]

The National Solar Radiation Database (NSRDB) is NREL’s flagship solar data resource. With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the NSRDB is a crucial public data asset. A fundamental input to accurate surface irradiance in the NSRDB is high quality cloud property data. Cloud properties are used in radiative transfer calculations and are sourced from satellite imagery. Improving the accuracy of cloud property inputs is a tractable method for improving the accuracy of the irradiance data in the NSRDB. For example, in July of 2018, an average location in the Continental United States is missing cloud property data for nearly one quarter of all daylight cloudy timesteps. This project aims to improve the cloud data inputs to the NSRDB by using machine learning techniques to exploit the NSRDB’s massive data resources. More accurate cloud property input data will yield more accurate surface irradiance data in the NSRDB, providing direct benefit to researchers at NREL and to public data users everywhere.

Buster, Grant↗
DOE OSTI2021

Data Reduction for Science: Brochure from the Advanced Scientific Computing Research Workshop

Data reduction for science holds promise for addressing the challenges of moving, storing, and processing massive data sets produced by the scientific community. Pursuing the PRDs outlined here will enable advances in data streaming, fast feedback and/or autonomous control of experiments, and faster time to scientific insight. These advances will result in a significant improvement in the ability to transport, store, process and interpret experimental, observational, and computational data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Rapid detection of rare events from in situ X-ray diffraction data using machine learning

High-energy X-ray diffraction methods can non-destructively map the 3D microstructure and associated attributes of metallic polycrystalline engineering materials in their bulk form. These methods are often combined with external stimuli such as thermo-mechanical loading to take snapshots of the evolving microstructure and attributes over time. However, the extreme data volumes and the high costs of traditional data acquisition and reduction approaches pose a barrier to quickly extracting actionable insights and improving the temporal resolution of these snapshots. This article presents a fully automated technique capable of rapidly detecting the onset of plasticity in high-energy X-ray microscopy data. The technique is computationally faster by at least 50 times than the traditional approaches and works for data sets that are up to nine times sparser than a full data set. This new technique leverages self-supervised image representation learning and clustering to transform massive data sets into compact, semantic-rich representations of visually salient characteristics ( e.g. peak shapes). These characteristics can rapidly indicate anomalous events, such as changes in diffraction peak shapes. It is anticipated that this technique will provide just-in-time actionable information to drive smarter experiments that effectively deploy multi-modal X-ray diffraction methods spanning many decades of length scales.

Zheng, Weijian↗
DOE OSTI2024

In situ Synchrotron X‐ray Metrology Boosted by Automated Data Analysis for Real‐time Monitoring of Cathode Calcination

Abstract Synchrotron X‐ray‐based in situ metrology is advantageous for monitoring the synthesis of battery materials, offering high throughput, high spatial and temporal resolution, and chemical sensitivity. However, the rapid generation of massive data poses a challenge to on‐site, on‐the‐fly analysis needed for real‐time process monitoring. Here, a weighted lagged cross‐correlation (WLCC) similarity approach is presented for automated data analysis, which merges with in situ synchrotron X‐ray diffraction metrology to monitor the calcination process of the archetypal nickel‐based cathode, LiNiO 2 . The WLCC approach, incorporating variables that account for peak shifts and width changes associated with structural transformations, enables rapid extraction of phase progression within 10 seconds from tens of diffraction patterns. Details are captured, from initial precursors to intermediates and the final layered LiNiO 2 , providing information for agile on‐site adjustments during experiments and complementing post hoc diffraction analysis by offering insights into early‐stage phase nucleation and growth. Expanding this data‐powered platform paves the way for real time calcination process monitoring and control, which is pivotal to quality control in battery cathode manufacturing.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Poeschel, Franz↗
DOE OSTI2025

TxDOT Road Elevation Model Dataset

This dataset provides three formats of Road Elevation Model (REM) data: 3D road line/polygon GeoPackage (GPKG), road lidar LAZ and COPC LAZ, and road digital surface model (DSM) GeoTIFF. Data are produced from the ~50TB TxGIO (formerly TNRIS) state lidar collections. This dataset is currently organized by maintenance section in each TxDOT district. Computation is done on GPU computing resources at Oak Ridge National Laboratory (ORNL), through a Strategic Partnership Project with UT Austin and an NSF ACCESS computing allocation award that enables fast massive data movement between TACC Corral and ORNL CADES/OLCF using Globus. In addition to this release from ORNL, a copy of this dataset can also be downloaded at https://web.corral.tacc.utexas.edu/nfiedata/road3d/.

13 HYDRO ENERGY↗
DOE OSTI2025

Compactly‐Supported Nonstationary Kernels for Computing Exact Gaussian Processes on Big Data

The Gaussian process (GP) is a widely used method for analyzing large-scale data sets, including spatio-temporal measurements of nonlinear processes that are now commonplace in the environmental sciences. Traditional implementations of GPs involve stationary kernels (also termed covariance functions) that limit their flexibility, and exact methods for inference that prevent application to data sets with more than about 10,000 points. Modern approaches to address stationarity assumptions generally fail to accommodate large data sets, while all attempts to address scalability focus on approximating the Gaussian likelihood, which can involve subjectivity and lead to inaccuracies. In this work, we explicitly derive an alternative kernel that can discover and encode both sparsity and nonstationarity. We embed the kernel within a fully Bayesian GP model and leverage high-performance computing resources to enable the analysis of massive data sets. We demonstrate the favorable performance of our novel kernel relative to existing exact and approximate GP methods across a variety of synthetic data examples. Furthermore, we conduct space–time prediction based on more than 1 million measurements of daily maximum temperature and verify that our results outperform state-of-the-art methods in the Earth sciences. More broadly, having access to exact GPs that use ultra-scalable, sparsity-discovering, nonstationary kernels allows GP methods to truly compete with a wide variety of machine learning methods.

Gaussian processes↗
DOE OSTI2022

KGML-ag: a modeling framework of knowledge-guided machine learning to simulate agroecosystems: a case study of estimating N<sub>2</sub>O emission using data from mesocosm experiments

Abstract. Agricultural nitrous oxide (N2O) emission accounts for a non-trivial fraction of global greenhouse gas (GHG) budget. To date, estimating N2O fluxes from cropland remains a challenging task because the related microbial processes (e.g., nitrification and denitrification) are controlled by complex interactions among climate, soil, plant and human activities. Existing approaches such as process-based (PB) models have well-known limitations due to insufficient representations of the processes or uncertainties of model parameters, and due to leverage recent advances in machine learning (ML) a new method is needed to unlock the “black box” to overcome its limitations such as low interpretability, out-of-sample failure and massive data demand. In this study, we developed a first-of-its-kind knowledge-guided machine learning model for agroecosystems (KGML-ag) by incorporating biogeophysical and chemical domain knowledge from an advanced PB model, ecosys, and tested it by comparing simulating daily N2O fluxes with real observed data from mesocosm experiments. The gated recurrent unit (GRU) was used as the basis to build the model structure. To optimize the model performance, we have investigated a range of ideas, including (1) using initial values of intermediate variables (IMVs) instead of time series as model input to reduce data demand; (2) building hierarchical structures to explicitly estimate IMVs for further N2O prediction; (3) using multi-task learning to balance the simultaneous training on multiple variables; and (4) pre-training with millions of synthetic data generated from ecosys and fine-tuning with mesocosm observations. Six other pure ML models were developed using the same mesocosm data to serve as the benchmark for the KGML-ag model. Results show that KGML-ag did an excellent job in reproducing the mesocosm N2O fluxes (overall r2=0.81, and RMSE=3.6 mgNm-2d-1 from cross validation). Importantly, KGML-ag always outperforms the PB model and ML models in predicting N2O fluxes, especially for complex temporal dynamics and emission peaks. Besides, KGML-ag goes beyond the pure ML models by providing more interpretable predictions as well as pinpointing desired new knowledge and data to further empower the current KGML-ag. We believe the KGML-ag development in this study will stimulate a new body of research on interpretable ML for biogeochemistry and other related geoscience processes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Detector Interface for Streaming, Control, and Open-source integration (DISCO) v1.0.0

This suite consists of a multi-package ecosystem featuring detector emulators, EPICS areaDetector drivers, and remote server frameworks designed for the Advanced Light Source (ALS). Engineered for high-bandwidth devices—including VFCCD, Timepix3, Timepix4, and related pixel detectors—the software simulates hardware, wraps vendor SDKs into remote-callable servers, and integrates with open-source control systems. Key Capabilities: Distributed SDK Architecture: Server packages wrap hardware-specific SDKs, allowing areaDetector drivers to execute remote framework calls. This isolates proprietary libraries from the EPICS IOC, enhancing stability and enabling distributed computing across beamline networks. Device Support: Custom drivers for VFCCD, the Timepix family, and similar sensors optimize the data path from hardware control to high-speed transport. Full-Stack Emulation: Sophisticated emulator packages allow end-to-end pipeline testing and software development without requiring physical hardware or beam time. Integrated Workflows: Supports high-bandwidth streaming for real-time analysis and robust, metadata-rich file-based workflows (e.g., HDF5/NeXus). By standardizing interfaces across heterogeneous hardware, this suite reduces technical debt. It provides the ALS with a scalable, open-source solution to manage massive data rates within a unified control environment.

Mahl, Johannes [Lawrence Berkeley National Laborat↗
DOE OSTI2021

Reaching new peaks for the future of the CMS HTCondor Global Pool

The CMS experiment at CERN employs a distributed computing infrastructure to satisfy its data processing and simulation needs. The CMS Submission Infrastructure team manages a dynamic HTCondor pool, aggregating mainly Grid clusters worldwide, but also HPC, Cloud and opportunistic resources. This CMS Global Pool, which currently involves over 70 computing sites worldwide and peaks at 350k CPU cores, is employed to successfully manage the simultaneous execution of up to 150k tasks. While the present infrastructure is sufficient to harness the current computing power scales, CMS latest estimates predict a noticeable expansion in the amount of CPU that will be required in order to cope with the massive data increase of the High-Luminosity LHC (HL-LHC) era, planned to start in 2027. This contribution presents the latest results of the CMS Submission Infrastructure team in exploring and expanding the scalability reach of our Global Pool, in order to preventively detect and overcome any barriers in relation to the HL-LHC goals, while maintaining high effciency in our workload scheduling and resource utilization.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

When Machine Learning Meets 2D Materials: A Review

The availability of an ever-expanding portfolio of 2D materials with rich internal degrees of freedom (spin, excitonic, valley, sublattice, and layer pseudospin) together with the unique ability to tailor heterostructures made layer by layer in a precisely chosen stacking sequence and relative crystallographic alignments, offers an unprecedented platform for realizing materials by design. However, the breadth of multi-dimensional parameter space and massive data sets involved is emblematic of complex, resource-intensive experimentation, which not only challenges the current state of the art but also renders exhaustive sampling untenable. To this end, machine learning, a very powerful data-driven approach and subset of artificial intelligence, is a potential game-changer, enabling a cheaper – yet more efficient – alternative to traditional computational strategies. It is also a new paradigm for autonomous experimentation for accelerated discovery and machine-assisted design of functional 2D materials and heterostructures. Here, the study reviews the recent progress and challenges of such endeavors, and highlight various emerging opportunities in this frontier research area.

2D materials↗
DOE OSTI2025

Survey-wide asteroid discovery with a high-performance computing enabled non-linear digital tracking framework

Modern astronomical surveys detect asteroids by linking together their appearances across multiple images taken over time. This approach faces limitations in detecting faint asteroids and handling the computational complexity of trajectory linking. Here, we present a novel method that adapts “digital tracking” – traditionally used for short-term linear asteroid motion across images – to work with large-scale synoptic surveys such as the Vera Rubin Observatory Legacy Survey of Space and Time (Rubin/LSST). Our approach combines hundreds of sparse observations of individual asteroids across their non-linear orbital paths to enhance detection sensitivity by several magnitudes. To address the computational challenges of processing massive data sets and dense orbital phase spaces, we developed a specialized high-performance computing architecture. We demonstrate the effectiveness of our method through experiments that take advantage of the extensive computational resources at Lawrence Livermore National Laboratory. This work enables the detection of significantly fainter asteroids in existing and future survey data, potentially increasing the observable asteroid population by orders of magnitude across different orbital families, from near-Earth objects (NEOs) to Kuiper belt objects (KBOs).

Asteroid discovery↗
DOE OSTI2025

The ladder and readout cables of intermediate silicon strip detector for sPHENIX

A new silicon-strip-type detector was developed for precise charged-particle tracking in the central rapidity region of heavy ion collisions. A new detector and collaboration at the Relativistic Heavy Ion Collider at Brookhaven National Laboratory is sPHENIX, which is a major upgrade of the PHENIX detector. The intermediate tracker (INTT) is part of the advanced tracking system of the sPHENIX detector complex together with a CMOS monolithic-active-pixel-sensor based silicon-pixel vertex detector, a time-projection chamber, and a micromegas-based detector. The INTT detector is barrel shaped and comprises 56 silicon ladders. Two different types of strip sensors of 78 micron pitch and 320 micron thick are mounted on each half of a silicon ladder. Each strip sensor is segmented into 8 × 2 and 5 × 2 blocks with lengths of 16 and 20 mm. Strips are read out with a silicon strip-readout (FPHX) chip. In order to transmit massive data from the FPHX to the down stream readout electronics card (ROC), a series of long and high speed readout cables were developed. This document focuses on the silicon ladder, the readout cables, and the ROC of the INTT. The radiation hardness is studied for some parts of the INTT devices in the last part of this document, since the INTT employed some materials from the technology frontier of the industry whose radiation hardness is not necessarily well known.

43 PARTICLE ACCELERATORS↗
DOE OSTI2020

The global energy footprint of information and communication technology electronics in connected Internet-of-Things devices

Information and communication technology (ICT) has been critical to the emergence of a new Internet-of-Things (IoT) era, as a rapidly increasing number and variety of IoT electronic devices in a wide range of end-use sectors require massive data communication in terms of volume, velocity/speed, and variety. Additionally, semiconductor technology advancements have played a key role in ICT and unlike the past ICT energy footprint analysis, which focused on the final electronic product, an attempt has been made to examine the global energy impacts for the period 2016–2025 based on the market forecasts of three major IoT IC types, i.e., sensors and actuators, connectivity, and processor ICs distributed among four major IoT device components.Total global IoT semiconductor primary energy demand is projected to increase from 2 EJ in 2016 to 35 EJ by 2025, resulting mainly from a substantial projected increase energy needed in the manufacturing of ICs, with growth in the use of sensors and energy-intensive next fab generation manufacturing. Unlike the trend in manufacturing energy, total global operational energy use is projected to decrease significantly with the development of smaller transistor size, low-power devices, and faster wireless data communication technology. Total operational energy is projected to significantly decrease from 118 TWh/year (1,2 EJ/year) to less than 1 TWh/year (0.003 EJ/year) in spite of a significantly higher anticipated growth in its market size during the 2016–2025 forecast period. Its annual share of total primary energy use is projected to reach an insignificant level of 0.01% by the end of 2025, consistent with the past limited electronic product life cycle analyses besides the analysis focus here limited to ICs in IoT devices.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Machine-Learning for Excited-State Dynamics

The primary objective of this computational chemistry sciences team is to design a machine learning NAMD environment that will utilize current petascale and future exascale computational capabilities to advance understanding of charge and energy flow in materials. Our machine-learning NAMD environment will 1) integrate advanced NAMD capabilities directly into electronic structure software (e.g., ABINIT, Quantum Espresso, VASP, etc.); 2) merge the preparatory tools of Pychemia into PYXAID and Avogadro environments so that massive data collection from NAMD simulations.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Machine learning analysis reveals relationship between pomacentrid calls and environmental cues

Sound production rates of fishes can be used as an indicator for coral reef health, providing an opportunity to utilize long-term acoustic recordings to assess environmental change. As acoustic datasets become more common, computational techniques need to be developed to facilitate analysis of the massive data files produced by long-term monitoring. Machine learning techniques demonstrate an advantage in the identification of fish sounds over manual sampling approaches. Here we evaluated the ability of convolutional neural networks to identify and monitor call patterns for pomacentrids (damselfishes) in a tropical reef region of the western Pacific. A stationary hydrophone was deployed for 39 mo (2014-2018) in the National Park of American Samoa to continuously record the local marine acoustic environment. A neural network was trained—achieving 94% identification accuracy of pomacentrids—to demonstrate the applicability of machine learning in fish acoustics and ecology. The distribution of sound production was found to vary on diel and interannual timescales. Additionally, the distribution of sound production was correlated with wind speed, water temperature, tidal amplitude, and sound pressure level. This research has broad implications for state-of-the-art acoustic analysis and promises to be an efficient, scalable asset for ecological research, environmental monitoring, and conservation planning.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Optimization of the Observing Cadence for the Rubin Observatory Legacy Survey of Space and Time: A Pioneering Process of Community-focused Experimental Design

Vera C. Rubin Observatory is a ground-based astronomical facility under construction, a joint project of the National Science Foundation and the U.S. Department of Energy, designed to conduct a multipurpose 10 yr optical survey of the Southern Hemisphere sky: the Legacy Survey of Space and Time. Significant flexibility in survey strategy remains within the constraints imposed by the core science goals of probing dark energy and dark matter, cataloging the solar system, exploring the transient optical sky, and mapping the Milky Way. The survey’s massive data throughput will be transformational for many other astrophysics domains and Rubin’s data access policy sets the stage for a huge community of potential users. To ensure that the survey science potential is maximized while serving as broad a community as possible, Rubin Observatory has involved the scientific community at large in the process of setting and refining the details of the observing strategy. The motivation, history, and decision-making process of this strategy optimization are detailed in this paper, giving context to the science-driven proposals and recommendations for the survey strategy included in this Focus Issue.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

Modeling Cosmic Reionization

The transformation of cold neutral intergalactic hydrogen into a highly ionized warm plasma marks the end of the cosmic dark ages and the beginning of the age of galaxies. The details of this process reflect the nature of the early sources of radiation and heat, the statistical characteristics of the large-scale structure of the Universe, the thermodynamics and chemistry of cosmic baryons, and the histories of star formation and black hole accretion. A number of massive data sets from new ground- and space-based instruments and facilities over the next decade are poised to revolutionize our understanding of primeval galaxies, the reionization photon budget, the physics of the intergalactic medium (IGM), and the fine-grained properties of hydrogen gas in the "cosmic web". In this review we survey the physics and key aspects of reionization-era modeling and describe the diverse range of computational techniques and tools currently available in this field.

79 ASTRONOMY AND ASTROPHYSICS↗