Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Scalable Knowledge Graph Analytics at 136 Petaflop/s

We are motivated by newly proposed methods for data mining large-scale corpora of scholarly publications, such as the full biomedical literature, which may consist of tens of millions of papers spanning decades of research. In this setting, analysts seek to discover how concepts relate to one another. They construct graph representations from annotated text databases and then formulate the relationship-mining problem as one of computing all-pairs shortest paths (APSP), which becomes a significant bottleneck. In this context, we present a new high-performance algorithm and implementation of the Floyd-Warshall algorithm for distributed-memory parallel computers accelerated by GPUs, which we call DSNAPSHOT (Distributed Accelerated Semiring All-Pairs Shortest Path). For our largest experiments, we ran DSNAPSHOT on a connected input graph with millions of vertices using 4, 096nodes (24,576GPUs) of the Oak Ridge National Laboratory's Summit supercomputer system. We find DSNAPSHOT achieves a sustained performance of 136×1015 floating-point operations per second (136petaflop/s) at a parallel efficiency of 90% under weak scaling and, in absolute speed, 70% of the best possible performance given our computation (in the single-precision tropical semiring or “min-plus” algebra). Looking forward, we believe this novel capability will enable the mining of scholarly knowledge corpora when embedded and integrated into artificial intelligence-driven natural language processing workflows at scale.

Kannan, Ramakrishnan {ramki}↗
DOE OSTI2024

ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability

Earth system predictability is challenged by the complexity of environmental dynamics and the multitude of variables involved. Current AI foundation models, although advanced by leveraging large and heterogeneous data, are often constrained by their size and data integration, limiting their effectiveness in addressing the full range of Earth system prediction challenges. To overcome these limitations, we introduce the Oak Ridge Base Foundation Model for Earth System Predictability (ORBIT), an advanced vision transformer model that scales up to 113 billion parameters using a novel hybrid tensor-data orthogonal parallelism technique. As the largest model of its kind, ORBIT surpasses the current climate AI foundation model size by a thousandfold. Performance scaling tests conducted on the Frontier supercomputer have demonstrated that ORBIT achieves 684 petaFLOPS to 1.6 exaFLOPS sustained throughput, with scaling efficiency maintained at 41% to 85% across 49,152 AMD GPUs. These breakthroughs establish new advances in AI-driven climate modeling and demonstrate promise to significantly improve the Earth system predictability.

Wang, Xiao↗
DOE OSTI2022

Accelerating Hydraulic Fracture Imaging by Deep Transfer Learning

Deep transfer learning has a great success story in computer vision (CV), natural language processing, and many other fields. In this communication, we are going to push forward the deep transfer learning to the hydraulic fracture imaging problem by proposing a two-step approach: 1) train a convolutional neural network (CNN) to reconstruct target geometries by a relatively large amount of approximated field patterns generated from a simplified model and 2) fine-tune the top layers of transferred CNN by a small amount of true field patterns generated through a full model. The advantages include the rapid generation of large amount of data through the simplified model and the high reconstruction accuracy through a careful design of the deep transfer learning. Here, the CNN trained through the deep transfer learning can accurately reconstruct the lateral extent and direction of fractures with unseen conductivity and white Gaussian noise, showing a notable acceleration/accuracy enhancement over the previous CNN trained by mixing a small number of true data into the approximated data as data augmentation.

02 PETROLEUM↗
DOE OSTI2024

Technical Impacts of Light-Duty and Heavy-Duty Transportation Electrification on a Coordinated Transmission and Distribution System

In this study, we propose a strategy to model the required spatiotemporal charging demand from light-duty (LD) and medium- and heavy-duty (MHD) electric vehicles (EVs) using actual transportation data by mapping the demand for the required EV charging to a realistic and coordinated distribution and transmission electric grid at the predicted times of the day to study their impact on the power system in a variety of load, weather, and EV penetration scenarios. This work is the first study that includes the actual weather data and transportation data with realistic and coordinated distribution and transmission grid data in a large industry-scale level study. The main goal of this study is to identify possible issues and required upgrades in the electric grid, caused by an increase in EV integration. The transmission case study is a large grid with 6717 buses over a Texas footprint, and the distribution grid is over Houston, a city in Texas, covering over three million customers. The resulting overloads and voltage violations experienced in the system are discussed, and required planning upgrades to avoid these issues are suggested.

AC optimal power flow (AC-OPF)↗
DOE OSTI2021

Science Capsule: Towards Sharing and Reproducibility of Scientific Workflows

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Ghoshal, Devarshi↗
DOE OSTI2022

Co-scheduling Ensembles of In Situ Workflows

Molecular dynamics (MD) simulations are widely used to study large-scale molecular systems. HPC systems are ideal platforms to run these studies, however, reaching the necessary simulation timescale to detect rare processes is challenging, even with modern supercomputers. To overcome the timescale limitation, the simulation of a long MD trajectory is replaced by multiple short-range simulations that are executed simultaneously in an ensemble of simulations. Analyses are usually co-scheduled with these simulations to efficiently process large volumes of data generated by the simulations at runtime, thanks to in situ techniques. Executing a workflow ensemble of simulations and their in situ analyses requires efficient co- scheduling strategies and sophisticated management of computational resources so that they are not slowing down each other. In this paper, we propose an efficient method to co-schedule simulations and in situ analyses such that the makespan of the workflow ensemble is minimized. We present a novel approach to allocate resources for a workflow ensemble under resource constraints by using a theoretical framework modeling the workflow ensemble’s execution. We evaluate the proposed approach using an accurate simulator based on the WRENCH simulation framework on various workflow ensemble configurations. Results demonstrate the significance of co-scheduling simulations and in situ analyses that couple data together to benefit from data locality, in which inefficient scheduling decisions can lead to slowdown in makespan up to a factor of 30.

Do, Tu Mai Anh↗
DOE OSTI2022

The Latent Dirichlet Allocation model applied to airborne LiDAR data: A case study on mapping forest degradation associated with fragmentation and fire in the Amazon region

1. LiDAR data are being increasingly used to provide a detailed characterization of the vertical profile of forests. This characterization enables the generation of new insights on the influence of environmental drivers and anthropogenic disturbances on forest structure as well as on how forest structure influences important ecosystem functions and services. Unfortunately, extracting information from LiDAR data in a way that enables the spatial visualization of forest structure, as well as its temporal changes, is challenging due to the high dimensionality of these data. 2. In this study, we show how the Latent Dirichlet Allocation model applied to LiDAR data (LidarLDA) can be used to identify forest structural types and how the relative abundance of these forest types changes throughout the landscape. The code to fit this model is made available through the open-source r package LidarLDA in github. We illustrate the use of LidarLDA both with simulated data and data from a large-scale fire experiment in the Brazilian Amazon region. 3. Using simulated data, we demonstrate that LidarLDA accurately identifies the number of forest types as well as their spatial distribution and absorptance probabilities. For the empirical data, we found that LidarLDA detects both landscape-level patterns in forest structure as well as the strong interacting effect of fire and forest fragmentation on forest structure based on the experimental fire plots. More specifically, LidarLDA reveals that proximity to forest edge exacerbates the impact of fires, and that burned forests remain structurally different from unburned areas for at least 7 years, even when burned only once. Importantly, LidarLDA generates insights on the 3D structure of forest that cannot be obtained using more standard approaches that just focus on top-of-the-canopy information (e.g. canopy height models based on LiDAR data). 4. By enabling the mapping of forest structure and its temporal changes, we believe that LidarLDA will be of broad utility to the ecological research community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

A gamma-ray pulsar timing array constrains the nanohertz gravitational wave background

We report after large galaxies merge, their central supermassive black holes are expected to form binary systems. Their orbital motion should generate a gravitational wave background (GWB) at nanohertz frequencies. Searches for this background use pulsar timing arrays, which perform long-term monitoring of millisecond pulsars at radio wavelengths. We used 12.5 years of Fermi Large Area Telescope data to form a gamma-ray pulsar timing array. Results from 35 bright gamma-ray pulsars place a 95% credible limit on the GWB characteristic strain of 1.0 × 10 -14 at a frequency of 1 year –1 . The sensitivity is expected to scale with t obs , the observing time span, as t$^{-13/6}_{obs}$. This direct measurement provides an independent probe of the GWB while offering a check on radio noise models.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Programming Strategies for Irregular Algorithms on the Emu Chick

The Emu Chick prototype implements migratory memory-side processing in a novel hardware system. Rather than transferring large amounts of data across the system interconnect, the Emu Chick moves lightweight thread contexts to near-memory cores before the beginning of each remote memory read. Previous work has characterized the performance of the Chick prototype in terms of memory bandwidth and programming differences from more typical, non-migratory platforms, but there has not yet been an analysis of algorithms on this system. This work evaluates irregular algorithms that could benefit from the lightweight, memory-side processing of the Chick and demonstrates techniques and optimization strategies for achieving performance in sparse matrix-vector multiply operation (SpMV), breadth-first search (BFS), and graph alignment across up to eight distributed nodes encompassing 64 nodelets in the Chick system. We also define and justify relative metrics to compare prototype FPGA-based hardware with established ASIC architectures. The Chick currently supports up to 68x scaling for graph alignment, 80 MTEPS for BFS on balanced graphs, and 50% of measured STREAM bandwidth for SpMV.

Hein, Eric↗
DOE OSTI2021

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing (HPC) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS) that used a Virtual Data Space (VDS) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

Quantum Transfer Learning to Boost Dementia Detection

Dementia is a devastating condition with profound implications for individuals, families, and healthcare systems. Early and accurate detection of dementia is critical for timely intervention and improved patient outcomes. While classical machine learning and deep learning approaches have been explored extensively for dementia prediction, these solutions often struggle with high-dimensional biomedical data and large-scale datasets, quickly reaching computational and performance limitations. To address this challenge, quantum machine learning (QML) has emerged as a promising paradigm, offering faster training and advanced pattern recognition capabilities. This work aims to demonstrate the potential of quantum transfer learning (QTL) to enhance the performance of a weak classical deep learning model applied to a binary classification task for dementia detection. Besides, we show the effect of noise on the QTL-based approach, investigating the reliability and robustness of this method. Using the OASIS 2 dataset, we show how quantum techniques can transform a suboptimal classical model into a more effective solution for biomedical image classification, highlighting their potential impact on advancing healthcare technology.

Bhowmik, Sounak [University of Tennessee, Knoxvill↗
DOE OSTI2021

An Intelligent Garbage Sorting System Based on Edge Computing and Visual Understanding of Social Internet of Vehicles

In order to enable Social Internet of Vehicles devices to achieve the purpose of intelligent and autonomous garbage classification in a public environment, while avoiding network congestion caused by a large amount of data accessing the cloud at the same time, it is therefore considered to combine mobile edge computing with Social Internet of Vehicles to give full play to mobile edge computing features of high bandwidth and low latency. At the same time, based on cutting-edge technologies such as deep learning, knowledge graph, and 5G transmission, the paper builds an intelligent garbage sorting system based on edge computing and visual understanding of Social Internet of Vehicles. First of all, for the massive multisource heterogeneous Social Internet of Vehicles big data in the public environment, different item modal data adopts different processing methods, aiming to obtain a visual understanding model. Secondly, using the 5G network, the model is deployed on the edge device and the cloud for cloud-side collaborative management, aiming to avoid the waste of edge node resources, while ensuring the data privacy of the edge node. Finally, the Social Internet of Vehicles devices is used to make intelligent decision-making on the big data of the items. First, the items are judged as garbage, and then the category is judged, and finally the task of grabbing and sorting is realized. The experimental results show that the system proposed in this paper can efficiently process the big data of Social Internet of Vehicles and make valuable intelligent decisions. At the same time, it also has a certain role in promoting the promotion of Social Internet of Vehicles devices.

Shen, Xuehao↗
DOE OSTI2021

ArborX

ArborX library tackles a problem of efficiently finding geometric objects that are close in space. Variations of this problem, such as finding the nearest neighbors of a point, or finding all objects within a certain distance, are inherent components of applications in many fields. The data may be large so that solving the problem efficiently may require significant computational resources, such as multiple processors or accelerators such as general purpose GPUs. ArborX' main advantage in its ability to solve large problems efficiently utilizing a combination of distributed and on-node parallelism. ArborX can be run efficiently on a wide variety of hardware, including GPUs from different vendors, which distinguishes it from other available libraries which typically choose only few of these. The other advantage is that it supports both types of user problems: spatial problems (useful for intersections and finding objects within certain distance), and nearest neighbor problems. ArborX also supports flexible interface in its interaction with a user. Particularly, it allows a user to call user's own function on a positive match, a functionality not rarely available in other libraries. ArborX implements construction and traversal algorithms using efficient tree structures, such as bounding volume hierarchy (BVH). At its core, it uses linear BVH for its low construction cost and sufficient quality. ArborX is written using C++, and is parallelized using the message passing interface (MPI) for the distributed communication, and the Kokkos library for on-node parallelism. This approach allows ArborX to be run on a wide variety of hardware, from common laptops and desktops to supercomputers while using the same codebase. ArborX also implements several advanced algorithms using geometric search, such as density-based clustering algorithm DBSCAN.

ECP↗
DOE OSTI2025

ArborX 2.0

ArborX library tackles a problem of efficiently finding geometric objects that are close in space. Variations of this problem, such as finding the nearest neighbors of a point, or finding all objects within a certain distance, are inherent components of applications in many fields. The data may be large so that solving the problem efficiently may require significant computational resources, such as multiple processors or accelerators such as general purpose GPUs. ArborX' main advantage in its ability to solve large problems efficiently utilizing a combination of distributed and on-node parallelism. ArborX can be run efficiently on a wide variety of hardware, including GPUs from different vendors, which distinguishes it from other available libraries which typically choose only few of these. The other advantage is that it supports both types of user problems: spatial problems (useful for intersections and finding objects within certain distance), and nearest neighbor problems. ArborX also supports flexible interface in its interaction with a user. Particularly, it allows a user to call user's own function on a positive match, a functionality not rarely available in other libraries. ArborX implements construction and traversal algorithms using efficient tree structures, such as bounding volume hierarchy (BVH). At its core, ArborX uses linear BVH for its low construction cost and sufficient quality. ArborX implements both spatial and nearest-neighbor traversal algorithms. ArborX also provides several clustering algorithms (minimum spanning tree, DBSCAN, HDBSCAN*), interpolation using minimum least squares and ray tracing. ArborX is written using C++, and is parallelized using the message passing interface (MPI) for the distributed communication, and the Kokkos library for on-node parallelism. This approach allows ArborX to be run on a wide variety of hardware, from common laptops and desktops to supercomputers while using the same codebase.

Prokopenko, Andrey [Oak Ridge National Laboratory ↗
DOE OSTI2020

Asynchronous I/O VOL Connector (AsyncVOL) v0.1

Asynchronous I/O is becoming increasingly popular with the large amount of data access required by scientific applications. They can take advantage of an asynchronous interface by scheduling I/O as early as possible and overlap computation or communication with I/O operations, which hides the cost associated with I/O and improves the overall performance.

Tang, Houjun↗
DOE OSTI2026

Generator Frequency Response Droop Monitoring Tool

Monitoring and analyzing the frequency response performance of power generation units is essential for maintaining reliable and secure power system operation. To address this need, an automation tool has been developed to provide a pipeline for processing historical power plant generation data, including large-scale SCADA archives. The tool performs end-to-end processing, including event detection, frequency response (FR) analysis in accordance with NERC standards, and estimation of speed governor droop characteristics. The tool is designed with a modular architecture, allowing individual components of the workflow to be extended, customized, or deployed independently. In addition, the tool provides an API that enables seamless integration with other production systems and operational analytics platforms.

Etingov, PavelV [Pacific Northwest National Labora↗
DOE OSTI2026

MCP-eGridGPT (MCP-Enabled Chatbot with Electrical Power System Analysis and Interactive Visualization Tool) [SWR-25-126]

This software is an advanced chatbot system that integrates the Model Context Protocol (MCP) to provide intelligent electrical power system analysis and automated visualization generation. The system enables users to interact with complex electrical engineering tools through natural language, automatically analyzes power system data for voltage violations and grid health assessment, and generates professional interactive HTML dashboards and reports. Key features include dynamic tool discovery from MCP servers, multi-LLM provider support, intelligent data interpretation using large language models, automated chart generation, and a web-based interface for real-time analysis. The software bridges sophisticated electrical engineering analysis with user-friendly interfaces, making power system diagnostics accessible through conversational AI.

Choi, Seong [National Laboratory of the Rockies (N↗
DOE OSTI2026

How Well Can CMIP6 Models Represent the Observed Influence of the Pacific and Indian Oceans on the Indian Summer Monsoon Rainfall?

This study evaluates the ability of CMIP6 climate models to simulate the observed effects of tropical Pacific and Indian Ocean sea surface temperature anomalies (SSTAs) on Indian summer monsoon rainfall (ISMR) variability. Using observational data and the large ensemble historical simulations of seven CMIP6 models from 1950 to 2014, we applied a cyclostationary linear inverse model (CS-LIM) to isolate the impacts of tropical Pacific SSTAs, Indian Ocean SSTAs and their interaction on the interannual variability of ISMR. Overall, CMIP6 models well reproduced the observed enhanced (reduced) ISMR variability from Pacific SSTAs (Indian Ocean SSTAs and the Indo-Pacific interaction), but with varying spatial patterns and magnitudes. While CESM2 and E3SM-2-0 showed the best agreement with observations for the effects of Pacific SSTAs and the Indo-Pacific interaction, respectively, CMIP6 models showed mixed results for the impacts from Indian Ocean SSTAs. Composite analysis of ISMR anomalies during the developing phases of pure and co-occurring El Niño-Southern Oscillation (ENSO) and Indian Ocean dipole (IOD) events revealed that the impacts from Pacific SSTAs were captured reasonably well by E3SM-2-0, CESM2, MIROC6, and MPI-ESM1-2-LR, while E3SM-2-0 also showed the best agreement with observations for the effects from the Indo-Pacific interaction. However, all models showed substantial biases in simulating the Indian Ocean SSTA impacts on ISMR, especially for pure El Niño events. Overall, this study provides new insights into how individual CMIP6 models simulate the isolated impacts from the tropical Pacific and Indian Oceans, which has important applications for improving ISMR predictions and interpreting ISMR future projections.

monsoon↗