Search NASA⌕ Search

SEARCH · Search NASA

Results for “Massive Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Global Archaeal Diversity Revealed Through Massive Data Integration: Uncovering Just Tip of Iceberg

The domain of Archaea has gathered significant interest for its ecological and biotechnological potential and its role in helping us to understand the evolutionary history of Eukaryotes. In comparison to the bacterial domain, the number of adequately described members in Archaea is relatively low, with less than 1000 species described. It is not clear whether this is solely due to the cultivation difficulty of its members or, indeed, the domain is characterized by evolutionary constraints that keep the number of species relatively low. Based on molecular evidence that bypasses the difficulties of formal cultivation and characterization, several novel clades have been proposed, enabling insights into their metabolism and physiology. Given the extent of global sampling and sequencing efforts, it is now possible and meaningful to question the magnitude of global archaeal diversity based on molecular evidence. To do so, we extracted all sequences classified as Archaea from 500 thousand amplicon samples available in public repositories. After processing through our highly conservative pipeline, we named this comprehensive resource the ‘Global Archaea Diversity’ (GAD), which encompassed nearly 3 million molecular species clusters at 97% similarity, and organized it into over 500 thousand genera and nearly 100 thousand families. Saline environments have contributed the most to the novel taxa of this previously unseen diversity. The majority of those 16S rRNA gene sequence fragments were verified by matches in metagenomic datasets from IMG/M. These findings reveal a vast and previously overlooked diversity within the Archaea, offering insights into their ecological roles and evolutionary importance while establishing a foundation for the future study and characterization of this intriguing domain of life.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

In Silico Human Mobility Data Science: Leveraging Massive Simulated Mobility Data (Vision Paper)

Human mobility data science using trajectories or check-ins of individuals has many applications. Recently, we have seen a plethora of research efforts that tackle these applications. However, research progress in this field is limited by a lack of large and representative datasets. The largest and most commonly used dataset of individual human trajectories captures fewer than 200 individuals, while datasets of individual human check-ins capture fewer than 100 check-ins per city per day. Thus, it is not clear if findings from the human mobility data science community would generalize to large populations. Since obtaining massive, representative, and individual-level human mobility data is hard to come by due to privacy considerations, the vision of this work is to embrace the use of data generated by large-scale socially realistic microsimulations. Informed by both real data and leveraging social and behavioral theories, massive spatially explicit microsimulations may allow us to simulate entire megacities at the person level. The simulated worlds, which do not capture any identifiable personal information, allow us to perform “in silico” experiments using the simulated world as a sandbox in which we have perfect information and perfect control without jeopardizing the privacy of any actual individual. In silico experiments have become commonplace in other scientific domains such as chemistry and biology, permitting experiments that foster the understanding of concepts without any harm to individuals. This work describes challenges and opportunities for leveraging massive and realistic simulated alternate worlds for in silico human mobility data science.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Big Data Analysis of Massive PMU Datasets: A Data Platform Perspective

The discovery of `event signatures' and useful insights from very large historical Phasor Measurement Unit (PMU) datasets is predicated on offline Big Data analysis approaches that rely on the generation of predictive features on a massive scale. This paper presents lessons learned from a data platform perspective towards reducing barriers to adoption of Big Data analytics against a real dataset of almost half a trillion data points drawn from over 400 PMUs distributed across the North American power grid. We demonstrate software abstractions and targeted performance optimizations that can lead to significant productivity gains for power systems researchers seeking to perform offline exploratory temporal analysis and modeling tasks, with a focus on feature generation. We describe how our optimized approach goes beyond a naive application of mainstream Big Data technologies, enabling feature generation tasks, that previously took days or even weeks, to now be completed in just a few hours.

Kumar, Vijay S.↗
DOE OSTI2020

Portable interactive visualization of large-scale simulations in geotechnical engineering using Unity3D

Development in large-scale geotechnical engineering simulation places tremendous demand for efficient visualization of such simulation data. This study presents a lightweight software tool, i.e. Geotechnical Interactive Visualization (GIV), as a solution to this challenge, which achieves efficient interactive visualization of large-scale simulations data in geotechnical engineering. Visualization data flow and algorithms specifically optimized for common geotechnical engineering applications are implemented in GIV. GIV can visualize geotechnical structure models with time-varying attributes attached to mesh with fixed topology, and also models with time-varying mesh topologies but no attributes attached, the two most common visualization tasks in geotechnical engineering. Furthermore, challenges for large-scale simulation data visualization, including parallel simulation data redundancy, massive data size, dynamic user interaction, and portability are overcome via specifically designed algorithms for simulation data preprocessing and optimized visualization modules using the powerful 3D rendering and interactive game engine Unity3D. Comparison of GIV with several widely used visualization tools for the visualization of large-scale idealized datasets and realistic geotechnical simulations highlights the visualization efficiency, smooth interactivity, and lightweight features of GIV.

42 ENGINEERING↗
DOE OSTI2023

Editorial: Edge computation and digital distribution networks

The development of digital technologies is penetrating all areas of energy revolution. Based on the in-depth integration of advanced digital technologies, distribution networks are gradually transforming into digital distribution networks (DDNs) with tremendous changes from the structure to the operation mode. DDNs is the digitalized appearance of the physical distribution network, in which ubiquitous connections and massive data are the basic characteristics (Huo et al., 2022). It is an important task to utilize the massive data and propose novel operation modes to construct more efficient and intelligent distribution networks (Jian et al., 2022). Among the advanced digital technologies in DDNs, edge computing has received wide attention (Zhao et al., 2022). It has superior performance in local sensing and intelligent computation, which can effectively relieve huge communication pressure. However, the limited computing resources and the complex computing tasks at the edge side significantly challenge the collaboration of distribution network regulation and advanced digital technologies (Hu et al., 2022). It is necessary to find out proper methods to utilize advanced digital technology to construct DDNs. This Research Topic is organized to introduce the recent progress in the construction, operation and advanced computational methods for DDNs. Finally, seven papers have been accepted, which can be sorted into the following three categories: 1) Evolution and technical features of DDNs, 2) Intelligent operation control of DDNs, 3) Advanced simulation for large-scale DDNs. The three sections below respectively introduce the major research and contributions of the papers covered in each category.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management

The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. Here, the ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2026

Expanding Access to Science Participation: A FAIR Framework for Petascale Data Visualization and Analytics

The massive data generated by scientists daily serve as both a major catalyst for new discoveries and innovations, as well as a significant roadblock that restricts access to the data. Here, our paper introduces a new approach to removing Big Data barriers and democratizing access to petascale data for the broader scientific community. Our novel data fabric abstraction layer allows user-friendly querying of scientific information while hiding the complexities of dealing with file systems or cloud services. We enable FAIR (Findable, Accessible, Interoperable, and Reusable) access to datasets such as NASA’s petascale climate datasets. Our paper presents an approach to managing, visualizing, and analyzing petabytes of data within a browser on equipment ranging from the top NASA supercomputer to commodity hardware like a laptop. Our novel data fabric abstraction utilizes state-of-the art progressive compression algorithms and machine-learning insights to power scalable visualization dashboards for petascale data. The result provides users with the ability to identify extreme events or trends dynamically, expanding access to scientific data and further enabling discoveries. We validate our approach by improving the ability of climate scientists to visually explore their data via three fully interactive dashboards. We further validate our approach by deploying the dashboards and simplified training materials in the classroom at a minority-serving institution. These dashboards, released in simplified form to the general public, contribute significantly to a broader push to democratize the access and use of climate data.

Computer science↗
DOE OSTI2023

Massive Trajectory Data Based on Patterns of Life

Individual human location trajectory and check-in data have been the driving force for human mobility research in recent years. However, existing human mobility datasets are very limited in size and representativeness. For example, one of the largest and most commonly used datasets of individual human location trajectories, GeoLife, captures fewer than two hundred individuals. To help fill this gap, this Data and Resources paper leverages an existing data generator based on fine-grained simulation of individual human patterns of life to produce large-scale trajectory, check-in, and social network data. In this simulation, individual human agents commute between their home and work locations, visit restaurants to eat, and visit recreational sites to meet friends. We provide large datasets of months of simulated trajectories for two example regions in the United States: San Francisco and New Orleans. In addition to making the datasets available, we also provide instructions on how the simulation can be used to re-generate data, thus allowing researchers to generate the data locally without downloading prohibitively large files.

Amiri, Hossein↗
DOE OSTI2024

MassIVE MSV000094084 - Data files for manuscript - Selected ion monitoring for orbitrap-based metabolomics: Please open README-data-for-SIM-paper-20240210.doc for details.

Orbitrap mass spectrometry in full scan mode enables the simultaneous detection of hundreds of metabolites and their isotope-labeled forms. Yet, sensitivity remains limiting for many metabolites, including low-concentration species, poor ionizers, and low-fractional-abundance isotope-labeled forms in isotope-tracing studies. Here, we explore selected ion monitoring (SIM) as a means of sensitivity enhancement. The analytes of interest are enriched in the orbitrap analyzer by using the quadrupole as a mass filter to select particular ions. In tissue extracts, SIM significantly enhances the detection of ions of low intensity, as indicated by improved signal-to-noise (S/N) ratios and measurement precision. In addition, SIM improves the accuracy of isotope-ratio measurements. SIM, however, must be deployed with care, as excessive accumulation in the orbitrap of similar m/z ions can lead, via space-charge effects, to decreased performance (signal loss, mass shift, and ion coalescence). Ion accumulation can be controlled by adjusting settings including injection time and target ion quantity. Overall, we suggest using a full scan to ensure broad metabolic coverage, in tandem with SIM, for the accurate quantitation of targeted low-intensity ions, and provide methods deploying this approach to enhance metabolome coverage. Please open README-data-for-SIM-paper-20240210.doc for details. [doi:10.25345/C5JH3DD3K] [dataset license: CC0 1.0 Universal (CC0 1.0)]

Rabinowitz, Joshua↗
DOE OSTI2024

CMOS-Based Single-Cycle in-Memory XOR/XNOR

Big data applications are on the rise, and so is the number of data centers. The ever-increasing massive data pool needs to be periodically backed up in a secure environment. Moreover, a massive amount of securely backed-up data is required for training binary convolutional neural networks for image classification. XOR and XNOR operations are essential for large-scale data copy verification, encryption, and classification algorithms. The disproportionate speed of existing compute and memory units makes the von Neumann architecture inefficient to perform these Boolean operations. Compute-in-memory (CiM) has proved to be an optimum approach for such bulk computations. The existing CiM-based XOR/XNOR techniques either require multiple cycles for computing or add to the complexity of the fabrication process. Here, we propose a CMOS-based hardware topology for single-cycle in-memory XOR/XNOR operations. Our design provides at least 2× improvement in the latency compared with other existing CMOS-compatible solutions. We verify the proposed system through circuit/system-level simulations and evaluate its robustness using a 5000-point Monte Carlo variation analysis. This all-CMOS design paves the way for practical implementation of CiM XOR/XNOR at scaled technology nodes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

SITCOMTN-165: Surface brightness profiles around massive galaxies in LSSTComCam data

Low surface brightness photometry of massive galaxies can probe the extended stellar halos and intracluster light but is often limited by background subtraction and masking systematics. Here we present radial surface brightness profiles for the brightest cluster galaxy of Abell 360 and 2dFGRS TGS323Z113 using Rubin Observatory Commissioning Camera (LSSTComCam) Data Preview 1 in the g, r, and z bands. Our results show that the LSSTComCam data can go down to ~28 mag arcsec^-2 in the r band. Comparison to DECaLS measurements processed with Legacyhalos shows good agreement of surface brightness profiles at small radii while the LSSTComCam data extend deeper at larger radii.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Machine Learning and Data Science to Advance Laboratory Earthquake Prediction and Illuminate the Mechanics of Precursors to Failure

Earthquakes represent one of our greatest natural hazards and in recent years human induced seismicity is adding to the threat. Even a modest improvement in the ability to forecast devastating large earthquakes or smaller shallow events associated with fluid injection could save thousands of lives and billions of dollars. Current efforts to forecast earthquakes are limited by knowledge of earthquake physics and hampered by a lack of reliable lab or field observations. However, recent work has provided a critical opportunity for advancement. We have found: 1) clear and consistent precursors prior to earthquake-like failure in the laboratory and 2) that lab earthquakes can be predicted using machine learning (ML). These works show that stick-slip failure events –the lab equivalent of earthquakes– are preceded by a cascade of micro-failure events that radiate elastic energy in a manner that foretells catastrophic failure. Remarkably, ML predicts the fault zone stress state, the failure time and in some cases the magnitude of lab earthquakes. In addition, the observations include clear precursors to failure in the form of changes in fault zone properties prior to lab earthquakes. Precursors have been observed in previous laboratory studies but their origin is poorly understood and their possible connection to ML based earthquake prediction is unknown. The work conducted under our project has dramatically expanded these efforts. We have developed an integrated data science approach to illuminate the physics of earthquake precursors and lab earthquake prediction. Our work has accelerated the development of ML, artificial intelligence (AI), and related data science approaches by providing massive data sets that are tightly connected to critical scientific problems and by bringing together leading subject matter experts and data scientists. Earthquake physics involves phenomena that are far from equilibrium. Our work has leveraged data science methods to illuminate these phenomena and investigate how they relate to earthquake prediction. In addition to a large database with many types of labeled events that is available to everyone, our work has advanced the fundamental understanding of seismic forecasting, earthquake physics, and fault rheology

58 GEOSCIENCES↗
DOE OSTI2022

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management platform↗
DOE OSTI2023

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data: Preprint

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management↗
DOE OSTI2020

Cybersecurity impact of the growth of data in sports

As the sporting industry continues to increase dependence on new and often massive data sets, they also risk exposing themselves, as well as the leagues, teams, venues, players, and fans, to risks posed by not protecting the confidentiality and integrity of this data. The industry, especially true for professional sports, demands an ever increasing supply of new forms of data to achieve their goals of enticing new fans; enhancing experience; expanding the fan experience from venues and television to the internet and mobile devices; measuring and evaluating the performance and safety of players; and increasing revenue. New methods have emerged to collect data about a variety of aspects of sports ranging from players to fans to teams to venues. Data collected during sporting events is combined with historical data to also help teams analyze and predict performance. To synthesize these predictive analytical and historical memories, sports teams and organizations mix a variety of computer technologies. System integration tends to focus on ensuring the availability of the data and reliability of the technologies but might introduce risks to the confidentiality (privacy) and integrity of the data. Recognizing the potential risks requires review and evaluation. Much of the data is collected for publicly providing information to the fans, so threats to confidentiality pose fewer risks. The greatest risk is violating the privacy of players, such a private performance test, or using technology to gain a competitive advantage.

Jenkins, Stephanie↗
DOE OSTI2024

Efficient distributed continual learning for steering experiments in real-time

Deep learning has emerged as a powerful method for extracting valuable information from large volumes of data. However, when new training data arrives continuously (i.e., is not fully available from the beginning), incremental training suffers from catastrophic forgetting (i.e., new patterns are reinforced at the expense of previously acquired knowledge). Training from scratch each time new training data becomes available would result in extremely long training times and massive data accumulation. Rehearsal-based continual learning has shown promise for addressing the catastrophic forgetting challenge, but research to date has not addressed performance and scalability. To fill this gap, we propose an approach based on a distributed rehearsal buffer that efficiently complements data-parallel training on multiple GPUs to achieve high accuracy, short runtime, and scalability. It leverages a set of buffers (local to each GPU) and uses several asynchronous techniques for updating these local buffers in an embarrassingly parallel fashion, all while handling the communication overheads necessary to augment input minibatches using unbiased, global sampling. We further propose a generalization of rehearsal buffers to support both classification and generative learning tasks, as well as more advanced rehearsal strategies (notably Dark Experience Replay, leveraging knowledge distillation). We illustrate this approach with a real-life HPC streaming application from the domain of ptychographic image reconstruction. Furthermore, we run extensive experiments on up to 128 GPUs of the ThetaGPU supercomputer to compare our approach with baselines representative of training-from-scratch (the upper bound in terms of accuracy) and incremental training (the lower bound). Results show that rehearsal-based continual learning achieves a top-5 validation accuracy close to the upper bound, while simultaneously exhibiting a runtime close to the lower bound.

Asynchronous data management↗