Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Data Analysis Approach for Large Data Volumes in a Connected Community

Recent advancements within smart neighborhoods where utilities are enabling automatic control of appliances such as heating, ventilation, and air conditioning (HVAC) and water heater (WH) systems are providing new opportunities to minimize energy costs through reduced peak load. This requires systematic collection, storage, management, and in-memory processing of large volumes of streaming data for fast performance. In this paper, we propose a multi-tier layered IoT software framework that enables effective descriptive and predictive data analysis for understanding live operation of the neighborhood, fault identification, and future opportunities for further optimization of load curves. We then demonstrate how we achieve live situational awareness of the connected neighborhood through a suite of visualization components. Finally, we discuss a few analytic dashboards that address questions such as peak load reductions obtained due to optimization, customer preference for automatic control of appliances (do they override the automatic control of HVAC?, etc.). 1 1 This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Chinthavali, Supriya↗
DOE OSTI2026

Electromagnetic Transient Modeling of Large Data Centers for Grid-Level Studies

The magnitude and complexity of electricity usage patterns from large data centers are having significant impacts on the operation and dynamics of the power grid; grid operators and planners require a range of specialized data center models to properly evaluate these impacts and specify technical solutions as needed. Towards addressing this need, Pacific Northwest National Laboratory (PNNL) has developed a library of electromagnetic transient (EMT) models for grid-level studies of data centers called the data center model library (DML). This report describes how the DML was created and how it may properly be used. The models present in the DML are generic models; subject matter expertise and additional technical data are needed to modify these models before they can represent any real data center. However, they will significantly reduce the level of effort required to develop site-specific models and can serve as a common starting point to guide industry towards a more refined consensus. Most of the models within DML are dedicated to representing the power electronics interfaces commonly used in modern data centers, such as double-conversion uninterruptible power supplies and single-phase power factor correction converters. These models are intended for use in grid-level studies and are a simplified aggregation of many small components. That said, background material on the physical and electrical design of large data centers is provided as companion material so that users can be aware of many of the details which have been omitted or streamlined as a matter of practical necessity. Additionally, guidance on the application of EMT analysis for data center interconnection studies is provided, which aids users in identifying when the DML is necessary and what sort of additional model development may be necessary for conducting real-world studies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Electromagnetic Transient Modeling of Large Data Centers for Grid-Level Studies: Beta Release

The magnitude and complexity of electricity usage patterns from large data centers are having significant impacts on the operation and dynamics of the power grid; grid operators and planners require a range of specialized data center models to properly evaluate these impacts and specify technical solutions as needed. Towards addressing this need, Pacific Northwest National Laboratory (PNNL) has developed a library of electromagnetic transient (EMT) models for grid-level studies of data centers called the data center model library (DML). This report describes how the DML was created and how it may properly be used. This report details the DML’s beta release, completed in July 2026. This is a revision and expansion of the alpha release, which was made available in January 2026 The models present in the DML are generic models; subject matter expertise and additional technical data are needed to modify these models before they can represent any real data center. However, they will significantly reduce the level of effort required to develop site-specific models and can serve as a common starting point to guide industry towards a more refined consensus. Most of the models within DML are dedicated to representing the power electronics interfaces commonly used in modern data centers, such as double-conversion uninterruptible power supplies and single-phase power factor correction converters. These models are intended for use in grid-level studies and are a simplified aggregation of many small components. That said, background material on the physical and electrical design of large data centers is provided as companion material so that users can be aware of many of the details which have been omitted or streamlined as a matter of practical necessity. Additionally, guidance on the application of EMT analysis for data center interconnection studies is provided, which aids users in identifying when the DML is necessary and what sort of additional model development may be necessary for conducting real-world studies.

electromagnetic transients↗
DOE OSTI2022

Organizing Large Data Sets for Efficient Analyses on HPC Systems

Upcoming exascale applications could introduce significant data management challenges due to their large sizes, dynamic work distribution, and involvement of accelerators such as graphical processing units, GPUs. In this work, we explore the performance of reading and writing operations involving one such scientific application on two different supercomputers. Our tests showed that the Adaptable Input and Output System, ADIOS, was able to achieve speeds over 1TB/s, a significant fraction of the peak I/O performance on Summit. We also demonstrated the querying functionality in ADIOS could effectively support common selective data analysis operations, such as conditional histograms. In tests, this query mechanism was able to reduce the execution time by a factor of five. More importantly, ADIOS data management framework allows us to achieve these performance improvements with only a minimal amount of coding effort.

Gu, Junmin↗
DOE OSTI2025

Quantifying Distribution System Resilience From Utility Data: Large Event Risk and Benefits of Investments

We focus on blackouts in electric distribution systems that have a large cost to customers. To quantify resilience to these events, we show how to calculate risk metrics from the historical outage data routinely collected by utilities' outage management systems. Risk is defined using a customer cost exceedance curve. The exceedance curve has a heavy tail that implies large fluctuations in large blackout costs, and this makes estimating the mean large cost in the usual way impractical. To avoid this problem, we use new resilience metrics describing the large event risk; these metrics are the probability of a large cost event, the annual log cost resilience index, and the average of the logarithm of the cost of large-cost events or the slope magnitude of the tail on a log–log exceedance curve. Resilience can be improved by planned investments to upgrade system components or speed up restoration. The benefits that these investments would have had if they had been made in the past can be quantified by “rerunning history” with the effects of the investment included, and then recalculating the large event risk to find the improvement in resilience. An example using utility data shows a 2% reduction in the probability of a large cost event due to 10% wind hardening and 6%–7% reduction due to 10% faster restoration in two different areas of a distribution utility. This new data-driven approach to quantify resilience and resilience investments is realistic and much easier to apply than complicated approaches based on modeling all the phases of resilience. Moreover, an appeal to improvements to past lived experience may well be persuasive to customers and regulators in making the case for resilience investments.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

System and method of managing large data files

Disclosed are systems and software that provide a high-performance, extensible file format and web API for remote data access and a visual interface for data viewing, query, and analysis. The described system can support storage of raw spectroscopic data such as neural recording data, MSI data, metadata, and derived analyses in a single, self-describing format that may be compatible by a large range of analysis software.

Bowen, Benjamin P.↗
DOE OSTI2022

Large-Data Omics Approaches in Modern Remediation

Forum papers are thought-provoking opinion pieces or essays founded in fact, sometimes containing speculation, on a civil engineering topic of general interest and relevance to the readership of the journal. The views expressed in this Forum article do not necessarily reflect the views of ASCE or the Editorial Board of the journal.

42 ENGINEERING↗
DOE OSTI2025

Using Large Language Models to help customers monitor global threat data

Large Language Models have proven adept at answering general knowledge questions. To make these generative AI tools useful to our mission customers for monitoring global threats, the data sciences team at Sandia is utilizing retrieval augmented generation (RAG) techniques to customize these models with local data. The local data we use consists of data such as research articles and patent abstracts that we've collected over the last several years using automated pipelines.

Herzer, John Andrew [Sandia National Laboratories ↗
DOE OSTI2020

Putting Our Industry's Data to Work: A Case Study of Large-Scale Data Aggregation: Preprint

With increasing deployment of Advanced Metering Infrastructure (AMI), Building Automation System (BAS) controls, Internet of Things (IoT) network devices, and data-driven evaluation, measurement, and verification studies, the building sector is currently generating a staggering amount of energy-related data. In the right hands, these data sets can contribute to increased comfort and energy savings for building occupants and a more reliable electrical grid; however, due to a combination of factors, including significant privacy concerns, much of the data that are presently generated and stored are not used outside of basic operational applications. In the past year, our team has dedicated over 2,000 person-hours to accessing building energy data for a project funded by the U.S. Department of Energy (DOE) Building Technologies Office. We sought whole-building or end-use (e.g., lighting) timeseries data at the individual-building or equipment level where possible or aggregated information, such as timeseries averages and quartiles by building type (e.g., office, retail, hospital), where sharing individual building information was not an option. We are additionally working with IoT and BAS data sets to derive information important to the project. We have assembled an extensive data set that will enable the development of publicly available end-use load profiles to benefit the U.S. building and electricity industries. Here we present an overview of the data set that we have assembled to date, the motivators and approaches that got us here, and the lessons we learned through our efforts. We also discuss work underway that presents additional options for future data access.

building energy data↗
DOE OSTI2021

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

The global nature of the ITER project along with its projected ~ petabyte per day data generation presents a unique challenge, but also an opportunity for the fusion community to rethink, optimize, and enhance our scientific discovery process. Recognizing this, collaborative research with computational scientists was undertaken over the past several years to create a framework for large-scale data movement across wide-area networks (WANs), to enable global near-real time analysis of fusion data. This would broaden the available computational resources for analysis/simulation, and increase the number of researchers actively participating in experiments. An official demonstration of this framework for fast, large data transfer and real-time analysis was carried out between the KSTAR tokamak in Daejeon, Korea and PPPL in Princeton, USA. Streaming large data transfer, with near real-time movie creation and analysis of the KSTAR Electron Cyclotron Emission imaging (ECEI) data, was performed using the I/O framework ADIOS, and comparisons made at PPPL with simulation results from the XGC1 code. These demonstrations were made possible utilizing an optimized network configuration at PPPL, which achieved over 8.8 Gbps (88% utilization) in throughput tests from NFRI to PPPL. This demonstration showed the feasibility for large-scale data analysis of KSTAR data, and provides a nascent framework to enable use of globally distributed computational and personnel resources in pursuit of scientific knowledge from the ITER experiment.

43 PARTICLE ACCELERATORS↗
DOE OSTI2022

Physics‐Informed Machine Learning Method for Large‐Scale Data Assimilation Problems

Abstract We develop a physics‐informed machine learning approach for large‐scale data assimilation and parameter estimation and apply it for estimating transmissivity and hydraulic head in the two‐dimensional steady‐state subsurface flow model of the Hanford Site given synthetic measurements of said variables. In our approach, we extend the physics‐informed conditional Karhunen‐Loéve expansion (PICKLE) method to modeling subsurface flow with unknown flux (Neumann) and varying head (time‐dependent Dirichlet) boundary conditions. We demonstrate that the PICKLE method is comparable in accuracy with the standard maximum a posteriori (MAP) method, but is significantly faster than MAP for large‐scale problems. Both methods use a mesh to discretize the computational domain. In MAP, the parameters and states are discretized on the mesh; therefore, the size of the MAP parameter estimation problem directly depends on the mesh size. In PICKLE, the mesh is used to evaluate the residuals of the governing equation, while the parameters and states are approximated by the truncated conditional Karhunen‐Loéve expansions with the number of parameters controlled by the smoothness of the parameter and state fields, and not by the mesh size. For a considered example, we demonstrate that the computational cost of PICKLE increases near linearly (as N 1.15 ) with the number of grid nodes N , while that of MAP increases much faster (as N 3.28 ). We also show that once trained for one set of Dirichlet boundary conditions (i.e., one river stage), the PICKLE method provides accurate estimates of the hydraulic head for any value of the Dirichlet boundary conditions (i.e., for any river stage).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

A semiparametric latent factor model for large scale temporal data with heteroscedasticity

Large scale temporal data have flourished in a vast array of applications, and their sophisticated structures, especially the heteroscedasticity among subjects with inter- and intra-temporal dependence, have fueled a great demand for new statistical models. In this paper, with covariate information, we consider a flexible model for large scale temporal data with subject-specific heteroscedasticity. Formally, the model employs latent semiparametric factors to simultaneously account for the subject-specific heteroscedasticity and the contemporaneous and/or serial correlations. The subject-specific heteroscedasticity is modeled as the product of the unobserved factor process and subject’s covariate effect, which is further characterized via additive models. For estimation, we propose a two-step procedure. First, the latent factor process and nonparametric loading are recovered through projection-based methods, and following, we estimate the regression components by approaches motivated from the generalized least squares. By scrupulously examining the non-asymptotic rates for recovering the factor process and its loading, we show the consistency and efficiency of estimated regression coefficients in the absence of prior knowledge of latent factor process and subject’s covariate effect. Here, the statistical guarantees remain valid even for finite time points that makes our method particularly appealing when the subjects significantly outnumber the observation time points. Using comprehensive simulations, we demonstrate the finite sample performance of our method, which corroborates the theoretical findings. Finally, we apply our method to a data set of air quality and energy consumption collected at 129 monitoring sites in the United States in 2015.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Design Considerations and Analysis of Multi-Level Erasure Coding in Large-Scale Data Centers

Multi-level erasure coding (MLEC) has seen large deployments in the field, but there is no in-depth study of design considerations for MLEC at scale. In this paper, we provide comprehensive design considerations and analysis of MLEC at scale. We introduce the design space of MLEC in multiple dimensions, including various code parameter selections, chunk placement schemes, and various repair methods. We quantify their performance and durability, and show which MLEC schemes and repair methods can provide the best tolerance against independent/correlated failures and reduce repair network traffic by orders of magnitude. To achieve this, we use various evaluation strategies including simulation, splitting, dynamic programming, and mathematical modeling. We also compare the performance and durability of MLEC with other EC schemes such as SLEC and LRC and show that MLEC can provide high durability with higher encoding throughput and less repair network traffic over both SLEC and LRC.

Wang, Meng↗
DOE OSTI2021

The Atacama Cosmology Telescope: Summary of DR4 and DR5 Data Products and Data Access

Two recent large data releases for the Atacama Cosmology Telescope (ACT), called DR4 and DR5, are available for public access. These data include temperature and polarization maps that cover nearly half the sky at arcminute resolution in three frequency bands; lensing maps and component-separated maps covering ~ 2,100 deg 2 of sky; derived power spectra and cosmological likelihoods; a catalog of over 4,000 galaxy clusters; and supporting ancillary products including beam functions and masks. The data and products are described in a suite of ACT papers; here we provide a summary. In order to facilitate ease of access to these data we present a set of Jupyter IPython notebooks developed to introduce users to DR4, DR5, and the tools needed to analyze these data. The data products (excluding simulations) and the set of notebooks are publicly available on the NASA Legacy Archive for Microwave Background Data Analysis (LAMBDA); simulation products are available on the National Energy Research Scientific Computing Center (NERSC).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Hydrogen-Based Energy Storage Systems for Large-Scale Data Center Applications

Global demand for data and data access has spurred the rapid growth of the data center industry. To meet demands, data centers must provide uninterrupted service even during the loss of primary power. Service providers seeking ways to eliminate their carbon footprint are increasingly looking to clean and sustainable energy solutions, such as hydrogen technologies, as alternatives to traditional backup generators. In this viewpoint, a survey of the current state of data centers and hydrogen-based technologies is provided along with a discussion of the hydrogen storage and infrastructure requirements needed for large-scale backup power applications at data centers.

08 HYDROGEN↗