SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Distributed heterogeneous compute infrastructure for the study of additive manufacturing systems

We present the current status of a large-scale computing framework to address the need of the multidisciplinary effort to study chemical dynamics. Specifically, we are enabling scientists to process and store experimental data, run large-scale computationally expensive high-fidelity physical simulation, and analyze these results using the state-of-the-art data analytics tools, machine learning, and uncertainty quantification methods using heterogeneous computing resources, such as CPU and GPU cluster. The framework can integrate or abstract out multiple domains based on roles. In order to develop this framework, we have leveraged an existing framework coupled with in-house heterogeneous computing resources. We present the results of using this framework on a single metadata triggered workflow to accelerate an additive manufacturing use case.

heterogeneous computing, hpc, workflows, reproduci↗

DOE OSTI2023

Scalable Volume Visualization for Big Scientific Data Modeled by Functional Approximation

Considering the challenges posed by the space and time complexities in handling extensive scientific volumetric data, various data representations have been developed for the analysis of large-scale scientific data. Multivariate functional approximation (MFA) is an innovative data model designed to tackle substantial challenges in scientific data analysis. It computes values and derivatives with high-order accuracy throughout the spatial domain, mitigating artifacts associated with zero- or first-order interpolation. However, the slow query time through MFA makes it less suitable for interactively visualizing a large MFA model. In this work, we develop the first scalable interactive volume visualization pipeline, MFA-DVV, for the MFA model encoded from large-scale datasets. Our method achieves low input latency through distributed architecture, and its performance can be further enhanced by utilizing a compressed MFA model while still maintaining a high-quality rendering result for scientific datasets. We conduct comprehensive experiments to show that MFA-DVV can decrease the input latency and achieve superior visualization results for big scientific data compared with existing approaches.

big scientific dataset↗

DOE OSTI2022

Differentially Private K -Means Clustering Applied to Meter Data Analysis and Synthesis

The proliferation of smart meters has resulted in a large amount of data being generated. It is increasingly apparent that methods are required for allowing a variety of stakeholders to leverage the data in a manner that preserves the privacy of the consumers. The sector is scrambling to define policies, such as the so called ‘15/15 rule’, to respond to the need. However, the current policies fail to adequately guarantee privacy. Here, in this paper, we address the problem of allowing third parties to apply K-means clustering, obtaining customer labels and centroids for a set of load time series by applying the framework of differential privacy. We leverage the method to design an algorithm that generates differentially private synthetic load data consistent with the labeled data. We test our algorithm’s utility by answering summary statistics such as average daily load profiles for a 2-dimensional synthetic dataset and a real-world power load dataset.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2024

Assessing and benchmarking the fidelity of posterior inference methods for astrophysics data analysis

In this era of large and complex astronomical survey data, interpreting, validating, and comparing inference techniques becomes increasingly difficult. This is particularly critical for emerging inference methods like Simulation-Based Inference (SBI), which offer significant speedup potential and posterior modeling flexibility, especially when deep learning is incorporated. We present a study to assess and compare the performance and uncertainty prediction capability of Bayesian inference algorithms – from traditional MCMC sampling of analytic functions to deep learning-enabled SBI. We focus on testing the capacity of hierarchical inference modeling in those scenarios. Before we extend this study to cosmology, we first use astrophysical simulation data to ensure interpretability. We demonstrate a probabilistic programming implementation of hierarchical and non-hierarchical Bayesian inference using simulations derived from the DeepBench software library, a benchmarking tool developed by our group that generates simple and controllable astrophysical objects from first principles. This study will enable astronomers and physicists to harness the inference potential of these methods with confidence.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

Overview of the distributed image processing infrastructure to produce the Legacy Survey of Space and Time

The Vera C. Rubin Observatory is preparing to execute the most ambitious astronomical survey ever attempted, the Legacy Survey of Space and Time (LSST). Currently the final phase of construction is under way in the Chilean Andes, with the Observatory’s ten-year science mission scheduled to begin in 2025. Rubin’s 8.4-meter telescope will nightly scan the southern hemisphere collecting imagery in the wavelength range 320–1050 nm covering the entire observable sky every 4 nights using a 3.2 gigapixel camera, the largest imaging device ever built for astronomy. Automated detection and classification of celestial objects will be performed by sophisticated algorithms on high-resolution images to progressively produce an astronomical catalog eventually composed of 20 billion galaxies and 17 billion stars and their associated physical properties. In this article we present an overview of the system currently being constructed to perform data distribution as well as the annual campaigns which reprocess the entire image dataset collected since the beginning of the survey. These processing campaigns will utilize computing and storage resources provided by three Rubin data facilities (one in the US and two in Europe). Each year a Data Release will be produced and disseminated to science collaborations for use in studies comprising four main science pillars: probing dark matter and dark energy, taking inventory of solar system objects, exploring the transient optical sky and mapping the Milky Way. Also presented is the method by which we leverage some of the common tools and best practices used for management of large-scale distributed data processing projects in the high energy physics and astronomy communities. We also demonstrate how these tools and practices are utilized within the Rubin project in order to overcome the specific challenges faced by the Observatory.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Mitigate: An Adaptive Network Data Anonymization Tool Using Condensation-Based Differential Privacy

Modern network devices collect a large amount of data that can be analyzed to identify bottlenecks, anomalies, cyber-attacks, etc. Therefore, there is often a need to analyze such collections of network data quite often by an external expert or by the research community. However, these collections of data contain sensitive, proprietary information. In order for the network data to be shared, it must first be anonymized. The overall objective of this project is to develop an innovative privacy management tool to anonymize network data and achieve sufficient privacy, acceptable data utility, and efficient data analysis at the same time. No existing anonymization methods can achieve all of these at the same time. The core of this technology is a differential private clustering algorithm that provides strong privacy protection, preserves data properties important for subsequent analysis, and allows the party receiving the anonymized data to conduct analysis directly on anonymized data without the need of decryption or any extra processing. The research carried out was to design, implement and verify a solution to this problem by completing the following tasks: 1) developing the core technology; 2) developing a context based method that automatically recommends fields that must be anonymized; 3) conducted experiments showing superior results using our approach compared to existing tools, and 4) developed an intuitive but basic user interface. The research that was conducted generated novel algorithmic techniques that utilize state-of-the-art methods such as condensation, differential privacy preservation, clustering, automated tuning based on contextual awareness, and recommendation techniques to specify columns to users for anonymization leading to optimal privacy that allows research analysis on the dataset. Experiments were conducted to evaluate the efficacy of these novel algorithmic techniques by performing analysis on original non-anonymized datasets, then conducting analysis on the same yet anonymized datasets and comparing the results of the analyses. Overall, the anonymized analysis results were within 1% of the original results, verifying that the generated technology not only guarantees a high level of privacy but also enables research analysis as if it were conducted on the original dataset. Potential applications of this technology include anonymization of any type of structured network datasets that contain sensitive identifiers, such as IP addresses, that can be used in multiple applications. For example, to create an AI or machine learning model for cyber security, e.g., to detect attacks, or for performance analysis, e.g., identify bottlenecks or predict performance. In addition, a market analysis that was conducted for potential applications of this technology identified a broader range of applications of our anonymization technology beyond the network sector that includes healthcare, banking, insurance, securities, finance (FISB), data brokering, cloud services, ad sales, and government.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Analytical estimation of the signal to noise ratio efficiency in axion dark matter searches using a Savitzky-Golay filter

The signal to noise ratio efficiency ϵ SNR in axion dark matter searches has been estimated using large-statistic simulation data reflecting the background information and the expected axion signal power obtained from a real experiment. This usually requires a lot of computing time even with the assistance of powerful computing resources. Employing a Savitzky-Golay filter for background subtraction, in this work, we estimated a fully analytical ϵ SNR without relying on large-statistic simulation data, but only with an arbitrary axion mass and the relevant signal shape information. Hence, our work can provide ϵ SNR using minimal computing time and resources prior to the acquisition of experimental data, without the detailed information that has to be obtained from real experiments. Axion haloscope searches have been observing the coincidence that the frequency independent scale factor ξ is approximately consistent with the ϵ SNR . This was confirmed analytically in this work, when the window length of the Savitzky-Golay filter is reasonably wide enough, i.e., at least 5 times the signal window.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2023

Data preservation in high energy physics

Data preservation is a mandatory specification for any present and future experimental facility and it is a cost-effective way of doing fundamental research by exploiting unique data sets in the light of the continuously increasing theoretical understanding. This document summarizes the status of data preservation in high energy physics. The paradigms and the methodological advances are discussed from a perspective of more than ten years of experience with a structured effort at international level. The status and the scientific return related to the preservation of data accumulated at large collider experiments are presented, together with an account of ongoing efforts to ensure long-term analysis capabilities for ongoing and future experiments. Transverse projects aimed at generic solutions, most of which are specifically inspired by open science and FAIR principles, are presented as well. A prospective and an action plan are also indicated.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2023

Machine learning from RANS and LES to inform coarse grid simulations

Nuclear system thermal hydraulic analysis has historically relied on computationally inexpensive 1D codes. However, such tools are unable to capture multiscale multidimensional effects in large nuclear reactor enclosures. On the other hand, simulations with higher fidelity can be too expensive for such purposes. One of the ways to reduce computational cost is to perform simulations on a coarse grid, which, unfortunately, introduces large discretization errors. In this paper, two high-to-low data-driven approaches are investigated: (1) a coarse grid turbulence model to predict eddy viscosity and (2) correction of errors in coarse grid velocity fields. The approaches aim to reduce grid- and turbulence model-induced errors in coarse grid Reynolds-averaged Navier–Stokes (RANS) simulations. Two sources of high-fidelity data, RANS and large eddy simulations (LES), are explored. To extract the eddy viscosity from the LES data, an inverse optimization problem is solved. However, the LES eddy viscosity is shown to be comparable to the RANS eddy viscosity in terms of error reduction. Therefore, the directly available RANS eddy viscosity was used to develop a coarse grid data-driven turbulence model. Additionally, error correction in velocity is used to reduce the remaining uncertainties and bring the results closer to reality. In conclusion, the performance of the frameworks is demonstrated for a scaled upper plenum of a gas-cooled reactor facility.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2022

An open science exploration of global 1-km simulations of the earth's atmosphere

We are making available a multi-petabyte data set for open science via a data hackathon. The participants bring their science to the computing and data facility, alleviating the need to transfer and manage the large volume of data. This accelerates the process of discovery science while increasing scientific throughput.

Anantharaj, Valentine↗

DOE OSTI2021

PA-BCI Meteorological dataset 1985-2018

This data package contains a QA/QC-ed meteorological time series dataset from 1985 to 2018, including relative humidity, temperature, solar radiation, rainfall, wind speed, and barometric pressure at Barro Colorado Island (PA-BCI) in Panama. The data were collected at the elevations of 42 m and 48 m at the BCI Lutz station, except for rainfall that was collected at the BCI Clearing station at the elevation of 1 m. Individual time series datasets were collected using electro-mechanical (electronic sensors, data loggers, chart recorders, etc.). The rainfall dataset was collected using the tipping bucket method, and the data could have been underestimated by 13-17% (likely the large values). The data is included as a single CSV file.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Uinta Basin CarbonSAFE II: Storage Complex Feasibility (Final Report)

The primary objective of this CarbonSAFE Phase II project was to establish the technical and commercial feasibility of a commercial-scale CO 2 geological storage complex for Deseret Power Electric Cooperative Bonanza Power Plant and other CO 2 sources in the northeast Uinta Basin, Utah, with the goal to securely store at least 50 million metric tons of captured CO 2 and accelerate CO 2 capture, utilization, and storage (CCUS) deployment. The project team established high-potential technical and commercial feasibility for a storage site within the east Uinta Basin (Utah), in the Cretaceous sandstones (Frontier, Dakota, and Buckhorn), Entrada Sandstone, Nugget Sandstone, and/or Weber Sandstone southwest of the Bonanza coal-fired power plant. This project collected and analyzed state-of-the-art data to characterize the storage complex consistent with Environmental Protection Agency (EPA) permitting standards. The team conducted extensive analog studies, outcrop mapping, and data sampling, which largely contributed to understanding the subsurface lithology and facies. Existing data were obtained and assessed from Utah Division of Oil, Gas, and Mining (DOGM), Utah Geological Survey (UGS), Colorado Geological Survey (CGS), U.S. Geological Survey (USGS), and EPA. These data were analyzed using state-of-the-art CCUS technologies for Societal Considerations, Site Characterization, Modeling and Simulations, Risk Assessment, Management and Monitoring, potential Underground Injection Control (UIC) Class VI Well Permitting, and Technical/Economic Feasibility. Through these high-resolution data collection and feasibility studies, this project was expected to provide a reference for initiating Underground Injection Control (UIC) and other commercial-scale geological storage permitting processes in the Western United States, ultimately contributing to the nation's decarbonization goals through low-risk, cost-effective commercial-scale carbon capture, utilization, and storage (CCUS) projects.

42 ENGINEERING↗

DOE OSTI2025

Data Quality Monitoring for the Hadron Calorimeters Using Transfer Learning for Anomaly Detection

The proliferation of sensors brings an immense volume of spatio-temporal (ST) data in many domains, including monitoring, diagnostics, and prognostics applications. Data curation is a time-consuming process for a large volume of data, making it challenging and expensive to deploy data analytics platforms in new environments. Transfer learning (TL) mechanisms promise to mitigate data sparsity and model complexity by utilizing pre-trained models for a new task. Despite the triumph of TL in fields like computer vision and natural language processing, efforts on complex ST models for anomaly detection (AD) applications are limited. In this study, we present the potential of TL within the context of high-dimensional ST AD with a hybrid autoencoder architecture, incorporating convolutional, graph, and recurrent neural networks. Motivated by the need for improved model accuracy and robustness, particularly in scenarios with limited training data on systems with thousands of sensors, this research investigates the transferability of models trained on different sections of the Hadron Calorimeter of the Compact Muon Solenoid experiment at CERN. The key contributions of the study include exploring TL’s potential and limitations within the context of encoder and decoder networks, revealing insights into model initialization and training configurations that enhance performance while substantially reducing trainable parameters and mitigating data contamination effects.

47 OTHER INSTRUMENTATION↗

DOE OSTI2023

Electrical Infrastructure Cost Model for Marine Energy Systems

The National Renewable Energy Laboratory's Electrical Infrastructure Cost Model is an Excel-based tool designed to estimate the electrical infrastructure costs of marine energy components and subsystems. It incorporates data collected from offshore wind projects, utility projects, and other relevant sources to provide accurate and comprehensive cost projections. With its user-friendly interface, the model allows users to input various parameters related to the system array, electrical cables, and substations. By leveraging industry data, cost trends, and technological advancements, the model generates outputs that include system array sizing, electrical cable specifications and costs, substation specifications and costs, and total electrical infrastructure costs. One of the notable strengths of the model is its flexibility in covering multiple-orders-of-magnitude scaled systems, accommodating projects ranging from proof-of-concept or pilot-scale installations to large-scale offshore systems. By collecting data largely from offshore wind reports and utility projects, the model incorporates real-world conditions and accounts for industry-specific factors. It incorporates cost trends and sizing relationships to deliver cost estimations for electrical infrastructure components, such as electrical cables and substation equipment.

16 TIDAL AND WAVE POWER↗

DOE OSTI2026

Atacama Cosmology Telescope: B-mode delensing with DR6 data and external tracers of large-scale structure

Large-scale -mode polarization of the cosmic microwave background (CMB) is a prime target for current and future experiments in search of primordial gravitational waves. With increasingly sensitive instruments being deployed, secondary -modes induced by the weak gravitational lensing of CMB photons are becoming an important limitation and need to be removed, a process known as delensing. In this work, we combine internally reconstructed CMB lensing maps from the Atacama Cosmology Telescope (ACT) data release 6 (DR6) with galaxy samples from unWISE and a map of the cosmic infrared background (CIB) fluctuations from Planck to produce a well-correlated tracer of the CMB lensing field. Our coadded tracer, shown to be 55%–85% correlated with the true lensing convergence at multipoles , is then convolved with ACT DR6 -mode polarization to yield a template of the lensing -modes. We assess its performance on a wide range of scales by using it to delens ACT DR6 and Planck -modes over 23% of the sky, removing around 39% of the lensing power at and 47% at , respectively. Our template achieves the highest delensing efficiency to date and will be useful for the analysis of early polarization maps from the Simons Observatory. We finally outline prospects for further improvements by including additional large-scale structure tracers from upcoming cosmological surveys.

Hertig, Emilie↗

DOE OSTI2023

Classical and quantum compression for edge computing: the ubiquitous data dimensionality reduction

Edge computing aims to address the challenges associated with communicating and transferring large amounts of data generated remotely to a data center in a timely and efficient manner. A central pillar of edge computing is local (i.e., at- or near-source) data processing capability so that data transfer to a data center for processing can be minimized. Data compression at the edge is therefore a natural component of edge workflows. Here we present a survey of data compression algorithms with a focus on edge computing. Not all compression algorithms can accommodate the data type heterogeneity, tight processing and communication time constraints, or energy efficiency requirement characteristics of edge computing. We discuss specific examples of compression algorithms that are being explored in the context of edge computing. We end our review with a brief survey of emerging quantum compression techniques that are of importance in quantum information processing, including the proposed concept of quantum edge computing.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data With Spatial Information

Efficient processing of large-scale time series data is an intricate problem in machine learning. Conventional sensor signal processing pipelines with hand engineered feature extraction often involve huge computational cost with high dimensional data. Deep recurrent neural networks have shown promise in automated feature learning for improved time-series processing. However, generic deep recurrent models grow in scale and depth with increased complexity of the data. This is particularly challenging in presence of high dimensional data with temporal and spatial characteristics. Consequently, this work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to efficiently process complex multi-dimensional time series data with spatial information. Here, the cellular recurrent architecture in the proposed model allows for location-aware synchronous processing of time series data from spatially distributed sensor signal sources. Extensive trainable parameter sharing due to cellularity in the proposed architecture ensures efficiency in the use of recurrent processing units with high-dimensional inputs. This study also investigates the versatility of the proposed DCRNN model for classification of multi-class time series data from different application domains. Consequently, the proposed DCRNN architecture is evaluated using two time-series datasets: a multichannel scalp EEG dataset for seizure detection, and a machine fault detection dataset obtained in-house. The results suggest that the proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2026

Speed to Power: Solutions for Accelerating Large Load Connections

Rapid growth in demand from data centers and other large loads is creating a range of new challenges for electricity planners, investors, system operators, and regulators, leading to bottlenecks that have slowed connection of large loads to the electric grid. In response, innovative solutions for accelerating large load connections are beginning to emerge across the U.S. Drawing on an extensive document and literature review, this report identifies more than 40 potential solutions for accelerating large load connections, organized into five functional areas: load forecasting, interconnection, resource planning and procurement, markets and operations, and cost allocation and ratemaking. The five functional areas provide a framework for organizing challenges and solutions to large load connection bottlenecks.

24 POWER TRANSMISSION AND DISTRIBUTION↗