SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Planck intermediate results

We present the NPIPE processing pipeline, which produces calibrated frequency maps in temperature and polarization from data from the Planck Low Frequency Instrument (LFI) and High Frequency Instrument (HFI) using high-performance computers. NPIPE represents a natural evolution of previous Planck analysis efforts, and combines some of the most powerful features of the separate LFI and HFI analysis pipelines. For example, following the LFI 2018 processing procedure, NPIPE uses foreground polarization priors during the calibration stage in order to break scanning-induced degeneracies. Similarly, NPIPE employs the HFI 2018 time-domain processing methodology to correct for bandpass mismatch at all frequencies. In addition, NPIPE introduces several improvements, including, but not limited to: inclusion of the 8% of data collected during repointing manoeuvres; smoothing of the LFI reference load data streams; in-flight estimation of detector polarization parameters; and construction of maximally independent detector-set split maps. For component-separation purposes, important improvements include: maps that retain the CMB Solar dipole, allowing for high-precision relative calibration in higher-level analyses; well-defined single-detector maps, allowing for robust CO extraction; and HFI temperature maps between 217 and 857 GHz that are binned into 0'.9 pixels (Nside = 4096), ensuring that the full angular information in the data is represented in the maps even at the highest Planck resolutions. The net effect of these improvements is lower levels of noise and systematics in both frequency and component maps at essentially all angular scales, as well as notably improved internal consistency between the various frequency channels. Based on the NPIPE maps, we present the first estimate of the Solar dipole determined through component separation across all nine Planck frequencies. The amplitude is (3366.6 ± 2.7) μK, consistent with, albeit slightly higher than, earlier estimates. From the large-scale polarization data, we derive an updated estimate of the optical depth of reionization of τ = 0.051 ± 0.006, which appears robust with respect to data and sky cuts. There are 600 complete signal, noise and systematics simulations of the full-frequency and detector-set maps. As a Planck first, these simulations include full time-domain processing of the beam-convolved CMB anisotropies. The release of NPIPE maps and simulations is accompanied with a complete suite of raw and processed time-ordered data and the software, scripts, auxiliary data, and parameter files needed to improve further on the analysis and to run matching simulations.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

Using Machine Learning to Predict Future Temperature Outputs in Geothermal Systems

Optimizing the power output, and economic value, of geothermal power plants over decades of operation is a major challenge in renewable energy. Optimizing the output requires the ability to predict the mass flow rates and the output temperatures of production wells based on the inputs of injection wells, as well as the time history of the system. Machine Learning (ML) that incorporates the known physics of geothermal systems is one possible solution to this challenge. In this work, we explore the ability of ML algorithms to predict future temperature outputs based on historical data. Considering the challenges with obtaining an empirical dataset from field data that is large enough to enable reliable ML, we propose an alternate approach: developing a high-fidelity reservoir model and using computational resources to build a dataset that enables ML. As a first step towards achieving this goal, we present preliminary results from applying ML to predict the temperature timeseries of simple modeled geothermal systems. We describe the application of relevant state-of-the-art ML approaches, such as the Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNN), to extract temporal structures in the model data. We assess the accuracy of the forecasts we obtain, compare the selected approaches, and share the lessons learned that would inform the process of training and utilizing ML algorithms for larger and more complex geothermal systems.

GEOTHERMAL ENERGY↗

DOE OSTI2020

Accelerate M-TIP on GPUs and deploy to Summit and NERSC-9 (against simulated data) WBS 2.2.4.05 ExaFEL, Milestone ADSE13-199

We evaluated the orientation matching step in the M-TIP SPI workflow for potential offloading to accelerators. We ported the code to GPUs, benchmarked it, optimized and down-selected the best versions. The accelerated version of the orientation matching code that was developed at LANL (LANL GPU v3) is 34-55X faster than sequential, 2.4-4.9X faster than the fastest OpenMP open source version we found (FAISS OpenMP) and 1.5-4X faster than the fastest GPU open source version we found (FAISS GPU). Summit single-node GPU versions were somewhat faster than Cori GPU. Image size plays a role; mid-range image sizes take more time. The LANL CUDA multi-node, multi-GPU implementation shows mostly linear strong scaling. I/O also plays a large role; splitting data into parts improves read time and burst buffers dramatically improve read times. This work will be integrated into the M-TIP workflow as part of the next milestone ADSE13-193.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Final Technical Report for "Cloud-based Low-Scaling Quantum Chemistry Simulations for Materials"

The objective of phase I was to develop the basic infrastructure for performing efficient hybrid DFT calculations on solid materials with a turnaround time amenable to the use in large-scale data-driven approaches. To fulfill the goal we have developed a novel algorithm that significantly reduces the cost of exchange matrix formation. The algorithm does so by making use of three ingredients (a) interpolative decomposition of the electron integrals (b) robust pseudospectral method and (c) occ-RI approach. Our published work demonstrates that the algorithm is orders of magnitude faster than any other hybrid-DFT method and is on the order of only 3-4 times slower than pure DFT. All steps of the algorithm developed during phase I can be accelerated to the point that, for large enough systems, the diagonalization of the Fock matrix will become the most expensive step. To treat such large systems we have created an interface to the ASCR-funded PEXSI library. The PEXSI method is used to compute the density matrix directly from the Fock matrix in a manner that preserves the sparsity of the local representation.

Shiozaki, Toru↗

DOE OSTI2022

Development of lean, efficient, and fast physics-framed deep-learning-based proxy models for subsurface carbon storage

In this work, we present deep-learning-based surrogate models for CCUS developed with four different algorithms and a physics-framed two-phase flow problem involving displacement of water by CO 2 . The deep-learning models were trained using 3D datasets describing the pressure plume, CO 2 saturation plume, and water extraction rate generated by numerical simulation. The hyperparameters defining the architecture of the neural networks were optimized to determine the slimmest network size and training parameters that give the most efficient performance at the least training cost. To develop a robust model that closely mimics the governing physical laws, the discretized form of the two-phase fluid transport equation was used to formulate the supervised deep-learning task. The algorithms investigated in this study predicted the data to above 95% accuracy, with the multi-layer perceptron model demonstrating the best performance by balancing training speed, prediction time, and prediction accuracy with lean network capacity. Furthermore, the surrogate models simultaneously predict reservoir pressure and CO 2 saturation in every grid block, including the surface well extraction rate and bottomhole pressure, at all simulation times for a given static model realization in just a few seconds on a standard desktop computer. A key outcome of this study is that limits can be placed on network design parameters to avoid over designing neural networks, with associated efficiencies in training and prediction times. This is very useful because large volumes of data may be generated in CCUS projects and over-design of neural network architectures imposes penalties that are antithetical to the goal of near-real time forecasting.

58 GEOSCIENCES↗

DOE OSTI2025

Hestia-SWIFL: hourly anthropogenic fossil fuel CO2 and heat on the 2km WRF grid, version 1.1

The Hestia-SWIFL version 1.1 anthropogenic heat (AH) and fossil fuel CO2 (FFCO2) emissions data product represent emissions due to the combustion of fossil fuel and cement production within the state of Arizona from 2019 to 2022. This product was developed as part of the Southwest Urban Corridor Integrated Field Laboratory (SW-IFL) project, which aims to provide new knowledge and tools that address extreme heat, air quality, climate change and related urban environmental issues by integrating high-resolution observations, modeling, and civic engagement. The emissions are generated using a bottom-up/engineering approach and are tied to results generated by the Vulcan Project version 4, an effort to quantify space/time-resolved FFCO2 & AH emissions for the entire United States landscape. A large number of data sources are combined to best estimate the emissions at fine scales such as air quality emissions data, traffic flow data, building information, sociodemographic information, and fuel statistics. The AH product provides emissions for two emissions sources (transportation and point source emissions) in units of Watts per hour per square meter (W/m2) per year (annual files) or per hour (hourly files). The FFCO2 product provides emissions from nine individual emission sectors as well as the total, and in units of tons of carbon (tC) per grid cell per year or per hour. The output made available here places the native spatial resolution of the Hestia FFCO2 & AH emissions data product (points, lines, and polygons) into a regularized 2km x 2km grid at hourly and annual temporal resolutions, and stored in netCDF files. The exact spatial extent is defined by the ASU Weather Research Forecast (WRF) simulation grid. All data are processed using R/Python pm high-performance computing system. 2-27-2026 updates: Bugs in airport hourly profile (both AH and FFCO2) and building spatial patterns (FFCO2 only) were fixed. Hourly emissions are reprocessed for all years to reflect those changes.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Technoeconomic Analysis of Infrastructure Buildout Scenarios

The success of CCUS deployment in the southeast will depend on the optimized pipeline network from CO 2 point sources to geologic sinks for CO 2 storage. An optimal pipeline framework will reduce both environmental impacts and pipeline building cost. However, finding an optimal pipeline/transport infrastructure design is a non-trivial task and requires simultaneous usage of large volumes of data, computational resources, and a state-of-the-art simulator. Such a transport infrastructure model must consider point sources for CO 2 capture and associated volumes, sinks for CO 2 storage, and transportation from source to sink via pipeline networks. These considerations must be addressed simultaneously and systemically in an optimization process, in which a defined objective function (i.e., capital, variable CO 2 capture, transport, and storage cost function) is required to be minimized with the consideration of practical constraints (e.g., CO 2 flow through pipelines is less than the maximum capacity). Although optimization has been widely used in subsurface resources production and CO 2 storage, its application in large scale CCUS infrastructure design is rarely reported in the literature. SimCCS, developed by Los Alamos National Laboratory, is an open-source CCUS pipeline infrastructure design toolset that facilitates the optimization of pipeline infrastructure networks.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2019

A simple json format for storing and exchanging code calculation and experimental results

Calculation codes and code systems give their results in a plethora of different formats and/or files. For instance, MCNP gives a user the MCNP specific output file, the mctal file, the ptrac file, etc. while PARTISN has a PARTISN specific output file and a number of binary files with various kinds of information about the calculation. However, the results we are interested in are always the same. These can be single values such as the effective multiplication factor k eff or the effective delayed neutron fraction β eff . It can be histogram data (such as particle spectra, reaction rates, sensitivity profiles, etc.) or even pointwise data (for example the nuclide composition as a function of time, etc.). It is also worthwhile to note that this would also apply to experimental data that we may wish to compare with as well. Calculation and experimental results can basically be split into two components: attributes (or metadata) and the actual result itself. Attributes represent information about the result that are not necessarily required for its’ interpretation but that are still good to have. Often, these attributes are what we will want to filter results on. This means that these attributes are often the answer to specific questions we may have about the result. The following is a non exhaustive list of such basic questions: what type of result is it: e.g. the effective multiplication factor k eff , the effective delayed neutron fraction β eff , etc.; which case does it relate to: e.g. a specific ICSBEP case such as PU-MET-FAST-001-001; which code and version produced this result: e.g. MCNP 6.2, etc.; which nuclear data library was used to obtain the result: e.g. ENDF/B-VIII.0 or JEFF 3.3; which nuclide and reaction does it apply to (for reaction rates and sensitivity profiles); which volume in the geometry was it calculated in? The second component of the calculation or experimental result is the actual numerical data of the result. It is this data that we want to compare, exchange, plot, interpret, etc. It consists of values with optional uncertainties and units. It also includes the underlying structure (if any structure is required to interpret the result). Depending on the type of result, it may be a single value (e.g. the effective multiplication factor), an array of values (e.g. a reaction rate or particle spectrum) or even a multidimensional array. In order to exchange results between different organisations, for instance for benchmarking, people often resort to spreadsheets or custom text files that need to be filled out. While this approach works well if little information is to be exchanged, it can become cumbersome when dealing with large amounts of data (like for instance particle spectra for thousands of cases). In addition, such an approach does not always capture the metadata we may be interested in to allow for automated processing of these files. The purpose of this report is to propose a standardised json structure to store any type of result and present an associated python interface to interact with these results directly. For such a structure to be useful, it must fulfill a specific set of functional requirements that we try to meet with this proposal. The format must allow for the calculation result to stand by itself. This means that we should not have to look in multiple places to be able to understand it. For example, if the result relates to a particle spectrum, the group structure in which the result is given should be stored with it. The format should be code agnostic. This means that we do not need to know where it came from to understand it. The format should be result type agnostic. This means that the format must be generic enough so that we can even use it for results we have not thought of yet. The format should be relatively easy to interact with through different scripting or programming languages (i.e. the machine interface) while still providing a clean structure that can be used directly by any human.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Expediting field-effect transistor chemical sensor design with neuromorphic spiking graph neural networks

Improving the sensitive and selective detection of analytes in a variety of applications requires accelerating the rational design of field-effect transistor (FET) chemical sensors. Achieving high-performance detection relies on identifying optimal probe materials that can effectively interact with target analytes, a process traditionally driven by chemical intuition and time-consuming trial-and-error methods. To address the difficulties in probe screening for FET sensor development, this work presents a methodology that combines neuromorphic machine learning (ML) architectures, specifically a hybrid spiking graph neural network (SGNN), with an enriched dataset of physicochemical properties through semi-automated data extraction using large language models. Achieving a classification accuracy of 0.89 in predicting sensor sensitivity categories, the SGNN model outperformed traditional ML techniques by leveraging its ability to capture both global physicochemical properties and sparse topological features through a hybrid modeling framework. Next-generation sensor design was informed by the actionable insights into the connections between material properties and sensing performance offered by the SGNN framework. Through virtual screening for the detection of per- and polyfluoroalkyl substances (PFAS) as a use case, the effectiveness of the SGNN model was further validated. Density functional theory simulations confirmed graphene as a promising active material for PFAS detection as suggested by the SGNN framework. By bridging gaps in predictive modeling and data availability, this integrated approach provides a strong foundation for accelerating advancements in FET sensor design and innovation.

Ferreira, Rodrigo Pires [Univ. of Chicago, IL (Uni↗

DOE OSTI2024

Spectrum and extension of the inverse-Compton emission of the Crab Nebula from a combined Fermi -LAT and H.E.S.S. analysis

The Crab Nebula is a unique laboratory for studying the acceleration of electrons and positrons through their non-thermal radiation. Observations of very-high-energy γ rays from the Crab Nebula have provided important constraints for modelling its broadband emission. We present the first fully self-consistent analysis of the Crab Nebula’s γ-ray emission between 1 GeV and ∼100 TeV, that is, over five orders of magnitude in energy. Using the open-source software package GAMMAPY, we combined 11.4 yr of data from the Fermi Large Area Telescope and 80 h of High Energy Stereoscopic System (H.E.S.S.) data at the event level and provide a measurement of the spatial extension of the nebula and its energy spectrum. We find evidence for a shrinking of the nebula with increasing γ-ray energy. Furthermore, we fitted several phenomenological models to the measured data, finding that none of them can fully describe the spatial extension and the spectral energy distribution at the same time. Especially the extension measured at TeV energies appears too large when compared to the X-ray emission. Our measurements probe the structure of the magnetic field between the pulsar wind termination shock and the dust torus, and we conclude that the magnetic field strength decreases with increasing distance from the pulsar. We complement our study with a careful assessment of systematic uncertainties.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2023

Measurement of the Higgs boson mass in the H → ZZ*→ 4ℓ decay channel using 139 fb –1 of s = 13 TeV pp collisions recorded by the ATLAS detector at the LHC

The mass of the Higgs boson is measured in the H→ZZ * →4ℓ decay channel. The analysis uses proton–proton collision data from the Large Hadron Collider at a centre-of-mass energy of 13 TeV recorded by the ATLAS detector between 2015 and 2018, corresponding to an integrated luminosity of 139 fb –1 . The measured value of the Higgs boson mass is 124.99 ± 0.18(stat.) ± 0.04(syst.) GeV. In final states with muons, this measurement benefits from an improved momentum-scale calibration relative to that adopted in previous publications. The measurement also employs an analytic model that takes into account the invariant-mass resolution of the four-lepton system on a per-event basis and the output of a deep neural network discriminating signal from background events. This measurement is combined with the corresponding measurement using 7 and 8 TeV pp collision data, resulting in a Higgs boson mass of 124.94 ± 0.17(stat.) ± 0.03(syst.) GeV.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2022

C3F: Collaborative Container-based Model Coupling Framework

Solving complex real-world grand challenge problems requires in-depth collaboration of researchers from multiple disciplines. Such collaboration often involves harnessing multiscale and multi-dimensional data and combining models from different fields to simulate systems. However, the progress on this front has been limited mainly due to significant gaps in domain knowledge and tools that are typically employed in silos of the domains. Researchers from different fields face considerable barriers to understanding and reusing each other’s data/models in order to collaborate effectively. For example, in solving the global sustainability problems, researchers from hydrology, climate science, agriculture, and economics need to run their respective models to study different components of the global and local food, energy and water systems while, at the same time, need to interact with other researchers and integrate the results of one model with another. Developing this kind of model coupling workflow calls for (1) a large amount of data being processed and exchanged across domains and organizations, (2) identifying and processing the output of one model to make it ready for integration into another model, (3) controlling the workflow dynamically so that it runs until a certain convergence condition or other criteria is met, and (4) close collaboration among the modelers to explore, tune, and test the configuration and data transformation needed to link the models. We have developed C3F, a flexible collaborative model coupling framework to help researchers accelerate their model integration and linking efforts by leveraging advanced cyberinfrastructure such as high-performance computing and virtual containers. In this paper, we describe our experience and lessons learned in developing this cyberinfrastructure solution to support the linking of Water Balance Model (WBM) and SIMPLE-G agricultural economic model in an NSF funded INFEWS project and a DOE-funded Program on Coupled Human and Earth Systems (PCHES) to study the implications of groundwater scarcity for food-energy-water systems. The C3F model coupling framework can be extended to facilitate other model linkages as well.

containerization↗

DOE OSTI2020

NCBI’s Virus Discovery Codeathon: Building “FIVE” —The Federated Index of Viral Experiments API Index

Viruses represent important test cases for data federation due to their genome size and the rapid increase in sequence data in publicly available databases. However, some consequences of previously decentralized (unfederated) data are lack of consensus or comparisons between feature annotations. Unifying or displaying alternative annotations should be a priority both for communities with robust entry representation and for nascent communities with burgeoning data sources. To this end, during this three-day continuation of the Virus Hunting Toolkit codeathon series (VHT-2), a new integrated and federated viral index was elaborated. This Federated Index of Viral Experiments (FIVE) integrates pre-existing and novel functional and taxonomy annotations and virus–host pairings. Variability in the context of viral genomic diversity is often overlooked in virus databases. As a proof-of-concept, FIVE was the first attempt to include viral genome variation for HIV, the most well-studied human pathogen, through viral genome diversity graphs. As per the publication of this manuscript, FIVE is the first implementation of a virus-specific federated index of such scope. FIVE is coded in BigQuery for optimal access of large quantities of data and is publicly accessible. Many projects of database or index federation fail to provide easier alternatives to access or query information. To this end, a Python API query system was developed to enhance the accessibility of FIVE.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Beyond microbial abundance: metadata integration enhances disease prediction in human microbiome studies

Multiple studies have highlighted the interaction of the human microbiome with physiological systems such as the gut, immune, liver, and skin, via key axes. Advances in sequencing technologies and high-performance computing have enabled the analysis of large-scale metagenomic data, facilitating the use of machine learning to predict disease likelihood from microbiome profiles. However, challenges such as compositionality, high dimensionality, sparsity, and limited sample sizes have hindered the development of actionable models. One strategy to improve these models is by incorporating key metadata from both the human host and sample collection/processing protocols. This remains challenging due to sparsity and inconsistency in metadata annotation and availability. In this paper, we introduce a machine learning-based pipeline for predicting human disease states by integrating host and protocol metadata with microbiome abundance profiles from 68 different studies, processed through a consistent pipeline. Our findings indicate that metadata can enhance machine learning predictions, particularly at higher taxonomic ranks like Kingdom and Phylum, though this effect diminishes at lower ranks. Our study leverages a large collection of microbiome datasets comprising 11,208 samples, therefore enhancing the robustness and statistical confidence of our findings. This work is a critical step toward utilizing microbiome and metadata for predicting diseases such as gastrointestinal infections, diabetes, cancer, and neurological disorders.

Mathematics and Computing↗

DOE OSTI2021

Machine learning enabled electron microscopy data analysis tool

This software is for materials image and spectral data analysis. It consists machine learning models that can segment a large amount of image data in a high-throughput way, and an interface for image data and spectral data analysis.

ZHONG, XIAOTING↗

DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository: Preprint

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗

DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository

access↗

DOE OSTI2023

A new coupling of a GPU-resident large-eddy simulation code with a multiphysics wind turbine simulation tool

The development of new wind farm control strategies can benefit from combined analysis of flow dynamics in the farm and the behavior of individual turbines within one simulation environment. In this work, we present such an environment by developing a new coupling between the large-eddy simulation (LES) code GRASP and the multiphysics wind turbine simulation tool OpenFAST via an actuator line model (ALM). In addition, the implementation of the recently proposed filtered actuator line model (FALM) within the coupling is described. The new ALM implementation is cross-verified with results from four other commonly used research LES codes. The results for the blade loads and the near wake obtained with the new coupling are consistent with the other codes. Deviations are observed in the far wake. The results further indicate that the FALM is able to reduce the lift and power overprediction from which the traditional ALM suffers on coarse LES grids. This new simulation environment paves the way for future wind farm simulations under realistic weather conditions by leveraging GRASP's ability to impose data from large-scale meteorological models as boundary conditions.

17 WIND ENERGY↗