SEARCH · Search NASA

Results for “Big data transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

53 records · Page 3

All sources NASA NTRS DOE OSTI

DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

NASA NTRS2022

Cosmological constraints on dark matter interactions with ordinary matter

Dark matter interactions with electrons or protons during the early Universe leave imprints on the cosmic microwave background and the matter power spectrum, and can be probed through cosmological and astrophysical observations. These interactions lead to momentum and heat exchange between the ordinary and dark matter components, which in turn results in a transfer of pressure from the ordinary to the dark matter. We explore these interactions using a diverse suite of data: cosmic microwave background anisotropies, baryon acoustic oscillations, the Lyman-α forest, and the abundance of Milky-Way subhalos. We derive constraints using model-independent parameterizations of the dark matter–electron and dark matter–proton interaction cross sections and map these constraints onto concrete dark matter models. Our constraints are complementary to other probes of dark matter interactions with ordinary matter, such as direct detection, big bang nucleosynthesis, various astrophysical systems, and accelerator based experiments. They exclude sufficiently large cross sections for a large range of dark matter masses, which cannot be accessed by direct-detection experiments due to the overburden from the Earth’s atmosphere or crust.

Manuel A. Buen-Abad↗

DOE OSTI2022

Cosmological constraints on dark matter interactions with ordinary matter

Dark matter interactions with electrons or protons during the early Universe leave imprints on the cosmic microwave background and the matter power spectrum, and can be probed through cosmological and astrophysical observations. These interactions lead to momentum and heat exchange between the ordinary and dark matter components, which in turn results in a transfer of pressure from the ordinary to the dark matter. Here, we explore these interactions using a diverse suite of data: cosmic microwave background anisotropies, baryon acoustic oscillations, the Lyman-α forest, and the abundance of Milky-Way subhalos. We derive constraints using model-independent parameterizations of the dark matter–electron and dark matter–proton interaction cross sections and map these constraints onto concrete dark matter models. Our constraints are complementary to other probes of dark matter interactions with ordinary matter, such as direct detection, big bang nucleosynthesis, various astrophysical systems, and accelerator-based experiments. They exclude sufficiently large cross sections for a large range of dark matter masses, which cannot be accessed by direct-detection experiments due to the overburden from the Earth’s atmosphere or crust.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Geo Thermal Cloud: Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

The project is motivated by the challenges, risks, and costs associated with geothermal exploration and production. Many processes and parameters impacting geothermal conditions are poorly understood. Diverse datasets are available to help characterize subsurface geothermal conditions (public and proprietary; satellite, airborne surveys, vegetation/water sampling, geological, geophysical, etc.). Yet, it is not clear how to properly leverage these datasets for geothermal exploration due to an incomplete understanding of how physical processes impacting subsurface geothermal conditions are represented in these observations. Recent advancements in machine learning (ML) provide great promise to resolve these issues. The tremendous challenges and risks of geothermal exploration and production bring the demand for novel ML methods and tools that can (1) analyze large field datasets, (2) assimilate model simulations (large inputs and outputs), (3) process sparse datasets, (4) perform transfer learning (between sites with different exploratory levels), (5) extract hidden geothermal signatures in the field and simulation data, (6) label geothermal resources and processes, (7) identify high-value data acquisition targets, and (8) guide geothermal exploration and production by selecting optimal exploration, production, and drilling strategies. Our goals and work under Phases 1 and 2 (as proposed) of this project address all these needs.

15 GEOTHERMAL ENERGY↗

DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗

DOE OSTI2024

GeoThermalCloud: Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

58 GEOSCIENCES↗

DOE OSTI2024

Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

The primary goals of this project are identifying hidden geothermal resources in the USA and designing profitable enhanced geothermal systems (EGS). Many non-obvious processes and parameters could characterize geothermal resources and could control the ultimate energy potential of geothermal fields. Diverse datasets (e.g., geology, geochemistry, geophysics, satellite, airborne geophysics) are available to help characterize geothermal resources, but this data is sparse and multi-scale. This has hindered attempts to leverage the datasets for geothermal exploration and profitable EGS design. Recent advancements in machine learning (ML) give promise to overcome these issues. Modern ML methods and tools can (1) analyze large datasets, (2) assimilate model ensembles that include a multitude of inputs and outputs, (3) process sparse datasets, (4) perform transfer learning between sites with different data quality, (5) extract hidden geothermal signatures from field and simulation data, (6) label geothermal resources and processes, (7) identify high-value data acquisition targets, and (8) guide geothermal exploration and production by selecting optimal exploration, production, and drilling strategies. In this work, we implement ML-based geothermal exploration and an enhanced geothermal systems (EGS) design tool to achieve the above goals. Our exploration tool is GeoThermalCloud (GTC) EGS design tool is GeoDT-ML. GTC (github.com/SmartTensors/GeoThermalCloud.jl) utilizes a LANL unsupervised ML platform called SmartTensors (https://tensors.lanl.gov/) to automate data analyses and interpretations by extracting hidden signatures to identify geothermal prospects. It enables the identification of critical measurements needed to identify geothermal resource signatures. GeoDT-ML (github.com/SmartTensors/GeoThermalCloud.jl/tree/master/) adds coupling to GeoDT (https://github.com/GeoDesignTool/GeoDT.git) for stochastic EGS design optimization and performance prediction. GeoDT-ML leverages recent advances in deep learning and high-performance computing. Contributors to this effort include LANL, PNNL, Google, Stanford, and Julia Computing.

15 GEOTHERMAL ENERGY↗

NASA NTRS2001

Telemetry Tracking & Control (TT&C) - First TDRSS, then Commercial GEO & Big LEO and Now Through LEO

The advent of low earth orbit (LEO) commercial communication satellites provides an opportunity to dramatically reduce Telemetry, Tracking and Control (TT&C) costs of launch vehicles, Unpiloted Aerial Vehicles (UAVs), Research Balloons and spacecraft by reducing or eliminating ground infrastructure. Personnel from the Goddard Space Flight Center's Wallops Flight Facility (GSFC\WFF) have successfully used commercial Geostationary Earth Orbit (GEO) and Big LEO communications satellites for Long Duration Balloon Flight TT&C. The Flight Modem is a GSFC\WFF Advanced Range Technology initiative (ARTI) designed to streamline TT&C capability in the user community of these scientific data gathering platforms at low cost. Making use of existing LEO satellites and adapting and ruggedized commercially available components; two-way, over the horizon communications may be established with these vehicles at great savings due to reduced infrastructure. Initially planned as a means for permitting GPS data for tracking and recovery of sounding rocket and balloon payloads, expectations are that the bandwidth can soon be expanded to allow more comprehensive data transfer. The system architecture which integrates antennas, GPS receiver, commercial satellite packet data modem and a single board computer with custom software is described and technical challenges are discussed along with the plan for their resolution. A three-phase testing and development plan is outlined and the current results are reported. Results and status of ongoing flight tests on aircraft and sounding rockets are reported. Future applications on these platforms and the potential for satellite support are discussed along with an analysis of cost effectiveness of this method vs. other tracking and data transmission schemes.

Morgan, Dwayne R.↗

NASA NTRSDate not supplied

Assessment of Sensor Footprint Size and Comparison of Commercial Smallsat Images

Science users of commercial satellite data build their studies on the properties of the satellite data they work with, pixel size being one of the determining factors. However, pixel size and nominal image footprint size can be different, impacting the scale of features discernable in the satellite images. Here, we assess geometric properties such as nominal image footprint size for Planet Labs' SuperDove series, MAXAR's WorldView series, and potentially BlackSky. Additionally, we assess temporal change in nominal footprint size for the SuperDove series. Nominal Image footprint size is assessed over the CalVal sites in Baotou, China; Shadnagar, India; and Big Spring, TX. Edge spread functions are constructed along the black/white transitions, from which the line spread functions are constructed and modulation transfer functions of the scene are estimated. Scene full width half maximum (FWHM), which represents sensor footprint size, is estimated from the line spread function. The average nominal footprint size is 3.3 pixels for SuperDove series, 1.5 pixels for WorldView-2, and 1.3 pixels for WorldView-3. The SuperDove series nominal footprint size improves with time in orbit, from an average of 3.4 pixels soon after launch to an average of 3.2 pixels one or more years after launch.

PlanetScope↗

NASA NTRS1982

Contrast reduction by the atmosphere and retrieval of nonuniform surface reflectance

A radiative transfer model is developed which gives the upward radiance at nadir for any 1-D Lambertian surface reflectance. This model is used to depict the atmospheric effect on the transmittance of contrast for any 1-D surface reflectance. Here by contrast we mean a general variation of the radiation field across the image. With the aid of this model an inversion algorithm is developed for retrieval of true surface reflectance from high resolution satellite data (e.g., Landsat). This inversion technique can be a useful tool for extraction of surface reflectance from satellite data in the case of a surface reflectance variable in one dimension only (e.g., seashore or near borders of big fields). A sensitivity study of the inversion procedure on the knowledge of atmospheric parameters and sensor calibration was performed. It is shown that this inversion technique is stable even in the presence of errors in the sensor calibration and the atmospheric parameters. The method was applied to Landsat data in two wavelengths. The results show reasonable dependence of the derived surface reflectance on the distance from the seashore.

Mekler, Y.↗

DOE OSTI2024

Convergence in simulating global soil organic carbon by structurally different models after data assimilation

Abstract Current biogeochemical models produce carbon–climate feedback projections with large uncertainties, often attributed to their structural differences when simulating soil organic carbon (SOC) dynamics worldwide. However, choices of model parameter values that quantify the strength and represent properties of different soil carbon cycle processes could also contribute to model simulation uncertainties. Here, we demonstrate the critical role of using common observational data in reducing model uncertainty in estimates of global SOC storage. Two structurally different models featuring distinctive carbon pools, decomposition kinetics, and carbon transfer pathways simulate opposite global SOC distributions with their customary parameter values yet converge to similar results after being informed by the same global SOC database using a data assimilation approach. The converged spatial SOC simulations result from similar simulations in key model components such as carbon transfer efficiency, baseline decomposition rate, and environmental effects on carbon fluxes by these two models after data assimilation. Moreover, data assimilation results suggest equally effective simulations of SOC using models following either first‐order or Michaelis–Menten kinetics at the global scale. Nevertheless, a wider range of data with high‐quality control and assurance are needed to further constrain SOC dynamics simulations and reduce unconstrained parameters. New sets of data, such as microbial genomics‐function relationships, may also suggest novel structures to account for in future model development. Overall, our results highlight the importance of observational data in informing model development and constraining model predictions.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Investigating resource-efficient neutron/gamma classification ML models targeting eFPGAs

There has been considerable interest and resulting progress in implementing machine learning (ML) models in hardware over the last several years from the particle and nuclear physics communities. A big driver has been the release of the Python package, hls4ml, which has enabled porting models specified and trained using Python ML libraries to register transfer level (RTL) code. So far, the primary end targets have been commercial field-programmable gate arrays (FPGAs) or synthesized custom blocks on application specific integrated circuits (ASICs). However, recent developments in open-source embedded FPGA (eFPGA) frameworks now provide an alternate, more flexible pathway for implementing ML models in hardware. These customized eFPGA fabrics can be integrated as part of an overall chip design. In general, the decision between a fully custom, eFPGA, or commercial FPGA ML implementation will depend on the details of the end-use application. In this work, we explored the parameter space for eFPGA implementations of fully-connected neural network (fcNN) and boosted decision tree (BDT) models using the task of neutron/gamma classification with a specific focus on resource efficiency. We used data collected using an AmBe sealed source incident on Stilbene, which was optically coupled to an OnSemi J-series silicon photomultiplier (SiPM) to generate training and test data for this study. We investigated relevant input features and the effects of bit-resolution and sampling rate as well as trade-offs in hyperparameters for both ML architectures while tracking total resource usage. The performance metric used to track model performance was the calculated neutron efficiency at a gamma leakage of 10 -3 . The results of the study will be used to aid the specification of an eFPGA fabric, which will be integrated as part of a test chip.

47 OTHER INSTRUMENTATION↗

DOE OSTI2020

State-of-the-art on research and applications of machine learning in the building life cycle

Fueled by big data, powerful and affordable computing resources, and advanced algorithms, machine learning has been explored and applied to buildings research for the past decades and has demonstrated its potential to enhance building performance. This study systematically surveyed how machine learning has been applied at different stages of building life cycle. By conducting a literature search on the Web of Knowledge platform, we found 9579 papers in this field and selected 153 papers for an in-depth review. The number of published papers is increasing year by year, with a focus on building design, operation, and control. However, no study was found using machine learning in building commissioning. There are successful pilot studies on fault detection and diagnosis of HVAC equipment and systems, load prediction, energy baseline estimate, load shape clustering, occupancy prediction, and learning occupant behaviors and energy use patterns. None of the existing studies were adopted broadly by the building industry, due to common challenges including (1) lack of large scale labeled data to train and validate the model, (2) lack of model transferability, which limits a model trained with one data-rich building to be used in another building with limited data, (3) lack of strong justification of costs and benefits of deploying machine learning, and (4) the performance might not be reliable and robust for the stated goals, as the method might work for some buildings but could not be generalized to others. Finally, findings from the study can inform future machine learning research to improve occupant comfort, energy efficiency, demand flexibility, and resilience of buildings, as well as to inspire young researchers in the field to explore multidisciplinary approaches that integrate building science, computing science, data science, and social science.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2026

Unified Modeling Architecture for Load Management in Extreme Heat: The New York City Case

Integration of renewable resources to meet growing energy demand is becoming a global priority under decarbonization mandates. This study contributes to ongoing efforts on this key subject by assessing the feasibility of using coastal-urban renewable energy resources, namely, offshore wind and rooftop photovoltaic systems, to meet electricity demand of New York City during the intense recent heat wave period of June 2025. A unified modeling framework, based on the urbanized weather research and forecasting model, is used to simulate climate, renewable resources, and energy demand variables. Findings show significant energy load mismatch of approximately 1150 GWh over the month, between the demand and the combined renewable generation outcome. Three storage integration scenarios are analyzed to mitigate the deficits, reducing said deficits by a minimum of approximately 9% over the duration of the month. This study provides a transferable modeling framework tool for evaluating renewable integration in dense urban environments that can be used by grid operators to support grid resilience during extreme heat events.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Characterization of shale using Helium and Argon at high pressures

In order to estimate the shale gas in place and the eventual recovery during shale gas operations, one of the key requirements is to accurately characterize the shale’s petrophysical and transport properties such as porosity, permeability, diffusivity, and storage capacity. Despite the many efforts reported in the technical literature aiming to provide an improved understanding of the complex pore structures and the associated fluid flow in gas shales, complete characterization of organic-rich shale samples still poses a big challenge. Here, we have characterized mass transfer and sorption in shale at different length scales using Helium (He) and Argon (Ar) as probe gases. Thermogravimetric analysis (TGA) with a shale cube of ~1 cm 3 in size and gas expansion experiments with a full-diameter core (3.5” in diameter) were used to measure sorption kinetics/isotherms and mass transfer, respectively. Both samples are from the same depth/location in the Marcellus shale formation. The TGA steady-state technique was initially used to generate excess sorption isotherms for Ar, while dynamic TGA experiments were used to study its sorption kinetics. The TGA experiments demonstrate that Ar, which has a similar sorption potential as Methane, but is generally assumed to be inert, adsorbs onto the surfaces of the mesoporous and microporous regions of the shale samples according to a Langmuir-type behavior. Helium expansion experiments, on the full-diameter core, were used to measure the overall porosity, on the basis that He is a non-sorbing and inert gas as compared to Ar. The He expansion experiments, furthermore, allow us to delineate the mass transfer of gas across the inherent hierarchy of pore sizes, including macropores (macro- and microcracks), mesopores and micropores. Similar expansion experiments were also performed with Ar to study the combined impact of mass transfer and sorption. A triple-porosity model (TPM) was utilized to interpret the He expansion experiments with the shale core and to extract (estimate) relevant transport parameters. We report and compare here the diffusivities and permeabilities of the whole core for both He and Ar, as calculated from the modeling and fitting of the experimental data. On the premise that the shale cube is representative of the matrix region of the core, the Ar sorption kinetics from the cube experiments were subsequently combined with the transport parameters extracted from the He experiments to predict the behavior of the Ar expansion test with the full-diameter core. An excellent agreement is observed between the model predictions and the experimental data. The experimental observations and their interpretation indicate that one must be cautious when using Ar to estimate the true porosity and permeability of shales. In addition, we demonstrate that He and Ar probe gases, when used in tandem, can be employed effectively as a tool to characterize shales in terms of mass transfer and sorption dynamics across scales.

04 OIL SHALES AND TAR SANDS↗

NASA NTRS2017

Smashing the Stovepipe: Leveraging the GMSEC Open Architecture and Advanced IT Automation to Rapidly Prototype, Develop and Deploy Next-Generation Multi-Mission Ground Systems

Satellite/Payload Ground Systems - Typically highly-customized to a specific mission's use cases - Utilize hundreds (or thousands!) of specialized point-to-point interfaces for data flows / file transfers Documentation and tracking of these complex interfaces requires extensive time to develop and extremely high staffing costs Implementation and testing of these interfaces are even more cost-prohibitive, and documentation often lags behind implementation resulting in inconsistencies down the road With expanding threat vectors, IT Security, Information Assurance and Operational Security have become key Ground System architecture drivers New Federal security-related directives are generated on a daily basis, imposing new requirements on current / existing ground systems - These mandated activities and data calls typically carry little or no additional funding for implementation As a result, Ground System Sustaining Engineering groups and Information Technology staff continually struggle to keep up with the rolling tide of security Advancing security concerns and shrinking budgets are pushing these large stove-piped ground systems to begin sharing resources - I.e. Operational / SysAdmin staff, IT security baselines, architecture decisions or even networks / hosting infrastructure Refactoring these existing ground systems into multi-mission assets proves extremely challenging due to what is typically very tight coupling between legacy components As a result, many "Multi-Mission" ops. environments end up simply sharing compute resources and networks due to the difficulty of refactoring into true multi-mission systems Utilizing continuous integration / rapid system deployment technologies in conjunction with an open architecture messaging approach allows System Engineers and Architects to worry less about the low-level details of interfaces between components and configuration of systems GMSEC messaging is inherently designed to support multi-mission requirements, and allows components to aggregate data across multiple homogeneous or heterogeneous satellites or payloads - The highly-successful Goddard Science and Planetary Operations Control Center (SPOCC) utilizes GMSEC as the hub for it's automation and situational awareness capability Shifts focus towards getting GS to a final configuration-managed baseline, as well as multi-mission / big-picture capabilities that help increase situational awareness, promote cross-mission sharing and establish enhanced fleet management capabilities across all levels of the enterprise.

GMSEC↗

DOE OSTI2022

Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints

Vast volumes of data are produced by today's scientific simulations and advanced instruments. These data cannot be stored and transferred efficiently because of limited I/O bandwidth, network speed, and storage capacity. Error-bounded lossy compression can be an effective method for addressing these issues: not only can it significantly reduce data size, but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific requirements or constraints for lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. For example, some datasets contain irrelevant data that should be isolated in particular and users often have intuition regarding value ranges, geospatial regions, and other data subsets that are crucial for subsequent analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not consider these constraints during compression, resulting in inferior compression ratios with respect to user's post hoc analysis, due to the fact that the data itself provides little or no value for post hoc analysis. In this work we address this issue by proposing an optimized framework that can preserve diverse constraints during the error-bounded lossy compression, e.g., cleaning the irrelevant data, efficiently preserving different precision for multiple value intervals, and allowing users to set diverse precision over both regular and irregular regions. We perform our evaluation on a supercomputer with up to 2,100 cores. Experiments with six real-world applications show that our proposed diverse constraints based error-bounded lossy compressor can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios compared with the traditional state-of-the-art compressor SZ. Furthermore, our experiments also demonstrate very good scalability in compression performance compared with the I/O throughput of the parallel file system.

97 MATHEMATICS AND COMPUTING↗