SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 757 records · Page 42

All sources NASA NTRS DOE OSTI

DOE OSTI2024

Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments

Structural biology experiments benefit significantly from state-of-the-art synchrotron data collection. One can acquire macromolecular crystallography (MX) diffraction data on large-area photon-counting pixel-array detectors at framing rates exceeding 1000 frames per second, using 200 Gbps network connectivity, or higher when available. In extreme cases this represents a raw data throughput of about 25 GB s −1 , which is nearly impossible to deliver at reasonable cost without compression. Our field has used lossless compression for decades to make such data collection manageable. Many MX beamlines are now fitted with DECTRIS Eiger detectors, all of which are delivered with optimized compression algorithms by default, and they perform well with current framing rates and typical diffraction data. However, better lossless compression algorithms have been developed and are now available to the research community. Here one of the latest and most promising lossless compression algorithms is investigated on a variety of diffraction data like those routinely acquired at state-of-the-art MX beamlines.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Probabilistic Look-ahead Contingency Analysis Integration with Commercial Tool and Practical Data

This paper presents an initial effort of integrating a smart sampling-based probabilistic look-ahead contingency analysis algorithm with General Electric (GE) Grid Solutions’ commercial energy management system (EMS) tool as a proof-of-concept for a seamless research tool integration using real world large-scale grid data. With the increasing impact of random forces such as variable generation and load, their stochastic behaviors cannot be ignored. However, the current practices are still dominated by deterministic tools. They are becoming increasingly inadequate for the future grid. The developed look-ahead contingency analysis algorithm incorporates forecast errors of variable energy and load to address the challenges brought by the increasing uncertainty of power system. The algorithm can reveal the potential violations caused by the variance of variable energy and load that are not normally detected by traditional deterministic approaches. To test its performance under practical environments ( real data with real commercial tool), significant efforts have been made to prepare test cases, modify GE EMS tool, and adapt an extreme value distribution algorithm to analyze the GE EMS’s violation-only outputs. The test results clearly demonstrate the effectiveness of the developed algorithm as new transformer violations that were not previously detected have been identified. This performance provides better situational awareness to engineers for their decision-making process under uncertainty. Moreover, with the discussion of computational performance and future work, this paper has shown a clear path for integrating the probabilistic algorithm with commercial tools to make us better equipped for the changing power system.

Modeling and simulation of power systems, constrai↗

DOE OSTI2026

PFLOTRAN modeling data and scripts associated with “Refining the Hydrogeologic Framework of a Large River Corridor Model Using Waterborne Transient Electromagnetics”

NOTE: The manuscript associated with this data package is currently in review. The data may be revised based on reviewer feedback. Upon manuscript acceptance, this data package will be updated with the final dataset and additional metadata. This data package is associated with the publication “Refining the Hydrogeologic Framework of a Large River Corridor Model Using Waterborne Transient Electromagnetics” submitted to Water Resources Research (Terry et al. 2025). The data package contains the groundwater modeling dataset from PFLOTRAN software. It includes the python script for mesh generation, boundary condition setting, PFLOTRAN input deck formation and postprocessing. It couples groundwater flow and species transport for Hanford Reach river corridor and pipelines the model generation and processing. This model can be used to easily generate the model and analysis for Hanford site. It can also be adjusted to other hydrologic area with ease. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. The data package consists of 6 folders: (1) “data” contains all necessary data as input and intermediate data for processing; (2) “mesh” contains all mesh related files to generate mesh in Hanford Reach river corridor; (3) “model_run” contains the generated script for PFLOTRAN modeling; (4) “notebooks” contains all the Python script to generate the model; (5) “output” contains all the output from the computation; (6) “postprocessing” contains the Python script to generate scientific figure for manuscript. All files are .csv (comma-separated values), .h5 (HDF5 format), .in (input files), .ipynb (Jupyter notebooks), .p (Python pickle), .png (images), .PNG (images), .py (Python scripts), .pyc (Python bytecode), .r (R scripts), .sh (shell scripts), .txt (text files), .vtu (3D mesh/visualization format), .xz (compressed archive), or .zip (compressed archive).

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

Improved Parametric Models for Explosion Pressure Signals Derived From Large Datasets

Accurate recording and characterization of explosion-induced pressure signals are key components of the forensic analysis of explosion events in the atmosphere. Parametric overpressure models based on several key waveform features (peak overpressure, positive pulse duration, and impulse) are widely used to estimate explosion energy in terms of trinitrotoluene equivalent yield. However, those models are often developed by a limited dataset, including only a few events or recordings at relatively short propagation distances. Here, we develop empirical waveform-parameter models based on a regression analysis of a large set of data curated from four chemical explosion experiments including 16 detonations. We measured peak overpressure and impulse for positive and negative phases from 1000 pressure signals recorded at local ranges (<20 km) with scaled distance up to 8000 m/kg 1/3 . Additionally, the measured waveform parameters showed large variation with respect to observing distances indicating the effects of atmospheric propagation. In this study, a second-order polynomial model was used in a least-squares regression to account for those propagation effects and to improve data fitting. In addition to model parameters for waveform features, we also determined range-dependent model uncertainties based on data variance. The model uncertainty represents the prediction error of our models and can be critical to evaluating the uncertainty of yield estimate.

58 GEOSCIENCES↗

DOE OSTI2022

Quantum Machine Learning Applications in High-Energy Physics

Some of the most significant achievements of the modern era of particle physics, such as the discovery of the Higgs boson, have been made possible by the tremendous effort in building and operating large-scale experiments like the Large Hadron Collider or the Tevatron. In these facilities, the ultimate theory to describe matter at the most fundamental level is constantly probed and verified. These experiments often produce large amounts of data that require storing, processing, and analysis techniques that continually push the limits of traditional information processing schemes. Thus, the High-Energy Physics (HEP) field has benefited from advancements in information processing and the development of algorithms and tools for large datasets. More recently, quantum computing applications have been investigated to understand how the community can benefit from the advantages of quantum information science. Nonetheless, to unleash the full potential of quantum computing, there is a need to understand the quantum behavior and, thus, scale up current algorithms beyond what can be simulated in classical processors. In this work, we explore potential applications of quantum machine learning to data analysis tasks in HEP and how to overcome the limitations of algorithms targeted for Noisy Intermediate-Scale Quantum (NISQ) devices.

Delgado, Andrea↗

DOE OSTI2026

An ultra-fast method for generating synthetic down-scattered neutron data for inertial confinement fusion implosions

In inertial confinement fusion experiments at the National Ignition Facility, asymmetries are probed by a variety of neutron diagnostics, including neutron imaging systems, real-time neutron activation diagnostics (RTNADs), and neutron spectrometers. It is often useful to generate synthetic data based on these diagnostics to validate and tune models. However, current methods of doing so using Monte Carlo particle tracing are time-consuming. In this paper, an ultra-fast method is presented for generating synthetic neutron images, RTNAD data, and spectrometry data using line integrals and 3D convolutions. While it does not contain as much physics as particle tracing codes, it is thousands of times faster and produces nearly identical data. This enables analysis techniques that depend on generating large amounts of synthetic data, which will prove very useful for the study of asymmetries going forward.

Deuterium↗

DOE OSTI2020

SRNL Analysis of ICCWR LCM and WAMS data for Corrosion and Cracking (FY2020 Progress Report)

3013 corrosion↗

DOE OSTI2021

FY21 Progress Report: SRNL Analysis of ICCWR LCM and WAMS data for Corrosion and Cracking

The development of algorithms for machine learning and data analysis for the 3013 Surveillance Program is a collaborative effort by the Savannah River National Laboratory (SRNL) and the University of South Carolina (USC). For corrosion detection, Laser Confocal Microscope (LCM) or Wide Area 3D Measurement System (WAMS) data is extracted from large binary files, with software written to convert the data to physical attributes (e.g., height, color and grayscale values; all as functions of a location in a plane projection). A user-friendly Matlab Graphical User Interface (GUI) that reads data from either LCM or WAMS files was developed to integrate input data with software developed for processing and evaluation. The GUI can selectively download binary data, interrogate data attributes, label data, flag significant features, execute Machine Learning (ML) algorithms, output parameters for trained ML algorithms, report ML model accuracy with respect to labeled data, and generate graphical representations for various analyses. Features can be called out by user-specified thresholds, manual labeling or machine learning algorithms when they have been completed. The ability to rapidly label data is important because of the volume of data required for training machine learning algorithms. The GUI has the flexibility to allow addition of improved ML algorithms, methods for data visualization, and statistical computations. Statistical analyses via the GUI include areas of pits within a defined range of pit depths, correlations between Red-Green-Blue (RGB) or grayscale intensity and relative surface height, covariances between values associated with features, and feature histograms. The development of supervised machine learning algorithms, however, has been hindered by a lack of training data. The machine learning algorithms for crack identification are being refined but require improvements to the true positive rate for crack detection. This shortcoming is an artifact of the limited training data currently available, perhaps more so than the structure of the neural networks. At present, the best results are had from a consensus over an ensemble of randomly generated Deep Neural Network (DNN) or Convolutional Neural Network (CNN) algorithms. Although the consensus accuracy method has yielded optimum true positive and true negative rates in excess of 80%, additional validation testing is necessary. In addition to the suite of LCM data that was initially used, and which represents the majority of the work presented in this report, WAMS image data was also reviewed at a preliminary level. The review included a comparison between image resolution and dynamic range for each method. WAMS (ZON file) image data was found to have a pixel pitch of 3.69μm compared to 1 μm for the LCM (vk4 file) data, which implies a lower resolution for the WAMS images. Conversely, the ratio of dynamic range of the WAMS data to the LCM data was approximately 41:20 for height data, suggesting that information from WAMS should more accurately determine the depth of pits. At present, the significance of the greater dynamic range of the WAMS data relative to the LCM data has not yet been evaluated.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2026

Data Mining of Groundwater to Identify MAGs with Methane, Propane and Toluene Monooxygenases

Whole genome sequencing datasets, involving more than 600 groundwater samples, from nine countries, were analyzed to identify metagenome assembled genomes (MAGs) containing full operons for propane monooxygenase, soluble methane monooxygease, toluene monooxygenase and particulate ammonia/methane monooxygenase. The enzymes encoded by these genes are a focus of interest because of their ability to degrade common groundwater contaminants. Due to the large amount of data, sequence analyses involved more than 80 individual KBase narratives. The approach followed the KBase tutorial called "Metagenome-Assembled Genome Extraction from a Compost Microbiome Enrichment" The generated MAGs were exported from each individual narrative into separate summary KBase narratives for each monooxygenase. Three KBase narratives were generated for particulate ammonia/methane monooxygenase, due to the large number of MAGs identified.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2022

Variability in observed stable water isotopes in snowpack across a mountainous watershed in Colorado

In this study, isotopic information from 81 snowpits was collected over a 5-year period in a large, Colorado watershed. Data spans gradients in elevation, aspect, vegetation, and seasonal climate. They are combined with overlapping campaigns for water isotopes in precipitation and snowmelt, and a land-surface model for detailed estimates of snowfall and climate at sample locations. Snowfall isotopic inputs, describe the majority of δ18O snowpack variability. Aspect is a secondary control, with slightly more enriched conditions on east and north facing slopes. This is attributed to preservation of seasonally enriched snowfall and vapour loss in the early winter. Sublimation, expressed by decreases in snowpack d-excess in comparison to snowfall contributions, increases at low elevation and when seasonal temperature and solar radiation are high. At peak snow accumulation, post-depositional fractionation appears to occur in the top 25 ± 14% of the snowpack due to melt-freeze redistribution of lighter isotopes deeper into the snowpack and vapour loss to the atmosphere during intermittent periods of low relative humidity and high windspeed. Relative depth of fractionation increases when winter daytime temperatures are high and winter precipitation is low. Once isothermal, snowpack isotopic homogenization and enrichment was observed with initial snowmelt isotopically depleted in comparison to snowpack and enriching over time. The rate of δ18O increase (d-excess decrease) in snowmelt was 0.02‰ per day per 100-m elevation loss. Isotopic data suggests elevation dictates snowpack and snowmelt evolution by controlling early snow persistence (or absence), isotopic lapse rates in precipitation and the ratio of energy to snow availability. Hydrologic tracer studies using stable water isotopes in basins of large topographic relief will require adjustment for these elevational controls to properly constrain stream water sourcing from snowmelt.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Energy scale and resolution for anti-$k_t$ jets with radius parameters $R$ = 0.2 and 0.6 measured in proton-proton collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector

Jets with different radius parameters R are an important tool for probing quantum chromodynamics processes at different angular scales. Jets with small R = 0.2 are instrumental in measurements of the substructure of large-R jets resulting from collimated hadronic decays of energetic W, Z, and Higgs bosons, top quarks, and of potential new resonances. This paper presents measurements of the energy scale, resolution, and associated uncertainties of jets with radius parameters R = 0.2 and 0.6, obtained using the ATLAS detector. The results are based on 37 fb -1 of proton–proton collision data from the Large Hadron Collider at a centre-of-mass energy of $\sqrt{s}$ = 13 TeV. A new in situ method for measuring jet energy scale differences between data and Monte Carlo simulations is presented. The systematic uncertainties in the jet energy scale for central jets $(|\eta | < 1.2)$ typically vary from 1% to about 5% as a function of $|\eta |$ at very low transverse momentum, $p_T$ of around 20 GeV for both R = 0.2 and 0.6 jets. The relative energy resolution ranges from (35 ± 6)% at $p_T$ = 20 GeV to (6 ± 0.5)% at $p_T$ = 300 GeV for central R = 0.2 jets, and is found to be slightly worse for R + 0.6 jets. Finally, the effect of close-by hadronic activity on the jet energy scale is investigated and is found to be well modelled by the ATLAS Monte Carlo simulations.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2022

EXFOR-NSR PDF database: a system for nuclear knowledge preservation and data curation

Current needs of nuclear science and technology include complete, well-documented, and easily verifiable nuclear data. The complete data records require supporting nuclear bibliography, presently stored in dedicated libraries, in addition, to actual data. Additionally, experimental nuclear reaction data (EXFOR) and Nuclear Science References (NSR) databases contain compilations based on primary (journals) and secondary (conference proceedings, theses, preprints, etc.) publications, and data received from authors via private communications. The secondary library materials and private communications often represent a bottleneck for nuclear data verification, compilation, evaluation, and dissemination activities. To address this issue, bibliographic materials were scanned into PDF (Portable Document Format) files and uploaded in a relational database. The traditional scope of nuclear databases that includes meta-data and numbers derived from data in specialized formats was broadened to accommodate the large volumes of original nuclear data publications. The complete PDF publication files were stored in a relational database as Binary Large OBjects (BLOB). This unique collection of nuclear data compilations and supporting publications generate many opportunities for machine learning applications. The Web interfaces for authorized and public access to the EXFOR-NSR nuclear publications database were implemented at the U.S. National Nuclear Data Center, https://www.nndc.bnl.gov/ and IAEA Nuclear Data Section, https://www-nds.iaea.org/ . The current system is complementary to major nuclear libraries and narrowly focused on nuclear data compilation and evaluation procedures. The contents of the PDF database, details of implementation, and Web interface are described. New capabilities for data curation, knowledge preservation, worldwide dissemination, and natural language processing (NLP) applications are given.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

A Deep Learning-Based Workflow for Fast Prediction of 3D State Variables in Geological Carbon Storage: A Dimension Reduction Approach

In this study, we used deep learning techniques, which are a form of artificial intelligence, to create fast and effective models for predicting how fluids flow in underground geological formations. This is important for managing geological carbon storage, a method used to fight climate change by storing carbon dioxide underground. The challenge lies in the complex nature of these underground spaces and the large amount of data needed to accurately simulate them. To overcome these issues, we developed a new workflow that reduces the data’s complexity before training the deep learning model and then reconstructs the predicted results in their original form. We also proposed a unique approach to handle the specific complexities found in 3D saturation fields, a crucial aspect of fluid flow prediction. We tested our method using real-world data from the Gulf of Mexico. Our results show that our approach not only accurately predicts fluid behavior but also significantly reduces computation time. This will greatly improve real-time decision-making and risk assessment in large-scale geological carbon storage operations.

Wang, Hongsheng↗

DOE OSTI2021

Real-time nuclear activation detectors for measuring neutron angular distributions at the National Ignition Facility (invited)

The Real Time Nuclear Activation Detector (RTNAD) array at NIF measures the distribution of 14 MeV neutrons emitted by deuterium-tritium (DT) fueled inertial confinement fusion implosions. The uniformity of the neutron distribution is an important indication of implosion symmetry and DT shell integrity. The array consists of 48 LaBr 3 (Ce) crystal gamma-ray spectrometers mounted outside the NIF target chamber, which continuously monitor the slow decay of the 909 keV gamma-ray line from activated 89 Zr located in Zr cups surrounding each crystal. The measured decay rate dramatically increases during a DT implosion in proportion to the number of 14 MeV neutrons striking each Zr cup. The neutrons produce activated 89 Zr through an (n, 2n) reaction on 90 Zr, which is insensitive to low energy neutrons. The neutron flux along the detector line-of-sight at shot time is determined by extrapolating the fitted 909 keV decay curve back to shot time. Automatic analysis algorithms were developed to handle the non-stop data stream. The large number of detectors and the high statistical accuracy of the array enable the spherical harmonic modes of the neutron angular distribution to be measured up to L ≤ 4 to provide a better understanding of implosion dynamics. In addition, these data combined with measurements of the down-scattered neutrons can be used to derive fuel areal density distributions. This paper will describe the RTNAD hardware and analysis procedures.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2022

Accurate and Rapid Forecasts for Geologic Carbon Storage via Learning-Based Inversion-Free Prediction

Carbon capture and storage (CCS) is one approach being studied by the U.S. Department of Energy to help mitigate global warming. The process involves capturing CO 2 emissions from industrial sources and permanently storing them in deep geologic formations (storage reservoirs). However, CCS projects generally target “green field sites,” where there is often little characterization data and therefore large uncertainty about the petrophysical properties and other geologic attributes of the storage reservoir. Consequently, ensemble-based approaches are often used to forecast multiple realizations prior to CO 2 injection to visualize a range of potential outcomes. In addition, monitoring data during injection operations are used to update the pre-injection forecasts and thereby improve agreement between forecasted and observed behavior. Thus, a system for generating accurate, timely forecasts of pressure buildup and CO 2 movement and distribution within the storage reservoir and for updating those forecasts via monitoring measurements becomes crucial. This study proposes a learning-based prediction method that can accurately and rapidly forecast spatial distribution of CO 2 concentration and pressure with uncertainty quantification without relying on traditional inverse modeling. The machine learning techniques include dimension reduction, multivariate data analysis, and Bayesian learning. The outcome is expected to provide CO 2 storage site operators with an effective tool for timely and informative decision making based on limited simulation and monitoring data.

58 GEOSCIENCES↗

DOE OSTI2022

Field Experience Detecting PV Underperformance in Real Time Using Existing Instrumentation

Maintenance at large-scale photovoltaic plants employs a mix of preventative and corrective maintenance practices. Large outages, such as an inverter tripping offline, are often easy to detect. More subtle sub-inverter faults and failures can accumulate and go unnoticed for months or years. A software-based fault detection method has been developed to analyze commonly measured data from large-scale PV plants for more timely detection of subtle underperformance. The method has been demonstrated on eight datasets from large-scale plants with high accuracy of detection. Results are validated using aerial infrared scanning. String outages are detected with a true positive rate of 73 percent and tracker issues are detected with a true positive rate of 88 percent. The developed method can be uniformly applied to photovoltaic plants across a range of scales and configurations to assess performance, quickly detect underperformance, and determine the source and location of failures. The results inform and improve operations and maintenance at PV plants, ultimately aiding in improved affordability, reliability, availability, and resiliency of solar electricity.

14 SOLAR ENERGY↗

DOE OSTI2021

Rapid and flexible segmentation of electron microscopy data using few-shot machine learning

Abstract Automatic segmentation of key microstructural features in atomic-scale electron microscope images is critical to improved understanding of structure–property relationships in many important materials and chemical systems. However, the present paradigm involves time-intensive manual analysis that is inherently biased, error-prone, and unable to accommodate the large volumes of data produced by modern instrumentation. While more automated approaches have been proposed, many are not robust to a high variety of data, and do not generalize well to diverse microstructural features and material systems. Here, we present a flexible, semi-supervised few-shot machine learning approach for segmentation of scanning transmission electron microscopy images of three oxide material systems: (1) epitaxial heterostructures of SrTiO 3 /Ge, (2) La 0.8 Sr 0.2 FeO 3 thin films, and (3) MoO 3 nanoparticles. We demonstrate that the few-shot learning method is more robust against noise, more reconfigurable, and requires less data than conventional image analysis methods. This approach can enable rapid image classification and microstructural feature mapping needed for emerging high-throughput characterization and autonomous microscope platforms.

36 MATERIALS SCIENCE↗

DOE OSTI2025

Evaluation of cloud height, optical thickness, and phase retrievals from the CHROMA algorithm applied to Sentinel-3 OLCI data

We previously developed the Cloud Height Retrieval from O 2 Molecular Absorption (CHROMA) algorithm for the Ocean Color Instrument (OCI) on the new NASA Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) mission. Here, we apply CHROMA to observations from the Ocean Land Colour Instrument (OLCI) to guide expectations for PACE, as it will take some time to obtain large-scale validation data for OCI. We use cloud top height (CTH), phase, and (for liquid clouds) cloud optical thickness (COT) data from the ground-based Atmospheric Radiation Measurement (ARM) network to evaluate the OLCI retrievals. We found that OLCI and Moderate Resolution Imaging Spectroradiometer (MODIS) CTH compare similarly well to the ARM reference. OLCI has a tendency to underestimate CTH as CTH increases, and algorithm assumptions about cloud geometric thickness may contribute to this. ARM COT from multifilter shadowband radiometers (MFRSR) and Sun photometers are well-correlated with one another, albeit with a roughly 30 % offset on average; OLCI and MODIS COT agree more closely with the MFRSR data. OLCI retrieval uncertainty estimates show skill at telling low-uncertainty cases from high-uncertainty ones, although CTH uncertainties are underestimated. Additionally, we compare the OLCI data to satellite retrievals based on thermal infrared measurements from MODIS and Sea and Land Surface Temperature Radiometer (SLSTR) data. Differences are broadly consistent with physical expectations based on the A-band vs. thermal techniques, although one key challenge in such aggregated comparisons is different cloud masking sensitivities and algorithm failure rates meaning additional sampling differences are introduced. We conclude by discussing the transition to and possible enhancements for PACE OCI.

Sayer, Andrew M. [Univ. of Maryland Baltimore Coun↗