Search NASA⌕ Search

SEARCH · Search NASA

Results for “process data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A Visual Analytics Framework for Reviewing Streaming Performance Data

Understanding and tuning the performance of extreme-scale parallel computing systems demands a streaming approach due to the computational cost of applying offline algorithms to vast amounts of performance log data. Analyzing large streaming data is challenging because the rate of receiving data and limited time to comprehend data make it difficult for the analysts to sufficiently examine the data without missing important changes or patterns. To support streaming data analysis, we introduce a visual analytic framework comprising of three modules: data management, analysis, and interactive visualization. The data management module collects various computing and communication performance metrics from the monitored system using streaming data processing techniques and feeds the data to the other two modules. The analysis module automatically identifies important changes and patterns at the required latency. In particular, we introduce a set of online and progressive analysis methods for not only controlling the computational costs but also helping analysts better follow the critical aspects of the analysis results. Finally, the interactive visualization module provides the analysts with a coherent view of the changes and patterns in the continuously captured performance data. Through a multi-faceted case study on performance analysis of parallel discrete-event simulation, we demonstrate the effectiveness of our framework for identifying bottlenecks and locating outliers.

Human-centered computing↗
DOE OSTI2023

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

dbprocessing

dbprocessing is a python framework for automating data processing pipelines. The dbprocessing package uses a configuration file to define the input, intermediate, and output data products for a particular data type as well as the processes that connect those data products. In addition, dbprocessing uses python scripts called inspectors to determine if a particular file matches a configured data product. If a particular data product is found, it is ingested into the dbprocessing database (sqlite3 or postgres) which then triggers all of the chained processes to the final data product. While the framework is written in python, it can run software in any language but may require a “wrapper” to translate the dbprocessing command line arguments to the form expected by the software.

Walker, Andrew↗
DOE OSTI2020

Damage Pseudo Cross Section Generator

This software is a python script for generating group damage pseudo-cross-sections for specific elements, and tabulating them in forms useful to python or to office-open workbooks (xlsx). The damage pseudo-cross-section is needed for calculating how much neutron radiation damage a structural material receives in a reactor measured in displacements per atom (DPA). The process of generating these pseudo-cross-sections is laborious as it requires running the nuclear data processing code, NJOY, for hundreds of isotopes, and then post-processing the data. This also allows many opportunities for human error to be involved. This script eliminates these problems by fully automating the process.

Gale, MicahD.↗
DOE OSTI2026

Data and scripts associated with “Moisture content modulates DOM thermodynamic regulation of oxygen consumption in drying streambed sediments”

This data package is associated with the publication “Moisture content modulates DOM thermodynamic regulation of oxygen consumption in drying streambed sediments” published in Scientific Reports (Garayburu-Caruso et al., 2026). The package contains processed data products and scripts used to quantify how drying and re-inundation of riverbed sediments influence dissolved organic matter (DOM) thermodynamic properties and their relationship with sediment oxygen (O₂) consumption across 33 stream sites in the contiguous United States. The data package contains DOM thermodynamic metrics (e.g., Gibbs free energy of carbon oxidation and thermodynamic efficiency), and O₂ consumption along with watershed-scale climate and land-cover metrics used as explanatory variables in the analyses. Underlying unprocessed and processed ultrahigh-resolution mass spectrometry data, oxygen consumption rates from laboratory moisture-manipulation experiments, within-sample environmental properties, sediment moisture content and contextual field measurements are archived separately at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/2428003 (Laan et al., 2024) and https://data.ess-dive.lbl.gov/datasets/doi:10.15485/1923689 (Forbes et al.,2023). A preliminary version of this data package was published in February 2026 at the time of manuscript submission. It was updated in June 2026, at the time of manuscript acceptance, to include the finalized data and additional metadata (readme, data dictionary, and file level metadata). For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. At the top level, the data package is organized into five main folders: (1) Data, (2)Figures, (3) Map, (4) GAM_Reulsts, and (5) src. The Data folder contains analysis-ready tabular files with oxygen consumption rates, DOM thermodynamic properties by site and treatment, site-level environmental variables, watershed-scale metrics, and other derived variables referenced in the manuscript. The Figures folder contains static image files associated with the main text and supplemental figures, while the Map folder includes spatial data and map-layer files used to create the sampling-location map. The GAM results folder contains the results for each of the general additive model (GAM).The src folder contains R scripts used to perform data processing, statistical analyses (including clustering, generalized additive models, and threshold analysis), and figure generation. This data package is associated with a GitHub repository found at https://github.com/WHONDRS-Hub/ECA_DOM_Thermodynamics.

Dissolved organic matter↗
DOE OSTI2023

Utah FORGE: Neubrex Well 16B(78)-32 Circulation Test Fiber Optics Monitoring Data and Reports - July, 2023

This dataset features Distributed Acoustic Sensing (DAS) and fiber optics monitoring data acquired by Neubrex Energy Services during the Utah FORGE Well 16B(78)-32 circulation test in July 2023. DAS and fiber optic monitoring data include absolute strain, strain change, strain change rate, distributed temperature sensing (DTS), and frequency band extraction (FBE). Data processing was completed between September and October 2023. The dataset includes the fiber optic sensing data collected during these operations, along with a data collection report, descriptions of the data, and instructions for reading the data files.

15 GEOTHERMAL ENERGY↗
DOE OSTI2023

Long–short-term memory encoder–decoder with regularized hidden dynamics for fault detection in industrial processes

The ability of recurrent neural networks (RNN) to model nonlinear dynamics of high dimensional process data has enabled data-driven RNN-based fault detection algorithms. Previous studies have focused on detecting faults by identifying the discrepancies in data distribution between the faulty and normal data, as reflected in prediction errors generated by RNN models. However, in industrial processes, variations in data distribution can also result from changes in normal control setpoints and compensatory control adjustments in response to disturbances, making it hard to differentiate between normal and faulty conditions. This paper proposes a fault detection method utilizing a long short-term memory (LSTM) encoder–decoder structure with regularized hidden dynamics and reversible instance normalization (RevIN) to compactly represent high-dimensional measurements for effective monitoring. During training, the hidden states of the model are regularized to form a low-dimensional latent space representation of the original multivariate time series data. As a result, the prediction errors of the latent states can be used to monitor the abnormal dynamic variations, while the reconstruction errors of the measured variables are used to monitor the abnormal static variations. Furthermore, the proposed indices can reflect operating conditions, even when the distribution of test data changes, which helps distinguish faults from normal adjustments and disturbances that controllers can settle. Here, data from numerical simulation and the Tennessee Eastman process are used to illustrate the effectiveness of the proposed fault detection method.

42 ENGINEERING↗
DOE OSTI2018

A programmable shared-memory system for an array of processing-in-memory devices

Processing in memory (PIM), the concept of integrating processing directly with memory has been attracting a lot of attention, since PIM can assist in overcoming the throughput limitation caused by data movement between CPU and memory. The challenge, however, is that it requires the programmers to have a deep understanding of the PIM architecture to maximize the benefits such as data locality and parallel thread execution on multiple PIM devices. In this study, we present AnalyzeThat, a programmable shared-memory system for parallel data processing with PIM devices. Thematic to AnalyzeThat is a rich PIM-aware data structure (PADS), which is an encapsulation that integrally ties together the data, the analysis tasks and the runtime needed to interface with the PIM device array. The PADS abstraction provides (i) a sophisticated key-value data container that allows programmers to easily store data on multiple PIMs, (ii) a suite of parallel operations with which users can easily implement data analysis applications, and (iii) a runtime, hidden to programmers, which provides the mechanisms needed to overlay both the data and the tasks on the PIM device array in an intelligent fashion, based on PIM-specific information collected from the hardware. We have developed a PIM emulation framework called AnalyzeThat. In conclusion, our experimental evaluation with representative data analytics applications suggests that the proposed system can significantly reduce the PIM programming effort without losing its technology benefits.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Overhead longwave infrared hyperspectral material identification using radiometric models

Material detection algorithms used in hyperspectral data processing are computationally efficient but can produce relatively high numbers of false positives. Material identification performed as a secondary processing step on detected pixels can help separate true and false positives. This paper presents a material identification processing chain for longwave infrared hyperspectral data of solid materials collected from airborne platforms. The algorithms utilize unwhitened radiance data and an iterative algorithm that determines the temperature, humidity, and ozone of the atmospheric profile. Pixel unmixing is done using constrained linear regression and Bayesian Information Criteria for model selection. The resulting product includes an optimal atmospheric profile and full radiance material model that includes material temperature, abundance values, and several fit statistics. A logistic regression method utilizing all model parameters to improve identification is also presented. This paper details the processing chain and provides justification for the algorithms used. Several examples are provided using modeled data at different noise levels.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Dataset for scientific paper "Simulated plant‑mediated oxygen input has strong impacts on fine‑scale porewater biogeochemistry and weak impacts on integrated methane fluxes in coastal wetlands", a modeling study based on field observation at the tidal salt marshes of the Parker River Estuary, Massachusetts, United States

This dataset is the raw and processed data for the paper "Simulated plant ‑ mediated oxygen input has strong impacts on fine ‑ scale porewater biogeochemistry and weak impacts on integrated methane fluxes in coastal wetlands". This study investigated how plant-mediated oxygen input affects subsurface biogeochemical reactions of organic carbon degradation and the resulting methane emissions of coastal wetlands by model simulation. We used the subsurface geochemical simulator PFLOTRAN for the modeling, which produced the simulated changes in porewater chemical substances and methane emissions over 10 days under different scenarios of plant-mediated oxygen input.Specifically, this dataset contains: 1) the input files for PFLOTRAN of all simulation runs conducted in this study. Those files are with an extension of ".in", containing information of the biogeochemical reaction network (stoichiometry, reaction rate, Monod constants, etc), fluid flow rate and oxygen concentration in the fluid which together simulated the plant-mediated oxygen input, the configuration of artificial reactions that simulated the methane fluxes, etc. The PFLOTRAN input files are text files, which can be opened by NotePad, but running these input files will require proper installation of PFLOTRAN (instruction: https://documentation.pflotran.org/user_guide/how_to/installation/installation.html). 2) the raw and processed model output from PFLOTRAN of all simulation runs, and 3) the python scripts used to process the raw model output, including random allocation of root cells, converting raw data into organized formats, calculating the methane fluxes based on the model output, data visualization, etc. The raw and processed model output from PFLOTRAN are in .spydata format, which can be viewed with Python. and 3) the python scripts for data processing and analysis are programming scripts, which can be opened with Python.This modeling work, in particular the model parameterization of root density and initial conditions of porewater concentrations of biogeochemical substances, was based on field measurements at the salt marsh of the Upper Parker River Estuary, Massachusetts, United States.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Utah FORGE: InSAR Data 2019

This dataset contains Interferometric Synthetic Aperture Radar (InSAR) data used for ground deformation monitoring during Phase 2C of the Utah FORGE project. The dataset includes measurements of the mean rate of range change and associated standard errors, provided in both CSV and NetCDF formats. Supporting materials include histograms, maps, and MATLAB scripts used for data processing, as well as unit vector information describing the satellite's sensor orientation. Compressed archives contain additional metadata and masked range change data for individual interferometric pairs. A README file is included to provide further details on the dataset's structure and contents.

15 GEOTHERMAL ENERGY↗
DOE OSTI2025

Pyrogenic Organic Matter Laboratory Experiment: Aerobic Respiration and Geochemistry from Variably Inundated Stream Sediments (v3)

This dataset supports a broader study examining the effects of variable inundation and pyrogenic organic matter on ecosystem respiration. The dataset provides data generated from a laboratory batch experiment investigating the interaction between variable inundation conditions (wet and dry sediment) and pyrogenic organic matter (burned and unburned treatments). The contents include time series dissolved oxygen, sediment geochemistry data, and field metadata (including qualitative information on instream and river corridor characteristics). This data package was originally published in November 2025. It was updated in April 2026 (v2; new and modified files) and May 2026 (v3; modified files). See the change history section in the readme for more details For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) file-level metadata; (2) data dictionary; (3) field metadata; (4) international generic sample number (IGSN) mapping file; (5) readme; (6) field protocol; (7) sample name metadata; (8) an environmental context picture for the dry and inundated sampling locations; and (9) a subfolder with sample data from the sediment incubation experiment. The sample data subfolder contains (1) dissolved organic carbon (DOC, measured as non-purgeable organic carbon, NPOC); (2) total nitrogen (TN); (3) gravimetric moisture; (4) partial pressure and production rates of carbon dioxide, methane, and nitrous oxide; (5) field wet sediment mass, dry sediment mass, water mass, and field wet sediment volume in incubation and sediment NPOC/TN vials; (6) methods codes; (7) respiration rates, pH, and temperature from after the incubation, raw time series dissolved oxygen and temperature, and a subfolder containing associated plots and scripts; (8) ions; (9) FTICR-MS methods; and (10) a subfolder of 12 Tesla (12T) FTICR-MS data. This folder contains the CoreMS processed data and three subfolders, one containing the .xml files, one containing the CoreMS output files, and the other containing instructions and scripts for processing the files in CoreMS (https://github.com/EMSL-Computing/CoreMS). All files are .csv, .pdf, .R, .xml, .html, .Rmd, .py, .cal, .json, or .jpg.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Leveraging Pre-Built Catalogs and Object-Level Scheduling to Eliminate I/O Bottlenecks in HPC Environments

Modern High-Performance Computing (HPC) environments face mounting challenges due to the shift from large to small file datasets, along with an increasing number of users and parallelized applications. As HPC systems rely on Parallel File Systems (PFS), such as Lustre for data processing, performance bottlenecks stemming from Object Storage Target (OST) contention have become a significant concern. Existing solutions, such as LADS with its object-level scheduling approach, fall short in large-scale HPC environments due to their inability to effectively address metadata I/O bottlenecks and the growing number of I/O processes. This study highlights the pressing need for a comprehensive solution that tackles both OST contention and metadata I/O challenges in diverse HPC workloads. To address these challenges, we propose SwiftLoad, an object-level I/O scheduling framework that leverages a metadata catalog to enhance the performance and efficiency of parallel HPC utilities. The adoption of the metadata catalog mitigates the metadata I/O bottlenecks that commonly occur in HPC utilities, a challenge that is particularly pronounced in object-level I/O scheduling. SwiftLoad addresses OST contention and the uneven distribution of I/O processes across different OSTs through mathematical modeling and incorporates a Loader Configuration Module to regulate the number of I/O processes. Evaluated with two representative utilities—data deduplication profiling and data augmentation—SwiftLoad achieved performance improvements of up to 5.63x and 11.0x, respectively, on a production supercomputer.

HPC↗
DOE OSTI2018

Graphene milling dynamics during helium ion beam irradiation

We explore the potential of the Helium Ion Microscope (HIM) as a tool for direct-write patterning of graphene and describe the underlying processes of graphene milling with image data processing. Controlled helium ion irradiation of suspended graphene conducted while monitoring the mill in-situ revealed the localized formation of nanopores, their growth, and coalescence. We also explore the effects of defects on the milling dynamics, and show that pre-exposed membranes rupture by cracking and rapid crack propagation at the edges of the growing defects. The mechanism for the rupturing process is described by local defect formation by excessive irradiation of helium ions, dictated by the scanning direction of the beam. As a result, these findings enrich fundamental understanding of the graphene milling process with a helium ion beam that, is necessary for high-resolution and high throughput patterning of graphene with nanoscale precision.

36 MATERIALS SCIENCE↗
DOE OSTI2018

Modeling of time-lapse seismic monitoring using CO 2 leakage simulations for a model CO 2 storage site with realistic geology: Application in assessment of early leak-detection capabilities

Time-lapse surface seismic surveys have been widely used at carbon sequestration sites for site characterization, monitoring subsurface CO 2 plume migration, and detecting potential CO 2 leakage from a storage reservoir. Monitoring in the first permeable unit directly above the primary seal is important for early detection of CO 2 leakage. Forward modeling of time-lapse seismic data can be used to assess the utility of the seismic method for CO 2 leakage detection. We develop a workflow for forward modeling of time-lapse seismic data, including constructing seismic velocity models using flow simulation outputs, modeling of pre-stack and post-stack synthetic seismic data following seismic data processing sequence and analysis of processed synthetic time-lapse seismic data. We apply the forward modeling and analysis workflow to assessing the detectability and the earliest detection time of seismic monitoring using the hypothetical CO 2 leakage scenarios for a model geologic storage site with realistic geology. We then derive the detection thresholds using the simulated normalized root-mean-square (NRMS) differences for the no-leakage case at a range of signal-to-noise ratios, representing the background noise levels in seismic data. We then compare NRMS differences triggered by the CO 2 leakage to the detection thresholds at each time step to quantify the detectability and the earliest detection time of seismic monitoring. We analyze the effects of the acquisition parameters and elastic parameters on the produced synthetic seismic data and earliest detection time. Our modeling results indicate that high signal-to-noise ratio is needed to detect the CO 2 leakage at the model site. Minimizing the background noise in seismic data is crucial for improving the detectability of the seismic method. Increasing the shot density or increasing the dominant frequency of the source wavelet is likely to increase the possibility of the leakage detection and reduce the time needed for the detection. The elastic parameters used in the rock physics modeling have significant effects on the resultant seismic velocity models and synthetic seismic data, highlighting their critical roles in understanding and interpreting time-lapse seismic reflection data associated with CO 2 monitoring, verification and accounting activities.

58 GEOSCIENCES↗
DOE OSTI2024

Connected Traffic Signal Coordination Optimization Framework through Network-Wide Adaptive Linear Quadratic Regulator–Based Control Strategy

Traffic congestion in metropolitan areas causes several significant challenges, such as longer travel times, decreased productivity, increased fuel consumption and vehicle emissions, and even severe injuries during crashes. Traffic signal control is a management approach to reduce traffic congestion and allocate the appropriate right of way for safety and mobility efficiency, both in temporal and spatial domains. Here, this study proposes a network-wide adaptive signal control coordination optimization framework based on the linear quadratic regulator algorithm. The traffic flow conditions driven by signal control inputs are formulated based on their network-wide state-space representation. After modeling traffic control regulation constraints, an adaptive linear quadratic regulator algorithm is designed to maximize the network-wide total throughput under the current conditions. Optimal signal control split time durations for multiple intersections in the network are derived by solving the algebraic Riccati equation. Furthermore, the recursive least square parameter estimation method is employed to quantify dynamic traffic condition changes. To verify the effectiveness of this proposed signal control framework, both simulation and real-world experimental tests are conducted for multiple intersections in downtown Chattanooga, Tennessee, United States. In preparation for real-world experimental tests, pipelines for real-time data processing implementation and historical traffic flow data analysis are conducted. The test results demonstrate that the proposed control framework achieves a decrease in travel time by up to 19.4%, total time spent (TTS) by up to 11.9%, and relative queue balance (RQB) by up to 15.6%. The research findings indicate that the proposed signal control framework can be generalized to handle large scale signal control optimization network-wide.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Online learning of quadratic manifolds from streaming data for nonlinear dimensionality reduction and nonlinear model reduction

Here, this work introduces an online greedy method for constructing quadratic manifolds from streaming data, designed to enable in situ analysis of numerical simulation data on the Petabyte scale. Unlike traditional batch methods, which require all data to be available upfront and take multiple passes over the data, the proposed online greedy method incrementally updates quadratic manifolds in one pass as data points are received, eliminating the need for expensive disk input/output operations as well as storing and loading data points once they have been processed. A range of numerical examples demonstrate that the online greedy method learns accurate quadratic manifold embeddings while being capable of processing data that far exceed common disk input/output capabilities and volumes as well as main-memory sizes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Overhead longwave infrared hyperspectral material identification using radiometric models

Material detection algorithms used in hyperspectral data processing are computationally efficient but can produce relatively high numbers of false positives. Material identification performed as a secondary processing step on detected pixels can help mitigate false positives. A material identification processing chain for longwave infrared hyperspectral data of solid materials collected from airborne platforms is presented. The algorithms utilize unwhitened radiance data and Nelder–Meade numerical optimization to estimate the temperature, humidity, and ozone levels of the atmospheric profile. Pixel unmixing is done using constrained linear regression and Bayesian information criteria for model selection. The resulting identification product includes an optimal atmospheric profile and a full radiance material model that includes material temperature, abundance values, and several fit statistics. A logistic regression method utilizing the model parameters to improve identification is also presented. Furthermore, several examples are provided using modeled data at several noise levels.

58 GEOSCIENCES↗