SEARCH · Search NASA

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

DOE OSTI2025

WHONDRS River Corridor Sediment and Water Geochemistry and In Situ Sensor Data from 7 Perennial and 7 Intermittent Streams across San Antonio, Texas (v3)

This dataset supports a broader study examining the effects of intermittency on sediment respiration. The dataset provides sediment and surface water geochemistry and in situ sensor data from 7 perennial and 7 intermittent streams in San Antonio, Texas. Each stream/site was visited both in summer during base flow (July-September 2023) and winter during peak flow (January-February 2024). Related data were collected and will be published separately in collaboration with A. Veach. The data package was originally published in April 2025. It was updated in June 2025 (v2; modified and new files) and September 2025 (v3; modified files). See the change history section in the readme for more details. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. This dataset is comprised of two folders of field photos and videos, one folder of raw Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) data and one main data folder containing (1) file-level metadata; (2) data dictionary; (3) field metadata; (4) readme; (5) international generic sample number (IGSN) mapping file; (6) field protocol; (7) a subfolder with sample data; and (8) a subfolder with sensor data. The sample data subfolder contains (1) surface water and sediment dissolved organic carbon (DOC, measured as non-purgeable organic carbon, NPOC) data and averages; (2) surface water and sediment total nitrogen data and averages; (3) sediment grain size data; (4) sediment iron (II) data and averages; (5) wet sediment mass, dry sediment mass, water mass, and wet sediment volume in incubation and sediment ICR vials; (7) sediment incubation respiration rate data and averages; (8) normalized respiration rate data and averages; (9) methods codes; (10) sediment percent carbon and nitrogen; (11) sediment X-ray diffraction (XRD) data; (12) gravimetric moisture and averages; (13) a subfolder with sediment incubation respiration data, scripts, and plots; (14) surface water and sediment FTICR methods; and (15) a subfolder of 9.4 Tesla (9.4T) FTICR-MS data. This folder contains five subfolders, one containing the sediment .xml data files, one containing the water .xml files, one containing the sediment CoreMS output files, one containing the water CoreMS output files, and the other containing instructions and scripts for processing the files in CoreMS (https://github.com/EMSL-Computing/CoreMS). The sensor data subfolder contains (1) a subfolder with miniDOT dissolved oxygen and temperature data and plots; (2) miniDOT dissolved oxygen and temperature summary data; and (3) miniDOT installation methods. All files are .csv, .pdf, .R, .xml, .d, .html, .Rmd, .py, .cal, .json, .jpg, .jpeg, .png, .mov, or .mp4. CORRECTION: The data processing methods for FTICR described in “v3_WHONDRS_AV1_Methods_Codes.csv” mistakenly indicate that users should process the data in Formultitude. The corrected description should read: “Both unprocessed and processed data are provided to allow users flexibility in data processing. Instructions and scripts for processing the data using CoreMS are included.” CORRECTION: Carbon and nitrogen content are reported as percentages. The current column headers "01395_C_percent_per_mg" and "01397_N_percent_per_mg" are incorrect. These should read "01395_C_percent" and "01397_N_percent" and will be corrected in the next version of this data package.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Public Data Set: Modeling of Injected Current Stream-Induced 3D Perturbations in Local Helicity Injection Plasmas

This public dataset contains openly-documented, machine readable digital research data corresponding to figures published in C.E. Schaefer et al., “Modeling of Injected Current Stream-Induced 3D Perturbations in Local Helicity Injection Plasmas,” accepted for publication in Physics of Plasmas.

Schaefer, Carolyn E [University of Wisconsin-Madis↗

DOE OSTI2026

HPC ODA Commons [SWR-26-003]

HPC ODA Commons is a community-driven platform for standardizing HPC operational data analytics. HPC sites generate enormous volumes of operational data - scheduler logs, accounting records, monitoring streams - but turning that data into actionable insight is needlessly hard. Each site builds bespoke parsers, schemas, and evaluation pipelines. Results can't be compared across institutions. Promising analytics ideas stay siloed because there's no shared language for describing the data, the experiments, or the outcomes. HPC ODA Commons fixes this by establishing community-governed contracts - versioned schemas, canonical artifacts, and benchmark recipes - that make ODA workflows discoverable, reproducible, and comparable. It pairs these standards with a practical, CLI-first toolkit that lets operators and researchers go from raw logs to standardized results without sending data off-cluster.

Menear, Kevin [National Laboratory of the Rockies ↗

DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Explore the source record for details and available documents.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Explore the source record for details and available documents.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Semi–Analytical Modeling of Transient Stream Drawdown and Depletion in Response to Aquifer Pumping

Analytical and semi–analytical models for stream depletion with transient stream stage drawdown induced by groundwater pumping are developed to address a deficiency in existing models, namely, the use of a fixed stream stage condition at the stream–aquifer interface. Here field data are presented to demonstrate that stream stage drawdown does indeed occur in response to groundwater pumping near aquifer–connected streams. A model that predicts stream depletion with transient stream drawdown is developed based on stream channel mass conservation and finite stream channel storage. The resulting models are shown to reduce to existing fixed–stage models in the limit as stream channel storage becomes infinitely large, and to the confined aquifer flow with a no–flow boundary at the streambed in the limit as stream storage becomes vanishingly small. The model is applied to field measurements of aquifer and stream drawdown, giving estimates of aquifer hydraulic parameters, streambed conductance, and a measure of stream channel storage. The results of the modeling and data analysis presented herein have implications for sustainable groundwater management.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Data and script associated with “Shifts in Rain-Snow Partitioning Drive Faster Water Transit Times in the US Pacific Northwest”

This data package contains the data and code to use and run the Water Tracer enabled version of the Weather Research and Forecasting Hydrologic model (WT-WRF-Hydro) with the Sequential Precipitation Input Tagging (SPIT) framework. It is associated with the publication “Shifts in Rain-Snow Partitioning Drive Faster Water Transit Times in the US Pacific Northwest” published in Scientific Reports (Butler et al., 2026; https://doi.org/10.1038/s41598-026-46539-1). We use the Continental U.S. (CONUSII; Rasmussen et al., 2021) dataset to force the model with an historical climate (2006–2013) and a future climate (2086–2093) with a representative carbon pathway (RCP) 8.5 scenario. We use the model to calculate water transit times in five headwater catchments within the U.S. Pacific Northwest. We also show key hydrologic and environmental variables that affect water transit times and changes in the future. Finally, we use observed data to validate the model such as stream water isotopes, snowpack characteristics, and stream discharge. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. The data package consists of 11 folders: (1) "Figures" contains the exported figures used in the manuscript; (2) "Model_Isotope_Date" contains the WT-WRF-Hydro isotope date used in model validation; (3) “Model_Outputs_Future” contains the WT-WRF-Hydro future climate outputs; (4) “Model_Outputs_Historical” contains the WT-WRF-Hydro historical climate outputs; (5) “Model_Outputs_Weights_Areas” contains the WT-WRF-Hydro weights per catchment used to calculate water transit times and isotopes in stream water; (6) “MODIS_data_scripts” contains data used to validate snow conditions in the study area; (7) “Observed_Flow_Data” contains the observed streamflow data used in model validation; (8) “Observed_Isotope_Data” contains the observed stream water isotope data used in model validation; (9) “Scripts” contains the Python scripts used to general results and the figures; (10) “Statistic_Outputs” contains the water transit time statistical outputs reported in this manuscript; (11) “Validation_SNOTEL” contains the SNOTEL data used in model validation. The files in this data package have the following file extensions: .tif, .txt, .csv, .pdf, .py, .jpg, and .png.

American River↗

DOE OSTI2026

Data Summarization and Inference at Scale

This is the final report for the DOE ASCR grant SC-0022260, Data Summarization and Inference at Scale, PI: Alex Pothen, Purdue University. The goal of the project was to solve data-intensive and compute-intensive problems in the physical sciences, engineering, information science, data science, etc. by designing and implementing new algorithms that could work with a subset of the data. The four subgoals were: (a) The solution of problems where the data is too large to be stored in the memory of a computer. In this streaming model of computation, the data arrives as a stream of elements to the computer, each element is processed as it arrives, and a decision is made to discard the data or to store it; only a small subset of the data proportional to the size of the output solution is stored, and when all the data has been streamed, a solution to the problem is computed from the stored subset. (b) The use of machine learning methods to compute solutions to data-intensive problems. The use of GPUs is critical to obtain high performance on machine learning tasks, but their memory sizes are smaller relative to that of CPUs. For large-scale problems, the data is sampled many times, and small samples are used with repetition, for robustness, to compute solutions to inference tasks. This sampling reduces the memory required to solve the problem, but attention is needed to avoid slow convergence to the solutions, and reduced accuracy of inference. We propose submodular optimization, Large Language Models, and physics-informed neural networks to enable GPU computations here. (c) Modeling and visualization of high-dimensional data using interpretable features. Clinical proteomic data sets from immunology for the detection of cancer and other diseases are temporal and high-dimensional, and algorithms for visualizing these data sets using clinically interpretable features are lacking. We propose methods that compute distances based on the optimal transportation problem and graph edit distances to address this problem. We also propose the use of optimal transport-based distances, spatial statistics, and network structure to classify image data sets, We apply these algorithms to electron micrographs of the peripheral nervous system in the digestive tract. (d) The design of data-intensive algorithms on emerging architectures, specifically, noisy, intermediate-scale quantum (NISQ) devices. Quantum computers offer the possibility of exploring large solution spaces due to the principle of superposition, but current quantum computers are limited by few qubits, short coherence times due to noise, poor interconections among the qubits, etc. We propose the use of the divide and conquer paradigm to solve large-scale problems, wherein collections of small subproblems are solved on the quantum devices, and the solutions to the subproblems are integrated into a solution for the original problem on a classical computer.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Distributed Wind Monitoring Best Practices

Accessible performance and operational data have been identified as a key enabler for distributed wind energy industry advancement. While utility-scale wind turbines benefit from reliable and continuous supervisory control and data acquisition (SCADA)-based monitoring platforms, monitoring of the U.S. fleet of distributed wind (DW) turbines has been more inconsistent, unreliable, and sometime difficult to access. Without fleet monitoring data, the industry will never understand and thus work to improve turbine under-performance and reliability issues. For the DW industry to scale up, attract investors, and boost credibility, fleetwide monitoring must be robust and reliable, select data must be made accessible to stakeholders, and the data must be in a format useful to users. To help move the industry toward a more standardized, accessible stream of monitoring data, this distributed wind monitoring best practices report attempts to cover topics including key monitoring channels, hardware, communication strategies, and accessibility. Strategic engagement with DW original equipment manufacturers (OEMs), service providers, lab and university researchers, testing organization, certification bodies, end users and solar photovoltaic (PV) monitoring experts has enabled a better understanding of the current state-of-the-art of monitoring and aided in articulating this set of best practices that will guide OEMs toward harmonized monitoring strategies, aimed at a future goal of achieving accessible performance and operational data for the entire fleet of U.S. distributed wind turbines.

17 WIND ENERGY↗

DOE OSTI2025

Stream bed topography in 2016 at the Lower Montane site in the East River Watershed, Colorado

This dataset contains stream bed topography data from the Lower Montane site in the East River Watershed, Colorado. It is intended to support hydro-biogeochemical analyses for the Watershed Function Scientific Focus Area (SFA). The stream bed was surveyed using a RTK-GPS (Real-Time Kinematic Global Positioning System) and a total station at hundreds of locations during two field campaigns, conducted in October 2015 and October 2016. The two surveys were merged into a single product under the assumption of minimal changes in stream bed topography between the years. A LiDAR (Light Detection and Ranging) derived ground surface elevation product from 2015 (see reference below) was used to interpolate the stream bed topography and estimate water depth at the time of the LiDAR survey. The dataset includes three GeoTIFF products, two *.csv data files, and three *.csv metadata files. Feel free to contact the author with any questions.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Data and scripts associated with “Non-random processes impacting organic matter chemistry are maximized in mid-order streams”

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Autonomous Anomaly Detection For Continuous Streams

The code implements the Isolation Forest (IFML) algorithm within the digital twin (DT) of the AGN-201 nuclear reactor. The DT captures real-time operational data including control rod positions, reactor power, and temperature. The IFML model isolates anomalies by detecting patterns that deviate from expected operational behavior. The algorithm recursively partitions the data and assigns anomaly scores based on the isolation of rare and different events. By tuning parameters specific to the reactor’s operational data, the IFML identifies deviations such as unauthorized material insertions or reactor reactivity shifts. The system streams data using LabView and integrates with the DeepLynx data warehouse for anomaly processing.

Trevino, Eduardo↗

DOE OSTI2025

The Role of Snowmelt and Subsurface Heterogeneity in Headwater Hydrology of a Mountainous Catchment in Colorado: A Model‐Data Integration Approach

Mountainous headwater streams are sustained by both snowmelt‐driven streamflow and groundwater discharge in the Upper Colorado River Basin. However, predicting headwater stream discharge magnitude and peak flow timing is challenging in mountainous terrains, where snowmelt rates vary with vegetation type and elevation, and heterogeneous subsurface physical properties influence groundwater storage and its release. We used a model‐data integration approach to investigate the roles of snowmelt and subsurface structure in stream discharge and groundwater level. We ran an ensemble of 100 integrated surface‐subsurface hydrologic models for a mountainous headwater catchment near Crested Butte, Colorado, USA. We also evaluated and calibrated these models against observed data sets, including snow depth measurements using distributed temperature probes, stream discharge, and groundwater levels. Calibration with multiple data sources using neural density estimators has further constrained uncertainty in subsurface properties and snowmelt rates. Results indicated that observed slower snowmelt rates in evergreen forests delayed the peak flow and baseflow onset. In upstream areas with lower subsurface permeability, water was stored within the subsurface but was not released as interflow or shallow groundwater flow, and thereby not contributing to downstream streamflow during recession limb periods. Double peaks in groundwater occurred in areas with spatial subsurface heterogeneity, in our case due to the contrast between granodiorite and Mancos shale. These process‐based insights into groundwater and snowmelt dynamics in mountainous headwaters will help improve predictions of headwater hydrology.

Wang, Lijing [University of Connecticut, Storrs, C↗

DOE OSTI2025

Applying Transfer Learning for Street-Scale Nuisance Flood Forecasting in Coastal-Urban Cities

An important challenge with Machine Learning (ML) is its transferability; that is, whether a ML model trained on one set of data can be applied to a second set of data without requiring a full re-training of the model. Transfer Learning (TL) addresses this challenge by transferring knowledge learned in the source domain (the data it was trained on) to the target domain (a second set of data that is statistically different but related, which the model was not trained on). This study investigates the use of TL for street-scale nuisance flood forecasting by exploring whether a ML model trained on data collected for one set of streets can effectively forecast flooding for another set of streets in the same city using TL. The envisioned use case is a city deploying a new flood depth monitoring sensor on a street and using TL to apply a ML model, trained on sensor data from an existing flood depth sensor network, to this new street. Eventually, the new flood depth sensor will have a sufficient dataset for training its own ML model, but TL can be used to fill the gap in time while this new dataset is being generated. This method is explored using a Long Short-Term Memory (LSTM) model trained on data for the flood-prone streets of Norfolk City, Virginia. The data used for training includes environmental time series (rainfall, tide), topographic features (Digital Elevation Model (DEM), Topographic Wetness Index (TWI), Depth To Water (DTW)), and street-scale flood depth time series obtained from a high-fidelity physics-based model, acting as a synthetic street-scale stream depth sensor dataset since actual stream depth sensor data is generally unavailable for most cities. A set of 180 flood-prone streets was used to train a base model, while another set of 180 flood-prone streets was used to re-train that model using different TL strategies. The results show that full-weight re-training proved most effective and minimal re-training of only the output layer was insufficient. The advantage of TL was most pronounced when target data was limited, meaning data collected at the new water depth sensor location included generally less than 18 flood events. As target data increased beyond 18 flood events, the benefit of TL diminished relative to training a ML model directly on the local flood events. These findings can assist cities as they implement street-scale flood sensing systems to create accurate forecasts for new sensing locations that do not yet have sufficient data records to train a local ML model.

Roy, Binata [Univ. of Virginia, Charlottesville, V↗

DOE OSTI2024

Exploring DAOS as a Burst Buffer for a 100 Gbps DAQ Real-Time Streaming System

We present an experimental evaluation of a burst buffer for a real-time DAQ streaming system designed to transmit instrument data to remote data centers. The system is based on EJ-FAT, a load balancing system capable of Nx 100Gbps streams, distributing data from event sources to processing nodes. We explore applying the DAOS system as a burst buffer to serve a number of purposes: improve resiliency, elasticity and add new functions into the processing pipeline. In the evaluation a sender transmits events over a 100Gbps network to a receiver integrated with DAOS to store the reassembled events using DAOS APIs. We evaluate the system for possible bottlenecks and provide end-to-end evaluation with a burst buffer using DAOS storage abstractions. We show that a receiver node can support 38.1 Gbps. This proves the viability of our approach and allows us to extend this work to investigate scale-out properties and new streaming optimizations.

Mei, Xinxin↗

DOE OSTI2025

The Artificial Scientist: in-Transit Machine Learning of Plasma Simulations

Large-scale simulations or scientific experiments produce petabytes of data per run. This poses massive challenges for I/O and storage when scientific analysis workflows are run manually offline. Unsupervised deep learning-based techniques to extract patterns and non-linear relations from these large amounts of data provide a way to build scientific understanding from raw data, reducing the need for manual pre-selection of analysis steps, but require exascale compute and memory to process the full dataset available. In this paper, we demonstrate a heterogeneous streaming workflow in which plasma simulation data is streamed directly to a Machine Learning (ML) application training a model on the simulation data in-transit, completely circumventing the capacity-constrained filesystem bottleneck. This workflow employs openPMD to provide a high level interface to describe scientific data and also uses ADIOS2, to transfer volumes of data that exceed the capabilities of the filesystem. We employ experience replay to avoid catastrophic forgetting in learning from this non-steady state process in a continual manner and adapt it to improve model convergence while learning in-transit. As a proof-of-concept, we approach the ill-posed inverse problem of predicting particle dynamics from radiation in a particle-incell (PIConGPU) simulation of the Kelvin-Helmholtz instability (KHI). We detail hardware-software co-design challenges as we scale PIConGPU to full Frontier, the Top-1 system as of June 2024 Top500 list.

Kelling, Jeffrey [Helmholtz-Zentrum Dresden Rossen↗

DOE OSTI2026

Data, model inputs, and analysis scripts associated with a manuscript on stream intermittency controls across spatial scales in Pacific Northwest watersheds

NOTE: The manuscript associated with this data package is currently in review. The data may be revised based on reviewer feedback. Upon manuscript acceptance, this data package will be updated with the final dataset and additional metadata. This data package is associated with the manuscript "Hydroclimatic Memory and Watershed Template Shape Stream Intermittency: Multi-scale Attribution Using Process-based Simulation and Explainable ML" by Niroula et al. (2026), submitted to Water Resources Research (WRR). The study investigates the dominant controls on stream intermittency across local, reach, and watershed scales using a coupled process-based simulation and explainable machine-learning framework. Long-term daily simulations from the Advanced Terrestrial Simulator (ATS) were used to generate wetness states and ponded-depth responses over river-corridor cells. These ATS outputs were then aggregated across scales and used to train XGBoost (eXtreme Gradient Boosting) models. SHAP (SHapley Additive exPlanations) was applied to quantify the relative importance of hydroclimatic forcings, watershed template attributes, and antecedent-memory effects in shaping intermittency behavior. The analysis is carried out for three contrasting Pacific Northwest watersheds: Oak Creek (OCW), American River Watershed (ARW), and H.J. Andrews (HJA). Across these testbeds, the package contains ATS-ready watershed inputs, ATS run configuration and selected output files, model-evaluation data products, intermittency-analysis datasets, machine-learning target-feature tables, SHAP outputs, and notebooks used to organize, analyze, and visualize results. At a high level, the package documents a workflow in which ATS provides the physically based simulation backbone and explainable machine learning is used as a post-processing attribution tool. The contents are intended to support interpretation of the manuscript figures and results, provide context for how intermittency metrics were generated at multiple scales, and preserve the key artifacts needed to understand and reuse the analysis workflow. The package contains a high-level directory summary file (`summary.txt`) and four main content folders (1) `evaluation_plots` contains evaluation figures and supporting evaluation datasets; (2) `intermittency_plots` contains intermittency-focused analysis notebook and prepared datasets; (3) `ml-training-and-shap_values_plots` contains ML training inputs, SHAP outputs, and figure-generation notebooks; and (4) `watershed_mesh_and_ats_input` contains ATS model setup materials, forcing inputs, geometry, and selected run files. More specifically, the `evaluation_plots` folder contains the notebook used for ATS evaluation plotting and site-specific evaluation datasets. These include evapotranspiration and water-balance products for three watersheds, as well as an Oak Creek field-measurement discharge file. The `intermittency_plots` folder contains the notebook used for intermittency analysis and the prepared datasets used to analyze intermittent and non-intermittent wetness behavior across the study watersheds. The `ml-training-and-shap_values_plots` folder contains notebooks and outputs for the machine-learning and explainability workflow. This includes the main XGBoost and SHAP notebook(s), a beeswarm plotting notebook, target-feature tables for machine-learning training, SHAP summary tables, and per-sample SHAP value archives. The `watershed_mesh_and_ats_input` folder contains ATS-related watershed inputs and supporting materials. This includes mesh and shape products, ATS-readable LAI and meteorological forcing inputs, selected ATS spinup and transient-run files, and a watershed workflow example notebook. Subdirectories are organized by watershed where applicable.All files are .cpg (codepage files), .csv (comma-separated values), .dbf (database files), .exo (Exodus mesh format), .h5 (HDF5 format), .ipynb (Jupyter notebooks), .pkl (Python pickle), .prj (projection files), .sh (shell scripts), .shp (shapefile geometry), .shx (shapefile index), .txt (text files), or .xml (markup data).

Advanced Terrestrial Simulator↗

DOE OSTI2026

Vulcan-Forge: Architecture and Design of a Multi-Modal Forensic Analysis Plugin for CALDERA

Forge and VULCAN together describe an open-architecture cybersecurity analysis ecosystem that unifies forensic artifact processing, detection engineering, and vulnerability intelligence within integrated platforms. Forge operates as a plugin for MITRE CALDERA, ingesting diverse evidence formats—including EVTX, PCAP/PCAPNG, CSV, JSON, YAML, XML, binaries, and archives—to construct a unified artifact graph enriched with severity scoring, TLP classification, and audit trails. It provides subsystems for artifact parsing, streaming structured-data visualization, NetworkMiner-based packet inspection, PE/.NET binary analysis, and LLM-assisted triage and rule generation, with outputs validated against CCCS-YARA and pySigma schemas. VULCAN complements this by serving as a cybersecurity analyst platform that integrates a Neo4j knowledge graph, Qdrant vector retrieval, SSVC-based triage, and a local LLM to deliver CVE intelligence and forensic analysis through a multi-source ingest pipeline drawing from NVD, CISA KEV, EPSS, MITRE ATT&CK, and CAPEC. Together, they bridge structured threat intelligence with automated forensic analysis and detection workflows.

97 MATHEMATICS AND COMPUTING↗