SEARCH · Search NASA

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Near-Real-Time Material Tracking: Combining Vis–NIR Spectroscopy with Flow Sensing for Accurate Nd(III) Quantification

A fiber-optic visible–near-infrared (vis–NIR) absorption spectroscopy and flow sensor system has been developed for near-real-time tracking of Nd mass in the effluent stream from a column in a fume hood. The approach leverages two unique data streams and a partial least-squares regression (PLSR) model trained on vis–NIR absorption spectra of Nd(III) (0–1.5 M) in 1 M HNO 3 . In-line volumetric flow rate and vis–NIR spectra are measured in sequence after a chromatography column. The time stamps from each data stream are then synchronized, which allows integrated volumes to be combined with Nd(III) molarities predicted by a PLSR model to accurately calculate the Nd mass flowing through the column. This integrated measurement provides instantaneous mass flow and accumulates these data over time to obtain the total mass processed. The methodology developed in this study contributes critical technical infrastructure to improve monitoring capabilities to support chemical separations and the production of strategic materials and isotopes.

Irvine, Sawyer B. [Oak Ridge National Laboratory (↗

DOE OSTI2022

EJFAT: Towards Intelligent Compute Destination Load Balancing

To handle increased data flow, Jefferson Lab (JLab) is partnering with ESnet for development of an AI/ML directed compute work Load Balancer (LB) of UDP streamed data. The LB is FPGA based featuring dynamically configurable, low latency and high throughput destination address switching. The LB provides integration of edge and core computing to support JLab experimental programs, the Electron-Ion Collider, as well as data centers of the future. In the ESnet/JLab FPGA Accelerated Transport (EJFAT) initiative, the function of the LB Data Plane (DP) is to redirect data streams to selectable (but unknown to sender) destination hosts based on current worload and within that host to destination ports as a function of sub- stream id. This effects hierarchical scaling, first across compute machines for processing over a series of events and second, across ports so different data source sub-streams may be assigned to different processors for further parallelization. The LB Control Plane (CP) programs the DP using compute farm telemetry to direct and balance workloads across a compute cluster as the operating conditions require. While Proportional/Integrative/Derivative (PID) controllers are often seen in similar applications, here we investigate the feasibility of a Reinforcement Learning (RL) based schedule manager running in the CP to provide dynamic updates to the DP scheduling policy.

Lawrence, David↗

DOE OSTI2020

My vehicle is a data mine

In this talk we explore how analysis of vehicle data provides information of individual vehicle behaviors, information of other vehicles in the flow of traffic, and insights into the behavior of drivers. Over the last two decades traditional passenger vehicles have been transformed from integrated two-port electrical nodes to cyber-physical systems of communicating computational nodes whose individual state and control variables are shared on a standard controller area network (CAN) bus. As driver assistance systems have crept into vehicles as safety features, driver behaviors can be observed through analysis of the data streams on the CAN bus as these new nodes communicate with one another. The properties of these data streams, as well as architectures and approaches to gather the data, are important to consider when drawing conclusions on the relevance of the data in making decisions at varying levels of the information hierarchy. We will demonstrate several technical challenges associated with these data collection processes, as well as preliminary results that demonstrate application relevance of the data to behavior, traffic, and systems domains.

42 ENGINEERING↗

DOE OSTI2026

IPC-Fusion (Infrastructure Perception and Control (IPC): Multisensor Data Fusion Software) [SWR-25-153]

As part of the National Laboratory of the Rockies' (NLR’s) Infrastructure Perception and Control Laboratory, the IPC-Fusion toolkit provides a probabilistic, scalable, multi-sensor fusion framework that integrates (late-stage fusion) heterogeneous object detection data from traffic sensors to enable robust, real-time tracking of roadway occupants. The algorithmic design of the toolkit is motivated by the need for creating a digital twin of traffic at the edge in a scalable and affordable manner. The software operates by combining object-level measurements (such as position and velocity) from a suite of sensors (such as radar, lidar, camera) using Kalman filtering and probabilistic data association techniques to overcome individual sensor limitations and achieve superior tracking performance in complex traffic zones. The framework addresses key challenges including heterogeneous measurement uncertainties, asynchronous data streams, varying spatiotemporal data resolutions, robust data association, and adaptive object lifecycle management. Validated on real-world traffic intersection data including vehicles and pedestrians, IPC-Fusion demonstrates enhanced tracking reliability across scenarios involving occlusions, sensor failures, and varying traffic densities, supporting the broader IPC initiative's goal of transforming transportation infrastructure through advanced perception capabilities for intelligent transportation systems, traffic safety applications, and autonomous vehicle support.

Sandhu, Rimple [National Laboratory of the Rockies↗

DOE OSTI2026

Curation and Dissemination of Complex Multi-Modal Datasets for Radiation Detection, Localization, and Tracking

The PANDAWN sensor network in Chicago, IL, is a state-of-the-art testbed for networked, multi-modal sensing. It integrates AI/data science methods into its operation, from data acquisition to automated data labeling and curation workflows. The curation and dissemination of diverse multi-modal datasets will enable the development of new radiological/nuclear (R/N) detection, localization, and tracking algorithms and methods relevant across the nonproliferation mission space. This article first introduces the PANDAWN sensor network and the features that make it stand out from previous multi-modal data acquisition efforts. We then review the various data streams acquired on the PANDAWN nodes and present the implementation of an automated data curation pipeline that includes the labeling of radiation and contextual data streams. Here, we finally provide a short overview of different studies that leveraged the curated datasets.

Data curation↗

DOE OSTI2021

krowkee

krowkee is a toolkit for scalably and efficiently summarizing many data streams in distributed memory. krowkee is intended for applications where one needs to summarize huge loosely structured data, such as matrices or graphs, where individual components such rows/columns or vertex adjacency information are impractical to store and directly inspect. krowkee ingests these objects as data streams - unstructured, arbitrarily ordered lists of updates - and accumulates summaries thereof in the form of data sketches.

Dunton, AlecM.↗

DOE OSTI2021

Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow

Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks, a key tool for reasoning about temporal completeness in infinite streams.First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches:• Computing a single correct answer, as in notifications.• Reasoning about a lack of data, as in dip detection.• Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models.• Safely and punctually garbage collecting obsolete inputs and intermediate state.• Surfacing a reliable signal of overall pipeline health.Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.

Akidau, Tyler↗

DOE OSTI2026

Towards a self-driving trigger at the LHC: adaptive response in real time

Real-time data filtering and selection—or trigger—systems at high-throughput scientific facilities such as the experiments at the Large Hadron Collider must process extremely high-rate data streams under stringent bandwidth, latency, and storage constraints. Yet these systems are typically designed as static, hand-tuned menus of selection criteria grounded in prior knowledge and simulation. In this work, we further explore the concept of a self-driving trigger, an autonomous data-filtering framework that reallocates resources and adjusts thresholds dynamically in real-time to optimize signal efficiency, rate stability, and computational cost as instrumentation and environmental conditions evolve. We introduce a benchmark ecosystem to emulate realistic collider scenarios and demonstrate real-time optimization of a menu including canonical energy sum triggers as well as modern anomaly-detection algorithms that target non-standard event topologies using machine learning. Using simulated data streams and publicly available collision data from the Compact Muon Solenoid experiment, we demonstrate the capability to dynamically and automatically optimize trigger performance under specific cost objectives without manual retuning. Our adaptive strategy shifts trigger design from static menus with heuristic tuning to intelligent, automated, data-driven control, unlocking greater flexibility and discovery potential in future high-energy physics analyses.

Emami, Shaghayegh [Michigan U.] (ORCID:00090007589↗

DOE OSTI2022

CANShield: Signal-based Intrusion Detection for Controller Area Networks

Modern vehicles rely on complex cyber-physical systems made up of hundreds of electronic control units (ECUs) connected through controller area network (CAN) buses. However, the CAN bus attack surface is increasing due to advanced features in automobiles, making it prone to injection attacks. The ordinary injection attacks disrupt the typical timing properties of the CAN data stream, and the rule-based intrusion detection systems (IDS) can easily detect them. However, advanced attackers can inject false data to the signal level, maintaining the regular pattern/frequency of the CAN messages. Such attacks can bypass the rule-based IDS or any anomaly-based IDS built on binary payload data. To make the vehicles robust against such intelligent attacks, we propose CANShield, a signal-based intrusion detection framework for the CAN bus that consists of three modules. A data preprocessing module handles the high-dimensional CAN data stream at the signal level and make them suitable for any machine learning model. A data analyzer module consists of multiple deep autoencoder networks, each analyzing the time series data from a different perspective. Finally, an attack detection module uses an ensemble method to make the final decision. Evaluation results on a standard signal-based dataset show the effectiveness of the CANShield in detecting five advanced attacks.

Shahriar, Md Hasan↗

DOE OSTI2026

The LCLStream Ecosystem for Multi-Institutional Dataset Exploration

We describe a new end-to-end experimental data streaming framework designed from the ground up to support new types of applications – AI training, extremely high-rate X-ray time-of-flight analysis, crystal structure determination with distributed processing, and custom data science applications and visualizers yet to be created. Throughout, we use design choices merging cloud microservices with traditional HPC batch execution models for security and flexibility. This project makes a unique contribution to the DOE Integrated Research Infrastructure (IRI) landscape. By creating a flexible, API-driven data request service, we address a significant need for high-speed data streaming sources for the X-ray science data analysis community. With the combination of data request API, mutual authentication web security framework, job queue system, high-rate data buffer, and complementary nature to facility infrastructure, the LCLStreamer framework has prototyped and implemented several new paradigms critical for future generation experiments.

Rogers, David [ORNL] (ORCID:0000000251871768)↗

DOE OSTI2024

Arroyo Stream Processing Toolset (arroyopy) v0.1.0

Processing event or streaming data presents several technological challenges. A variety of technologies are often used by scientific user facilities. ZMQ is used to stream data and messages in a peer-to-peer fashion. Message brokers like Kafka, Redis Pubsub, EPICS PVA and RabbitMQ are often employed to route and pass messages from instruments to processing workflows. Arroyopy provides an API and structure to flexibly integrate with these tools and incorporate arbitrarily complex processing workflows, letting the hooks to the workflow code be independent of the connection code and hence reusable at a variety of instruments.

Chavez Esparza, Tanny Andrea [Lawrence Berkeley Na↗

DOE OSTI2018

ShareAnalytics

ShareAnalytics is a lightweight extensible platform for collaborative data analytics and facility characterization via Non-intrusive Load Monitoring (NILM). It consists of three core components: a full stack web application, a dashboard tool for analyzing streaming data and a High Performance Computing (HPC) cluster leveraging slurm for resource management to perform real time analysis. The entire framework is implemented in Docker for ease of deployment and to allow the user to test the platform before making a significant investment in hardware if they require additional computing resources. While many other platforms for analyzing streaming data exists, ShareAnalytics is unique in that HPC capabilities and facility characterization via scored analytics are fully integrated into the platform.

Oesch, TimothyS↗

DOE OSTI2021

Wind Energy Forecasting with the Weather Research and Forecasting Model

This was a collaborative effort between Lawrence Livermore National Security, LLC as manager and operator of Lawrence Livermore National Laboratory (LLNL) and Siemens Energy, Inc. (Siemens) to develop a wind resource forecasting tool. LLNL was to develop an independent high-resolution mesoscale modeling capability forecasting tool that could be implemented in conjunction with existing wind farm control and monitoring software to provide forecasting of wind resources using local observations of winds and temperature. Research with LLNL’s state-of-the-art large-eddy simulation meteorological prediction model, based on the community WRF model and innovative turbulence parameterizations, would improve that model’s applicability to large wind farms offshore and in complex terrain. The modeling capability would include uncertainty quantification. Finally, the application of the modeling tool and existing global climate change predictions would enable the delineation of the likely effects of climate change on wind resources. Siemens was to provide high time resolution hub-height wind speed and other meteorological data streams, including temperature profiles from wind farms, for LLNL to incorporate into the modeling system, to validate and tune this forecasting model for their locations of interest. These data streams would also be used for longer-term studies of correlations of wind resources to climate oscillations to indicate how long-term climate change trends may affect the available wind resource. Siemens would also provide information and observations of turbine wakes for incorporation into the modeling tool. By implementing state-of-the-art turbulence parameterizations into a simulation model and/or ensembles of simulation models, and by integrating real-time hub height wind speed and other meteorological datastreams from wind farms into that model or ensemble of models, LLNL would develop a forecasting tool that could be implemented by Siemens as an add-on to existing wind farm control and monitoring software to provide owners with useful resource forecasting. The desired outcome was that the accuracy level of the output would be sufficient to substantiate power output commitments. The final deliverable for this work would consist of a document outlining the algorithms and software tools that could be integrated into Siemens Wind Park Supervisor.

17 WIND ENERGY↗

DOE OSTI2023

MDDC Multi-Length Scale Data Architecture Contribution Report – PNNL, INL, ANL, LANL and ORNL

This report offers a comprehensive view of data streams currently generated at Pacific Northwest National Laboratory, Idaho National Laboratory, Argonne National Laboratory, Los Alamos National Laboratory, and Oak Ridge National Laboratory set to integrate into the evolving Multi-Dimensional Data Correlation framework at Oak Ridge National Laboratory. Developed by the Advanced Materials and Manufacturing Technologies program, the Multi-Dimensional Data Correlation framework serves as a cutting-edge software to manage data relevant to advanced manufacturing and material behavior in advanced reactors. The report defines data streams, highlights their generation methods and visualization methods both for experimental and computational aspects relevant to the Advanced Materials and Manufacturing Technologies project. A logical next step for this work is to integrate the MDDC framework into PNNL’s, INL’s, ANL’s, LANL’s and ORNL’s fabrication, experimentation, and modelling workflows. This would require setting up the MDDC framework at PNNL, INL, ANL, and LANL and integrating it into the data collection and storage for these different activities.

36 MATERIALS SCIENCE↗

DOE OSTI2020

The Manufacturing Data and Machine Learning Platform: Enabling Real-time Monitoring and Control of Scientific Experiments via IoT

IoT devices and sensor networks present new opportunities for measuring, monitoring, and guiding scientific experiments. Sensors, cameras, and instruments can be combined to provide previously unachievable insights into the state of ongoing experiments. However, IoT devices can vary greatly in the type, volume, and velocity of data they generate, making it challenging to fully realize this potential. Indeed, synergizing diverse IoT data streams in near-real time can require the use of machine learning (ML). In addition, new tools and technologies are required to facilitate the collection, aggregation, and manipulation of sensor data in order to simplify the application of ML models and in turn, fully realize the utility of IoT devices in laboratories. Here we will demonstrate how the use of the Argonne-developed Manufacturing Data and Machine Learning (MDML) platform can analyze and use IoT devices in a manufacturing experiment. MDML is designed to standardize the research and operational environment for advanced data analytics and AI-enabled automated process optimization by providing the infrastructure to integrate AI in cyber-physical systems for in situ analysis. We will show that MDML is capable of processing diverse IoT data streams, using multiple computing resources, and integrating ML models to guide an experiment.

Machine Learning↗

DOE OSTI2020

KAZRARSCL-c0-Cloud Boundaries subset

The KAZR-ARSCL VAP provides cloud boundaries and best-estimate time-height fields of radar moments. The VAP merges corrected, but uncalibrated, KAZR moments from all active radar modes with cloud base and cloud mask observations from the micropulse lidar (MPL), cloud base from the ceilometer, as well information from soundings, rain gauge, and microwave radiometer instruments to produce two data streams, one with best-estimate cloud base and cloud layer boundaries, and another which also includes best-estimate time-height fields of radar moments. This DOI is for the data stream that contain cloud layer boundaries only. Please note that the reflectivity used in this level c0 product is uncalibrated.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

WHONDRS 2016 Sediment Organic Matter Characterization Data from Streams across HJ Andrews Experimental Forest, Oregon

This dataset supports a broader synoptic effort to map morphological, hydrological, chemical, and biological conditions across a fifth-order mountain stream network. Samples were generated through a collaborative synoptic sampling effort in 2016. The dataset provides sediment Fourier Transform Ion Cyclotron Resonance Mass Spectrometry (FTICR-MS) from 60 sites across the HJ Andrews Experimental Forest, Oregon (https://andrewsforest.oregonstate.edu). Related data were collected as part of the event and were published separately in collaboration with other team members. The data are available at http://www.hydroshare.org/resource/ea6c0832885a46c3939e7bb22e48e754 and are described within https://doi.org/10.5194/essd-11-1567-2019 (Ward et al., 2019). The hydroshare data package contains processed FTICR-MS data from the samples included in this data package. The data were processed via Formultitude (previously called Formularity; https://github.com/PNNL-Comp-Mass-Spec/Formultitude). However, we have re-processed the data using Core-MS and included it in this data package. Additional related data collected in 2025 from a similar effort can be found at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/3023310 and http://www.hydroshare.org/resource/b274c4a234bf4b12b7cb8a54a696c629. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) a folder of sample data; (2) data dictionary; (3) file-level metadata; (4); (5) coordinates; and (6) readme. The sample data subfolder contains 12 Tesla (12T) FTICR-MS data. This folder contains the processed data and three subfolders, one containing the .xml files, one containing the CoreMS output files, and the other containing instructions and scripts for processing the files in CoreMS (https://github.com/EMSL-Computing/CoreMS). All files are .csv, .pdf, .R, .xml, .Rmd, .py, .cal, or .json.

Biogeochemistry↗

DOE OSTI2023

Assimilation of multiple datasets results in large differences in regional- to global-scale NEE and GPP budgets simulated by a terrestrial biosphere model

In spite of the importance of land ecosystems in offsetting carbon dioxide emissions released by anthropogenic activities into the atmosphere, the spatiotemporal dynamics of terrestrial carbon fluxes remain largely uncertain at regional to global scales. Over the past decade, data assimilation (DA) techniques have grown in importance for improving these fluxes simulated by terrestrial biosphere models (TBMs), by optimizing model parameter values while also pinpointing possible parameterization deficiencies. Although the joint assimilation of multiple data streams is expected to constrain a wider range of model processes, their actual benefits in terms of reduction in model uncertainty are still under-researched, also given the technical challenges. In this study, we investigated with a consistent DA framework and the ORCHIDEE-LMDz TBM–atmosphere model how the assimilation of different combinations of data streams may result in different regional to global carbon budgets. To do so, we performed comprehensive DA experiments where three datasets (in situ measurements of net carbon exchange and latent heat fluxes, spaceborne estimates of the normalized difference vegetation index, and atmospheric CO 2 concentration data measured at stations) were assimilated alone or simultaneously. We thus evaluated their complementarity and usefulness to constrain net and gross C land fluxes. We found that a major challenge in improving the spatial distribution of the land C sinks and sources with atmospheric CO 2 data relates to the correction of the soil carbon imbalance.

54 ENVIRONMENTAL SCIENCES↗