Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Simulation of an inductively coupled plasma with a two-dimensional Darwin particle-in-cell code

A two-dimensional particle-in-cell code for the simulation of low-frequency electromagnetic processes in laboratory plasmas has been developed. The code uses the Darwin method omitting the electromagnetic wave propagation. The Darwin method separates the electric field into solenoidal and irrotational parts. The solenoidal electric field is calculated with a new algorithm based on the equation for the electric field vorticity. The system of linear equations in the new algorithm is readily solved using a standard iterative method. The irrotational electric field is the electrostatic field calculated with the direct implicit algorithm. The code is verified by reproducing the two-stream instability, electron electromagnetic waves, and shear Alfvén waves. The code is applied to simulate an inductively coupled plasma with the driving current flowing around the plasma region. In this simulation, a ring of dense plasma forms at the initial stage but then the density becomes maximal in the center and decays monotonically toward the walls. The skin effect is in the transitional mode between local and non-local, and the electron velocity distribution function is non-Maxwellian.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

PvaPy streaming framework for real-time data processing

User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.

EPICS pvAccess↗
DOE OSTI2025

Data Assimilation for Robust UQ Within Agent-Based Simulation on HPC Systems

Agent-based simulation provides a powerful tool for in silico system modeling. However, these simulations do not provide built-in methods for uncertainty quantification (UQ). Within these types of models a typical approach to UQ is to run multiple realizations of the model then compute aggregate statistics. This approach is limited due to the compute time required for a solution. When faced with an emerging biothreat, public health decisions need to be made quickly and solutions for integrating near real-time data with analytic tools are needed. We propose an integrated Bayesian UQ framework for agent-based models based on sequential Monte Carlo sampling. Given streaming or static data about the evolution of an emerging pathogen this Bayesian framework provides a distribution over the parameters governing the spread of a disease through a population. These estimates of the spread of a disease may be provided to public health agencies seeking to abate the spread. By coupling agent-based simulations with Bayesian modeling in a data assimilation, our proposed framework provides a powerful tool for modeling dynamical systems in silico. We propose a method which reduces model error and provides a range of realistic possible outcomes. Moreover, our method addresses two primary limitations of ABMs: the lack of UQ and an inability to assimilate data. Our proposed framework combines the flexibility of an agent-based model with UQ provided by the Bayesian paradigm in a workflow which scales well to HPC systems. We provide algorithmic details and results on a simulated outbreak with both static and streaming data.

Spannaus, Adam [ORNL] (ORCID:0000000225213657)↗
DOE OSTI2026

rustpix

rustpix is a high-performance, open-source Rust library with first-class Python bindings (via PyO3) for processing pixel-detector data in neutron imaging. It targets time-stamping detectors such as Timepix3 (TPX3) at ORNL's Spallation Neutron Source (VENUS beamline), where each detected neutron deposits charge across a cluster of pixels within a very high-rate event stream (96M+ hits/sec). rustpix parses TPX3 event data in parallel using memory-mapped I/O, offers four interchangeable clustering algorithms (ABS adjacency-based search, DBSCAN, graph/union-find connected components, and a parallel grid method), and extracts weighted, super-resolved centroids to produce neutron-event lists. A streaming architecture lets it process files larger than available memory. rustpix is distributed as a pip-installable Python package (with NumPy integration), Rust crates, a command-line tool, and an interactive GUI; it writes HDF5, Apache Arrow, and CSV; and it is designed to extend to TPX4 and other detector types. Released as open-source under the MIT License.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),↗
DOE OSTI2026

Towards a self-driving trigger at the LHC: adaptive response in real time

Real-time data filtering and selection—or trigger—systems at high-throughput scientific facilities such as the experiments at the Large Hadron Collider must process extremely high-rate data streams under stringent bandwidth, latency, and storage constraints. Yet these systems are typically designed as static, hand-tuned menus of selection criteria grounded in prior knowledge and simulation. In this work, we further explore the concept of a self-driving trigger, an autonomous data-filtering framework that reallocates resources and adjusts thresholds dynamically in real-time to optimize signal efficiency, rate stability, and computational cost as instrumentation and environmental conditions evolve. We introduce a benchmark ecosystem to emulate realistic collider scenarios and demonstrate real-time optimization of a menu including canonical energy sum triggers as well as modern anomaly-detection algorithms that target non-standard event topologies using machine learning. Using simulated data streams and publicly available collision data from the Compact Muon Solenoid experiment, we demonstrate the capability to dynamically and automatically optimize trigger performance under specific cost objectives without manual retuning. Our adaptive strategy shifts trigger design from static menus with heuristic tuning to intelligent, automated, data-driven control, unlocking greater flexibility and discovery potential in future high-energy physics analyses.

Emami, Shaghayegh [Michigan U.] (ORCID:00090007589↗
DOE OSTI2025

Developing ML/AI Methods for High-Throughput Characterization of Multiple-Sensor Streams of Tokamak Dynamics for High-Speed Control (Final Report)

This project evaluated and developed new mathematical and algorithmic techniques capable of handling (in real-time) the growing amounts of data generated by modern fusion research. While existing numerical linear algebra (NLA) methods provide the backbone to classical data analysis and algorithms, these methods fundamentally do not port to distributed architectures nor do they allow low-latency data reduction for control. Motivated by the needs for modern fusion reactors, this project explored and implemented new numerical methods to characterize plasma dynamics, respond in real-time to discharge evolution, and to process massive-scale data accurately and rapidly more fully. This project links expertise in multiple-sensor diagnostics of tokamak plasma dynamics from Columbia University’s Plasma Physics Laboratory with expertise in massive-scale data reduction and extreme data control algorithms at Columbia University’s Data Science Institute. This interdisciplinary project (i) applied machine learning methods, (ii) implemented a properly-trained neural-network for very fast processing of high-speed plasma videography, and (ii) developed the applied mathematical methods, based on randomized-NLA (rNLA) routines, for data analysis, reduction, and real-time control. The Columbia University High Beta Tokamak-Extended Pulse (HBT-EP) facility provided data to test new algorithms and partnership with Columbia University's Data Sciences Institute evaluated the broader use of new algorithms for many challenging control applications.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Tactical Analysis for Calculating Contextual Risk at Boundaries: Summary of Laboratory Directed Research & Development Effort

The Tactical Analysis for Calculating Contextual Risk at Boundaries (TACCRAB) tool is an innovative digital twin (DT) platform and automated risk algorithm designed to transform operational decision-making in structured screening environments, with an initial focus on Southern Border Land Ports of Entry (POEs). The invention provides integration points for advanced artificial intelligence, predictive modeling, and real-time data analysis to produce a comprehensive risk management tool that enables proactive, data-informed security strategies. The core inventive features of TACCRAB center on its unique risk algorithm, which dynamically calculates contextual risk by synthesizing historical data, near real-time streaming data from the checkpoints themselves, and AI-generated predictions. Unlike traditional risk assessment methods, TACCRAB utilizes a DT to provide comprehensive operational insights, allowing stakeholders to visualize, simulate, and optimize checkpoint configurations with unprecedented speed and contextual awareness. TACCRAB's key innovation lies in its ability to combine multiple complex inputs - including technology detection probabilities, resource availability, screening pathway characteristics, and threat actor behavioral patterns - into a unified risk calculation and update these inputs based on changing operational and environmental conditions. By leveraging a DT that continuously updates and learns from linked data, TACCRAB can suggest adaptive mitigation strategies that minimize risk while maintaining operational efficiency. Particularly novel is the platform's approach to decision support, which goes beyond static risk assessment. The DT provides dynamic metrics such as wait times, resource allocation effectiveness, and potential emerging threat scenarios, enabling users to view sophisticated, relevant what-if simulations and optimize checkpoint operations in near real-time. The system's architecture allows for generalized application across different screening environments, such as secure facilities, ports of entry, and soft targets, making it a versatile tool for security and operational management. The invention distinguishes itself through its comprehensive integration of predictive modeling, AI-driven pattern discovery, and user-friendly interface design. By combining these elements, TACCRAB transforms complex risk data into actionable insights, supporting decision-makers at various organizational levels - from booth agents making split-second screening decisions to checkpoint managers optimizing the day's resource allocation to strategic planners managing long-term investments.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2026

Heterogeneous Multi-Domain Dataset Synthesis to Facilitate Privacy and Risk Assessments in Smart City IoT

The emergence of the Smart Cities paradigm and the rapid expansion and integration of Internet of Things (IoT) technologies within this context have created unprecedented opportunities for high-resolution behavioral analytics, urban optimization, and context-aware services. However, this same proliferation intensifies privacy risks, particularly those arising from cross-modal data linkage across heterogeneous sensing platforms. To address these challenges, this paper introduces a comprehensive, statistically grounded framework for generating synthetic, multimodal IoT datasets tailored to Smart City research. The framework produces behaviorally plausible synthetic data suitable for preliminary privacy risk assessment and as a benchmark for future re-identification studies, as well as for evaluating algorithms in mobility modeling, urban informatics, and privacy-enhancing technologies. As part of our approach, we formalize probabilistic methods for synthesizing three heterogeneous and operationally relevant data streams—cellular mobility traces, payment terminal transaction logs, and Smart Retail nutrition records—capturing the behaviors of a large number of synthetically generated urban residents over a 12-week period. The framework integrates spatially explicit merchant selection using K-Dimensional (KD)-tree nearest-neighbor algorithms, temporally correlated anchor-based mobility simulation reflective of daily urban rhythms, and dietary-constraint filtering to preserve ecological validity in consumption patterns. In total, the system generates approximately 116 million mobility pings, 5.4 million transactions, and 1.9 million itemized purchases, yielding a reproducible benchmark for evaluating multimodal analytics, privacy-preserving computation, and secure IoT data-sharing protocols. To show the validity of this dataset, the underlying distributions of these residents were successfully validated against reported distributions in published research. We present preliminary uniqueness and cross-modal linkage indicators; comprehensive re-identification benchmarking against specific attack algorithms is planned as future work. This framework can be easily adapted to various scenarios of interest in Smart Cities and other IoT applications. By aligning methodological rigor with the operational needs of Smart City ecosystems, this work fills critical gaps in synthetic data generation for privacy-sensitive domains, including intelligent transportation systems, urban health informatics, and next-generation digital commerce infrastructures.

IoT↗
DOE OSTI2025

Moment-based adaptive time integration for thermal radiation transport

Here, in this paper we develop a framework for moment-based adaptive time integration of deterministic multifrequency thermal radiation transpot (TRT). We generalize our recent semi-implicit-explicit (IMEX) integration framework for gray TRT to multifrequency TRT, and also introduce a semi-implicit variation that facilitates higher-order integration of TRT, where each stage is implicit in all components except opacities. To appeal to the broad literature on adaptivity with Runge–Kutta methods, we derive new embedded methods for four asymptotic preserving IMEX Runge–Kutta schemes we have found to be robust in our previous work on TRT and radiation hydrodynamics. We then use a moment-based high-order-low-order representation of the transport equations. Due to the high dimensionality, memory is always a concern in simulating TRT. We form error estimates and adaptivity in time purely based on temperature and radiation energy, for a trivial overhead in computational cost and memory usage compared with the base second order integrators. We then test the adaptivity in time on the tophat and Larsen problem, demonstrating the ability of the adaptive algorithm to naturally vary the timestep across 4–5 orders of magnitude, ranging from the dynamical timescales of the streaming regime to the thick diffusion limit.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Intelligent Experiments through Real-Time AI: Fast Data Processing and Autonomous Detector Control for High-Energy Nuclear Experiments

The aim of this project is to develop software and hardware for fast real-time data processing and autonomous detector control and calibration for the sPHENIX and the future EIC experiments. Below summarizes Georgia Tech team efforts in the past year: 1. We developed a real-time clustering algorithm and FPGA-based pipeline architecture for processing fired pixel data from ALPIDE sensors in sPHENIX experiments. Our Columnar Clustering Co-Design introduces a hardware-aware, stream-friendly approach that segments pixel data by column pairs using a Column Pair Clustering (CPC) strategy, followed by Cluster Stitching to merge adjacent subclusters. Implemented in Vitis HLS, the pipeline comprises five stages—read-in, subclustering, stitching, analysis, and write-out—connected by tagged HLS streams with custom end-of-event signaling for robust synchronization. We designed a pipelined dataflow model optimized for throughput, low latency, and minimal buffering, enabling scalable clustering across events of arbitrary size. Our system maintains spatial precision via center-of-mass and shape key extraction and efficiently handles edge cases such as fragmented or nested clusters. Compared against DBSCAN in both software and hardware, our approach demonstrates competitive performance under FPGA constraints. 2. We also conducted a comprehensive algorithm-to-hardware co-design of connected component analysis tailored for sPHENIX experiments, focusing on real-time, low-latency processing using FPGAs and High-Level Synthesis (HLS). Starting from a Python-based particle tracking pipeline, the team translated the core logic—graph traversal via DFS and Union-Find—into an HLS-compatible C++ model, replacing dynamic memory and recursion with static arrays and pipelined control flow. The final design includes a fully streamed and dataflow-compatible Union-Find kernel optimized across five iterations, incorporating loop pipelining, array partitioning, AXI/FIFO interface tuning, and function flattening. Experimental results show up to 14.8× speedup over the CPU baseline, reducing per-graph latency to 1.58 μs and demonstrating strong resource efficiency with only ~7k LUTs and zero BRAM usage. The design maintains functional correctness against the Python reference using a Python-based C-simulation framework and Mean Squared Error metrics. This work validates the potential of HLS-driven FPGA designs for edge-level HEP data acquisition, laying a scalable foundation for future integration with real-time detector pipelines and multi-graph processing systems.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Contaminant Investigation and Pre‐Processing Opportunities for Textile‐To‐Textile Recycling

Millions of metric tons of textiles are landfilled or incinerated each year in the United States, with less than 1% of textiles recycled into new clothing or fabrics. To counter this trend, a growing number of companies and researchers are exploring how a circular economy can be applied to support textile‐to‐textile recycling. A significant barrier they face comes down to quickly and efficiently extracting pure feedstock material from post‐consumer garments that feature a mix of natural and synthetic fibers. Textile recyclers prefer pure feedstocks, as working with mixed sources typically means lower throughput, higher risk of equipment failure, and diminished business margins. To facilitate a circular economy for textiles, methods, and technologies are needed that can efficiently separate out materials and contaminants from end‐of‐life textiles to increase the flow of pure feedstocks to recyclers. This paper summarizes findings from interviews with a cross section of textile recyclers and from a review of literature to define basic feedstock requirements. In addition to our qualitative research, we deconstruct a bale of post‐consumer textiles and analyze them using computer‐vision imaging, Fourier transform infrared spectroscopy (FTIR), and machine learning. The resulting data are used to set system‐level design inputs for an automated contaminant removal system to process post‐consumer clothing into appropriate feedstocks for recycling. To set the system's levels for automated real‐time near‐infrared analysis, we identify the minimum percentage of primary material that any single garment in a load of used clothing must contain for the average of the full output stream to meet the target purity levels of recyclers. Here, the envisioned automated system can also address undesirable trace materials that might contaminate the processed stream by using imaging cameras coupled with artificial intelligence to identify sections of clothing for de‐trimming. Proof‐of‐concept machine learning algorithms are evaluated to locate and identify trims or garment areas with hidden contaminant materials. Integrating these methods into automated textile cutting systems can provide a cost‐effective means for increasing feedstock purity from used clothing, which can advance circularity for textiles by helping recyclers to reach production volumes and quality targets that were not possible solely with manual dismantling operations.

Parsons, Ryan [Rochester Institute of Technology, ↗
DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗
DOE OSTI2025

Multiscale drivers of extreme southern California flooding: ENSO, MJO, North Pacific jet, and atmospheric rivers

Extreme rainfall and flooding, driven by a powerful atmospheric river (AR) and a persistent Madden-Julian Oscillation (MJO), hit Southern California in February 2024 during the 2023–2024 El Niño, affecting over 10 million people. ARs are key contributors to extreme rainfall and flooding along the U.S. West Coast. Although the AR-MJO link has been documented, its spatio-temporal variability remains a major forecasting and risk-management challenge. Combining precipitation, stream gauge and demographic data, we quantify the physical drivers and population exposure to this extreme event. Leveraging a Lagrangian MJO precipitation tracking algorithm, we unravel the multiscale interactions responsible for the AR’s development. El Niño favored a large, long-lived MJO that interacted with the North Pacific Jet (NPJ) over more than three weeks. The MJO convective outflow modulated the NPJ by inducing negative potential vorticity advection along the tropopause. The ensuing NPJ extension and acceleration induced explosive cyclogenesis, whose AR-driven moisture transport resulted in extreme rainfall.

Atmospheric dynamics↗
DOE OSTI2024

Unconventional Quantum Advantages for Computation (U-QuAC)

While quantum computing offers the promise of exponential advantages, limited quantum speedups are known, especially for practical applications. To open new avenues for quantum advantages, we propose Unconventional Quantum Advantages for Computation (U-QuACs), with respect to unconventional resources such as space (number of bits or quantum bits of memory required to solve a problem), accuracy of solution, communication, or energy consumption. We focus on space-efficient quantum algorithms, where we seek to design algorithms that solve a problem using much less space than the total size of the input. A natural setting in which space is critical is the streaming model of computation, where the input data arrives sequentially in pieces that must each be processed individually. Streaming is motivated by a variety of problems including analysis of internet traffic or social networks. We design the first exponential quantum space advantage for a natural streaming problem, which also constitutes the first quantum advantage for approximating a discrete optimization problem, albeit with respect to space.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Differentiable vertex fitting for jet flavor tagging

This work explores the use of differentiable programming to integrate domain knowledge, in the form of domain specific software, into neural networks to develop scientific machine learning systems. We propose a differentiable vertex fitting algorithm that estimates the crossing point of multiple curves. In the high energy physics setting, these curves are defined by particle equations of motion and the crossing point represents the origin of particle production. This differentiable vertex fitting algorithm can be seamlessly integrated into neural networks, and we show its utility and efficacy in the high energy physics application of the classification of jets, i.e., collimated streams of particles in particle detectors whose originating parent particle we aim to classify. We demonstrate how differentiable vertex fitting can be integrated into larger transformer-based models for jet flavor tagging and show improvements in heavy flavor jet classification when compared to baseline models. Published by the American Physical Society 2024

Smith, Rachel E. C. (ORCID:0000000335851262)↗
DOE OSTI2026

Qubit lattice algorithm simulations of the scattering of a bounded two dimensional electromagnetic pulse from an infinite planar dielectric interface

Qubit lattice algorithm (QLA) simulations are performed for a two-dimensional spatially bounded pulse propagating onto a plane interface between two dielectric slabs. QLA is an initial value scheme that consists of a sequence of unitary collision and streaming operators, with appropriate potential operators, that recover Maxwell equations in inhomogeneous dielectric media to the second order in the lattice discreteness. For the case of total internal reflection, there is transient energy transfer into the second medium due to the evanescent fields as the Poynting unit vector of the pulse is rotated from its incident to reflected direction. Because of the finite spatial extent of the pulse, a self-consistent Goos–Hanchen-type displacement along the interface is found without imposing any explicit interface boundary conditions on the fields. For normal incidence, the standard Fresnel coefficients are recovered for appropriately averaged QLA fields. Energy is conserved at all times to seven significant figures.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2026

Melter Tests to Define LAW Halide Concentrations, Phase 2

The present work represents the second of two phases to address the formation of secondary sulfate phases while processing high waste loading LAW formulations with elevated halide, chromium, and phosphate feed concentrations over a wide range of LAW waste types, including those with high potassium concentrations. The Test Plan for the present work outlines a series of tests designed to assess the formation of secondary phases over a range of sulfur and halide concentrations for two LAW streams and associated glass formulations as well as crucible testing to identify additives that have the potential to suppress the formation of secondary phases for wastes with high chlorine contents. The combined results from both phases of testing and from previous tests with these LAW waste streams will be used as inputs to future work to develop concentration limits for these components in the LAW glass property composition models and to support the modification and updating of the current WTP LAW glass formulation algorithm.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2024

Relevant biochar characteristics influencing compressive strength of biochar-cement mortars

To counteract the contribution of CO 2 emissions by cement production and utilization, biochar is being harnessed as a carbon-negative additive in concrete. Increasing the cement replacement and biochar dosage will increase the carbon offset, but there is large variability in methods being used and many researchers report strength decreases at cement replacements beyond 5%. This work presents a reliable method to replace 10% of the cement mass with a vast selection of biochars without decreasing ultimate compressive strength, and in many cases significantly improving it. By carefully quantifying the physical and chemical properties of each biochar used, machine learning algorithms were used to elucidate the three most influential biochar characteristics that control mortar strength: initial saturation percentage, oxygen-to-carbon ratio, and soluble silicon. These results provide additional research avenues for utilizing several potential biomass waste streams to increase the biochar dosage in cement mixes without decreasing mechanical properties.

97 MATHEMATICS AND COMPUTING↗