Results for “Streaming data” — Page 6

DOE OSTI2025

EJFAT Scientific Perspective

Presented new computing model to the test by deploying the EJFAT system alongside a data-stream processing framework running the production-level CLAS12 event reconstruction application. In this experiment, a continuous stream of CLAS12 Level-1 identified events was processed in real-time using the EJFAT load balancer, distributing the workload across 90 computing nodes located across the U.S. This marks the first-ever large-scale, real-time distributed data stream processing experiment, demonstrating that scientific data-streaming pipelines can efficiently scale across four dimensions, thanks to EJFAT’s advanced hardware and software capabilities.

Gyurjyan, Vardan [Thomas Jefferson National Accele↗

DOE OSTI2020

The influence of environmental microseismicity on detection and interpretation of small-magnitude events in a polar glacier setting

Glacial environments exhibit temporally variable microseismicity. To investigate how microseismicity influences event detection, we implement two noise-adaptive digital power detectors to process seismic data from Taylor Glacier, Antarctica. We add scaled icequake waveforms to the original data stream, run detectors on the hybrid data stream to estimate reliable detection magnitudes and compare analytical magnitudes predicted from an ice crack source model. We find that detection capability is influenced by environmental microseismicity for seismic events with source size comparable to thermal penetration depths. When event counts and minimum detectable event sizes change in the same direction (i.e. increase in event counts and minimum detectable event size), we interpret measured seismicity changes as ‘true’ seismicity changes rather than as changes in detection. Generally, one detector (two degree of freedom (2dof)) outperforms the other: it identifies more events, a more prominent summertime diurnal signal and maintains a higher detection capability. We conclude that real physical processes are responsible for the summertime diurnal inter-detector difference. One detector (3dof) identifies this process as environmental microseismicity; the other detector (2dof) identifies it as elevated waveform activity. Our analysis provides an example for minimizing detection biases and estimating source sizes when interpreting temporal seismicity patterns to better infer glacial seismogenic processes.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Towards Resilient Near Real-Time Analysis Workflows in Fusion Energy Science

Nuclear fusion holds the promise of an endless source of energy. Several research experiments across the world and joint modeling and simulation efforts between the nuclear physics and high performance computing communities are actively preparing the operation of the International Thermonuclear Experimental Reactor (ITER). Both experimental reactors and their simulated counterparts generate data that must be analyzed quickly and in a resilient way to support decision making for the configuration of subsequent runs or prevent a catastrophic failure. However, the cost if the traditional techniques used to improve the resilience of analysis workflows, i.e., replicating datasets and computational tasks, becomes prohibitive with explosion of the volume of data produced by modern instruments and simulations. Therefore, we advocate in this paper for an alternate approach based on data reduction and data streaming. The rationale is that by allowing for a reasonable, controlled, and guaranteed loss of accuracy it becomes possible to transfer smaller amounts of data, shorten the execution time of analysis workflows, and lower the cost of replication to increase resilience. We develop our research and development roadmap towards resilient near real-time analysis workflows in fusion energy science and present early results showing that data streaming and data reduction is a promising way to speed up the execution and improve the resilience of analysis workflows.

Suter, Fred↗

DOE OSTI2021

Estimating Watershed Subsurface Permeability From Stream Discharge Data Using Deep Neural Networks

Subsurface permeability is a key parameter in watershed models that controls the contribution from the subsurface flow to stream flows. Since the permeability is difficult and expensive to measure directly at the spatial extent and resolution required by fully distributed watershed models, estimation through inverse modeling has had a long history in subsurface hydrology. The wide availability of stream surface flow data, compared to groundwater monitoring data, provides a new data source to infer soil and geologic properties using integrated surface and subsurface hydrologic models. As most of the existing methods have shown difficulty in dealing with highly nonlinear inverse problems, we explore the use of deep neural networks for inversion owing to their successes in mapping complex, highly nonlinear relationships. We train various deep neural network (DNN) models with different architectures to predict subsurface permeability from stream discharge hydrograph at the watershed outlet. The training data are obtained from ensemble simulations of hydrographs corresponding to an permeability ensemble using a fully-distributed, integrated surface-subsurface hydrologic model. The trained model is then applied to estimate the permeability of the real watershed using its observed hydrograph at the outlet. Our study demonstrates that the permeabilities of the soil and geologic facies that make significant contributions to the outlet discharge can be more accurately estimated from the discharge data. Their estimations are also more robust with observation errors. Compared to the traditional ensemble smoother method, DNNs show stronger performance in capturing the nonlinear relationship between permeability and stream hydrograph to accurately estimate permeability. Our study sheds new light on the value of the emerging deep learning methods in assisting integrated watershed modeling by improving parameter estimation, which will eventually reduce the uncertainty in predictive watershed models.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Data Center Facility Monitoring with Physics Aware Approach

U.S. Department of Energy's National Renewable Energy Laboratory (NREL) hosts one of the world's most energy-efficient HPC data centers; this system uses component-level warm-water liquid cooling to efficiently remove heat from the data center and capture it for reuse in the building or rejection to the atmosphere. Given the complexity of this system, building data-driven tools for holistically monitoring and operating the entire data center is a priority for ensuring maximal efficiency and resiliency. In this advanced smart facility, over one million metrics are recorded per minute using state-of-the-art streaming data architecture and software to capture and process the state of the system in real time. Here we detail two efforts to effectively analyze, visualize, and interpret this large volume streaming data. We have developed a novel, flexible system for identifying and visualizing individual metric anomalies and component performance across the data center through automatic metadata extraction and physically-motivated visualization for quick interpretation. Additionally, to directly connect system maintenance to data stream processing we explore a physics informed multi-metric drift and anomaly detection application to detect scale-build up in heat exchangers.

anomaly detection↗

DOE OSTI2019

Minimum shift keying based protection mode capable wavelength division multiplexed passive optical network optical overlay system

We explored the functioning of an optical overlay of multicast and unicast data streams on a survival wavelength division multiplexed passive optical network using intensity and minimum shift keying modulation for both single and multiuser. In optical line terminal, using a sinusoidal clock, optical subcarriers are generated not only to carry the multicast and unicast data streams simultaneously but also in case of fiber failure, protection mode is enabled to carry the data streams smoothly. Here, 10 Gbps downstream multicast in normal mode and unicast signals both in normal and protection mode are transmitted error-free through 30 km dispersion shifted fiber. In both the cases 10 Gbps upstream signal is modulated and sent back to the optical line terminal. For single user nearly similar power penalty is witnessed for downstream unicast as well as re-modulated upstream data in both the conditions and also when multicast data is disabled. The power penalties between single and multiuser for both the modes are found to be approximately 0.06, 0.42 and 0.7 dB for unicast MSK, multicast and upstream respectively.

36 MATERIALS SCIENCE↗

DOE OSTI2022

Real-time GIS Programming and Geocomputation

Streaming data generated continuously from sensor networks, mobile devices, social media platforms and other edge devices have posed significant challenges to existing computing platforms for achieving both high throughput and low latency data processing in addition to scalable computing. This entry introduces a real-time computing and programming platform for time-critical GIS (Geographic Information System) applications. In this platform, advanced streaming data processing software, such as Apache Kafka and Spark Streaming, are integrated to enable data analytics in real-time. This computing platform can also be extended to integrate GeoAI (Geospatial Artificial Intelligence) based machine learning models to leverage both historical and streaming data to achieve real-time prediction and intelligent geospatial analytics. Two real-time geospatial applications in terms of flood simulation and climate data visualization are introduced to demonstrate how real-time programming and computing can help tackle real-world problems with important societal impacts.

Liu, Yan↗

DOE OSTI2024

Continual learning in the presence of repetition

Continual learning (CL) provides a framework for training models in ever-evolving environments. Although re-occurrence of previously seen objects or tasks is common in real-world problems, the concept of repetition in the data stream is not often considered in standard benchmarks for CL. Unlike with the rehearsal mechanism in buffer-based strategies, where sample repetition is controlled by the strategy, repetition in the data stream naturally stems from the environment. This report provides a summary of the CLVision challenge at CVPR 2023, which focused on the topic of repetition in class-incremental learning. The report initially outlines the challenge objective and then describes three solutions proposed by finalist teams that aim to effectively exploit the repetition in the stream to learn continually. The experimental results from the challenge highlight the effectiveness of ensemble-based solutions that employ multiple versions of similar modules, each trained on different but overlapping subsets of classes. This report underscores the transformative potential of taking a different perspective in CL by employing repetition in the data stream to foster innovative strategy design.

Class-incremental learning↗

DOE OSTI2022

Learning Global Proliferation Expertise Evolution Using AI-Driven Analytics and Public Information

Detecting and anticipating global proliferation expertise and capability evolution from unstructured, noisy, and incomplete public data streams is a highly desired, but extremely challenging task. Here, in this article, we present our pioneering data-driven approach to support the non-proliferation mission to detect and explain the evolution of proliferation expertise and capability development globally from terabytes of publicly available information (PAI), focusing on our knowledge extraction pipeline and descriptive analytics. We first discuss how we fuse nine open-source data streams, including multilingual data, to convert 4 TB of unstructured data to structured knowledge and encode dynamically evolving proliferation expertise representations—content and context graphs. For this, we rely on natural language processing (NLP) and deep learning (DL) models to perform information extraction, topic modeling, and distributed text representation (aka embedding) learning. We then present interactive, usable, and explainable descriptive analytics to refine domain knowledge and present it in a human-understandable form. Finally, we introduce future work avenues that will leverage our dynamic knowledge representations and descriptive analytics to enable predictive and prescriptive inferences to achieve real-time domain understanding and contextual reasoning about global proliferation expertise and capability evolution.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

DOE OSTI2017

Why wait? Let's start computing while data is still on the wire

In this era of Big Data, computing useful information from data is becoming increasingly complicated, particularly due to the ever increasing volumes of data that need to travel over the network to data centers to be stored and processed, all highly expensive operations in the long haul. In this paper we suggest that we can do computing and analysis of data "on the wire," i.e., while data is still in transit. The nature of these computations include analysis, visualization, pattern recognition, and prediction, or forecasting, on the streaming data. We follow a service function chaining architecture to implement this, assuming that the data packets arrive within a single network administrative domain. As a demonstration of this new computing paradigm, we present three examples. Firstly, we demonstrate pattern recognition and data visualization on streaming forex data, which can be used for lucrative trading in the forex market. In our second example, we analyze and learn user buying patterns from clickstream data streaming from multiple websites. Finally, we monitor solar sensors for a zero reading while the packets are still on their way to the data center, to schedule any maintenance and requisite repairs with no time delay.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

LHC physics dataset for unsupervised New Physics detection at 40 MHz

In the particle detectors at the Large Hadron Collider, hundreds of millions of proton-proton collisions are produced every second. If one could store the whole data stream produced in these collisions, tens of terabytes of data would be written to disk every second. The general-purpose experiments ATLAS and CMS reduce this overwhelming data volume to a sustainable level, by deciding in real-time whether each collision event should be kept for further analysis or be discarded. We introduce a dataset of proton collision events that emulates a typical data stream collected by such a real-time processing system, pre-filtered by requiring the presence of at least one electron or muon. This dataset could be used to develop novel event selection strategies and assess their sensitivity to new phenomena. In particular, we intend to stimulate a community-based effort towards the design of novel algorithms for performing unsupervised new physics detection, customized to fit the bandwidth, latency and computational resource constraints of the real-time event selection system of a typical particle detector.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2019

Stream splitting moving target defense

Systems and methods for utilizing stream splitting Moving Target Defense (MTD) to provide enhanced computer system communication system security by splitting a data stream in to a plurality of paths is described. In some implementations, Stream splitting MTD, involves splitting a single data stream (e.g., TCP stream) into a plurality of discrete units, then sending and receiving those discrete units from and to different (ideally geographically disparate) receiving servers, with the stream being reassembled on the receiving end. The plurality of discrete units of data include resequencing data. The size of each discrete unit may vary depending on the specific implementation, even down to small unit sizes (e.g., a single packet).

Evans, Nathaniel↗

DOE OSTI2026

WHONDRS Surface Water Geochemistry and Organic Matter Characterization Data from Streams Distributed across Latin America

This dataset supports a broader study examining global transferability of stream biogeochemistry and was generated in collaboration with the MicroSudAqua (µSudAqua) network (https://microsudaqua.netlify.app/en/). The dataset provides surface water geochemistry (dissolved organic carbon, total dissolved nitrogen, cations) and organic matter characterization (FTICR-MS) from streams in Argentina, Brazil, Chile, and Colombia. Samples were collected across stream orders (1st to 6th order) within five basins. Related data were collected and will be published separately in collaboration with the µSudAqua network. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to this readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) a folder of field photos; (2) a folder of surface water sample data, (3) a folder of raw Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) data; (4) file-level metadata; (5) data dictionary; (6) field metadata; (7) readme; (8) international generic sample number (IGSN) mapping file; and (9) field protocol. The sample data subfolder contains (1) dissolved organic carbon (DOC, measured as non-purgeable organic carbon, NPOC) data and averages; (2) total dissolved nitrogen data and averages; (3) anions and averages; (4) methods codes; (5) FTICR-MS methods; and (15) a subfolder of 9.4 Tesla (9.4T) FTICR-MS data. This folder contains the processed data and three subfolders, one containing the .xml files, one containing the water CoreMS output files, and the other containing instructions and scripts for processing the files in CoreMS (https://github.com/EMSL-Computing/CoreMS). All files are .csv, .pdf, .R, .xml, .d, .html, .Rmd, .py, .cal, .json, .jpg, .jpeg, .png, .mov, or .mp4.

Anions↗

DOE OSTI2026

Machine Learning Digital Twin for Lithium Ion Battery State of Health Predictions

A digital twin system has been established to model the long term degradation of the state of health of lithium ion batteries. Two data streams result from the computational model of the system and from the physical experiment measurements. A machine learning pipeline has been developed at the nexus of these data streams. Leveraging the unique data sources in multiple transfer learning approaches has lead to the development of multiple cell specific machine learning digital twins. Our discussion on this first of its kind technology will cover challenges in data ingestion, scalability, architecture and orchestration, and research findings.

25 - ENERGY STORAGE↗

DOE OSTI2023

915rwpprecipcor

Radar Wind Profiler (RWP) has operational modes: short duration high resolution (referred to as 'high' or 'high mode') and long duration low resolution (referred to as 'low' or 'low mode'). The motivation for this is to create a VAP that quality controls (noise, sig echo filtering), merges, co-grids, and performs initial analyses on these RWP modes. This is done to streamline downstream VAP development on the RWP, as well as enable incorporation of echo properties into the ARSCL chain. The proposed VAP will use the RWPPRECIP b0/b1 level data stream as input. Additional consideration for the incorporation of other ARM data streams to improve this VAP was performed, but not included at this time.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Geochemistry and Multiomics Data Differentiate Streams in Pennsylvania Based on Unconventional Oil and Gas Activity

Unconventional oil and gas (UOG) extraction is increasing exponentially around the world, as new technological advances have provided cost-effective methods to extract hard-to-reach hydrocarbons. While UOG has increased the energy output of some countries, past research indicates potential impacts in nearby stream ecosystems as measured by geochemical and microbial markers. Here, we utilized a robust data set that combines 16S rRNA gene amplicon sequencing (DNA), metatranscriptomics (RNA), geochemistry, and trace element analyses to establish the impact of UOG activity in 21 sites in northern Pennsylvania. These data were also used to design predictive machine learning models to determine the UOG impact on streams. We identified multiple biomarkers of UOG activity and contributors of antimicrobial resistance within the order Burkholderiales. Furthermore, we identified expressed antimicrobial resistance genes, land coverage, geochemistry, and specific microbes as strong predictors of UOG status. Of the predictive models constructed (n = 30), 15 had accuracies higher than expected by chance and area under the curve values above 0.70. The supervised random forest models with the highest accuracy were constructed with 16S rRNA gene profiles, metatranscriptomics active microbial composition, metatranscriptomics active antimicrobial resistance genes, land coverage, and geochemistry (n = 23). The models identified the most important features within those data sets for classifying UOG status. These findings identified specific shifts in gene presence and expression, as well as geochemical measures, that can be used to build robust models to identify impacts of UOG development.

16S rRNA↗

DOE OSTI2024

DEVELOPMENT AND DEMONSTRATION TESTBED FOR THE REMOTE OPERATIONS AND MONITORING OF MICROREACTORS

The nuclear industry is rapidly developing many advanced-reactor concepts for near-term deployment in both traditional and non-traditional nuclear-powered applications. One such category of advanced reactor is the microreactor, a class of reactor with less than 20MWth power output, intended for applications where the economics or logistics of traditional power sources are difficult. This includes applications such as remote communities, mining sites, defense installations, or humanitarian and disaster-relief missions. One key enabling feature for the successful deployment of microreactors is a remote operations capability. Remote operations provide monitoring and control capabilities which can significantly reduce staffing costs by eliminating the need for licensed operators at each reactor facility and improve the economic viability for microreactor deployment. A remote concept of operations is not currently an established capability in the nuclear industry. In addition, no demonstration microreactor is expected to complete construction or go critical until at least 2026. This leaves two major capability gaps: the successful demonstration of a remote concept of operations for microreactors and a test bed suitable for said demonstration. Both gaps must be addressed in order to advance the remote concepts of nuclear operation and, more broadly, microreactors themselves from paper to reality. This paper aims to fill these gaps and describes a test bed that would support development and deployment of a remote concept of nuclear operations, initial experimental results from that test bed, and the application of the test bed and experimental results for a digital-twin-based remote concept of operations underdevelopment at Idaho National Laboratory (INL). The platform chosen as a remote concept of nuclear operations test bed is the Single Primary Heat Extraction and Removal Emulator, known as SPHERE, located at INL. SPHERE is a small-scale non-nuclear test bed that emulates thermal behavior of a microreactor. The small-scale and non-nuclear nature of SHPERE limit safety concerns associated with remote operations while still providing the physical response representative of a microreactor. A network connection was added to SPHERE that enables remote-monitoring capability. This allows for real-time data streaming to networked workstations, data historians, and human-machine interfaces (HMIs). These are all critical components in a remote concept of operations, thus providing a robust development and demonstration platform. An initial experiment was performed using the SPHERE remote operations testbed. This included running a comprehensively instrumented SPHERE through a series of steady-state and transient operating scenarios in both normal and abnormal operating conditions, all while streaming live test data to a remote HMI and data warehouse. This initial experiment served three purposes: (1) characterizing the response of SPHERE, (2) demonstrating the remote connection to SPHERE, and (3) providing a baseline data set for development of a digital-twin-based remote concept of operations that is under development at INL.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2024

DEVELOPMENT AND DEMONSTRATION TESTBED FOR THE REMOTE OPERATIONS AND MONITORING OF MICROREACTORS

The nuclear industry is rapidly developing many advanced-reactor concepts for near-term deployment in both traditional and non-traditional nuclear-powered applications. One such category of advanced reactor is the microreactor, a class of reactor with less than 20MWth power output, intended for applications where the economics or logistics of traditional power sources are difficult. This includes applications such as remote communities, mining sites, defense installations, or humanitarian and disaster-relief missions. One key enabling feature for the successful deployment of microreactors is a remote operations capability. Remote operations provide monitoring and control capabilities which can significantly reduce staffing costs by eliminating the need for licensed operators at each reactor facility and improve the economic viability for microreactor deployment. A remote concept of operations is not currently an established capability in the nuclear industry. In addition, no demonstration microreactor is expected to complete construction or go critical until at least 2026. This leaves two major capability gaps: the successful demonstration of a remote concept of operations for microreactors and a test bed suitable for said demonstration. Both gaps must be addressed in order to advance the remote concepts of nuclear operation and, more broadly, microreactors themselves from paper to reality. This paper aims to fill these gaps and describes a test bed that would support development and deployment of a remote concept of nuclear operations, initial experimental results from that test bed, and the application of the test bed and experimental results for a digital-twin-based remote concept of operations underdevelopment at Idaho National Laboratory (INL). The platform chosen as a remote concept of nuclear operations test bed is the Single Primary Heat Extraction and Removal Emulator, known as SPHERE, located at INL. SPHERE is a small-scale non-nuclear test bed that emulates thermal behavior of a microreactor. The small-scale and non-nuclear nature of SHPERE limit safety concerns associated with remote operations while still providing the physical response representative of a microreactor. A network connection was added to SPHERE that enables remote-monitoring capability. This allows for real-time data streaming to networked workstations, data historians, and human-machine interfaces (HMIs). These are all critical components in a remote concept of operations, thus providing a robust development and demonstration platform. An initial experiment was performed using the SPHERE remote operations testbed. This included running a comprehensively instrumented SPHERE through a series of steady-state and transient operating scenarios in both normal and abnormal operating conditions, all while streaming live test data to a remote HMI and data warehouse. This initial experiment served three purposes: (1) characterizing the response of SPHERE, (2) demonstrating the remote connection to SPHERE, and (3) providing a baseline data set for development of a digital-twin-based remote concept of operations that is under development at INL.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗