Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Architecture Of A Multi-channel Data Streaming Device With An Fpga As A Coprocessor

The design of a data acquisition system often involves the integration of a Field Programmable Gate Array (FPGA) with analog front-end components to achieve precise timing and control. Reuse of these hardware systems can be difficult since they need to be tightly coupled to the communications interface and timing requirements of the specific ADC used. A hybrid design exploring the use of FPGA as a coprocessor to a traditional CPU in a dataflow architecture is presented. Reduction in the volume of data and gradual transitioning of data processing away from a hard real-time environment are both discussed. Chief design concerns, including data throughput and precise synchronization with external stimuli, are addressed. The discussion is illustrated by the implementation of a multi-channel digital integrator, a device based entirely on commercial off-the-shelf (COTS) equipment.

Nogiec, Jerzy M.↗
DOE OSTI2022

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Poeschel, Franz↗
DOE OSTI2018

A minimal set of physiomarkers in continuous high frequency data streams predict adult sepsis onset earlier

Purpose: We report that sepsis is a life-threatening condition with high mortality rates and expensive treatment costs. To improve short- and long-term outcomes, it is critical to detect at-risk sepsis patients at an early stage. Methods: A data-set consisting of high-frequency physiological data from 1161 critically ill patients was analyzed. 377 patients had developed sepsis, and had data at least 3 h prior to the onset of sepsis. A random forest classifier was trained to discriminate between sepsis and non-sepsis patients in real-time using a total of 132 features extracted from a moving time-window. The model was trained on 80% of the patients and was tested on the remaining 20% of the patients, for two observational periods of lengths 3 and 6 h prior to onset.Results: The model that used continuous physiological data alone resulted in sensitivity and F1 score of up to 80% and 67% one hour before sepsis onset. On average, these models were able to predict sepsis 294.19 ± 6.50 min (5 h) before the onset. Conclusions: Lastly, the use of machine learning algorithms on continuous streams of physiological data can allow for early identification of at-risk patients in real-time with high accuracy.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

EdgeAI: Machine learning via direct attached accelerator for streaming data processing at high shot rate x-ray free-electron lasers

We present a case for low batch-size inference with the potential for adaptive training of a lean encoder model. We do so in the context of a paradigmatic example of machine learning as applied in data acquisition at high data velocity scientific user facilities such as the Linac Coherent Light Source-II x-ray Free-Electron Laser. We discuss how a low-latency inference model operating at the data acquisition edge can capitalize on the naturally stochastic nature of such sources. We simulate the method of attosecond angular streaking to produce representative results whereby simulated input data reproduce high-resolution ground truth probability distributions. By minimizing the mean-squared error between the decoded output of the latent representation and the ground truth distributions, we ensure that the encoding layers and resulting latent representation maintains full fidelity for any downstream task, be it classification or regression. We present throughput results for data-parallel inference of various batch sizes, some with throughput exceeding 100 k images per second. We also show in situ training below 10 s per epoch for the full encoder–decoder model as would be relevant for streaming and adaptive real-time data production at our nation’s scientific light sources.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1977

Continuous data stream FIFO magnetic bubble domain shift register

There is described a simple first-in, first out (FIFO) magnetic bubble domain shift register which has continuous data storage capability. In a preferred embodiment, two parallel detector branches are associated with a main storage loop. The bubbles in the storage loop are replicated toward each detector branch by passive replicators in the storage loop. Annihilators associated with each of the replicators and each of the detectors are arranged an appropriate distance from the replicators so that selective energization of the annihilators permits certain bits to propagate to one detector and other bits to propagate to the other detector. Connection of the detectors in an appropriate bridge circuit permits full data rate reclamation.

Chen, Thomas T.↗
DOE OSTI2016

Epidemic forecasting is messier than weather forecasting: The role of human behavior and internet data streams in epidemic forecast

Mathematical models, such as those that forecast the spread of epidemics or predict the weather, must overcome the challenges of integrating incomplete and inaccurate data in computer simulations, estimating the probability of multiple possible scenarios, incorporating changes in human behavior and/or the pathogen, and environmental factors. In the past 3 decades, the weather forecasting community has made significant advances in data collection, assimilating heterogeneous data steams into models and communicating the uncertainty of their predictions to the general public. Epidemic modelers are struggling with these same issues in forecasting the spread of emerging diseases, such as Zika virus infection and Ebola virus disease. While weather models rely on physical systems, data from satellites, and weather stations, epidemic models rely on human interactions, multiple data sources such as clinical surveillance and Internet data, and environmental or biological factors that can change the pathogen dynamics. We describe some of similarities and differences between these 2 fields and how the epidemic modeling community is rising to the challenges posed by forecasting to help anticipate and guide the mitigation of epidemics. Here, we conclude that some of the fundamental differences between these 2 fields, such as human behavior, make disease forecasting more challenging than weather forecasting.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS2018

Enriching the Twitter Stream Increasing Data Mining Yield and Quality Using Machine Learning

Social media data streams are important sources of real-time and historical global information for science applications. At the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we are exploring the Twitter data stream for its potential in augmenting the validation program of NASA Earth science missions, specifically the Global Precipitation Measurement (GPM) mission. We have implemented a tweet processing infrastructure that outputs classified precipitation tweets. Inputs are "passive" tweets, along with a smaller number of tweets from "active" participants, i.e., those knowingly contributing to our effort. The "active" tweets, presumably of higher quality, enrich the Twitter stream. "Active" sources include data scraped from other social media (e.g., public Facebook posts) and data from existing crowdsourcing programs (e.g., mPING reports). In addition, there is likely relevant precipitation information in images and documents that are the end points of links often included in tweets. Information derived from these "active" sources could then be tweeted into the Twitter stream, thus enriching its quality. The objective of our current work is to mine these tweet­ linked images and documents, using neural networks, to increase the information content and quality related to precipitation. For images, we classified them as either precipitation-related or not. For training and validation, we used images obtained via the Google custom search API. We created two models: (1) by training a simple Convolutional Neural Network and (2) by using transfer learning principles to adapt a pre-trained object recognition model. For documents, both those linked to tweets and the tweet contents, we trained Hierarchical Attention Networks to determine precipitation occurrence, type, and intensity. For training and validation, we used a keyword-filtered tweet data set labelled with ground truth data from Dark Sky (an API to retrieve weather-related labels) and the National Severe Storms Laboratory's Multi­ Radar/Multi-Sensor (MRMS) system. Our results demonstrated the efficacy of our machine learning approaches for enriching the Twitter stream, to derive information potentially useful for validation of earth science satellite data.

Albayrak, Arif↗
DOE OSTI2022

Search for long-lived particles decaying into muon pairs in proton-proton collisions at $\sqrt{s}$ = 13 TeV collected with a dedicated high-rate data stream

A search for long-lived particles decaying into muon pairs is performed using proton-proton collisions at a center-of-mass energy of 13 TeV, collected by the CMS experiment at the LHC in 2017 and 2018, corresponding to an integrated luminosity of 101 fb -1 . The data sets used in this search were collected with a dedicated dimuon trigger stream with low transverse momentum thresholds, recorded at high rate by retaining a reduced amount of information, in order to explore otherwise inaccessible phase space at low dimuon mass and nonzero displacement from the primary interaction vertex. No significant excess of events beyond the standard model expectation is found. Upper limits on branching fractions at 95% confidence level are set on a wide range of mass and lifetime hypotheses in beyond the standard model frameworks with the Higgs boson decaying into a pair of long-lived dark photons, or with a long-lived scalar resonance arising from a decay of a b hadron. The limits are the most stringent to date for substantial regions of the parameter space. These results can be also used to constrain models of displaced dimuons that are not explicitly considered in this paper.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS2001

Chip for CCSDS Compatible Serial Data Streams

A configurable service processor for telemetry ground stations is totally implemented in VLSI/ASIC hardware and finds use in spacecraft systems and other communications systems that operate according to CCSDS and CCSDS-like protocols. The service processor performs the traditional functions of data extraction at very high data and packet rates.

Jason T Dowling↗
DOE OSTI2025

A Digital Twin for an Inverter-Based Resource Power Plant: Real-time data streaming unlocks situation awareness

Here, this study presents the development and successful implementation of a digital twin specifically designed for a grid-connected IBR power plant. By integrating a reduced-order model of the IBR system and dynamically updating the grid impedance with real-time data, the digital twin effectively captures and replicates the behavior of the physical system. Its accuracy and reliability are validated through critical test scenarios, including a three-phase fault and a line-tripping event. The results confirm that the digital twin closely emulates its physical counterpart, demonstrating its strong potential for real-time analysis, system monitoring, and predictive decision making in modern power systems.

Digital twins↗
DOE OSTI2022

Investigation of Multiple Data Streams for Gearbox Bearing Fault Prediction Through Machine-Learning Models

Operations and maintenance (O&M) cost of wind plant accounts up to 30% of total energy cost, which can be reduced through continuous monitoring and successfully detecting incipient wind turbine failures. To accomplish this, condition monitoring and predictive maintenance systems are being implemented in wind industry to support O&M decision making. A wide range of approaches for condition monitoring and fault prediction have been developed. These approaches generally use historical data of wind turbines collected by Supervisory Control and Data Acquisition (SCADA) system to identify patterns that lead to failure. These SCADA data show the overall condition of a wind turbine and can be leveraged to detect when the turbine's performance is degrading and to identify if a fault is developing. However, it becomes challenging to predict the failure of a specific wind turbine gearbox bearing, because the SCADA data are often not directly linked to the component. To bridge the gap, we have investigated features calculated from SCADA data using physics-based models and the gearbox design over the years. The damaged metric we used in the physics domain is frictional energy. Combining these physics domain variables with SCADA data as inputs to various machine learning models for gearbox bearing fault prediction, we have demonstrated the benefits of leveraging both physics and data domain models. It was an attempt to improve frictional-energy-based damage metric by adding data domain inputs, as we had learned that the frictional-energy-based damage metric alone is not sufficient to single out failed bearings from healthy. As condition monitoring data (either vibration or oil debris data) has become available at more and more wind plants, we would like to evaluate whether by adding the condition monitoring data can help further improve the performance of frictional-energy-based damage metric for gearbox bearing fault prediction. Both cases by modeling through various machine learning algorithms are discussed in this study along with some observations.

fault prediction↗
DOE OSTI2011

Reply to ''Comment on 'Optical rogue waves in telecommunication data streams'''

We feel that the main claim made in the abstract of the preceding Comment [Phys. Rev. A 84, 067801 (2011)] is wrong. Using results obtained in our paper [Phys. Rev. A 83, 061801 (2011)], we prove that rogue waves with amplitudes much larger than the average level can be observed during a short period of time in purely linear propagation regimes in optical fiber systems.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2010

Shotgun metagenomic data streams: surfing without fear

Timely information about bio-threat prevalence, consequence, propagation, attribution, and mitigation is needed to support decision-making, both routinely and in a crisis. One DNA sequencer can stream 25 Gbp of information per day, but sampling strategies and analysis techniques are needed to turn raw sequencing power into actionable knowledge. Shotgun metagenomics can enable biosurveillance at the level of a single city, hospital, or airplane. Metagenomics characterizes viruses and bacteria from complex environments such as soil, air filters, or sewage. Unlike targeted-primer-based sequencing, shotgun methods are not blind to sequences that are truly novel, and they can measure absolute prevalence. Shotgun metagenomic sampling can be non-invasive, efficient, and inexpensive while being informative. We have developed analysis techniques for shotgun metagenomic sequencing that rely upon phylogenetic signature patterns. They work by indexing local sequence patterns in a manner similar to web search engines. Our methods are laptop-fast and favorable scaling properties ensure they will be sustainable as sequencing methods grow. We show examples of application to soil metagenomic samples.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2018

Flexible Bandwidth Scheduling for Streaming Data Movement Over Dedicated Networks

A wide range of scientific disciplines are generating large amounts of data at a high speed, which must be transferred to remote sites for real-time processing. Reserving bandwidths over dedicated channels in high-performance networks (HPNs) within a specified time interval has proved to be an effective solution to such high-demanding data transfer. Given a bandwidth reservation request, if the desired bandwidth within the specified time interval cannot be satisfied, most of the existing scheduling algorithms simply reject the request, which would immediately terminate the application. One reasonable approach to mitigate this issue is to provide an alternative bandwidth reservation option to schedule the desired bandwidth within the time interval closest to the specified one. We propose a flexible bandwidth reservation algorithm that considers both the best and alternative bandwidth reservation options for a given request. Extensive simulations are conducted to show the superior performance of the proposed scheduling algorithm compared with a heuristic approach adapted from existing scheduling algorithms.

Zuo, Liudong↗
DOE OSTI2023

Online real-time learning of dynamical systems from noisy streaming data

Abstract Recent advancements in sensing and communication facilitate obtaining high-frequency real-time data from various physical systems like power networks, climate systems, biological networks, etc. However, since the data are recorded by physical sensors, it is natural that the obtained data is corrupted by measurement noise. In this paper, we present a novel algorithm for online real-time learning of dynamical systems from noisy time-series data, which employs the Robust Koopman operator framework to mitigate the effect of measurement noise. The proposed algorithm has three main advantages: (a) it allows for online real-time monitoring of a dynamical system; (b) it obtains a linear representation of the underlying dynamical system, thus enabling the user to use linear systems theory for analysis and control of the system; (c) it is computationally fast and less intensive than the popular extended dynamic mode decomposition (EDMD) algorithm. We illustrate the efficiency of the proposed algorithm by applying it to identify the Van der Pol oscillator, the chaotic attractor of the Henon map, the IEEE 68 bus system, and a ring network of Van der Pol oscillators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Calculation of Nuclear Reactor Cooling Tower Performance With Limited Data Streams

Monitoring of cooling tower performance in a nuclear reactor facility is necessary to ensure safe operation; however, instrumentation for measuring performance characteristics can be difficult to install and may malfunction or break down over long duration experiments. This paper describes employing a thermodynamic approach to quantify cooling tower performance, the Merkel model, which requires only five parameters, namely, inlet water temperature, outlet water temperature, liquid mass flowrate, gas mass flowrate, and wet bulb temperature. Using this model, a general method to determine cooling tower operation for a nuclear reactor was developed in situations when neither the outlet water temperature nor gas mass flowrate are available, the former being a critical piece of information to bound the Merkel integral. Furthermore, when multiple cooling tower cells are used in parallel (as would be in the case of large-scale cooling operations), only the average outlet temperature of the cooling system is used as feedback for fan speed control, increasing the difficulty of obtaining the outlet water temperature for each cell. To address these shortcomings, this paper describes a method to obtain individual cell outlet water temperatures for mechanical forced-air cooling towers via parametric analysis and optimization. In this method, the outlet water temperature for an individual cooling tower cell is acquired as a function of the liquid-to-gas ratio (L/G). Leveraging the tight tolerance on the average outlet water temperature, an error function is generated to describe the deviation of the parameterized L/G to the highly controlled average outlet temperature. The method was able to determine the gas flowrate at rated conditions to be within 3.9% from that obtained from the manufacturer’s specification, while the average error for the four individual cooling cell outlet water temperatures were 1.6 °C, -0.5 °C, -1.0 °C, and 0.3 °C.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗