SEARCH · Search NASA

Results for “Streaming Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

DOE OSTI2017

Enhanced Data Authentication System v. 2.0

EDAS is a system, comprised on hardware and software, that plugs in to an existing data stream, and branches all data for transmission to a secondary observer computer. The EDAS Junction box, which inserts into the data stream, has Java software that forms these data into packets, digitally signs, encrypts, and sends these packets to a safeguards inspector computer. Further, there is a second Java program running on the secondary observer computer that receives data from the EDAS Junction Box to decrypt, authenticate, and store incoming packets. Also, there is a stand-alone Java program that is used to configure the EDAS Junction Box.

Thomas, Maikael A.↗

DOE OSTI2012

Xrootd monitoring for the CMS experiment

During spring and summer of 2011, CMS deployed Xrootd-based access for all US T1 and T2 sites. This allows for remote access to all experiment data on disk in the US. It is used for user analysis, visualization, running of jobs at computing sites when data is not available at local sites, and as a fail-over mechanism for data access in jobs. Monitoring of this Xrootd infrastructure is implemented on three levels. Basic service and data availability checks are performed by Nagios probes. The second level uses Xrootd's summary data stream, this data is aggregated from all sites and fed into a MonALISA service providing visualization and storage. The third level uses Xrootd's detailed monitoring stream, which includes detailed information about users, opened files and individual data transfers. A custom application was developed to process this information. It currently provides a real-time view of the system usage and can store data into ROOT files for detailed analysis. Detailed monitoring allows us to determine dataset popularity and to detect abuses of the system, including sub-optimal usage of the Xrootd protocol and the ROOT prefetching mechanism.

Bauerdick, L. A.T.↗

DOE OSTI2016

Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1

Handling large streaming data is essential for various applications such as network traffic analysis, social networks, energy cost trends, and environment modeling. However, it is in general intractable to store, compute, search, and retrieve large streaming data. This software addresses a fundamental issue, which is to reduce the size of large streaming data and still obtain accurate statistical analysis. As an example, when a high-speed network such as 100 Gbps network is monitored, the collected measurement data rapidly grows so that polynomial time algorithms (e.g., Gaussian processes) become intractable. One possible solution to reduce the storage of vast amounts of measured data is to store a random sample, such as one out of 1000 network packets. However, such static sampling methods (linear sampling) have drawbacks: (1) it is not scalable for high-rate streaming data, and (2) there is no guarantee of reflecting the underlying distribution. In this software, we implemented a dynamic sampling algorithm, based on the recent technology from the relational dynamic bayesian online locally exchangeable measures, that reduces the storage of data records in a large scale, and still provides accurate analysis of large streaming data. The software can be used for both online and offline data records.

Sim, Alex↗

DOE OSTI2023

Contextually aware roadside radiation measurement testbed

Here we demonstrate a contextually aware multimodal roadside radiation measurement detection testbed for traffic monitoring applications in nuclear nonproliferation. Many variables in traffic such as vehicle or cargo size, mass, speed, shape, and distance of closest approach can have significant impacts on the radiation measured from a vehicle-transported radiation source. These factors can lead to uncertainties in the analysis of the radiation source, especially for lower-strength radiation sources of interest. Our testbed, known as the Multimodal Measurement System (MMS) uses non-radiation sensors including magnetometers, geophones, radiofrequency receivers, cameras, and LiDAR to extract contextual information about vehicles passing by the system. These contextual data can then be fused with data from radiation measurements to increase the system’s sensitivity and accuracy in nuclear threat detection applications. This work describes the instrumentation of the MMS and its data acquisition pipeline. Furthermore, we describe the pre-analysis performed on the raw multimodal data streams for data fusion, and the high-level machine learning analyses for detection and characterization. The variety of sensors within the MMS provides a valuable testbed that can be used to identify the combinations of contextual sensors that provide the greatest improvements to radiation source detection and characterization within the restrictions for various proliferation detection applications. The MMS is also modular so that additional combinations of sensors can be explored in the future.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2020

Cyber‐physical system testbed for power system monitoring and wide‐area control verification

The electric power system is intrinsically a cyber‐physical system (CPS) with power flowing inthe physical system and information flowing in the cyber‐network. Testbeds arecrucial for understanding the cyber‐physical interactions and provideenvironments for prototyping novel applications. This study proposes afour‐layer architecture for CPS testbeds with emphases on communication networkemulation and networked physical components. A configurable software‐definednetwork is employed to bridge physical components with wide‐area applicationsfor closed‐loop control. In order to distribute physically coupled devices intomultiple software simulations, this study proposes a data broker setup based ona distributed messaging environment to achieve low‐latency data streaming. Thedecoupled design with data streaming allows for building testbed components asmodules and running them in a distributed manner. Case studies verify the databroker setup for low‐latency sensing and actuation, as well as the communicationemulation setup for the desired network latency. Also illustrated is a replayattack scenario using synchrophasors in the Western Electricity CoordinatingCouncil (WECC) 181‐bus system for demonstrating the closed‐loop cyber‐physicalsimulation capability of the testbed.

Cui, Hantao↗

DOE OSTI2020

JANA2 Framework for Event Based and Triggerless Data Processing

Development of the second generation JANA2 multi-threaded event processing framework is ongoing through an LDRD initiative grant at Jefferson Lab. The framework is designed to take full advantage of all cores on modern many-core compute nodes. JANA2 efficiently handles both traditional hardware triggered event data and streaming data in online triggerless environments. Development is being done in conjunction with the Electron Ion Collider development. Anticipated to be the next large scale Nuclear Physics facility constructed. The core framework is written in modern C++ but includes an integrated Python interface. The status of development and summary of the more interesting features are presented.

Lawrence, David↗

DOE OSTI2024

Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows

The need for real-time processing to enable automated decision making and experimental steering has driven a shift from high-performance computing workflows on a centralized system to a distributed approach that integrates remote data sources, edge devices, and diverse compute facilities. Under this paradigm, data can be processed close to the source where it is generated, thus reducing latency and bandwidth usage. System resilience is thus a key challenge, requiring distributed workflows to survive component failures and to meet stringent quality-of-service requirements, which results in the need to mitigate anomalies such as congestion and low availability of resources. To address these challenges, we propose Diaspora, a unified resilience framework that is inspired by event-driven communication patterns used in public clouds. Specifically, we propose an event fabric that extends across sites, facilities, and computations to provide timely, reliable, and accurate information about data, application, and resource status. On top of the event fabric, we build resilience-enabling services that combine QoS-aware data streaming, resilient data views, resilient compute and data resources, and anomaly detection and prediction, all of which collectively enhance workflow resilience for these scientific cases.

Rao, Nageswara↗

DOE OSTI2025

Real Time Phasor Analytics (RTPA) and RTPA-SCR System Strength Online Tool

This presentation showcases the Real-Time Phasor Analytics (RTPA) framework for monitoring inertia and assessing system strength in power grids. RTPA is an open-source tool designed to standardize access to data from Power Management Units (PMUs) and Phasor Data Concentrators (PDCs). It facilitates real-time connectivity to multiple PDCs in accordance with the IEEE C37.118-2 standard and supports asynchronous data stream integration. Additionally, RTPA can simulate a PDC server streaming C37.118-2 data and provides Python bindings for seamless interaction with the framework, eliminating the need for direct Rust programming.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2018

Epi Archive: Automated Synthesis of Global Notifiable Disease Data

ObjectiveLANL has built software that automatically collects global notifiable disease data, synthesizes the data, and makes it available to humans and computers within the Biosurveillance Ecosystem (BSVE) as a novel data stream. These data have many applications including improving the prediction and early warning of disease events.IntroductionMost countries do not report national notifiable disease data in a machine-readable format. Data are often in the form of a file that contains text, tables and graphs summarizing weekly or monthly disease counts. This presents a problem when information is needed for more data intensive approaches to epidemiology, biosurveillance and public health.While most nations likely store incident data in a machine-readable format, governments are often hesitant to share data openly for a variety of reasons that include technical, political, economic, and motivational issues1.A survey conducted by LANL of notifiable disease data reporting in over fifty countries identified only a few websites that report data in a machine-readable format. The majority (>70%) produce reports as PDF files on a regular basis. The bulk of the PDF reports present data in a structured tabular format, while some report in natural language.The structure and format of PDF reports change often; this adds to the complexity of identifying and parsing the desired data. Not all websites publish in English, and it is common to find typos and clerical errors.LANL has developed a tool, Epi Archive, to collect global notifiable disease data automatically and continuously and make it uniform and readily accessible.MethodsWe conducted a survey of the national notifiable disease reporting systems notating how the data are reported and in what formats. We determined the minimal metadata that is required to contextualize incident counts properly, as well as optional metadata that is commonly found.The development of software to regularly ingest notifiable disease data and make it available involves three or four main steps: scraping, detecting, parsing and persisting.Scraping: we examine website design and determine reporting mechanisms for each country/website, as well as what varies across the reporting mechanisms. We then designed and wrote code to automate the downloading of the data for each country. We store all artifacts presented as files (PDF, XLSX, etc.) in their original form, along with appropriate metadata for parsing and data provenance.Detecting: This step is required when parsing structured non-machine-readable data such as tabular data in PDF files. We combined the Nurminen methodology of PDF table detection with in-house heuristics to find the desired data within PDF reports2.Parsing: We determined what to extract from each dataset and parsed these data into uniform data structures, correctly accommodating the variations in metadata (e.g., time interval definitions) and the various human languages.Persisting: We store the data in the Epi Archive database and make it available on the internet and through the BSVE. The data is persisted into a structured and normalized SQL database.ResultsThe Epi Archive tool currently contains national and/or subnational notifiable disease data from twenty nations. When a user accesses the Epi Archive site, they are prompted with four fields: country, subregion, disease of interest, and date duration. Upon form submission, a time series is generated from the users’ specifications. The generated graph can then be downloaded into a CSV file if a user is interested in performing personal analysis. Additionally, the data from Epi Archive can be reached through a REST API (Representational State Transfer Application Programming Interface).ConclusionsLANL, as part of a currently funded DTRA effort, is automatically and continually collecting global notifiable disease data. While 20 nations are in production, more are being brought online in the near future. These data are already being utilized and will have many applications including improving the prediction and early warning of disease events.References[1] van Panhuis WG, Paul P, Emerson C, et al. A systematic review of barriers to data sharing in public health. BMC Public Health. 2014. 14:1144. doi:10.1186/1471-2458-14-1144[2] Nurminen, Anssi. "Algorithmic extraction of data in tables in PDF documents." (2013).

Khalsa, Hari S.↗

DOE OSTI2019

Soft decision analyzer and method

A soft decision analyzer system is operable to interconnect soft decision communication equipment and analyze the operation thereof to detect symbol wise alignment between a test data stream and a reference data stream in a variety of operating conditions.

Steele, Glen F.↗

DOE OSTI2015

A MGy radiation-hardened sensor instrumentation link for nuclear reactor monitoring and remote handling

Decommissioning, dismantling and remote handling applications in nuclear facilities all require robotic solutions that are able to survive in radiation environments. Recently raised safety, radiation hardness and cost efficiency demands from both the nuclear regulatory and the society impose severe challenges in traditional methods. For example, in case of the dismantling of the Fukushima sites, solutions that survive accumulated doses higher than 1 MGy are mandatory. To allow remote operation of these tools in nuclear environments, electronics were used to be shielded with several centimeters of lead or even completely banned in these solutions. However, shielding electronics always leads to bulky and heavy solutions, which reduces the flexibility of robotic tools. It also requires longer repair time and produces extra waste further in a dismantling or decommissioning cycle. In addition, often in current reactor designs, due to size restrictions and the need to inspect very tight areas there are limitations to the use of shielding. A MGy radiation-hardened sensor instrumentation link developed by MAGyICS provides a solution to build a flexible, easy removable and small I and C module with MGy radiation tolerance without any shielding. Hereby it removes all these pains to implement electronics in robotic tools. The demonstrated solution in this poster is developed for ITER Remote Handling equipments operating in high radiation environments (>1 MGy) in and around the Tokamak. In order to obtain adequately accurate instrumentation and control information, as well as to ease the umbilical management, there is a need of front-end electronics that will have to be located close to those actuators and sensors on the remote handling tool. In particular, for diverter remote handling, it is estimated that these components will face gamma radiation up to 300 Gy/h (in-vessel) and a total dose of 1 MGy. The radiation-hardened sensor instrumentation link presented here, consists of multiple in-house developed, state-of-the-art building blocks. They have all been assessed and characterized up to a dose higher than 1 MGy. The first block in the signal chain is a smaller than 1 uV offset, low noise instrumentation amplifier that has a programmable gain between 8 and 256. This amplifier is followed by a high precision 16 bit ADC. Its main function is to digitize the amplified signal coming from the amplifier. The output of the ADC is a serial digital data stream. The next block is a 8 channel digital multiplexer. It converts the 8 digital data streams into 1 digital data stream. Hereby the instrumentation solution reduces the number of cables from 16 to 1 (8* 2 analogue differential signals). In addition, the multiplexer modules can be combined with other modules to reduce even further the number of cables. Every instrumentation solution requires a stable high precision voltage reference. Therefore also a bandgap reference has been developed and assessed under gamma irradiation. A low jitter, 10 MHz clock generator has been developed and qualified to clock the ADC and the multiplexer with high accuracy. Finally, an on-chip radiation-hard temperature sensor is also included. A complete remote, real-time test setup was prepared by MAGyICS in cooperation with Fusion for Energy to qualify the sensor instrumentation link at SCK-CEN. It is qualified by closely following the ESCC22900 space standard for electronics used in a radiation environment. The main benefit of the sensor instrumentation solution discussed here is that it can be directly employed in a MGy-level accumulated dose radiation environment, therefore it can digitize and multiplex sensor readout values early in the signal chain. Hereby the sensor values are not distorted by external interferences on the long transmission cable. Moreover it allows readout and digitize multiple low-bandwidth sensors ( pressure and temperature sensors, thermocouples, angular resolvers and LVDTs). Hence these instrumentation modules can be used to follow the trend of the digitalization in the nuclear industry. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2018

Improving Quality of Observational Streaming Medical Data by Using Long Short-Term Memory Networks (LSTMs)

We present an exploration of the encoder-decoder structured Long Short-Term Memory Network (LSTM) as a detector of the anomalous missing observations in streaming medical data by using the difference between the LSTM-reconstructed and observed values as the anomaly detector. We experiment with time-series data from bedside monitoring devices from the available Medical Information Mart for Intensive Care Database (MIMIC). Our results show that not only encoder-decoder LSTM approach works well for detecting the difference between anomalous and normal missing observations in streaming medical data, but also has an imputation potential for the missing data.

Bowie, Michael B.↗

DOE OSTI2019

Open Humans: A platform for participant-centered research and personal data exploration

Background: Many aspects of our lives are now digitized and connected to the internet. As a result, individuals are now creating and collecting more personal data than ever before. This offers an unprecedented chance for human-participant research ranging from the social sciences to precision medicine. With this potential wealth of data comes practical problems (e.g., how to merge data streams from various sources), as well as ethical problems (e.g., how best to balance risks and benefits when enabling personal data sharing by individuals). Results: To begin to address these problems in real time, we present Open Humans, a community-based platform that enables personal data collections across data streams, giving individuals more personal data access and control of sharing authorizations, and enabling academic research as well as patient-led projects. We showcase data streams that Open Humans combines (e.g., personal genetic data, wearable activity monitors, GPS location records, and continuous glucose monitor data), along with use cases of how the data facilitate various projects. Conclusions: Open Humans highlights how a community-centric ecosystem can be used to aggregate personal data from various sources, as well as how these data can be used by academic and citizen scientists through practical, iterative approaches to sharing that strive to balance considerations with participant autonomy, inclusion, and privacy.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2022

Automated Shift Detection in Sensor-Based PV Power and Irradiance Time Series: Preprint

PV power and irradiance sensor-based measurements are prone to error, resulting in issues such as abrupt time series data shifts. These shifts, which are usually unintentional, may be caused by software or hardware configuration changes on a PV system, and do not reflect an actual change in overall system performance. Locating these shifts and segmenting the associated time series aids in more accurate future PV analysis. In this research, an offline changepoint detection (CPD) algorithm that automatically detects these abrupt data shifts in sensor-based time series is introduced. Data shift periods in 101 daily PV power and irradiance time series were labeled manually by two solar experts. These data streams represent sensor-based measurements, and display a variety of data shift behaviors. A changepoint detection algorithm was tuned using the 101 labeled data streams, with each model configuration's ability to detect labeled changepoints benchmarked using metrics such as F1-score, recall, and Rand Index. Best performing models on seasonality-corrected data streams include the Pruned Exact Linear (PELT) method, the Binary Segmentation method, and the Bottom-Up method, all scoring an average F1-score of 0.76 or greater at detecting labeled changepoints within a 30-day window for the labeled data sets. To promote further research in this space, we are releasing the labeled data shift sets on U.S. Department of Energy's (DOE) DuraMAT Data Hub, and the associated algorithm in the Python PVAnalytics package.

changepoint detection↗

DOE OSTI2022

Automated Shift Detection in Sensor-Based PV Power and Irradiance Time Series

changepoint detection↗

DOE OSTI2022

Automated Shift Detection in Sensor-Based PV Power and Irradiance Time Series

PV power and irradiance sensor-based measurements are prone to error, resulting in issues such as time series data shifts. In this research, a changepoint detection (CPD) algorithm that automatically detects data shifts in sensor-based time series is introduced. Data shift periods in 101 daily PV power and irradiance time series were labeled manually by two solar experts. These data streams represent sensor-based measurements, and display a variety of data shift behaviors. A changepoint detection algorithm was tuned using the 101 labeled data streams, with each model configuration's ability to detect labeled changepoints benchmarked using metrics such as F1-score, recall, and Rand Index. Best performing models on seasonality-corrected data streams include the Pruned Exact Linear (PELT) method, the Binary Segmentation method, and the Bottom-Up method, all scoring an average F1-score of 0.76 or greater at detecting labeled changepoints within a 30-day window across the labeled data sets. Pending approval, we plan to release the labeled data sets for this research on NREL's DuraMAT Data Hub, and the associated algorithm in the Python PVAnalytics package. By supplying the training sets and algorithm, we hope to encourage further development in this research space.

data shift↗

DOE OSTI2022

Overview and How-to Tutorial Videos for Using NEWTS Data

Overview and How-to Tutorial Videos for Using NEWTS Data Video 1: Overview of NEWTS Database Video 2: How-to tutorial for EPA Flue Gas Desulfurization (FGD) Effluent NEWTS dataset Video 3: How-to tutorial for USGS Produced Waters NEWTS dataset Video 4: How-to tutorial for EPA Ash NEWTS dataset Video 5: How-to tutorial for Quillinan, et al 2018 DOE Geothermal Technology Office REE dataset Video 6: Tutorial video on navigating the NEWTS Dashboard, with an overview of NEWTS and navigating between the NEWTS Dashboard and Datasets (https://netl-doe.maps.arcgis.com/apps/dashboards/a5fa4192f7c6478dab3d6180d9c30b84) Video 7: Additional tutorial video on navigating the NEWTS Dashboard and investigating specific data points in the Dashboard and Datasets Video 8: Re-record of recent webinar giving an overview of the NEWTS Database and Dashboard, including interacting with the NEWTS Dashboard, locating specific data points, and finding the relevant streams in the NEWTS Database and datasets on EDX. Includes overview of the datasets, case studies, and steps for taking stream data from the database and modeling stream data in OLI Studio and Geochemist's Workbench. Note: Video 3 tutorial is also applicable to the USGS Brackish Water NEWTS dataset.

Aqueous Chemistry↗

DOE OSTI2025

A Data-Agnostic, Continuous Machine Learning Framework for Application in High Energy Physics and Beyond: Phase 1 Final Scientific/Technical Report

This Phase 1 effort has focused on the development of continual learning frameworks for use in machine learning, specifically in the applied context of High Energy Physics (HEP). Machine learning (ML) is a transformative technology by which computers, typically through the use of neural networks, are able to perform tasks with proficiency that rivals or surpasses that of human users. Model Degradation & Catastrophic Forgetting are two undesired phenomena which can occur in ML where the performance of a model degrades when either deployed on novel data streams, or trained on novel data which are sufficiently different than the data the models were initially trained on. A natural example where these sorts of effects can be observed is in the performance of detectors in harsh environments, where the detector signature may change over the lifetime of the detector as it ages and deteriorates — precisely what occurs in the experiments conducted in HEP. Real world HEP data is therefore an excellent test-ground and use-case for Continual Learning paradigms, which are techniques used in ML to counteract these problems. Ensemble learning is one such technique, where multiple smaller models are trained on subsets of the overall data and are ensembled together during inference. The intuition behind this technique is that, although there are shifts in the distributions which govern the incoming data streams, these shifts are not expected to be homogeneous or global. If a sufficient diversity in solutions within the various sub-models has been achieved, then at least one sub-model is expected to retain its performance within the overall ensemble. One further strength of this approach is that the architectures of the various models do not need to be identical, and in fact even different modalities of data can naturally be combined in this way. This work focused on applying ensemble learning techniques to derive results using two main datasets, anomaly detection in HEP data & time-series forecasting in semiconductor manufacturing data. Semiconductor manufacturing involves data with surprising similarity to that of HEP (e.g. wafer maps look very similar to digi-occupancy maps) and Cerium Lab’s prominence within the semiconductor industry makes semiconductor manufacturing a natural opportunity for commercialization of this work. Our efforts have led to two strong results. The first is that we evaluated the proposed ensembling techniques using previously proposed machine learning architectures for use in anomaly detection, namely AutoEncoder based models and their derivatives. We also developed new architectures which have not been evaluated in this context before. In fact, this work marks the first use of Vision Transformers for anomaly detection in HEP. Second, we demonstrated that ensemble learning significantly improves model performance in scenarios prone to degradation, validating its effectiveness across both HEP and semiconductor datasets. These results further support ensemble learning as a powerful strategy for mitigating catastrophic forgetting and maintaining robust performance in evolving data environments.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗