Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Scalable Performance Awareness for In Situ Scientific Applications

Part of the promise of exascale computing and the next generation of scientific simulation codes is the ability to bring together time and spatial scales that have traditionally been treated separately. This enables creating complex coupled simulations and in situ analysis pipelines, encompassing such things as "whole device" fusion models or the simulation of cities from sewers to rooftops. Unfortunately, the HPC analysis tools that have been built up over the preceding decades are ill suited to the debugging and performance analysis of such computational ensembles. In this paper, we present a new vision for performance measurement and understanding of HPC codes, MonitoringAnalytics (MONA). MONA is designed to be a flexible, high performance monitoring infrastructure that can perform monitoring analysis in place or in transit by embedding analytics and characterization directly into the data stream, without relying upon delivering all monitoring information to a central database for post-processing. It addresses the trade-offs between the prohibitively expensive capture of all performance characteristics and not capturing enough to detect the features of interest. We demonstrate several uses of MONA; capturing and indexing multi-executable performance profiles to enable later processing, extraction of performance primitives to enable the generation of customizable benchmarks and performance skeletons, and extracting communication and application behaviors to enable better control and placement for the current and future runs of the science ensemble. Relevant performance information based on a system for MONA built from ADIOS and SOSflow technologies is provided for DOE science applications and leadership machines.

Wolf, Matthew D.↗
DOE OSTI2017

A lightweight network anomaly detection technique

While the network anomaly detection is essential in network operations and management, it becomes further challenging to perform the first line of detection against the exponentially increasing volume of network traffic. In this paper, we develop a technique for the first line of online anomaly detection with two important considerations: (i) availability of traffic attributes during the monitoring time, and (ii) computational scalability for streaming data. The presented learning technique is lightweight and highly scalable with the beauty of approximation based on the grid partitioning of the given dimensional space. With the public traffic traces of KDD Cup 1999 and NSL-KDD, we show that our technique yields 98.5% and 83% of detection accuracy, respectively, only with a couple of readily available traffic attributes that can be obtained without the help of post-processing. Finally, the results are at least comparable with the classical learning methods including decision tree and random forest, with approximately two orders of magnitude faster learning performance.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Representative Day Selection Using Statistical Bootstrapping for Accelerating Annual Distribution Simulations

Capturing technical and economic impacts of solar photovoltaics (PV) and other distributed energy resources (DERs) on electric distribution systems can require high-time resolution (e.g. 1 minute), long-duration (e.g. 1 year) simulations. However, such simulations can be computationally prohibitive, particularly when including complex control schemes in quasi-steady-state time series (QSTS) simulation. Various approaches have been used in the literature to down select representative time segments (e.g. days), but typically these are best suited for lower time resolutions or consider only a single data stream (e.g. PV production) for selection. We present a statistical approach that combines stratified sampling and bootstrapping to select representative days while also providing a simple method to reassemble annual results. We describe the approach in the context of a recent study with a utility partner. This approach enables much faster QSTS analysis by simulating only a subset of days, while maintaining accurate annual estimates.

14 SOLAR ENERGY↗
DOE OSTI2021

PMU-data-driven Event Classification in Power Transmission Grids

This paper presents an event classification in transmission grids. The convolutional neural network (CNN)-based classifier is proposed to capture the temporal similarity of time-synchronized data stream from phasor measurement units (PMUs). The proposed CNN is trained using Bayesian optimization to search for the best hyperparameters. The effectiveness of the proposed event classification is validated through the real-world dataset from the U.S. transmission grids. This dataset includes line outage, transformer outage, frequency, and oscillation events. The validation process also includes different PMU outputs, such as voltage magnitude, phase angle, current magnitude, frequency, and rate of change of frequency (ROCOF). The results show that ROCOF gives the best classification performance compared to other PMU outputs. In addition, it is shown that the classifier trained with a larger dataset has higher accuracy. Moreover, the superiority of the proposed method is validated through comparison with other state-of-the-art classification methods.

Niazazari, Iman↗
DOE OSTI2025

Free-Space Optical Channel Turbulence Prediction: A Machine Learning Approach

Channel turbulence is a formidable obstacle for freespace optical (FSO) communication. Anticipation of turbulence levels is highly important for mitigating disruptions but has not been demonstrated without dedicated, auxiliary hardware. We show that machine learning (ML) can be applied to raw FSO data streams to rapidly predict channel turbulence levels with no additional sensing hardware. FSO was conducted through a controlled channel in the lab under six distinct turbulence levels, and the efficacy of using ML to classify turbulence levels was examined. ML-based turbulence level classification was found to be > 98% accurate with multiple ML training parameters. Classification effectiveness was found to depend on the timescale of changes between turbulence levels but converges when turbulence stabilizes over about a one minute timescale.

Zobaer Islam, Md [School of Electrical and Compute↗
DOE OSTI2020

Simulating the Autonomous Future: A Look at Virtual Vehicle Environments and How to Validate Simulation Using Public Data Sets

The rapid evolution of autonomous vehicles (AVs) has exposed the need for fast-paced development and testing processes of a variety of perception, planning, and control algorithms. To expedite development, the AV industry and researchers leverage virtual vehicle environments to simulate a range of test scenarios that may otherwise be costly or difficult to conduct on a real test track. However, the various virtual environments may have different results depending on the fidelity of various simulation features, such as vehicle dynamics, sensor simulation, and environment recreation. Herein, this tutorial article examines a proposed framework for constructing, parameterizing, and validating a virtual vehicle environment using an existing AV data set. First, an overview of several open source and commercially available simulation tools, including their associated workflows, for scene and scenario creation is presented. Next, various open AV data sets are examined to inform the data set selection for the validation framework. Then, an example workflow of recreating a real-world scene from the selected data set in a simulation tool with various emulated sensors parameterized to match the data set is demonstrated. Finally, an example AV-perception algorithm is subjected to data streams from virtual and real-world environments and suggested metrics for analyzing the results are discussed.

42 ENGINEERING↗
DOE OSTI2021

Event Cause Analysis in Distribution Networks using Synchro Waveform Measurements

This paper presents a machine learning method for event cause analysis to enhance situational awareness in distribution networks. The data streams are captured using time-synchronized high sampling rates synchro waveform measurement units (SWMU). The proposed method is formulated based on a machine learning method, the convolutional neural network (CNN). This method is capable of capturing the spatiotemporal feature of the measurements effectively and perform the event cause analysis. Several events are considered in this paper to encompass a range of possible events in real distribution networks, including capacitor bank switching, transformer energization, fault, and high impedance fault (HIF). The dataset for our study is generated using the real time digital simulator (RTDS) to simulate real-world events. The event cause analysis is performed using only one cycle of the voltage waveforms after the event is detected. The simulation results show the effectiveness of the proposed machine learning-based method compared to the state-of-the-art classifiers.

Niazazari, Iman↗
DOE OSTI2022

Real-time Event Detection Using Rank Signatures of Real-world PMU Data

Timely detection of power system events is a crucial task, which can facilitate the implementation of remedial actions to improve reliability, resiliency, and security of the system. Meanwhile, the widespread deployment of phasor measurement units (PMUs) makes it possible to develop data-driven event detection techniques. However, relying purely on data without incorporating domain knowledge for the event detection task in power systems poses substantial security and stability risks due to issues associated with data misinterpretation and model accuracy. In this regard, we propose a real-time event detection method using real-world PMU data by incorporating domain knowledge to adequately capture the event signatures. Specifically, we track the change in rank signatures of PMU data to accurately localize the events. To optimize the detection process, we incorporate an offline Bayesian optimization algorithm to tune the parameters by efficiently searching for the best values. The experiments using the real-world PMU dataset from a U.S. interconnection show that the proposed event detection approach can efficiently detect the events from PMU data streams with high accuracy.

Ghasemkhani, Amir↗
DOE OSTI2024

Enabling DER visibility using a distributed dissemination network

The electrical grid is currently undergoing a series of rapid transformational changes that have resulted in the introduction of new actors and operational schemes that have fragmented the data and control planes. To help address the issue, this paper describes the implementation of a sensor-oriented, distributed data dissemination network that seeks to eliminate data silos. The implementation is based on the DGSS architecture previously described in [1]. The developed product seeks to facilitate the seamless integration of multi-operator, multi-origin, multi-domain sensor data by using a distributed systems approach. The proposed solution decouples the sensor’s data streams from the application-specific infrastructure and migrates them into a software-defined databus that can be configured to suit the end application’s demands. To further validate DGSS capabilities, a DER oriented use case has been developed.

Sensor Dissemination Networks, Enhanced DER visibi↗
DOE OSTI2019

DeepWelding: a Deep Learning Enhanced Approach to GTAW Using Multi-source Sensing Images

Deep learning has great potential to reshape manufacturing industries. In this paper, we present DeepWelding, a novel framework that applies deep learning techniques to improve gas tungsten arc welding (GTAW) process monitoring and penetration detection using multi-source sensing images. The framework is capable of analyzing multiple types of optical sensing images synchronously and consists of three deep learning enhanced consecutive phases: image preprocessing, image selection, and weld penetration classification. Specifically, we adopted generative adversarial networks (pix2pix) for image denoising and classic convolutional neural networks (AlexNet) for image selection. Both pix2pix and AlexNet delivered satisfactory performance. However, five individual neural networks with heterogeneous architectures demonstrated inconsistent generalization capabilities in the classification phase when holding out multi-source images generated with specific experimental settings. Therefore, two ensemble methods combining multiple neural networks were designed to improve the model performance on unseen data collected from different experimental settings. In conclusion, we also found that the quality of model prediction was heavily influenced by the data stream collection environment. We think these findings are beneficial for the broad intelligent welding community.

42 ENGINEERING↗
DOE OSTI2024

Real-Time Ethernet Interface for NSTX-U’s Thomson Scattering Diagnostic (2023)

Here, the multipoint Thomson scattering (MPTS) diagnostic system at the National Spherical Torus Experiment Upgrade (NSTX-U) facility is undergoing an upgrade to operate in real-time and interface with the plasma control system (PCS) for NSTX-U. Previous prototyping efforts have shown that spectral analysis and rapid calculations of electron temperature and density are possible on a real-time Linux machine when using up to a 100-Hz laser pulse repetition rate. A remaining challenge was transferring the real-time data to NSTX-U’s PCS, which utilizes the front panel data port (FPDP) protocol. The original proposed method was to convert the real-time data into analog values, but a new solution was developed to keep the output format digital by using an Ethernet controller with a field-programmable gate array (FPGA). This article focuses on a new input module that has been developed to accept incoming user datagram protocol (UDP) packets sent over Ethernet, convert into FPDP format, and integrate into the existing data stream under NSTX-U’s real-time framework.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Proximal remote sensing: an essential tool for bridging the gap between high‐resolution ecosystem monitoring and global ecology

Summary A new proliferation of optical instruments that can be attached to towers over or within ecosystems, or ‘proximal’ remote sensing, enables a comprehensive characterization of terrestrial ecosystem structure, function, and fluxes of energy, water, and carbon. Proximal remote sensing can bridge the gap between individual plants, site‐level eddy‐covariance fluxes, and airborne and spaceborne remote sensing by providing continuous data at a high‐spatiotemporal resolution. Here, we review recent advances in proximal remote sensing for improving our mechanistic understanding of plant and ecosystem processes, model development, and validation of current and upcoming satellite missions. We provide current best practices for data availability and metadata for proximal remote sensing: spectral reflectance, solar‐induced fluorescence, thermal infrared radiation, microwave backscatter, and LiDAR. Our paper outlines the steps necessary for making these data streams more widespread, accessible, interoperable, and information‐rich, enabling us to address key ecological questions unanswerable from space‐based observations alone and, ultimately, to demonstrate the feasibility of these technologies to address critical questions in local and global ecology.

Plant Sciences↗
DOE OSTI2023

Approaches for Synthesis and Deployment of Controller Models on Automated Vehicles for Car-following in Mixed Autonomy

This paper describes the software design patterns and vehicle interfaces that were employed to transition vehicle controllers from simulation environments to open-road field experiments. The approach relies on a life cycle that utilizes model-based design and code generation, along with agile software development, and both software and hardware-in-the-loop testing, with additional safety margins. Autonomous designs should consider the dynamics of mixed autonomy in traffic to safely operate among humans. The software that provides a vehicle’s behavior intelligence is often developed through simulation, which may have a mismatch between dynamics, or as a result of a reinforcement learning workflow, which may be a black box with challenges to analyze. In each of these cases, it is important to have research interfaces that provide strongly typed data streams accessible to researchers who are not software experts while continuing to satisfy safety and liveness constraints. This paper describes how we design the hardware platform interfaces and software design process for a mixed autonomy traffic experiment with a leader-follower scenario. Controller synthesis for these vehicles requires clearly articulated vehicle interfaces and software design patterns for successful onboard deployment. Testing strategies for such controllers are also described before algorithms are transitioned to full-scale field experiments with safety operators for the vehicles. Testing strategies include software-in-the-loop simulation testing, hardware-in-the-loop simulation, ghost-car testing, and read-only testing in live traffic. With our approach, we were not only able to validate our controller synthesized in scripts and simulation, but also able to scale deployment to multiple vehicles.

Bhadani, Rahul↗
DOE OSTI2025

Development of Emergency Vehicle Preemption Strategies on Smart Corridors in a Digital Twin Environment

Emergency Response Vehicles (ERVs), such as firetrucks, ambulances, etc. operate with the purpose of saving lives and mitigating property damage. As such, ERV travel-time reductions may result in significant benefits to the community. A common strategy to improve travel times is Emergency Vehicle Preemption (EVP). EVP seeks to reduce ERV delays by providing the right-of-way to ERVs as they approach an intersection. This study proposes a new Dynamic Preemption strategy that determines the need for preemption prior to the ERV reaching the vicinity of the intersection, utilizing real-time data streams. This paper evaluates the effectiveness of some existing and proposed preempt control strategies using a digital twin testbed consisting of a series of signalized intersections on an urban arterial in Georgia. The best EVP strategy maximizes the improvement in ERV travel time while minimizing the adverse effect of preemption on the traffic in conflicting directions. Therefore, this study evaluates both the positive impact of EVP on the ERV as well as the adverse impact on the cross-street traffic. The study found that the potential exists for significant improvements in ERV travel time with the proposed Dynamic Preemption strategy, with minimal impact to the conflicting traffic. For the simulation corridor there was a 20% reduction in the ERV travel times with the implementation of the Dynamic Preemption strategy, compared to traditional EVP practices.

Roy, Somdut [Georgia Institute of Technology]↗
DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗
DOE OSTI2016

UCNB_Analyzer

The purpose of this software is to interpret and analyze data taken using the NI PXIe-5171R digitizer based data acquisition system for the UCNB and Nab experiments. The detection and data acquisition systems are identical for the 2 experiments, with some differences in analysis requirements. The software converts raw binary files produced by the NI DAQ into ROOT TTree format, performs waveform analysis using trapezoidal filter algorithms, pulse fitting, and noise analysis routines, and applies variable criteria to identify valid events in the data stream. The software will be used to perform analysis of the events for multi-channel coincidences, timing and energy studies, and event rates under different experimental conditions.

Broussard, LeahJ↗
DOE OSTI2018

Sonar

Sonar contains software for managing a HPC performance-monitoring software stack. This includes streaming data into a distributed Kafka message queue, storing/querying from a distributed Cassandra database, and front-end tools for secure access, visualization, and analytics of facility-wide performance data.

Gimenez, AlfredoA.↗
DOE OSTI2019

bokeh streaming [SWR 19-08]

Simple examples of bokeh streaming data. The plots are intended to be generic and able to infer some of the plot data from the structure of the data sent. The receiver accepts zlib compressed pickled python dictionaries.

Simpson, Jeffery↗