Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,045 records · Page 58
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Neutrino beam bunch structure reconstruction with precision timing in the ICARUS liquid argon time projection chamber

The ICARUS detector has been operating smoothly since 2021 as the far detector in the Short Baseline Neutrino (SBN) program at Fermilab, collecting neutrino interactions from both the Booster Neutrino Beam (BNB) and off-axis from the Neutrinos at the Main Injector (NuMI) beam. Analysis of neutrino interactions in ICARUS requires mitigation of substantial cosmogenic backgrounds. This is achieved by using an external Cosmic Ray Tagger (CRT) and a Photomultiplier Tube (PMT) system embedded in the liquid argon. The intrinsic neutrino beam bunch structure, inherited time structure from the Radio Frequency (RF) system used to accelerate the protons, can be resolved at the ICARUS detector using precise timing information. Located at shallow depth, ICARUS is exposed to a high flux of cosmic rays that can be mistaken for neutrino interactions. To mitigate this background, the CRT and a 3-meter-thick concrete overburden were installed. To better model backgrounds, ICARUS makes use of an overlay technique where simulated neutrino events are superimposed on detector beam-off data. PMTs installed within a Time Projection Chamber detect argon scintillation light emitted by high energy charged particles passing through the chamber and provide the event timing of neutrino interactions. The nanosecond-level beam bunch structure is reconstructed with the PMT system and can be used to further understand backgrounds and enhance neutrino physics capabilities. In this thesis, I will discuss background mitigation techniques using precision timing and present a novel technique to select neutrino events from our unbiased data stream using the beam bunch structure.

Heggestuen, Anna [Colorado State U.] (ORCID:000000↗
DOE OSTI2026

EXCLUSIVE NEUTRAL PION ELECTROPRODUCTION CROSS SECTION MEASUREMENTSWITHANEUTRALPARTICLE SPECTROMETER

Deep Virtual Compton Scattering (DVCS), the exclusive electron-proton scattering process ep ¿e'p'¿, provides access to generalized parton distributions (GPDs), which correlate information about the longitudinal momentum and transverse spatial structure of quarks inside the nucleon. Experiment E12-13-010 in Hall C at Jefferson Lab was designed to take high-precision measurements of the DVCS cross section over an extended kinematic range using the newly commissioned Neutral Particle Spectrometer (NPS). The NPS features a high-resolution electromagnetic calorimeter and a streaming data acquisition system optimized for operation at high luminosities. This thesis presents the detector and analysis work carried out to support the NPS DVCS program. In particular, it focuses on the hardware design, calibration, and performance of the calorimeter. A development of a waveform reconstruction analysis of the calorimeter signals enabled improved extraction of pulse amplitudes and times. The waveform analysis was also extended to operate in a multithreaded environment, substantially reducing processing time for large datasets. Analysis of exclusive neutral pion electroproduction events in the calorimeter gives a strong validation of the calorimeter’s performance and resolution. Together these developments establish a foundation for future analyses and extraction of the DVCS cross section and its use in constraining the GPDs.

Kerver, Mitchell [Old Dominion Univ., Norfolk, VA ↗
DOE OSTI2022

Data from: “Significant stream chemistry response to temperature variations in a high-elevation mountain watershed”

High-elevation mountain regions, central to global freshwater supply, are experiencing more rapid warming than low-elevation locations. High-elevation streams are therefore potentially critical indicators for earth system and water chemistry response to warming. Here we present concerted hydroclimatic and biogeochemical data from Coal Creek, Colorado in the central Rocky Mountains at elevations of 2700 to 3700 m, where air temperatures have increased by about 2 °C since 1980. We analyzed water chemistry every other day from 2016 to 2019. Water chemistry data indicate distinct responses of different solutes to inter-annual hydroclimatic variations. Specifically, the concentrations of solutes from rock weathering are stable inter-annually. Solutes that are active in soils, including dissolved organic carbon, vary dramatically, with double to triple peak concentrations occurring during snowmelt and in warm years. We advocate for consistent and persistent monitoring of high elevation streams to record early glimpse of earth surface response to warming.1) Coal Creek stream were sampled and measured to answer the question if there is a significant stream chemistry response to climate change in high-elevation watershed2) The file contains data regarding discharge, water temperature and stream chemistry data for 2016-2019 year. Stream chemistry includes dissolved organic and inorganic carbon, chloride, nutrients, cations. The data collection efforts were carried out by the DOE-funded "Advancing a Watershed Hydro-Biogeochemical Theory: Linking Water Travel Time and Reaction Rates Under Changing Climate" project and Watershed Function SFA.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Investigating the impacts of solid phase extraction on dissolved organic matter optical signatures and the pairing with high‐resolution mass spectrometry data across a freshwater stream network

Abstract Advancing our understanding of dissolved organic matter (DOM) chemistry in aquatic systems necessitates the integration of data streams from multiple analytical platforms. Some measurements require pretreatment with solid phase extraction (SPE), while others are performed directly on whole water samples. Evidence has suggested that SPE will be biased against select DOM fractions, leading to concerns over the ability to establish data linkages across platforms with variable needs for SPE pretreatment, such as those from optical measurements and those that provide high‐resolution molecular information. Here, we directly addressed this concern by assessing the impact of SPE on DOM optical properties through excitation–emission matrices with parallel factor analysis (PARAFAC) for 47 samples across a stream network within a single watershed reflective of variable DOM sources. PARAFAC data was further paired with molecular information obtained by Fourier transform ion cyclotron resonance mass spectrometry (FTICR‐MS). A comparison of PARAFAC models first revealed no systematic qualitative differences in major components between whole water DOM and DOM isolated by SPE (SPE‐DOM); however, quantitative biases against select components were observed. Further linkages with FTICR‐MS data revealed that the molecular fingerprint associated with each PARAFAC component was consistent between the whole water DOM and SPE‐DOM. Our results suggest that bulk scale linkages across these analytical platforms could be inferred irrespective of the observed quantitative biases resulting from SPE for samples within this example watershed. This work represents a key step toward the systematic evaluation of linkages between optical and high‐resolution mass spectrometry datasets in freshwater lotic environments.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗
DOE OSTI2016

Online anomaly detection for multi‐source VMware using a distributed streaming framework

Summary Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Such non‐conformant patterns typically correspond to samples of interest and are assigned to different labels in different domains, such as outliers, anomalies, exceptions, and malware. A daunting challenge is to detect anomalies in rapid voluminous streams of data. This paper presents a novel, generic real‐time distributed anomaly detection framework for multi‐source stream data. As a case study, we investigate anomaly detection for a multi‐source VMware‐based cloud data center, which maintains a large number of virtual machines (VMs). This framework continuously monitors VMware performance stream data related to CPU statistics (e.g., load and usage). It collects data simultaneously from all of the VMs connected to the network and notifies the resource manager to reschedule its CPU resources dynamically when it identifies any abnormal behavior from its collected data. A semi‐supervised clustering technique is used to build a model from benign training data only. During testing, if a data instance deviates significantly from the model, then it is flagged as an anomaly. Effective anomaly detection in this case demands a distributed framework with high throughput and low latency. Distributed streaming frameworks like Apache Storm, Apache Spark, S4, and others are designed for a lower data processing time and a higher throughput than standard centralized frameworks. We have experimentally compared the average processing latency of a tuple during clustering and prediction in both Spark and Storm and demonstrated that Spark processes a tuple much quicker than storm on average. Copyright © 2016 John Wiley & Sons, Ltd.

Solaimani, Mohiuddin↗
DOE OSTI2015

Statistical Techniques For Real-time Anomaly Detection Using Spark Over Multi-source VMware Performance Data

Anomaly detection refers to the identi cation of an irregular or unusual pat- tern which deviates from what is standard, normal, or expected. Such deviated patterns typically correspond to samples of interest and are assigned different labels in different domains, such as outliers, anomalies, exceptions, or malware. Detecting anomalies in fast, voluminous streams of data is a formidable chal- lenge. This paper presents a novel, generic, real-time distributed anomaly detection framework for heterogeneous streaming data where anomalies appear as a group. We have developed a distributed statistical approach to build a model and later use it to detect anomaly. As a case study, we investigate group anomaly de- tection for a VMware-based cloud data center, which maintains a large number of virtual machines (VMs). We have built our framework using Apache Spark to get higher throughput and lower data processing time on streaming data. We have developed a window-based statistical anomaly detection technique to detect anomalies that appear sporadically. We then relaxed this constraint with higher accuracy by implementing a cluster-based technique to detect sporadic and continuous anomalies. We conclude that our cluster-based technique out- performs other statistical techniques with higher accuracy and lower processing time.

Apache Spark↗
DOE OSTI2018

Detecting outliers in streaming time series data from ARM distributed sensors

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

Lu, Yuping↗
DOE OSTI2017

Final Report: Sampling-Based Algorithms for Estimating Structure in Big Data.

The purpose of this project was to develop sampling-based algorithms to discover hidden struc- ture in massive data sets. Inferring structure in large data sets is an increasingly common task in many critical national security applications. These data sets come from myriad sources, such as network traffic, sensor data, and data generated by large-scale simulations. They are often so large that traditional data mining techniques are time consuming or even infeasible. To address this problem, we focus on a class of algorithms that do not compute an exact answer, but instead use sampling to compute an approximate answer using fewer resources. The particular class of algorithms that we focus on are streaming algorithms , so called because they are designed to handle high-throughput streams of data. Streaming algorithms have only a small amount of working storage - much less than the size of the full data stream - so they must necessarily use sampling to approximate the correct answer. We present two results: * A streaming algorithm called HyperHeadTail , that estimates the degree distribution of a graph (i.e., the distribution of the number of connections for each node in a network). The degree distribution is a fundamental graph property, but prior work on estimating the degree distribution in a streaming setting was impractical for many real-world application. We improve upon prior work by developing an algorithm that can handle streams with repeated edges, and graph structures that evolve over time. * An algorithm for the task of maintaining a weighted subsample of items in a stream, when the items must be sampled according to their weight, and the weights are dynamically changing. To our knowledge, this is the first such algorithm designed for dynamically evolving weights. We expect it may be useful as a building block for other streaming algorithms on dynamic data sets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Kernelized approaches to streaming compression of scientific data

In this paper three algorithms are developed for the streaming compression of scientific data. The algorithms presented are reliant on the theory of vector-valued reproducing kernel Hilbert spaces and operator valued kernel. Further, the scientific data is modeled as a snapshot of time dependent vector field F(x, t) over a manifold M and the recovery of the data is framed as a learning problem. These processes are then appropriately modified and ana lyzed for the streaming scenario in which data is generated without the ability to revisit past entries.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1984

Free-jet feasibility study of a thermal acoustic shield concept for AST/VCE application-dual stream nozzles. Comprehensive data report. Volume 2: Laser velocimeter and suppressor. Base pressure data

Acoustic and diagnostic data that were obtained to determine the influence of selected geometric and aerodynamic flow variables of coannular nozzles with thermal acoustic shields are summarized in this comprehensive data report. A total of 136 static and simulated flight acoustic test points were conducted with 9 scale-model nozzles. Aerodynamic laser velocimeter measurements were made for four selected plumes. In addition, static pressure data in the chute base region of the suppressor configurations were obtained to assess the influence of the shield stream on the suppressor base drag.

Janardan, B. A.↗
DOE OSTI2023

Data and scripts associated with "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams."

This data package is associated with the publication "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams." in review at Limnology and Oceanography (Roley et al. 2023). This study focuses on understanding ecosystem metabolism for the Hanford Reach of the Columbia River in Washington state, a free-flowing stretch with a substantial discharge. Large rivers have been overlooked compared to small and medium rivers due to the challenges associated with measurements. Our study presents novel ways to address these challenges and highlights that metabolism patterns in large rivers differ from those observed in small-medium rivers and requires the application of knowledge and tools beyond those implemented for smaller rivers.This data package includes the data and R scripts for the analyses described in Roley et al. 2023. It includes dissolved oxygen and temperature data from a dissolved oxygen HOBO sensor, light data collected from the National Solar Radiation Database (https://nsrdb.nrel.gov/) and hydrologic variables estimated from the MASS-1 model (Niehus et al.; 2014). It also includes metabolism estimates (gross primary production, ecosystem respiration, and net ecosystem production) estimated via streamMetabolizer (Appling et al.; 2018). All analyses in the paper can be replicated with these data and scripts.The data package is comprised of one main data folder. The folder includes (1) file-level metadata (flmd); (2) a data dictionary (dd) for each data file; (3) data files; and (4) R scripts for metabolism estimates and data analysis. All files are .R, .csv, or .pdf.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Signatures of a Tidally Induced Spiral Arm at the Anticenter of the Milky Way and a Kinematically Extended Anticenter Stream Using DESI Data Release 2

Using the Dark Energy Spectroscopic Instrument (DESI) Milky Way Survey, we examine the six-dimensional space of the anticenter region of the Milky Way stellar disk (150° < Galactic longitude < 220°) using 61,883 main-sequence turnoff stars. We focus on two well-known stellar overdensities in the anticenter: the Monoceros Ring (MRi) and Anticenter Stream (ACS). We find that the MRi overdensity has kinematic signatures consistent with a tidally induced spiral arm, a type of dynamic spiral arm created by an interaction with a satellite galaxy, most likely the Sagittarius dwarf spheroidal galaxy (Sgr). We use the kinematics of the MRi to calculate the two most recent passage times of Sgr, finding 0.25 ± 0.09 Gyr and 1.10 ± 0.23 Gyr from the present day. We validate that the ACS is kinematically decoupled from the MRi because they are moving in opposite radial and vertical directions. We find that the kinematics associated with the ACS extends beyond our defined overdensity. The features we see in the ACS region are likely part of a broader distribution of stars with the same kinematic signature as detected in other places, like the vertical wave in the outer disk and phase spiral.

Lambert, Mika [University of California, Santa Cru↗
DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗
DOE OSTI2024

Machine Learning for Joint Quality Control

The use of lightweight material combinations has been highly demanded in manufacturing automotive structures. However, making robust dissimilar material joints of such lightweight materials is still challenging. A significant barrier to achieving high-quality and repeatable joint performance is a deficient understanding of the relationship between the welding process, joint attributes, and joint performance. In this context, welding factors refer to material, equipment, environment, and process parameters, while joint features comprise specific microstructural attributes of the weld such as nugget size, heat affected zone (HAZ) topology, intermetallic layer thickness, and sheet thickness reduction. Joint performance is quantified in terms of strength (e.g., tensile shear, coach peel, cross-tension), weld size, and hardness, among other factors. While there have been many attempts to establish this process-structure-property relationship by developing a model derived from the associated physics and first principles, the complexity of the joining processes compounded by the complex interactions with different materials in an automotive assembly line environment, has hindered the usefulness of such attempts. The complexity is further exacerbated using different stacking materials, especially comprising dissimilar material combinations. In practice, the common approach has been the laborious process of creating welds, characterizing them, and then physically testing them through experimentation. With the emergence of artificial intelligence (AI) methods, an alternative pathway to eliciting the desired process-structure-property relationship at an accelerated pace is to use a data-driven approach by employing machine-learning (ML) techniques. This approach is benefitted by the availability of large streams of data, generated through years of research and testing by original equipment manufacturers, in the form of material, process, environmental, equipment, microstructural, and bulk-scale performance information from multimodal, multiscale sensors making measurements from laboratory-scale to production-scale processes. During Phase I efforts, which ended in fiscal year (FY) 2021, the Oak Ridge National Laboratory and Pacific Northwest National Laboratory (ORNL/PNNL) team demonstrated the effectiveness of different ML/AI frameworks in modeling complex relationships between resistance spot welding (RSW) process parameters, weld attributes, and joint properties using a subset of data from General Motors (GM). In FY 2022, the project team further refined and expanded their respective ML models to analyze additional welds with new weld stack-ups and materials to enhance the ML model predictive capability. ORNL extended its unified deep neural networks (DNN) ML training and prediction framework with new data streams of process parameters, and PNNL extended its model describing RSW process parameters’ associations with weld attributes. In FY 2023, the project team completed the development of the AI/ML architecture for analyzing aluminum/steel joints manufactured by GM via RSW and transitioned into the inline welding quality monitoring task for steel/steel RSW joints provided by GM.

36 MATERIALS SCIENCE↗