Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2008

Precision Measurement of Black Hole Binary Dynamics: Analyzing the LISA Data Stream

One of the richest potential sources of insight into fundamental physics that LISA will be capable of observing is the inspiral of supermassive black hole binaries (BHBs). However, the data analysis challenge presented by the LISA data stream is quite unlike the situation for present day gravitational wave detectors. In order to make the precision measurements necessary to achieve LISA's science goals, the BHB signal must be distinguished from a data stream that not only contains instrumental noise, but potentially thousands of other signals as well, so that the "background" we wish to separate out to focus on the BHB signal is likely to be highly nonstationary and nongaussian, as well as being of scientific interest in its own right. In addition, whereas the theoretical templates that we calculate in order to ultimately estimate the parameters can afford to be somewhat inaccurate and still be effective for present day and near future detectors, this is not the case for LISA, and extremely high fidelity of the theoretical templates for high signal-to-noise signals will be required to prevent theoretical errors from dominating the parameter estimates. NVe, will describe efforts in the community of LISA data analysts to address the challenges regarding the specific issue of BHB signals. These efforts include using a Markov Chain Monte Carlo approach with the freedom to model the BHB and the other signals present in the data stream simultaneously, rather than trying to remove other signals and risk biasing the remaining data. The Mock LISA Data Challenge is a community of LISA scientists who generate rounds of simulated LISA noise with increasingly difficult signal content, and invite the LISA data analysis community to exercise their methods, or develop new methods, in an attempt to extract the parameters for the signals embedded in the mock data. In addition to practical approaches such ,is this to assess the level of parameter accuracy, one can apply the Fisher matrix formalism to assess both the statistical errors from noise and the theoretical errors

McWilliams, Sean T.↗
DOE OSTI2016

Dimensionality Reduction Algorithms for Big Nuclear Data Streams with Application to SCALE Cross Sections

A central problem in performing accurate nuclear reactor simulations is the ability to capture the changes in the few group microscopic cross sections as functions of system state, e.g. moderator density, fuel temperature, or neutron spectrum, and time and/or burnup of the fuel. For most reactor simulation scenarios, this results in huge data stream, which must be interpolated as needed by downstream calculations (e.g., core-wide simulation). This is typically accomplished by using sophisticated interpolating functions to capture the few-group variations over the wide range of conditions expected during operation. To address handling this big data, this paper explores the use of dimensionality reduction (DR) techniques to identify the true dimensionality of the data prior to its use in the calculations. This could significantly reduce the computational cost required for data processing/manipulation, thus reducing a significant portion of the total calculations time. By way of an example, in the current version of the ORIGEN-ARP module, the assembly-averaged microscopic cross sections at different conditions, e.g., fuel enrichments, irradiation history and water density, which are calculated using the TRITON sequence and interpolated on-demand by the ARP module to model assembly depletion for a wide range of assembly types (e.g. Westinghouse 17x17 and GE 9x9) over a wide range of conditions. This approach typically results in ∼1 GB per assembly type, which although not insurmountable, can be impractical for distribution. and it is the premise of this paper that most of this data is heavily correlated and hence can take great benefit of condensation via DR. To demonstrate DR, the proposed algorithm is applied to the nuclear data streams generated by SCALE 6.1. Using the depletion sequence (TRITON), microscopic cross sections are calculated for a mixed oxide (MOX) fuel assembly design at a representative range of fuel enrichments and fuel burnup. In conventional core simulators, multi dimensional interpolation of these cross sections is used to provides neutronics feedback due to local conditions without returning to expensive TRITON calculations for every change. This paper shows that dimensionality reduction can help compress the size of the cross section data without sacrificing the accuracy of their reconstruction at all conditions of interest. This work explores the application of dimensionality reduction algorithms to nuclear data streams typically employed in support of conventional full-core reactor calculations which rely on interpolation of pre-calculated microscopic cross sections. In particular, analysis of the few group MOX cross sections revealed that a large degree of dependencies exist between the few-group cross sections, which can be revealed via range-finding algorithms. Specifically, correlations between different reaction types for the same isotope have been identified, and correlations between different enrichments and correlations across different isotopes have been identified. This paper demonstrates that the reduction errors can be quantified, and they are uncorrelated with the range of variations for the various cross sections. In future work, we will focus on identifying the smallest rank that takes into account all types of correlations across the range of conditions used to model cross section variations. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

A dictionary learning algorithm for compression and reconstruction of streaming data in preset order

There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

Abstract We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Uncertainty guided online ensemble for non-stationary data streams in fusion science

Machine Learning (ML) is poised to play a pivotal role in the development and operation of next-generation fusion devices. Fusion data shows non-stationary behavior with distribution drifts, resulted by both experimental evolution and machine wear-and-tear. ML models assume stationary distribution and fail to maintain performance when encountered with such non-stationary data streams. Online learning techniques have been leveraged in other domains, however it has been largely unexplored for fusion applications. In this paper, we investigate online learning for continuous adaptation to drifting data streams in the prediction of Toroidal Field (TF) coils deflection at the DIII-D fusion facility. We further address the short-term performance degradation inherent to standard online learning, which arises because ground truth is unavailable at prediction time. To mitigate this issue, we propose an uncertainty-guided online ensemble framework. The method leverages the Deep Gaussian Process Approximation (DGPA) for calibrated uncertainty estimation and uses these uncertainty measures to guide a meta-algorithm that aggregates predictions from learners trained over different historical horizons. Our results show that online learning reduces prediction error by 80% compared to a static model. The online ensemble and the proposed uncertainty-guided ensemble further reduce error by approximately 6%, and 10% respectively, relative to standard single-model online learning, while also providing calibrated uncertainty estimates to support operational decision-making.

AI↗
DOE OSTI2017

The Impact of Streaming Data on Sensemaking with Mixed-Initative Visual Analytics

Visual data analysis helps people gain insights into data via interactive visualizations. People generate and test hypotheses and questions about data in context of the domain. This process can generally be referred to as sensemaking. Much of the work on studying sensemaking (and creating visual analytic techniques in support of it) has been focused on static datasets. However, how do the cognitive processes of sensemaking change when data are changing? Further, what implication for design does this create for mixed-initiative visual analytics systems? This paper presents the results of a user study analyzing the impact of streaming data on sensemaking. To perform this study, we developed a mixed-initiative visual analytic prototype, the Streaming Canvas, that affords the analysis of streaming text data. We compare the sensemaking process of people using this tool for a static and streaming dataset. We present the results of this study and discuss the implications on future visual analytic systems that combine machine learning and interactive visualization to help people make sense of streaming data.

visualization↗
DOE OSTI2019

A Software Defined Network Design for Analyzing Streaming Data in Transit

Year after year, network traffic keeps reaching newhighs as unprecedented volumes of data flow between an ever-increasing number of sources and destinations. As part of the Analysis on the Wire project, we aim to develop a network-centric approach for streaming data processing to facilitate scientific data analysis and reduce the overhead in sending big data to a data center. This work discusses our design for programmable network devices to augment network capabilities for streaming data processing, including efforts in progress at Brookhaven National Laboratory and the challenges faced to date.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Scenario driven data modelling: a method for integrating diverse sources of data and data streams

A system and method of integrating diverse sources of data and data streams is presented. The method can include selecting a scenario based on a topic, creating a multi-relational directed graph based on the scenario, identifying and converting resources in accordance with the scenario and updating the multi-directed graph based on the resources, identifying data feeds in accordance with the scenario and updating the multi-directed graph based on the data feeds, identifying analytical routines in accordance with the scenario and updating the multi-directed graph using the analytical routines and identifying data outputs in accordance with the scenario and defining queries to produce the data outputs from the multi-directed graph.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2011

KAZRARSCL-All-inclusive data stream

The KAZR-ARSCL VAP provides cloud boundaries and best-estimate time-height fields of radar moments. The VAP merges corrected KAZR moments from all active radar modes with cloud base and cloud mask observations from the micropulse lidar (MPL), cloud base from the ceilometer, as well information from soundings, rain gauge, and microwave radiometer instruments to produce two data streams, one with best-estimate cloud base and cloud layer boundaries, and another which also includes best-estimate time-height fields of radar moments. This DOI is for the data stream that contains all variables produced by the VAP.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

KAZRARSCL-c0-All-inclusive data stream

The KAZR-ARSCL VAP provides cloud boundaries and best-estimate time-height fields of radar moments. The VAP merges corrected, but uncalibrated, KAZR moments from all active radar modes with cloud base and cloud mask observations from the micropulse lidar (MPL), cloud base from the ceilometer, as well information from soundings, rain gauge, and microwave radiometer instruments to produce two data streams, one with best-estimate cloud base and cloud layer boundaries, and another which also includes best-estimate time-height fields of radar moments. This DOI is for the data stream that contains all variables produced by the VAP. Please note that reflectivity in this level c0 product is uncalibrated.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Constraining DALECv2 using multiple data streams and ecological constraints: analysis and application

We use a variational method to assimilate multiple data streams into the terrestrial ecosystem carbon cycle model DALECv2 (Data Assimilation Linked Ecosystem Carbon). Ecological and dynamical constraints have recently been introduced to constrain unresolved components of this otherwise ill-posed problem. We recast these constraints as a multivariate Gaussian distribution to incorporate them into the variational framework and we demonstrate their advantage through a linear analysis. By using an adjoint method we study a linear approximation of the inverse problem: firstly we perform a sensitivity analysis of the different outputs under consideration, and secondly we use the concept of resolution matrices to diagnose the nature of the ill-posedness and evaluate regularisation strategies. We then study the non-linear problem with an application to real data. Finally, we propose a modification to the model: introducing a spin-up period provides us with a built-in formulation of some ecological constraints which facilitates the variational approach.

58 GEOSCIENCES↗
NASA NTRS2003

Lessons Learned From Developing A Streaming Data Framework for Scientific Analysis

We describe the development and usage of a streaming data analysis software framework. The framework is used for three different applications: Earth science hyper-spectral imaging analysis, Electromyograph pattern detection, and Electroencephalogram state determination. In each application the framework was used to answer a series of science questions which evolved with each subsequent answer. This evolution is summarized in the form of lessons learned.

Wheeler. Kevin R.↗
DOE OSTI2019

MetaStrider: Architectures for Scalable Memory-centric Reduction of Sparse Data Streams

Reduction is an operation performed on the values of two or more key-value pairs that share the same key. Reduction of sparse data streams finds application in a wide variety of domains such as data and graph analytics, cybersecurity, machine learning, and HPC applications. However, these applications exhibit low locality of reference, rendering traditional architectures and data representations inefficient. This article presents MetaStrider, a significant algorithmic and architectural enhancement to the state-of-the-art, SuperStrider. Furthermore, these enhancements enable a variety of parallel, memory-centric architectures that we propose, resulting in demonstrated performance that scales near-linearly with available memory-level parallelism.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2011

Optical rogue waves in telecommunication data streams

Large broadening of short optical pulses due to fiber dispersion leads to a strong overlap in information data streams resulting in statistical deviations of the local power from its average. We present a theoretical analysis of rare events of high-intensity fluctuations--optical freak waves--that occur in fiber communication links using bit-overlapping transmission. Although the nature of the large fluctuations examined here is completely linear, as compared to commonly studied freak waves generated by nonlinear effects, the considered deviations inherit from rogue waves the key features of practical interest--random appearance of localized high-intensity pulses. We use the term ''rogue wave'' in an unusual context mostly to attract attention to both the possibility of purely linear statistical generation of huge amplitude waves and to the fact that in optics the occurrence of such pulses might be observable even with the standard Gaussian or even rarer-than-Gaussian statistics, without imposing the condition of an increased probability of extreme value events.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2025

Online learning of quadratic manifolds from streaming data for nonlinear dimensionality reduction and nonlinear model reduction

Here, this work introduces an online greedy method for constructing quadratic manifolds from streaming data, designed to enable in situ analysis of numerical simulation data on the Petabyte scale. Unlike traditional batch methods, which require all data to be available upfront and take multiple passes over the data, the proposed online greedy method incrementally updates quadratic manifolds in one pass as data points are received, eliminating the need for expensive disk input/output operations as well as storing and loading data points once they have been processed. A range of numerical examples demonstrate that the online greedy method learns accurate quadratic manifold embeddings while being capable of processing data that far exceed common disk input/output capabilities and volumes as well as main-memory sizes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Production Application Performance Data Streaming for System Monitoring

In this article, we present an approach to streaming collection of application performance data. Practical application performance tuning and troubleshooting in production high-performance computing (HPC) environments requires an understanding of how applications interact with the platform, including (but not limited to) parallel programming libraries such as Message Passing Interface (MPI). Several profiling and tracing tools exist that collect heavy runtime data traces either in memory (released only at application exit) or on a file system (imposing an I/O load that may interfere with the performance being measured). Although these approaches are beneficial in development stages and post-run analysis, a systemwide and low-overhead method is required to monitor deployed applications continuously. This method must be able to collect information at both the application and system levels to yield a complete performance picture. In our approach, an application profiler collects application event counters. A sampler uses an efficient inter-process communication method to periodically extract the application counters and stream them into an infrastructure for performance data collection. We implement a tool-set based on our approach and integrate it with the Lightweight Distributed Metric Service (LDMS) system, a monitoring system used on large-scale computational platforms. LDMS provides the infrastructure to create and gather streams of performance data in a low overhead manner. We demonstrate our approach using applications implemented with MPI, as it is one of the most common standards for the development of large-scale scientific applications. We utilize our tool-set to study the impact of our approach on an open source HPC application, Nalu. Our tool-set enables us to efficiently identify patterns in the behavior of the application without source-level knowledge. We leverage LDMS to collect system-level performance data and explore the correlation between the system and application events. Finally, we demonstrate how our tool-set can help detect anomalies with a low latency. We run tests on two different architectures: a system enabled with Intel Xeon Phi and another system equipped with Intel Xeon processor. Our overhead study shows our method imposes at most 0.5% CPU usage overhead on the application in realistic deployment scenarios.

59 BASIC BIOLOGICAL SCIENCES↗