Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

CLAS12 remote data-stream processing using ERSAP framework

Implementing a physics data processing application is relatively straightforward with the use of current containerization technologies and container image runtime services, which are prevalent in most high-performance computing (HPC) environments. However, the process is complicated by the challenges associated with data provisioning and migration, impacting the ease of workflow migration and deployment. Transitioning from traditional file-based batch processing to data-stream processing workflows is suggested as a method to streamline these workflows. This transition not only simplifies file provisioning and migration but also significantly reduces the necessity for extensive disk space. Data-stream processing is particularly effective for real-time processing during data acquisition, thereby enhancing data quality assurance. This paper introduces the integration of the JLAB CLAS12 event reconstruction application within the ERSAP data-stream processing framework that facilitates the execution of streaming event reconstruction at a remote data center and enables the return streaming of reconstructed events to JLAB while circumventing the need for temporary data storage throughout the process.

Gyurjyan, Vardan↗

Enhancing disease surveillance with novel data streams: challenges and opportunities

Novel data streams (NDS), such as web search data or social media updates, hold promise for enhancing the capabilities of public health surveillance. In this paper, we outline a conceptual framework for integrating NDS into current public health surveillance. Our approach focuses on two key questions: What are the opportunities for using NDS and what are the minimal tests of validity and utility that must be applied when using NDS? Identifying these opportunities will necessitate the involvement of public health authorities and an appreciation of the diversity of objectives and scales across agencies at different levels (local, state, national, international). We present the case that clearly articulating surveillance objectives and systematically evaluating NDS and comparing the performance of NDS to existing surveillance data and alternative NDS data is critical and has not sufficiently been addressed in many applications of NDS currently in the literature.

60 APPLIED LIFE SCIENCES↗

Biosurveillance Data Stream Framework: A Novel Approach to Characterization and Evaluation

Multiple data sources are used in a variety of biosurveillance systems. With the advent of new technologies, globalization, high performance computing, and "big data" opportunities, there are seemingly unlimited potential data streams that could be useful in biosurveillance. Data streams have not been universally defined in either the literature or by specific biosurveillance systems. Furthermore, the definitions and framework that we have developed enable a characterization methodology that facilitates understanding of data streams and can be universally applicable for use in evaluating and understanding a wide range of biosurveillance activities- filling a gap recognized in both the public health and biosurveillance communities.

59 BASIC BIOLOGICAL SCIENCES↗

A Systematic Evaluation of Traditional and Non-Traditional Data Streams for Integrated Global Biosurveillance (Final Report)

Living in a closely connected and highly mobile world presents many new mechanisms for rapid disease spread and in recent years, global disease surveillance has become a high priority. In addition, much like the contribution of non-traditional medicine to curing diseases, non-traditional data streams are being considered of value in disease surveillance. Los Alamos National Laboratory (LANL) was tasked by the Defense Threat Reduction Agency (DTRA)-Joint Science and Technology Office (JSTO) to determine the relevance of data streams for an integrated global biosurveillance system through the use of defined metrics and methodologies. The long-term, broad objective of this project was to provide information and analysis that can be leveraged from existing and developing national and international disease surveillance systems and methodologies, to create a global disease monitoring system, ultimately providing decision makers with timely information to prepare for and mitigate the spread of disease. Specifically, this project evaluated data streams that are currently being used in surveillance systems and data streams that had the potential for being used in surveillance to enable early disease detection. LANL’s focus was on infectious disease surveillance and the term “biosurveillance” will be used to refer to this scope hereafter. Data stream evaluation was conducted using two different methods: Multi criteria decision analysis (MCDA) using a tool called Logical Decisions ® that assigns utility scores to data streams based on weighted metrics and assigned values specific to data stream categories; and a Surveillance Window concept developed at LANL that assigns a window or windows of time specific to a disease within which information coming from various data streams can be determined to have utility. A cross method analysis was performed between the surveillance window based evaluation and the MCDA-based evaluation to identify data stream categories that showed high utility for both methods. In addition, algorithms that can integrate useful data streams to facilitate early disease detection were also examined. This project provided an understanding of data streams relevant to early warning, early detection and monitoring of disease outbreaks. The report describes the results of LANL’s evaluation of 16 data stream categories using the two methods, and recommendations for next steps. The project evolved significantly over its course, and significant improvements in our approaches were made compared to the initial proposed processes. A key outcome of this effort was that LANL was able to bring together several diverse entities involved in disease surveillance and lay the foundation for new collaborations that straddled military and civilian health surveillance. The robust evaluation framework developed by LANL has generated significant interest and there is interest in adopting this framework for various applications both within disease surveillance as well as other public health initiatives. Finally, LANL developed the Biosurveillance Resource Directory (BRD), a relational database that underwent pilot testing by members of the human, plant and animal disease surveillance community. The BRD is intended to be a global resource to facilitate rapid information access.

60 APPLIED LIFE SCIENCES↗

Classification and Fusion of Two Disparate Data Streams and Nuclear Dissolutions Application

We consider two streams of data or measurements with disparate qualities and time resolutions that need to be classified. The first stream consists of higher quality data at a coarser time resolution, and the other consists of lower quality data at a finer time resolution. We present a fuser-switch method that fuses the set of classifiers of each stream separately and switches between them. We show that this method provides classification decisions at a finer time resolution with superior detection and false alarm probabilities compared to individual classifiers, under the statistical independence and time resolution ratio conditions. When classifiers are trained using machine learning methods, we show that this superior performance is guaranteed with a confidence probability specified by the classifiers' generalization equations. We use these results to provide analytical foundations for previous practical results that achieved significant performance improvements in classifying Pu/Np target dissolution events at a radiochemical processing facility.

Rao, Nageswara↗

Streaming data analytics via message passing with application to graph algorithms

The need to process streaming data, which arrives continuously at high-volume in real-time, arises in a variety of contexts including data produced by experiments, collections of environmental or network sensors, and running simulations. Streaming data can also be formulated as queries or transactions which operate on a large dynamic data store, e.g. a distributed database. We describe a lightweight, portable framework named PHISH which enables a set of independent processes to compute on a stream of data in a distributed-memory parallel manner. Datums are routed between processes in patterns defined by the application. PHISH can run on top of either message-passing via MPI or sockets via ZMQ. The former means streaming computations can be run on any parallel machine which supports MPI; the latter allows them to run on a heterogeneous, geographically dispersed network of machines. We illustrate how PHISH can support streaming MapReduce operations, and describe streaming versions of three algorithms for large, sparse graph analytics: triangle enumeration, subgraph isomorphism matching, and connected component finding. Lastly, we also provide benchmark timings for MPI versus socket performance of several kernel operations useful in streaming algorithms.

97 MATHEMATICS AND COMPUTING↗

Resource Management for Processing Wide Area Data Streams on Supercomputers

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

data streaming↗

Advancing a Framework to Enable Characterization and Evaluation of Data Streams Useful for Biosurveillance

In recent years, biosurveillance has become the buzzword under which a diverse set of ideas and activities regarding detecting and mitigating biological threats are incorporated depending on context and perspective. Increasingly, biosurveillance practice has become global and interdisciplinary, requiring information and resources across public health, One Health, and biothreat domains. Even within the scope of infectious disease surveillance, multiple systems, data sources, and tools are used with varying and often unknown effectiveness. Evaluating the impact and utility of state-of-the-art biosurveillance is, in part, confounded by the complexity of the systems and the information derived from them. We present a novel approach conceptualizing biosurveillance from the perspective of the fundamental data streams that have been or could be used for biosurveillance and to systematically structure a framework that can be universally applicable for use in evaluating and understanding a wide range of biosurveillance activities. Moreover, the Biosurveillance Data Stream Framework and associated definitions are proposed as a starting point to facilitate the development of a standardized lexicon for biosurveillance and characterization of currently used and newly emerging data streams. Criteria for building the data stream framework were developed from an examination of the literature, analysis of information on operational infectious disease biosurveillance systems, and consultation with experts in the area of biosurveillance. To demonstrate utility, the framework and definitions were used as the basis for a schema of a relational database for biosurveillance resources and in the development and use of a decision support tool for data stream evaluation.

47 OTHER INSTRUMENTATION↗

A Block-Based MC-SURE Algorithm for Denoising Sensor Data Streams

We propose a strategy to automatically denoise sensor data streams corrupted with noise that can be approximated as additive white Gaussian noise. The proposed block-based method is adapted from the Monte-Carlo-SURE (MC-SURE) algorithm which enables the blind optimization of the denoising parameter of a wide class of filters. Our framework is formulated by identifying and addressing the challenges that arise when the MC-SURE algorithm is applied in an on-line data processing setting, where latency (and the length of data blocks) must be constrained. The strategy has been tested using real datasets. Our results indicate that the proposed method can be effectively used to handle the denoising of real sensor streaming data that reasonably fit the Gaussian model assumption.

97 MATHEMATICS AND COMPUTING↗

From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures

In this paper, we investigate three cross-facility data streaming architectures, Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS). We examine their architectural variations in data flow paths and deployment feasibility, and detail their implementation using the Data Streaming to HPC (DS2HPC) architectural framework and the SciStream memory-to-memory streaming toolkit on the production-grade Advanced Computing Ecosystem (ACE) infrastructure at Oak Ridge Leadership Computing Facility (OLCF). We present a workflow-specific evaluation of these architectures using three synthetic workloads derived from the streaming characteristics of scientific workflows. Through simulated experiments, we measure streaming throughput, round-trip time, and overhead under work sharing, work sharing with feedback, and broadcast and gather messaging patterns commonly found in AI-HPC communication motifs. Our study shows that DTS offers a minimal-hop path, resulting in higher throughput and lower latency, whereas MSS provides greater deployment feasibility and scalability across multiple users but incurs significant overhead. PRS lies in between, offering a scalable architecture whose performance matches DTS in most cases.

George, Anjus [ORNL] (ORCID:0000000179737061)↗

ERSAP: Toward Better NP Data-Stream Analytics With Flow-Based Programming

This paper presents an reactive, actor-model and FBP paradigm based framework that we develop to design data-stream processing applications for HEP and NP. This framework encourages a functional decomposition of the overall data processing application into small mono-functional artifacts. Artifacts that are easy to understand, develop, deploy and debug. The fact that these artifacts (actors) are programmatically independent they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages of this approach is fault tolerance where independent actors can come and go on the data-stream without forcing the entire application to crash. Furthermore, it also makes it is easy to locate the faulty actor in the data pipeline. Due the fact that the actors are loosely coupled, and that the data (inevitably) carries the context, they can run on heterogeneous environments, utilizing different accelerators. This paper describes the main design concepts of the framework and presents a ?proof of concept? application design and deployment results obtain processing on-beam calorimeter streaming data.

Gyurjyan, V.↗

Metabolomic data streaming for biology-dependent data acquisition

During the last 10 years, metabolomics has emerged as a powerful technology to interrogate cellular biochemistry at the global level. Although much of the success has been driven by advances in mass spectrometry, developments in bioinformatic resources for data processing have been equally important. Additionally, the widely used metabolomic software XCMS, in particular, has undergone substantial improvements since its first introduction in 2005 (ref. 1). In addition to improved algorithms for peak picking, retention time alignment and data visualization, XCMS has transitioned from a command-line interface requiring expertise with the R programming language, to a web-based platform with a graphical user interface2. This web-based platform, which we call XCMS Online, enables thousands of users to upload their metabolomic data and perform cloud-based processing.

59 BASIC BIOLOGICAL SCIENCES↗

A new stepwise carbon cycle data assimilation system using multiple data streams to constrain the simulated land surface carbon cycle

Here, large uncertainties in land surface models (LSMs) simulations still arise from inaccurate forcing, poor description of land surface heterogeneity (soil and vegetation properties), incorrect model parameter values and incomplete representation of biogeochemical processes. The recent increase in the number and type of carbon cycle-related observations, including both in situ and remote sensing measurements, has opened a new road to optimize model parameters via robust statistical model–data integration techniques, in order to reduce the uncertainties of simulated carbon fluxes and stocks. In this study we present a carbon cycle data assimilation system that assimilates three major data streams, namely the Moderate Resolution Imaging Spectroradiometer (MODIS)-Normalized Difference Vegetation Index (NDVI) observations of vegetation activity, net ecosystem exchange (NEE) and latent heat (LE) flux measurements at more than 70 sites (FLUXNET), as well as atmospheric CO 2 concentrations at 53 surface stations, in order to optimize the main parameters (around 180 parameters in total) of the Organizing Carbon and Hydrology in Dynamics Ecosystems (ORCHIDEE) LSM (version 1.9.5 used for the Coupled Model Intercomparison Project Phase 5 (CMIP5) simulations). The system relies on a stepwise approach that assimilates each data stream in turn, propagating the information gained on the parameters from one step to the next. Overall, the ORCHIDEE model is able to achieve a consistent fit to all three data streams, which suggests that current LSMs have reached the level of development to assimilate these observations. The assimilation of MODIS-NDVI (step 1) reduced the growing season length in ORCHIDEE for temperate and boreal ecosystems, thus decreasing the global mean annual gross primary production (GPP). Using FLUXNET data (step 2) led to large improvements in the seasonal cycle of the NEE and LE fluxes for all ecosystems (i.e., increased amplitude for temperate ecosystems). The assimilation of atmospheric CO 2 , using the general circulation model (GCM) of the Laboratoire de Météorologie Dynamique (LMDz; step 3), provides an overall constraint (i.e., constraint on large-scale net CO 2 fluxes), resulting in an improvement of the fit to the observed atmospheric CO 2 growth rate. Thus, the optimized model predicts a land C (carbon) sink of around 2.2 PgC yr -1 (for the 2000–2009 period), which is more compatible with current estimates from the Global Carbon Project (GCP) than the prior value. The consistency of the stepwise approach is evaluated with back-compatibility checks. The final optimized model (after step 3) does not significantly degrade the fit to MODIS-NDVI and FLUXNET data that were assimilated in the first two steps, suggesting that a stepwise approach can be used instead of the more “challenging” implementation of a simultaneous optimization in which all data streams are assimilated together. Most parameters, including the scalar of the initial soil carbon pool size, changed during the optimization with a large error reduction. This work opens new perspectives for better predictions of the land carbon budgets.

58 GEOSCIENCES↗

Contextualizing Data Streams for Infectious Disease Surveillance

To aid in developing a global biosurveillance program, it is critical to develop a framework to capture and understand the myriad of data streams and evaluate them in context of surveillance goals. Toward this goal, Los Alamos National Laboratory has developed a new method of evaluating the effectiveness of data stream types through the use of a novel concept called the surveillance window, a technique that integrates operational systems analysis, surveillance system analysis and epidemiological analysis. This study provides a simple, yet elegant methodology for which to ground truth known and emerging data streams for utility in integrated biosurveillance efforts.

59 BASIC BIOLOGICAL SCIENCES↗

Anomaly Detection, Localization and Classification using Drifting Synchrophasor Data Streams

With ongoing automation and digitization of the electric power system, several Phasor Measurement Units(PMUs) have been deployed for monitoring and control. PMU data can have multiple anomalies, and many of the researchers in the past have concentrated on training machine/deep learning algorithms offline for anomaly detection over PMU data (i.e., not in real time). These machine/deep learning algorithms, when trained offline on a sample rather than a population of the dataset, fail to consider the dynamic behavior of the power grid in real-time, resulting in low accuracy. Considering the dynamic behavior of the power grid (e.g., change in load, generation, distributed energy resources (DERs) switching, network, controls), the definition of data anomalies varies in time and requires online training. A fundamental challenge is to enable online (i.e., real-time) training of machine/deep learning algorithms for anomaly detection over streaming PMU data. While machine/deep learning is often desirable to manage data streams, training a deep learning algorithm over streaming PMU data is nontrivial due to changes in data statistics caused by dynamic streaming data. This paper proposes PMUNET: a novel device-level deep learning-based data-driven approach for anomaly detection, localization, and classification over streaming PMU data, using online learning and multivariate data-drift detection algorithm .Two variants of PMUNET, Dynamic data Change Driven Learning (DCDL) and Continuity Driven Learning (CDL), are proposed and compared. DCDL aims to train the deep learning algorithm whenever the definition of anomaly changes due to the power grid dynamics. On the other hand, CDL continuously trains the deep learning algorithm over the PMU data-stream. The experimental results verify that DCDL outperforms CDL and other efficient anomaly detection methods over multiple events such as faults and load/ generator/capacitor/DERs variations/switching for IEEE 14 and 39 Bus test system as well as real PMU industrial data. The result verifies that DCDL variant of PMUNET improves over existing approach with a gain of 2% - 10% in terms of accuracy, false-positive rate, and false-negative rate.

adversarial deep learning↗

Dimensionality Reduction Algorithms for Big Nuclear Data Streams with Application to SCALE Cross Sections

A central problem in performing accurate nuclear reactor simulations is the ability to capture the changes in the few group microscopic cross sections as functions of system state, e.g. moderator density, fuel temperature, or neutron spectrum, and time and/or burnup of the fuel. For most reactor simulation scenarios, this results in huge data stream, which must be interpolated as needed by downstream calculations (e.g., core-wide simulation). This is typically accomplished by using sophisticated interpolating functions to capture the few-group variations over the wide range of conditions expected during operation. To address handling this big data, this paper explores the use of dimensionality reduction (DR) techniques to identify the true dimensionality of the data prior to its use in the calculations. This could significantly reduce the computational cost required for data processing/manipulation, thus reducing a significant portion of the total calculations time. By way of an example, in the current version of the ORIGEN-ARP module, the assembly-averaged microscopic cross sections at different conditions, e.g., fuel enrichments, irradiation history and water density, which are calculated using the TRITON sequence and interpolated on-demand by the ARP module to model assembly depletion for a wide range of assembly types (e.g. Westinghouse 17x17 and GE 9x9) over a wide range of conditions. This approach typically results in ∼1 GB per assembly type, which although not insurmountable, can be impractical for distribution. and it is the premise of this paper that most of this data is heavily correlated and hence can take great benefit of condensation via DR. To demonstrate DR, the proposed algorithm is applied to the nuclear data streams generated by SCALE 6.1. Using the depletion sequence (TRITON), microscopic cross sections are calculated for a mixed oxide (MOX) fuel assembly design at a representative range of fuel enrichments and fuel burnup. In conventional core simulators, multi dimensional interpolation of these cross sections is used to provides neutronics feedback due to local conditions without returning to expensive TRITON calculations for every change. This paper shows that dimensionality reduction can help compress the size of the cross section data without sacrificing the accuracy of their reconstruction at all conditions of interest. This work explores the application of dimensionality reduction algorithms to nuclear data streams typically employed in support of conventional full-core reactor calculations which rely on interpolation of pre-calculated microscopic cross sections. In particular, analysis of the few group MOX cross sections revealed that a large degree of dependencies exist between the few-group cross sections, which can be revealed via range-finding algorithms. Specifically, correlations between different reaction types for the same isotope have been identified, and correlations between different enrichments and correlations across different isotopes have been identified. This paper demonstrates that the reduction errors can be quantified, and they are uncorrelated with the range of variations for the various cross sections. In future work, we will focus on identifying the smallest rank that takes into account all types of correlations across the range of conditions used to model cross section variations. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

A dictionary learning algorithm for compression and reconstruction of streaming data in preset order

There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.

97 MATHEMATICS AND COMPUTING↗