Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

Social Sensor Analytics: Making Sense of Network Models in Social Media

Social networks can be thought of as noisy sensor networks mapping real world information to the web. Owing to the extensive body of literature in sensor network analysis, this work sought to apply several novel and traditional methods in sensor network analysis for the purposes of efficiently interrogating social media data streams from raw data. We carefully revisit our definition of a social media signal from previous work both in terms of time-varying features within the data and the networked nature of the medium. Further, we detail our analysis of global patterns in Twitter over the months of November 2013 and June 2014, detect and categorize events, and illustrate how these analyses can be used to inform graph-based models of Twitter, namely using a recent network influence model called PhySense: similar to PageRank but tuned to behavioral analysis by leveraging a sociologically inspired probabilistic model. We ultimately identify forms of information dissemination via analysis of time series and dynamic graph spectra and corroborate these findings through manual investigation of the data as a requisite step in modeling the diffusion process with PhySense. We hope to sufficiently characterize global behavior in a medium such as Twitter as a means of learning global model parameters one may use to predict or simulate behavior on a large scale. We have made our time series and dynamic graph analytical code available via a GitHub repository https://github.com/cpatdowling/salsa and our data are available upon request.

event detection↗

Advanced Computational Framework for Environmental Management ZEM, Version 1.x

Typically environmental management problems require analysis of large and complex data sets originating from concurrent data streams with different data collection frequencies and pedigree. These big data sets require on-the-fly integration into a series of models with different complexity for various types of model analyses where the data are applied as soft and hard model constraints. This is needed to provide fast iterative model analyses based on the latest available data to guide decision-making. Furthermore, the data and model are associated with uncertainties. The uncertainties are probabilistic (e.g. measurement errors) and non-probabilistic (unknowns, e.g. alternative conceptual models characterizing site conditions). To address all of these issues, we have developed an integrated framework for real-time data and model analyses for environmental decision-making called ZEM. The framework allows for seamless and on-the-fly integration of data and modeling results for robust and scientifically-defensible decision-making applying advanced decision analyses tools such as Bayesian- Information-Gap Decision Theory (BIG-DT). The framework also includes advanced methods for optimization that are capable of dealing with a large number of unknown model parameters, and surrogate (reduced order) modeling capabilities based on support vector regression techniques. The framework is coded in Julia, a state-of-the-art high-performance programing language (http://julialang.org). The ZEM framework is open-source and can be applied to any environmental management site. The framework will be open-source and released under GPL V3 license.

Vesselinov, VelimirV.↗

IPC-Fusion (Infrastructure Perception and Control (IPC): Multisensor Data Fusion Software) [SWR-25-153]

As part of the National Laboratory of the Rockies' (NLR’s) Infrastructure Perception and Control Laboratory, the IPC-Fusion toolkit provides a probabilistic, scalable, multi-sensor fusion framework that integrates (late-stage fusion) heterogeneous object detection data from traffic sensors to enable robust, real-time tracking of roadway occupants. The algorithmic design of the toolkit is motivated by the need for creating a digital twin of traffic at the edge in a scalable and affordable manner. The software operates by combining object-level measurements (such as position and velocity) from a suite of sensors (such as radar, lidar, camera) using Kalman filtering and probabilistic data association techniques to overcome individual sensor limitations and achieve superior tracking performance in complex traffic zones. The framework addresses key challenges including heterogeneous measurement uncertainties, asynchronous data streams, varying spatiotemporal data resolutions, robust data association, and adaptive object lifecycle management. Validated on real-world traffic intersection data including vehicles and pedestrians, IPC-Fusion demonstrates enhanced tracking reliability across scenarios involving occlusions, sensor failures, and varying traffic densities, supporting the broader IPC initiative's goal of transforming transportation infrastructure through advanced perception capabilities for intelligent transportation systems, traffic safety applications, and autonomous vehicle support.

Sandhu, Rimple [National Laboratory of the Rockies↗

Tropical Rainfall Measuring Mission (TRMM). Phase B: Data capture facility definition study

The National Aeronautics and Aerospace Administration (NASA) and the National Space Development Agency of Japan (NASDA) initiated the Tropical Rainfall Measuring Mission (TRMM) to obtain more accurate measurements of tropical rainfall then ever before. The measurements are to improve scientific understanding and knowledge of the mechanisms effecting the intra-annual and interannual variability of the Earth's climate. The TRMM is largely dependent upon the handling and processing of the data by the TRMM Ground System supporting the mission. The objective of the TRMM is to obtain three years of climatological determinations of rainfall in the tropics, culminating in data sets of 30-day average rainfall over 5-degree square areas, and associated estimates of vertical distribution of latent heat release. The scope of this study is limited to the functions performed by TRMM Data Capture Facility (TDCF). These functions include capturing the TRMM spacecraft return link data stream; processing the data in the real-time, quick-look, and routine production modes, as appropriate; and distributing real time, quick-look, and production data products to users. The following topics are addressed: (1) TRMM end-to-end system description; (2) TRMM mission operations concept; (3) baseline requirements; (4) assumptions related to mission requirements; (5) external interface; (6) TDCF architecture and design options; (7) critical issues and tradeoffs; and (8) recommendation for the final TDCF selection process.

Source record↗

Data compression using Chebyshev transform

The present invention is a method, system, and computer program product for implementation of a capable, general purpose compression algorithm that can be engaged on the fly. This invention has particular practical application with time-series data, and more particularly, time-series data obtained form a spacecraft, or similar situations where cost, size and/or power limitations are prevalent, although it is not limited to such applications. It is also particularly applicable to the compression of serial data streams and works in one, two, or three dimensions. The original input data is approximated by Chebyshev polynomials, achieving very high compression ratios on serial data streams with minimal loss of scientific information.

Cheng, Andrew F.↗

Application of IRIG 106 Digital Data Recorder Standards for Flight Test

During flight tests, data is acquired from different sources and in different formats. Sensor data, aircraft avionics bus data, time, voice and video are essential for analysis of these flight tests. IRIG 106 standards exist for recording and transferring these data types. However, to enhance interoperability of flight test data between (multinational) test teams, a standard was needed to encapsulate all these data types. For this purpose the Telemetry Group of the Range Commanders Council (RCC) first published in 2004 the IRIG 106 Chapter 10 Standard, “Solid-State On-Board Recorder Standard”. This standard is evolving up to present. This AGARDograph gives an introduction to the standard, explains how the standard is positioned among other relevant standards, and describes the equipment and their application in flight tests with some examples. The topics covered include IRIG 106 Chapter 10 compatible recording of (multiple) data streams, synchronization of data, data recorders, data retrieval, data processing, and examples of flight test programs. Further development of the standard, originally published for on-board recorders, is described for extending the requirements for ground-based data recorders. The last chapter addresses a number of issues related to the application of the Chapter 10 standard with references to information and possible solutions. This document shows how data acquisition and data processing in accordance to IRIG 106 Chapter 10 standard can be applied in NATO flight tests and will thereby contribute to collaboration and cost efficiency.

flight test instrumentation↗

NASA Open Science Data Repository: Open Science for Life in Space

Space biology and health data are critical for the success of deep space missions and sustainable human presence off-world. At the core of effectively managing biomedical risks is the commitment to open science principles, which ensure that data are findable, accessible, interoperable, reusable, reproducible and maximally open. The 2021 integration of the Ames Life Sciences Data Archive with GeneLab to establish the NASA Open Science Data Repository significantly enhanced access to a wide range of life sciences, biomedical-clinical, and mission telemetry data alongside existing ‘omics data from GeneLab. This paper describes the new database, its architecture, and new data streams supporting diverse data types and enhancing data submission, retrieval, and analysis. Features include the Biological Data Management Environment for improved data submission, a new user interface, controlled data access, an enhanced API, and comprehensive public visualization tools for environmental telemetry, radiation dosimetry data, and ‘omics analyses. By fostering global collaboration through its Analysis Working Groups and training programs, the Open Science Data Repository promotes widespread engagement in space biology, ensuring transparency and inclusivity in research. It supports the global scientific community in advancing our understanding of spaceflight's impact on biological systems, ensuring humans will thrive in future deep space missions.

OSDR↗

Mind the Gap: Addressing Data Gaps and Assessing Noise Mismodeling in LISA

Due to the sheer complexity of the Laser Interferometer Space Antenna (LISA) space mission, data gaps arising from instrumental irregularities and/or scheduled maintenance are unavoidable. Focusing on merger-dominated massive black hole binary signals, we test the appropriateness of the Whittle-likelihood on gapped data in a variety of cases. From first principles, we derive the likelihood valid for gapped data in both the time and frequency domains. Cheap-to-evaluate proxies to p-p plots are derived based on a Fisher-based formalism, and verified through Bayesian techniques. Our tools allow to predict the altered variance in the parameter estimates that arises from noise mismodeling, as well as the information loss represented by the broadening of the posteriors. The result of noise mismodeling with gaps is sensitive to the characteristics of the noise model, with strong low-frequency (red) noise and strong high-frequency (blue) noise giving statistically significant fluctuations in recovered parameters. We demonstrate that the introduction of a tapering window reduces statistical inconsistency errors, at the cost of less precise parameter estimates. We also show that the assumption of independence between inter-gap segments appears to be a fair approximation even if the data set is inherently coherent. However, if one instead assumes fictitious correlations in the data stream, when the data segments are actually independent, then the resultant parameter recoveries could be inconsistent with the true parameters. The theoretical and numerical practices that are presented in this work could readily be incorporated into global-fit pipelines operating on gapped data.

LISA↗

Telemetry Analysis and Visualization Suite v. 2.2.0

This software provides a comprehensive set of tools for acquiring, extracting and analyzing any serial or packetized data stream, especially telemetry data. These tools enable real-time data acquisition, display, distribution and immediate post-test analysis.

Nguyen, Tram↗

ZEM: Integrated Framework for Real-Time Data and Model Analyses for Robust Environmental Management Decision Making - 16438

Typically environmental management (EM) problems require analysis of large and complex data sets originating from concurrent data streams with different data collection frequencies and pedigree. These big data sets require on-the-fly analysis, assimilation and integration into a series of models with different complexity for various types of model-based analyses. This is needed to provide fast iterative model analyses based on the latest available data to guide decision-making. Furthermore, the data and model are associated with uncertainties. The uncertainties are probabilistic (e.g. measurement errors) and non-probabilistic (unknowns, e.g. alternative conceptual models characterizing site conditions). To address all of these issues, we have developed an integrated framework for real-time data and model analyses for environmental decision-making called ZEM. The framework allows for seamless and on-the-fly integration of data and modeling results for robust and scientifically-defensible decision-making applying advanced decision analyses tools such as Bayesian-Information-Gap Decision Theory (BIG-DT). The framework also includes advanced methods for optimization that are capable of dealing with a large number of unknown model parameters, and surrogate (reduced order) modeling capabilities based on support vector regression techniques. The framework is predominantly coded in Julia, a state-of-the-art high-performance programing language (http://julialang.org) and incorporates the MADS computational tools (Model Analysis and Decision Support; http://mads.lanl.gov). We demonstrate the application of ZEM for data and model analyses at the LANL Chromium site. The ZEM framework will be released as open-source code and can be applied to any environmental management site. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Regulation of Carbon Sequestration and Water Use in an Ozark Forest: Proposing a New Strategically Located Ameriflux Tower Site in Missouri

by June 14, 2004, the MOFLUX site was fully instrumented and data streams started to flow. A primary accomplished deliverable for the project period was the data streams of CO 2 and water vapor fluxes and numerous meteorological variables (from which prepared datasets have been submitted to the AmeriFlux data archive for 2004-2006, Additionally, measurements of leaf biochemistry and physiology, biomass inventory, tree allometry, successional trends other variables were obtained.

54 ENVIRONMENTAL SCIENCES↗

Curation and Dissemination of Complex Multi-Modal Datasets for Radiation Detection, Localization, and Tracking

The PANDAWN sensor network in Chicago, IL, is a state-of-the-art testbed for networked, multi-modal sensing. It integrates AI/data science methods into its operation, from data acquisition to automated data labeling and curation workflows. The curation and dissemination of diverse multi-modal datasets will enable the development of new radiological/nuclear (R/N) detection, localization, and tracking algorithms and methods relevant across the nonproliferation mission space. This article first introduces the PANDAWN sensor network and the features that make it stand out from previous multi-modal data acquisition efforts. We then review the various data streams acquired on the PANDAWN nodes and present the implementation of an automated data curation pipeline that includes the labeling of radiation and contextual data streams. Here, we finally provide a short overview of different studies that leveraged the curated datasets.

Data curation↗

Anomaly Detection in Dynamic Networks

Anomaly detection in dynamic communication networks has many important security applications. These networks can be extremely large and so detecting any changes in their structure can be computationally challenging; hence, computationally fast, parallelisable methods for monitoring the network are paramount. For this reason the methods presented here use independent node and edge based models to detect locally anomalous substructures within communication networks. As a first stage, the aim is to detect changes in the data streams arising from node or edge communications. Throughout the thesis simple, conjugate Bayesian models for counting processes are used to model these data streams. A second stage of analysis can then be performed on a much reduced subset of the network comprising nodes and edges which have been identified as potentially anomalous in the first stage. The first method assumes communications in a network arise from an inhomogeneous Poisson process with piecewise constant intensity. Anomaly detection is then treated as a changepoint problem on the intensities. The changepoint model is extended to incorporate seasonal behavior inherent in communication networks. This seasonal behavior is also viewed as a changepoint problem acting on a piecewise constant Poisson process. In a static time frame, inference is made on this extended model via a Gibbs sampling strategy. In a sequential time frame, where the data arrive as a stream, a novel, fast Sequential Monte Carlo (SMC) algorithm is introduced to sample from the sequence of posterior distributions of the change points over time. A second method is considered for monitoring communications in a large scale computer network. The usage patterns in these types of networks are very bursty in nature and don’t fit a Poisson process model. For tractable inference, discrete time models are considered, where the data are aggregated into discrete time periods and probability models are fitted to the communication counts. In a sequential analysis, anomalous behavior is then identified from outlying behavior with respect to the fitted predictive probability models. Seasonality is again incorporated into the model and is treated as a changepoint model on the transition probabilities of a discrete time Markov process. Second stage analytics are then developed which combine anomalous edges to identify anomalous substructures in the network.

97 MATHEMATICS AND COMPUTING↗

Calibration of the ER-2 meteorological measurement system

The Meteorological Measurement System (MMS) on the high altitude ER-2 aircraft was developed specifically for atmospheric research. The MMS provides accurate measurements of pressure, temperature, wind vector, position (longitude, latitude, altitude), pitch, roll, heading, angle of attack, angle of sideslip, true airspeed, aircraft eastward velocity, northward velocity, vertical acceleration, and time, at a sample rate of 5/s. MMS data products are presented in the form of either 5 or 1 Hz time series. The 1 Hz data stream, generally used by ER-2 investigators, is obtained from the 5 Hz data stream by filtering and desampling. The method of measurement of the meteorological parameters is given and the results of their analyses are discussed.

Bowen, Stuart W.↗

The Role of the Consequence Management Home Team in the Fukushima Daiichi Response

The Consequence Management Home Team (CMHT) is a U.S. Department of Energy/National Nuclear Security Administration asset and played an important role in the U.S. response effort to the Fukushima Daiichi accident, ranging from the early days of the response to a continued involvement in supporting late phase efforts. Each stage of their work had distinct characteristics in terms of management of incoming data streams and creation of products. The CMHT assisted a variety of response organizations with modeling; radiological operations planning; field monitoring techniques; and the analysis, interpretation, and distribution of radiological data. In the Fukushima Daiichi response, the CMHT grew to include a broader range of support than was historically planned. Through their work, the social and economic impacts of a nuclear or radiological incident were minimized. The CMHT was an integral component of the response in Japan and acted as the central point from which all of the data and products flowed.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

krowkee

krowkee is a toolkit for scalably and efficiently summarizing many data streams in distributed memory. krowkee is intended for applications where one needs to summarize huge loosely structured data, such as matrices or graphs, where individual components such rows/columns or vertex adjacency information are impractical to store and directly inspect. krowkee ingests these objects as data streams - unstructured, arbitrarily ordered lists of updates - and accumulates summaries thereof in the form of data sketches.

Dunton, AlecM.↗

A study of high density bit transition requirements versus the effects on BCH error correcting coding

Several methods for increasing bit transition densities in a data stream are summarized, discussed in detail, and compared against constraints imposed by the 2 MHz data link of the space shuttle high rate multiplexer unit. These methods include use of alternate pulse code modulation waveforms, data stream modification by insertion, alternate bit inversion, differential encoding, error encoding, and use of bit scramblers. The psuedo-random cover sequence generator was chosen for application to the 2 MHz data link of the space shuttle high rate multiplexer unit. This method is fully analyzed and a design implementation proposed.

Ingels, F.↗

Efficient Databases for MPC Microdata (Final Report)

The purpose of this grant was to develop the theory and practice of high-performance databases for massive streamed datasets. Over the last three years, we have developed fast indexing technology, that is, technology for rapidly ingesting data and storing that data so that it can be efficiently queried and analyzed. During this project we developed the technology so that high-bandwidth data streams can be indexed and queried efficiently. Our technology has been proven to work data sets composed of tens of billions of rows when the data streams arrives at over 40,000 rows per second. We achieved these numbers even on a single disk driven by two cores. Our work comprised (1) new write-optimized data structures with better asymptotic complexity than traditional structures, (2) implementation, and (3) benchmarking. We furthermore developed a prototype of TokuFS, a middleware layer that can handle microdata I/O packaged up in an MPI-IO abstraction.

97 MATHEMATICS AND COMPUTING↗