SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40

All sources NASA NTRS DOE OSTI

DOE OSTI2024

A Co-Registered In-Situ and Ex-Situ Dataset of Electrical, Acoustic, and CT Characteristics from Wire Arc Additive Manufacturing Process

Recent progress in sensing techniques and data analytics tools have significantly accelerated the development of Wire Arc Additive Manufacturing (WAAM) systems. This data centric approach emphasizes leveraging available data throughout the production process to optimize performance. Integration of extensive data analysis provides the opportunity to improve precision, reduce waste, and enhance the quality of produced parts. This method relies on AI/ML models and optimization techniques, which are developed using the data collected from various sources, including in-situ sensors, ex-situ imaging, and manufacturing process parameters. The quality and diversity of this data, along with the alignment between different data streams (achieved through spatiotemporal registration) are critical for the successful development of AI/ML and optimization models. In this work, we present a spatiotemporally registered dataset generated during the WAAM process of deposition of a rectangular block. The dataset includes the comprehensive description of deposition process, process parameters, in-situ collected welding characteristics, acoustic data, and X-Ray Computed Tomography analysis data for the build. Dataset A Co-Registered In-Situ and Ex-Situ Dataset of Electrical, Acoustic, and CT Characteristics from Wire Arc Additive Manufacturing Process has arisen under UT-Battelle, LLC’s Prime Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy (DOE) to manage and operate the Oak Ridge National Laboratory. UT-Battelle, LLC will not assert any rights under United States law or under the Prime Contract it has in the dataset against any user of the dataset, including any copyrights or patent rights. UT-Battelle, LLC requests that attribution to the dataset is provided as academically appropriate.

42 ENGINEERING↗

DOE OSTI2022

Effectiveness of Privacy Techniques in Smart Metering Systems

Smart grid technologies enable timely energy billing for residential homes. The ability to react to energy demands during peak hours allows energy providers to conserve power and operate efficiently. However, these data streams are also susceptible to privacy attacks within the energy company and from outside hackers. We implemented four different privacy models: k-anonymous, l-diversity, t-closeness, and ε-differential privacy. We demonstrate the models’ effectiveness using a real-world dataset composed of 15 different residential households with energy consumption data spanning over a year.

Peralta-Peterson, Martin↗

NASA NTRSDate not supplied

AN INTRODUCTION TO THE GEONEX LEVEL-1G PRODUCTS: TOP-OF-ATMOSPHERE REFLECTANCE AND BRIGHTNESS TEMPERATURE

This paper introduces the GeoNEX (Geostationary-NASA Earth eXchange) Level-1G products of top-of-atmosphere (TOA) reflectance and brightness temperature. The products use data streams from the latest geostationary (GEO) sensors including the GOES-16/17 ABI and the Himawari-8/9 AHI. The GeoNEX processing pipeline starts by converting digital numbers to physical quantities with the latest radiometric calibration information. It integrates algorithms to automatically detect and remove residual geolocation errors, to estimate the pixel-wise data-acquisition time, and to accurately calculate the solar illumination angles for each pixel in the domain at every time step. The outputs are reprojected to a globally tiled common grid in geographic coordinates designed to facilitate inter-comparisons and/or synergies between the GeoNEX products and existing Earth observation datasets from polar-orbiting satellites. Therefore, the GeoNEX L1G products provide accurate and consistent TOA reflectance and brightness temperature datasets for scientific analyses and downstream product development.

Geostationary satellite, GOES-16, Himawari-8, NASA↗

DOE OSTI2016

Data Always Getting Bigger—A Scalable DOI Architecture for Big and Expanding Scientific Data

The Atmospheric Radiation Measurement (ARM) Data Archive established a data citation strategy based on Digital Object Identifiers (DOIs) for the ARM datasets in order to facilitate citing continuous and diverse ARM datasets in articles and other papers. This strategy eases the tracking of data provided as supplements to articles and papers. Additionally, it allows future data users and the ARM Climate Research Facility to easily locate the exact data used in various articles. Traditionally, DOIs are assigned to individual digital objects (a report or a data table), but for ARM datasets, these DOIs are assigned to an ARM data product. This eliminates the need for creating DOIs for numerous components of the ARM data product, in turn making it easier for users to manage and cite the ARM data with fewer DOIs. In addition, the ARM data infrastructure team, with input from scientific users, developed a citation format and an online data citation generation tool for continuous data streams. This citation format includes DOIs along with additional details such as spatial and temporal information.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1983

Spacelab high density digital recorders

The design and performance of the high-density digital recorder (HDDR) developed for use at the NASA centers (KSC, JSC, and GSFC) and at the JPL to store and retrieve 50-Mb/s PCM data streams from the Spacelab experiments are reported. The recording reproduction, and transport requirements are reviewed; and the design solutions adopted in the final version of the HDDR are described, incuding three-position-modulation and Y-phase encoding, microprocessor-controlled automatic bit synchronization and equalization, cyclic-redundancy-check error detection and correction, clock regeneration, data and clock variations, tape-speed control, and EEE-488 remote control. Reliable performance, with bit error rates 1 in 10 to the 10th forward and 1 in 10 to the 9th reverse or better and packing density up to 50 percent greater than that obtainable using conventional codes, is reported after 1.5 years of service.

Blais, R. A.↗

NASA NTRSDate not supplied

Data Science and the Knowledge Discovery Adventure

This talk will cover the important steps involved in the data science and knowledge discovery process: • Initial fact gathering (interview domain experts, review reports, articles, state-of-the-art) • Identify the problem (prediction, classification, statistical analysis, etc.) • Survey supporting data sources • Understand the data (numerical, categorical, text, sampling rate, data quality issues, etc.) • Selecting relevant features and sources • Acquire the data (set up agreements with the data stewards, APIs to download, etc.) • Merge data sources (temporal, spatial, common key, other ontologies...) • Feature Engineering (non linear domain knowledge or physics-based relationships) • Build data processing pipeline (may need to tap into data stream, develop parallel processing algorithm, federated learning etc.) • Build model and test (tune hyper-parameters, cross validation.) • Analyze/Validate results (do the results make sense. Does it answer the original question). • Deploy/Publish (Monitor and assess benefits)

Data science↗

DOE OSTI2022

Online Distribution System State Estimation via Stochastic Gradient Algorithm

Distribution network operation is becoming more challenging because of the growing integration of intermittent and volatile distributed energy resources (DERs). This motivates the development of new distribution system state estimation (DSSE) paradigms that can operate at fast timescale based on real-time data stream of asynchronous measurements enabled by modern information and communications technology. To solve the real-time DSSE with asynchronous measurements effectively and accurately, this paper formulates a weighted least squares DSSE problem and proposes an online stochastic gradient algorithm to solve it. The performance of the proposed scheme is analytically guaranteed and is numerically corroborated with realistic data on IEEE 123-bus feeder.

distribution system state estimation↗

NASA NTRS1990

A burst compression and expansion technique for variable-rate users in satellite-switched TDMA networks

A burst compression and expansion technique is described for asynchronously interconnecting variable-data-rate users with cost-efficient ground terminals in a satellite-switched, time-division-multiple-access (SS/TDMA) network. Compression and expansion buffers in each ground terminal convert between lower rate, asynchronous, continuous-user data streams and higher-rate TDMA bursts synchronized with the satellite-switched timing. The technique described uses a first-in, first-out (FIFO) memory approach which enables the use of inexpensive clock sources by both the users and the ground terminals and obviates the need for elaborate user clock synchronization processes. A continous range of data rates from kilobits per second to that approaching the modulator burst rate (hundreds of megabits per second) can be accommodated. The technique was developed for use in the NASA Lewis Research Center System Integration, Test, and Evaluation (SITE) facility. Some key features of the technique have also been implemented in the gound terminals developed at NASA Lewis for use in on-orbit evaluation of the Advanced Communications Technology Satellite (ACTS) high burst rate (HBR) system.

Budinger, James M.↗

NASA NTRS1990

A burst compression and expansion technique for variable-rate users in satellite-switched TDMA networks

A burst compression and expansion technique is described for asynchronously interconnecting variable-data-rate users with cost-efficient ground terminals in a satellite-switched, time-division-multiple-access (SS/TDMA) network. Compression and expansion buffers in each ground terminal convert between lower rate, asynchronous, continuous-user data streams and higher-rate TDMA bursts synchronized with the satellite-switched timing. The technique described uses a first-in, first-out (FIFO) memory approach which enables the use of inexpensive clock sources by both the users and the ground terminals and obviates the need for elaborate user clock synchronization processes. A continuous range of data rates from kilobits per second to that approaching the modulator burst rate (hundreds of megabits per second) can be accommodated. The technique was developed for use in the NASA Lewis Research Center System Integration, Test, and Evaluation (SITE) facility. Some key features of the technique have also been implemented in the ground terminals developed at NASA Lewis for use in on-orbit evaluation of the Advanced Communications Technology Satellite (ACTS) high burst rate (HBR) system.

Budinger, James M.↗

DOE OSTI2015

Text Stream Trend Analysis using Multiscale Visual Analytics with Applications to Social Media Systems

In this paper, we introduce a new visual analytics system, called Matisse, that allows exploration of global trends in textual information streams with specific application to social media platforms. Despite the potential for real-time situational awareness using these services, interactive analysis of such semi-structured textual information is a challenge due to the high-throughput and high-velocity properties. Matisse addresses these challenges through the following contributions: (1) robust stream data management, (2) automated sen- timent/emotion analytics, (3) inferential temporal, geospatial, and term-frequency visualizations, and (4) a flexible drill-down interaction scheme that progresses from macroscale to microscale views. In addition to describing these contributions, our work-in-progress paper concludes with a practical case study focused on the analysis of Twitter 1% sample stream information captured during the week of the Boston Marathon bombings.

interaction↗

DOE OSTI2019

MSE based drilling optimization using neural network simulaton

The method disclosed receives a data stream from an MWD system and determines the response of a specific energy (SE) relationship and a rate of penetration (ROP) relationship respectively to variables controllable by the operator, in order to enable operation at a lowest SE, or a highest Rate-of-Penetration (ROP) to SE ratio. The method utilizes artificial neural networks trained by MWD data to deduce a depth-of-cut and torque based on relationships manifesting between the various data points collected, and an SE equation and a predicted ROP is evaluated over a series of probable operating points. The method continuously gathers and analyzes MWD data during the drilling operation and allows an operator to manage the controllable parameters such that operation at the lowest SE or highest ROP or ROP to SE ratio can be achieved during the drilling operation.

Zhang, Wu↗

NASA NTRS2006

System and method for forward error correction

A system and method are provided for transferring a packet across a data link. The packet may include a stream of data symbols which is delimited by one or more framing symbols. Corruptions of the framing symbol which result in valid data symbols may be mapped to invalid symbols. If it is desired to transfer one of the valid data symbols that has been mapped to an invalid symbol, the data symbol may be replaced with an unused symbol. At the receiving end, these unused symbols are replaced with the corresponding valid data symbols. The data stream of the packet may be encoded with forward error correction information to detect and correct errors in the data stream.

Cole, Robert M.↗

NASA NTRS2007

System and method for transferring data on a data link

Cole, Robert M.↗

DOE OSTI2015

CORE (Common Operating Response Environment) Software Technology Suite

Agencies that oversee complex, multi-stakeholder programs need efficient, secure ways to link people and knowledge within and across organizations. The Common Operating Response Environment (CORE), a software suite developed by PNNL researchers does just that. The CORE tool—which is customizable for a multitude of uses—facilitates situational awareness by integrating diverse data streams without the need to reformat them, summarizing that information, and providing users with the information they need to rapidly understand and appropriately respond to situations. It is mobile device-ready, has a straightforward interface for ease of use across organizations and skill sets, and is incredibly configurable to the needs of each specific user, whether they require data summaries for high-level decision makers or tactical maps, operational data, or weather information for responders in the field. Information can be input into CORE and queried in a variety of ways—using customized forms, reports, visuals, or other organizational templates—according to the needs of each user’s organization, teams, and business processes. CORE data forms, for instance, could be accessed and used in real-time to capture information about vessels being inspected for nuclear material.

42 ENGINEERING↗

DOE OSTI2014

Situ

Rapidly discovering novel and sophisticated cyber attacks and providing situation aware-ness to analysts are unsolved problems in cyber security. We have developed a platform that scores events in real-time based on probabilistic models to define how typical an event is. This anomaly detection approach is based on unsupervised, probabilistic modeling of data at multiple scales. Scoring events based on multiple scales allows the system to 1) score anomalousness at different levels to detect important events that would otherwise be hidden, and 2) explain to users why an event is anomalous, not just that it is. The system was designed to address several challenges: 1) scaling to very high volume, heterogeneous, streaming data, and 2) minimizing the time from observation to discovery to understanding. The prototype has the real-time framework for pushing scored events to a web-based visualization.

Source record↗

DOE OSTI2014

Combined impact of lead, cadmium, polychlorinated biphenyls and non-chemical risk factors on blood pressure in NHANES

High blood pressure is associated with exposure to multiple chemical and non-chemical risk factors, but epidemiological analyses to date have not assessed the combined effects of both chemical and non-chemical stressors on human populations in the context of cumulative risk assessment. We developed a novel modeling approach to evaluate the combined impact of lead, cadmium, polychlorinated biphenyls (PCBs), and multiple non-chemical risk factors on four blood pressure measures using data for adults aged ≥20 years from the National Health and Nutrition Examination Survey (1999–2008). We developed predictive models for chemical and other stressors. Structural equation models were applied to account for complex associations among predictors of stressors as well as blood pressure. Models showed that blood lead, serum PCBs, and established non-chemical stressors were significantly associated with blood pressure. Lead was the chemical stressor most predictive of diastolic blood pressure and mean arterial pressure, while PCBs had a greater influence on systolic blood pressure and pulse pressure, and blood cadmium was not a significant predictor of blood pressure. The simultaneously fit exposure models explained 34%, 43% and 52% of the variance for lead, cadmium and PCBs, respectively. The structural equation models were developed using predictors available from public data streams (e.g., U.S. Census), which would allow the models to be applied to any U.S. population exposed to these multiple stressors in order to identify high risk subpopulations, direct intervention strategies, and inform public policy. - Highlights: • We evaluated joint impact of chemical and non-chemical stressors on blood pressure. • We built predictive models for lead, cadmium and polychlorinated biphenyls (PCBs). • Our approach allows joint evaluation of predictors from population-specific data. • Lead, PCBs and established non-chemical stressors were related to blood pressure. • Framework allows cumulative risk assessment in specific geographic settings.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Automatic Traffic Queue-End Identification using Location-Based Waze User Reports

Traffic queues, especially queues caused by non-recurrent events such as incidents, are unexpected to high-speed drivers approaching the end of queue (EOQ) and become safety concerns. Though the topic has been extensively studied, the identification of EOQ has been limited by the spatial-temporal resolution of traditional data sources. This study explores the potential of location-based crowdsourced data, specifically Waze user reports. It presents a dynamic clustering algorithm that can group the location-based reports in real time and identify the spatial-temporal extent of congestion as well as the EOQ. The algorithm is a spatial-temporal extension of the density-based spatial clustering of applications with noise (DBSCAN) algorithm for real-time streaming data with an adaptive threshold selection procedure. Here, the proposed method was tested with 34 traffic congestion cases in the Knoxville, Tennessee area of the United States. It is demonstrated that the algorithm can effectively detect spatial-temporal extent of congestion based on Waze report clusters and identify EOQ in real-time. The Waze report-based detection are compared to the detection based on roadside sensor data. The results are promising: The EOQ identification time of Waze is similar to the EOQ detection time of traffic sensor data, with only 1.1 min difference on average. In addition, Waze generates 1.9 EOQ detection points every mile, compared to 1.8 detection points generated by traffic sensor data, suggesting the two data sources are comparable in respect of reporting frequency. The results indicate that Waze is a valuable complementary source for EOQ detection where no traffic sensors are installed.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗