Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

New Los Alamos space instrument goes for a spin

Scientists and engineers at Los Alamos National Laboratory are using a unique centrifuge facility to evaluate a flight-ready telemetry system for evaluating a nuclear weapons test missile launch. "The telemetry unit was designed and manufactured by Los Alamos' Intelligence and Space Research Division," said Telemetry Program Manager Myles Fitzgerald. "Telemetry is a data stream of information on temperature, acceleration, vibration, strain, all the conditions a system encounters during a test flight." Because a test missile flight ultimately ends with the test system completely destroyed, telemetry data from the flight must be collected by receiving stations along the flight path. "The purpose of the centrifuge test is to subject the electronics to a high gravitational load (G-load) that's representative of what the system will experience when re-entering the Earth's atmosphere," said weapons test engineer Alex Cusick.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2020

Automated Integration of Continental-Scale Observations in Near-Real Time for Simulation and Analysis of Biosphere–Atmosphere Interactions

The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

Durden, David J.↗
DOE OSTI2016

Adaptive Sequential Monte Carlo for Multiple Changepoint Analysis

Process monitoring and control requires detection of structural changes in a data stream in real time. This paper introduces an efficient sequential Monte Carlo algorithm designed for learning unknown changepoints in continuous time. The method is intuitively simple: new changepoints for the latest window of data are proposed by conditioning only on data observed since the most recent estimated changepoint, as these observations carry most of the information about the current state of the process. The proposed method shows improved performance over the current state of the art. Another advantage of the proposed algorithm is that it can be made adaptive, varying the number of particles according to the apparent local complexity of the target changepoint probability distribution. This saves valuable computing time when changes in the changepoint distribution are negligible, and enables re-balancing of the importance weights of existing particles when a significant change in the target distribution is encountered. The plain and adaptive versions of the method are illustrated using the canonical continuous time changepoint problem of inferring the intensity of an inhomogeneous Poisson process, although the method is generally applicable to any changepoint problem. Performance is demonstrated using both conjugate and non-conjugate Bayesian models for the intensity. Lastly, appendices to the article are available online, illustrating the method on other models and applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Machine learning on FPGA for event selection

Real-time data processing is a frontier field in experimental particle physics. The application of FPGAs at the trigger level is used by many current and planned experiments (CMS, LHCb, Belle2, PANDA). Usually they use conventional processing algorithms. LHCb has implemented Machine Learning (ML) elements for real-time data processing with a triggered readout system that runs most of the ML algorithms on a computer farm. The work described in this article aims to test the ML-FPGA algorithms for streaming data acquisition. Herein, there are many experiments working in this area and they have a lot in common, but there are many specific solutions for detector and accelerator parameters that are worth exploring further. This report describes the purpose of the work and progress in evaluating the ML-FPGA application.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

An Integrated Platform for Collaborative Data Analytics

While collaboration among data scientists is a key to organizational productivity, data analysts face significant barriers to achieving this end, including data sharing, accessing and configuring the required computational environment, and a unified method of sharing knowledge. Each of these barriers to collaboration is related to the fundamental question of knowledge management “how can organizations use knowledge more effectively?”. In this paper, we consider the problem of knowledge management in collaborative data analytics and present ShareAL, an integrated knowledge management platform, as a solution to that problem. The ShareAL platform consists of three core components: a full stack web application, a dashboard for analyzing streaming data and a High Performance Computing (HPC) cluster for performing real time analysis. Prior research has not applied knowledge management to collaborative analytics or developed a platform with the same capabilities as ShareAL. ShareAL overcomes the barriers data scientists face to collaboration by providing intuitive sharing of data and analytics via the web application, a shared computing environment via the HPC cluster and knowledge sharing and collaboration via a real time messaging application.

Oesch, T↗
DOE OSTI2025

Modular Autonomous Experimentation for Biological Applications (Full Report)

The Modular Autonomous Research System (MARS) was developed to address the pressing need for faster, more reliable, and more adaptable scientific discovery. Traditional experimentation is limited by manual labor, long cycle times, and fragmented data streams, which constrain the ability to explore complex chemical and materials design spaces. To overcome these limitations, we created an integrated, modular platform that combines laboratory robotics, diverse measurement instruments, and a central data infrastructure with artificial intelligence–driven decision-making. The system links liquid handling robots, robotic arms, and optical plate readers into a closed loop where experiments are executed automatically, data is analyzed in real time, and subsequent experimental conditions are adaptively chosen to maximize information gain. Over the course of the project, MARS was validated on two primary test cases—spectroscopic metal–ligand binding assays and peptide-directed mineralization—which highlighted the system’s ability to handle uncertainty and variability in experimental measurements. To further demonstrate modularity and extensibility, we also established additional testbeds in electrochemistry for catalyst discovery and electrolyte formulation for advanced batteries. The results show that MARS can reliably conduct autonomous campaigns with minimal human intervention, adapt to distinct scientific domains, and provide a scalable model for future self-driving laboratories. This work establishes new capabilities for modular, uncertainty-aware automation and directly supports the need for advanced, data-driven research platforms capable of accelerating discovery across a wide range of scientific and national security missions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Deep learning at the edge enables real-time streaming ptychographic imaging

Abstract Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the oversampling constraints, allowing low-dose imaging using orders of magnitude less data than required by traditional methods.

36 MATERIALS SCIENCE↗
DOE OSTI2011

CLARA: A Contemporary Approach to Physics Data Processing

In traditional physics data processing (PDP) systems, data location is static and is accessed by analysis applications. In comparison, CLARA (CLAS12 Reconstruction and Analysis framework) is an environment where data processing algorithms filter continuously flowing data. In CLARA's domain of loosely coupled services, data is not stored, but rather flows from one service to another, mutating constantly along the way. Agents, performing event processing, can then subscribe to particular data/events at any stage of the data transformation, and make intricate decisions (e.g. particle ID) by correlating events from multiple, parallel data streams and/or services. This paper presents a PDP application development framework based on service oriented and event driven architectures. This system allows users to design (Java, C++, and Python languages are supported) and deploy data processing services, as well as dynamically compose PDP applications using available services. The PDP service bus provides a layer on top of a distributed pub-sub middleware implementation, which allows complex service composition and integration without writing code. Examples of service creation and deployment, along with the CLAS12 track reconstruction application design will be presented.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2018

Biologically inspired approaches for biosurveillance anomaly detection and data fusion

This study developed and tested biologically inspired computational methods to detect anomalous signals in data streams that could indicate a pending outbreak or bio-weapon attack. Current large-scale biosurveillance systems are plagued by two principal deficiencies: (1) timely detection of disease-indicating signals in noisy data and (2) anomaly detection across multiple channels. Anomaly detectors and data fusion components modeled after human immune system processes were tested against a variety of natural and synthetic surveillance datasets. A pilot scale immune-system-based biosurveillance system performed at least as well as traditional statistical anomaly detection data fusion approaches. Machine learning approaches leveraging Deep Learning recurrent neural networks were developed and applied to challenging unstructured and multimodal health surveillance data. Within the limits imposed of data availability, both immune systems and deep learning methods were found to improve anomaly detection and data fusion performance for particularly challenging data subsets.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

Data Analytics for Electrical Distribution Systems with Micro PMUs (GMLC 1.4.9 Technical Report)

This report describes how electrical utilities with residential, industrial, commercial and government customers make decisions using Data Analytics (DA) applications and discusses how OpenFMB was leveraged to share that data among disparate types of devices for DA. OpenFMB is a data standard developed by utilities for distributed communication and control of power system assets. Here we explored how OpenFMB can be used for sharing analytics and what those analytics could be for a set of representative grid devices. A review of DA applications performed by electrical utilities and customers is presented. An overview of how the testbed we used was built and configured is summarized. Several experiments which were carried out to explore the capabilities of OpenFMB alongside other data sharing mechanisms for gathering sensor measurements and sharing analytics is also discussed. And finally a comparison of OpenFMB and LBNL’s SPARK framework is summarized. Today’s electrical utilities focus on monitoring the real-time customer consumptions, offering new real-time tariff models based on customer usage habits (rewards and punishments), providing energy peak during peak load times (plan energy generation and distribution), improving the load management for transformers and power lines, and controlling the energy and grid resources efficiently. On the other hand, the DA applications for residential customers are based on modifying the customer electricity usage habit to save money, making accurate and timely decisions for electrical billing. However, the DA applications for industrial, commercial and government customers focus on creating efficient programs to improve the environmental condition (reducing CO₂ emission), developing better load forecasting for industrial and/or commercial activities. In this project, a testbed was built to explore how the OpenFMB framework can be used to distribute data among various types of representative grid edge devices. Micro PMUs (Phasor Measurement Unit) were used to collect and transmit data to an OpenPDC server alongside relays with a framework configured with a system using OpenFMB to collect and transmit data. The testbed was composed of a physical asset based on a 60 Hz AC radial power system constructed using SI-GRID components. The power system consisted of a 3-phase 14 volts line-to-ground (24 volts line-to-line) power source, three 3-phase line emulators, a programmable resistive load bank, two smart relays, two micro PMUs, and a (SCADA) and Phasor Data Concentrator (PDC) computer. This testbed configuration allowed us to perform breaker operation tests collecting data from the micro PMUs and from the relays’ sensors. During the micro PMU breaker operation tests we collected the measured and estimated fundamental voltages, currents, frequency as well as the total true, reactive and apparent power. The displacement power factor from two micro PMUs located in parallel power lines was also used. The software steps for the implementation of the Synchrophasor system are explained in detail in this report. The setting (1-2-3) and data collection (4-5) phases were performed with a free download open source software. In the setting phase, the micro PMUs were set (1), the data stream for each micro PMU was validated, and the connection and configuration files were created (2). The PDC computer was set using the connection and configuration files of the micro PMUs (3). In the data collection phase, the binary data from the micro PMUs was collected (4), and the binary data files of the micro PMUs was converted to analyze the test results (5). OpenFMB was configured to use protocol translators to read Modbus and transform the reading into the OpenFMB data model to be read for DA applications.Our experiments showed that by leveraging the OpenFMB data model and the micro PMUs together we could obtain better situational awareness of network faults, which in some cases could lead to failures in the system and would be less perceivable otherwise. For example, in one experiment the fundamental RMS current angle, and the fundamental displacement power factor (estimated by the ratio between the measured real and apparent total power) had a disturbance at the micro PMU connected to the beginning of the parallel power line because the breaker was opened without the operators or SCADA system’s knowledge, and no current was available. Based on the breaker state scenarios and micro PMU locations, the DA applications using the current angle and displacement power factor from the micro PMUs should be studied in detail to avoid a non-desired measurement that could affect the decision from the DA. Also, the sample rates for PMU, Relay and micro PMU devices were compared. The DA application improved the control and operation between the electrical utilities and their customers (residential, industrial, transportation, commercial and government). The higher the sample rate, the better the quality of the measurement for DA estimations.

42 ENGINEERING↗
NASA NTRSDate not supplied

Leveraging VIPER Geotechnical and Mobility Insights for Endurance Rover and Traverse Development

A new generation of lunar surface rover exploration is rapidly approaching. This is evidenced by the rise of commercially developed rovers from private companies (e.g., Lunar Outpost’s MAPP) and government agencies, like NASA’s Volatiles Investigating Polar Exploration Rover (VIPER), the Lunar Terrain Vehicle (LTV), and the Endurance/Intrepid rover concepts, and the INSPIRE mission concept. The drive distance requirements for off-world vehicles are also rapidly increasing. For example, LTV driving requirements dictate a 20 km range on a single battery charge, with an overall lifetime requirement of 13,000 km over a minimum 10-year service life. Similarly, NASA's Endurance-A mission concept requires a 2000 km vehicle traverse distance, which is nearly 50 times further than the USSR's Lunokhod 2, the longest lunar rover mission to date. Previous planetary surface missions on the Moon and Mars have revealed harsh mobility conditions: aggressive rocky terrains on Mars severely damaged the Curiosity rover’s wheels, soft lunar regolith caused the Lunokhod 2 rover to sink and NASA’s Apollo 15 LRV (Lunar Roving Vehicle) spun out due to poor traction, which required manual correction. This demonstrates that mobility challenges like wheel degradation and soft soil driving performance must be accounted for in a mission’s design phase to meet increasingly aggressive mission durations and mobility goals. The success of these long duration missions requires highly reliable and durable mobility platforms in conjunction with a more developed understanding of the geotechnical properties of the lunar surface. The VIPER mission is uniquely posed to inform future missions about mobility performance, wheel durability, and lunar surface soil mechanics of sunlit and shadowed regions through use of data streams that already exist for other mission purposes. VIPER’s mission to the lunar south pole will provide a new perspective into off-world mobility challenges including traversal into permanently shadowed regions. Mobility data from VIPER’s mission will represent a first glimpse into the surface mobility challenges of the lunar South Pole. Utilization of VIPER’s existing capabilities including cameras, inertial measurement unit (IMU) data, motor currents/torque, drill data, and the rover’s suspension state and kinematics provide the opportunity to determine potentially useful information for future rover and traverse designs, as well as on-Earth testing practices. Data products collected during VIPER’s mission could inform design considerations for the Endurance rover much earlier than NASA’s standard “lessons learned” approach as VIPER has a highly tactical mission operations structure. This work outlines the specific links between possible data products from the VIPER mission and their potential impact on the Endurance/Intrepid mission concepts in their early design phases. From VIPER, we will also gain insights for future development of onboard mobility software as VIPER is a way-point driven rover which requires more constant human operation on Earth than some other rover missions (non-lunar surface). Primary VIPER data products and insights of interest include wheel sinkage, wheel slip, wheel degradation, surface soil shear strength, and terrain slope estimation. VIPER data products will be made available via the Planetary Data System’s Geosciences and Cartography and Imaging Sciences nodes, conforming to the PDS4 standard.

mobility↗
DOE OSTI2013

A digitizer based compact digital spectrometer for ion beam analysis using field programmable gate arrays and various energy algorithms

We report on the implementation of a compact multi-detector fully digital spectrometer and data acquisition system at a nuclear microprobe for ion beam analysis and imaging. The spectrometer design allows for system scalability with no restriction on the number of detectors. It consists of four-channel high-speed digitizer modules for detector signal acquisition and one low-speed digital-to-analog converter (DAC) module with two DAC channels and additional general purpose inputs/outputs to control ion beam scanning and data acquisition. Each digitizer module of the spectrometer provides its own Field Programmable Gate Array (FPGA) as digital signal processing unit to analyze detector signals as well as to synchronize the ion beam position in hard real-time. With the customized FPGA designs for all modules, all calculation intensive tasks are executed inside the modules, which reduces significantly the data stream to and CPU load on the control computer. To achieve an optimal energy resolution for all detector/preamplifier pulse shape characteristics, a user-definable infinite impulse response filter with high throughput for energy determination was implemented. The new spectrometer has an online data analysis feature, a compact size, and is able to process any type of detector signals such as particle induced x-ray emission, Rutherford backscattering spectrometry, or scanning transmission ion microscopy.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2012

Development of a Real-Time, High-Speed Distribution Level Data Acquisition System

With the development of smart grids and the deployment of their enabling technologies, improved data acquisition will be needed at the distribution level to understand the full impact of these changes. With this in mind, NREL has developed a high-speed measurement and data collection network targeted specifically at the distribution level. This network is based around adaptable, rugged measurement devices designed for deployment at a variety of low and medium voltage locations below the sub-station. Each of these devices is capable of real-time data transmission via an Internet connection. Additionally, several analysis and visualization applications have been developed around the incoming data streams.

14 SOLAR ENERGY↗
DOE OSTI2015

Performance analysis of data intensive cloud systems based on data management and replication: a survey

As we delve deeper into the ‘Digital Age’, we witness an explosive growth in the volume, velocity, and variety of the data available on the Internet. For example, in 2012 about 2.5 quintillion bytes of data was created on a daily basis that originated from myriad of sources and applications including mobiledevices, sensors, individual archives, social networks, Internet of Things, enterprises, cameras, software logs, etc. Such ‘Data Explosions’ has led to one of the most challenging research issues of the current Information and Communication Technology era: how to optimally manage (e.g., store, replicated, filter, and the like) such large amount of data and identify new ways to analyze large amounts of data for unlocking information. It is clear that such large data streams cannot be managed by setting up on-premises enterprise database systems as it leads to a large up-front cost in buying and administering the hardware and software systems. Therefore, next generation data management systems must be deployed on cloud. The cloud computing paradigm provides scalable and elastic resources, such as data and services accessible over the Internet Every Cloud Service Provider must assure that data is efficiently processed and distributed in a way that does not compromise end-users’ Quality of Service (QoS) in terms of data availability, data search delay, data analysis delay, and the like. In the aforementioned perspective, data replication is used in the cloud for improving the performance (e.g., read and write delay) of applications that access data. Through replication a data intensive application or system can achieve high availability, better fault tolerance, and data recovery. In this paper, we survey data management and replication approaches (from 2007 to 2011) that are developed by both industrial and research communities. The focus of the survey is to discuss and characterize the existing approaches of data replication and management that tackle the resource usage and QoS provisioning with different levels of efficiencies. Moreover, the breakdown of both influential expressions (data replication and management) to provide different QoS attributes is deliberated. Furthermore, the performance advantages and disadvantages of data replication and management approaches in the cloud computing environments are analyzed. Open issues and future challenges related to data consistency, scalability, load balancing, processing and placement are also reported.

Malik, Saif Ur Rehman↗
DOE OSTI2024

HERMES

HERMES - High-speed Event Retrieval and Management for Enhanced Spectral imaging code. This code is meant to unpack and process neutron imaging data from the TPX3Cam made by Amsterdam Scientific Instruments. The TPX3Cam utilizes the Timepix3 chip in a single photon counting mode of image acquisition. The photon counting data streaming off the TPX3Cam needs to be process and analyzed in order to create final images. Therefore we are developing both python and cpp codes to allow for users of the TPX3Cam to efficiently analyze data and create images.

Long, Alexander↗
DOE OSTI2021

Data Reduction for Science: Brochure from the Advanced Scientific Computing Research Workshop

Data reduction for science holds promise for addressing the challenges of moving, storing, and processing massive data sets produced by the scientific community. Pursuing the PRDs outlined here will enable advances in data streaming, fast feedback and/or autonomous control of experiments, and faster time to scientific insight. These advances will result in a significant improvement in the ability to transport, store, process and interpret experimental, observational, and computational data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Status of the data acquisition, trigger, and slow control systems of the Mu2e experiment at Fermilab

The Mu2e experiment at the Fermilab will search for a coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10,000 over existing limits. In this work, the Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq framework as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates several framework components — an artdaq-based DAQ, an art-based event processing, and an EPICS-based detector control system (DCS), and provides a uniform multi-user interface to its components through a web browser. Data streams from the Mu2e tracker and calorimeter are handled by the artdaq-based DAQ and processed by a one-level software trigger implemented within the art framework. Events accepted by the trigger have their data combined, post-trigger, with the separately read out data from the Mu2e Cosmic Ray Veto system. Foundation of the Mu2e DCS, EPICS – an Experimental Physics and Industrial Control System – is an open-source platform for monitoring, controlling, alarming, and archiving. A prototype of the TDAQ and the DCS systems has been built and tested over the last three years at Fermilab’s Feynman Computing Center, and now the production system installation is underway. This work presents their status and focus on the installation plans and procedures for racks, workstations, network switches, gateway computers, DAQ hardware, slow controls implementation, and testing.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Unsupervised anomaly clustering via offset alignment in multivariate grid sensing data

Modern industries increasingly rely on multi-sensor technologies to acquire complex, high-dimensional data streams, enabling advanced monitoring and control systems. One critical application is online anomaly detection in electrical smart grids, where multivariate and multimodal sensing technologies play a vital role. However, detecting anomalies in such time-series data is challenging due to their inherent temporal dependencies and stochastic behavior. Traditional approaches based on supervised and semi-supervised learning methods depend on labeled datasets, which are often unavailable in real-world scenarios. While unsupervised methods have emerged as promising alternatives, these methods are highly susceptible to noise and outliers commonly present in sensing applications. Furthermore, deep learning-based anomaly detection methods, despite their performance, are often criticized for their black-box nature, limiting their applicability in safety-critical and online environments where interpretability and explainability are paramount. In this work, we propose an unsupervised anomaly clustering method leveraging a cyclic alignment-based offset detection algorithm for multivariate time-series signals. The proposed method is applied to multivariate data collected from vibrational, voltage, and magnetic field sensors deployed in a local grid substation. Our results demonstrate the robustness of the algorithm in accurately clustering various anomalies/events across different sensing modalities. Additionally, we compare the effectiveness of the proposed approach against a simple pattern-based anomaly detection method, which performs well for univariate data but fails to generalize to multivariate and multimodal time-series data.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗