Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Mu2e DAQ and slow control systems

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the feld of an aluminum nucleus with a sensitivity improvement by a factor of 10,000 over existing limits. The Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, fltering, and processing. otsdaq is an online DAQ software suite with a focus on fexibility and scalability and provides a multiuser, web-based, interface accessible through a web browser. The data stream from the detector subsystems is read by a software flter algorithm that selects events which are combined with the data fux coming from a Cosmic Ray Veto System. The Detector Control System (DCS) has been developed using the Experimental Physics and Industrial Control System (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS System has been integrated into otsdaq. A prototype of the TDAQ and the DCS systems has been built at Fermilab’s Feynman Computing Center. In this paper, we report on the progress of the integration of this prototype in the online otsdaq software.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Data acquisition and slow control interface for the Mu2e experiment

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10000 over existing limits. The Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, filtering, and processing. otsdaq is an online DAQ software suite with a focus on flexibility and scalability and provides a multi-user, web-based, interface accessible through a web browser. The data stream from the detector subsystems is read by a software filter algorithm that selects events which are combined with the data flux coming from a cosmic ray veto system. The Detector Control System (DCS) has been developed using the Experimental Physics and Industrial Control System (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS system has been integrated into otsdaq. A prototype of the TDAQ and the DCS systems has been built at Fermilab's Feynman Computing Center. In this study, we report on the progress of the integration of this prototype in the online otsdaq software.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2018

Integrated Multi-Scale Data Analytics and Machine Learning for the Distribution Grid

We consider the field of machine learning and where it is both useful, and not useful, for the distribution grid and buildings interface. While analytics, in general, is a growing field of interest, and often seen as the golden goose in the burgeoning distribution grid industry, its application is often limited by communications infrastructure, or lack of a focused technical application. Overall, the linkage of analytics to purposeful application in the grid space has been limited. In this paper we consider the field of machine learning as a subset of analytical techniques, and discuss its ability and limitations to enable the future distribution grid. To that end, we also consider the potential for mixing distributed and centralized analytics and the pros and cons of these approaches. There is an exponentially expanding volume of measured data being generated on the distribution grid, which, with appropriate application of analytics, may be transformed into intelligible, actionable information that can be provided to the right actors - such as grid and building operators, at the appropriate time to enhance grid or building resilience, efficiency, and operations against various metrics or goals - such as total carbon reduction or other economic benefit to customers. While some basic analysis into these data streams can provide a wealth of information, computational and human boundaries on performing the analysis are becoming significant, with more data and multi-objective concerns. Efficient applications of analysis and the machine learning field are being considered in the loop. This paper describes benefits and limits of present machine-learning applications for use on the grid and presents a series of case studies that illustrate the potential benefits of developing advanced local multi-variate analytics machine-learning-based applications.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2017

Integrated Multi-Scale Data Analytics and Machine Learning for the Distribution Grid and Building-to-Grid Interface

This white paper introduces the application of advanced data analytics to the modernized grid. In particular, we consider the field of machine learning and where it is both useful, and not useful, for the particular field of the distribution grid and buildings interface. While analytics, in general, is a growing field of interest, and often seen as the golden goose in the burgeoning distribution grid industry, its application is often limited by communications infrastructure, or lack of a focused technical application. Overall, the linkage of analytics to purposeful application in the grid space has been limited. In this paper we consider the field of machine learning as a subset of analytical techniques, and discuss its ability and limitations to enable the future distribution grid and the building-to-grid interface. To that end, we also consider the potential for mixing distributed and centralized analytics and the pros and cons of these approaches. Machine learning is a subfield of computer science that studies and constructs algorithms that can learn from data and make predictions and improve forecasts. Incorporation of machine learning in grid monitoring and analysis tools may have the potential to solve data and operational challenges that result from increasing penetration of distributed and behind-the-meter energy resources. There is an exponentially expanding volume of measured data being generated on the distribution grid, which, with appropriate application of analytics, may be transformed into intelligible, actionable information that can be provided to the right actors – such as grid and building operators, at the appropriate time to enhance grid or building resilience, efficiency, and operations against various metrics or goals – such as total carbon reduction or other economic benefit to customers. While some basic analysis into these data streams can provide a wealth of information, computational and human boundaries on performing the analysis are becoming significant, with more data and multi-objective concerns. Efficient applications of analysis and the machine learning field are being considered in the loop.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2018

Advanced Data Structures for Improved Cyber Resilience and Awareness in Untrusted Environments: LDRD Report

This report summarizes the work performed under the project "Advanced Data Structures for Improved Cyber Resilience and Awareness in Untrusted Environments." The goal of the project was to design, analyze, and test new data structures for cybersecurity applications. We had two major thrusts: 1) using new/improved write-optimized data structures and/or algorithms to better man- age and analyze high-speed massive-volume cyberstreams, and 2) adding security features to data structures at minimum cost. Write optimization allows data structures to better use secondary memory to store and search a larger amount of useful information. Secondary memory is large compared to main memory, but data movement to and from secondary memory must be carefully managed to run quickly enough to keep up with fast streams. The first thrust included managing cyberstreams in parallel, both multi-threaded and distributed, and improving the benchmarking infrastructure for testing new streaming data structures. We considered both (near) real-time discovery of particular patterns, and improved logging for improved forensics. We considered two kinds of security-feature problem. The first was high-performance history- independent external-memory data structures. These provide certain protections to data if a disk is stolen. We also prove some trade-offs between speed and security in this setting. The second data-security problem is more secure data look-up in secret-shared data bases. This report summarizes the project's major accomplishments, with the background to under- stand these accomplishments. It gathers the abstracts and references for the six refereed publications that have appeared as part of this work. We summarize several accomplishments that will be submitted for publication. We then archive one piece of partial work that is not likely to be published in the near future: validation of history-independent data structure implementations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Evaluation of a streaming readout solution for Jefferson Lab experiments

Jefferson Lab (JLab) operates the CEBAF accelerator that provides a high energy (up to 12 GeV) and high current (up to 150 µA) polarized electron beam for fixed target experiments. JLab's mission is to provide substantial progress in understanding the Quantum Chromo Dynamics (QCD), the current theory of the spectrum and the dynamics of hadronic matter. Given the broad scientific program a general-purpose detector, CLAS12, has been designed and constructed. Such a detector requires a sophisticated trigger and current experiments use an on-line FPGA-based system that relies upon custom firmware and electronics both of which are difficult reconfigure from one experiment to the next. To overcome these challenges transitioning to a streaming readout has been proposed since it would allow a more flexible, easier to debug, software trigger to be developed. To facilitate this JLab is gaining experience with streaming readout systems and is actively working to design and test a streaming data acquisition architecture. Significant work has already been done to make existing front-end electronics compatible with operation in streaming mode. Software-wise, much work remains to be done but an existing framework, TriDAS, which was developed for underwater neutrino telescopes can, in principle, be used with the existing JLab hardware. A test stand is already in place in the INDRA DAQ development lab at JLab. This paper will present the development of this TriDAS based test system and it?s operation in-beam using parts of the CLAS12 detector to evaluate streaming readout for a large general purpose detector.

Battaglieri, Marco↗
NASA NTRS1976

Definition and fabrication of an airborne scatterometer radar signal processor

A hardware/software system which incorporates a microprocessor design and software for the calculation of normalized radar cross section in real time was developed. Interface is provided to decommutate the NASA ADAS data stream for aircraft parameters used in processing and to provide output in the form of strip chart and pcm compatible data recording.

Source record↗
NASA NTRS1985

User data management

The primary objective is to identify, develop, and demonstrate key data management technologies to support user access to Space Station data. To accomplish this objective, there are several technical challenges which must be addressed. The first is how to provide routine customer access to high volume, dynamic and distributed data bases. This access will encompass the functions of mission and payload planning and operations, data processing and analysis, and data archive and distribution. Secondly, there must be some analysis of architectures for handling high volume data streams like those expected from the Space Station. This analysis will examine the use of packetized versus non-packetized data formats, modular expansion capabilities, real-time versus non-real-time data processing, and the interfaces and architecture required to support telescience operations. The task will also determine benchmarks of performance capabilities for technology operations, such as varied data base structures, data access procedures, distributed data base design, and data base machines. Information is provided here in outline form.

Source record↗
NASA NTRSDate not supplied

Do Better Satellite Precipitation Algorithms Improve Landslide Hazard Assessment?

Satellites make it possible to estimate precipitation in near real time. Given the challenges of achieving global coverage by other means, these data are used widely. However, few systems for landslide hazard assessment rely on satellite precipitation estimates. This could be due in part to perceptions of accuracy, although latency, spatial resolution, and other factors may also be important. We test whether recent changes to data streams from the Global Precipitation Measurement mission (GPM) have improved its potential for use in landslide prediction. Specifically, we examine data produced by the Integrated Multi-satellitERetrievals for the GPM (IMERG) algorithm, which was upgraded to version 7 this year. IMERG relies upon other algorithms, including the Goddard Profiling Algorithm (GPROF) and the GPM Combined Radar-Radiometer Algorithm (CORRA). Many changes have been made during the switch from IMERG version 6 to version 7. These include upgrading CORRA and GPROF to version 7, to improve the accuracy of precipitation in frozen, mountainous, and coastal areas. The measured intensity of some storms has been enhanced with a new algorithm, the Scheme for Histogram Adjustment with Ranked Precipitation Estimates in the Neighborhood. Combined with many others, these changes to IMERG should improve its utility for landslide hazard assessment in a variety of contexts. To test this idea, we retrain the global Landslide Hazard Assessment for Situational Awareness (LHASA) model twice—first with data from IMERG version 6B and second with 7B. Since current daily rainfall is the most important variable in determining outcomes predicted by LHASA, it should reflect changes made to that input. First, we grid the landslides at a daily, thirty-arcsecond resolution. This serves as the response variable. At each of these sites current and antecedent rainfall are extracted, along with antecedent snow mass and soil moisture, slope, and PGA. In addition, one million grid cells are selected at random points to represent conditions under which landslides (probably) do not occur. After merging these data, we hold back 20% of the dataset for validation purposes and train a machine-learning model with the rest. We assess both the model’s overall ability to identify landslides and its ability to predict specific large landslide disasters.

Thomas A Stanley↗
DOE OSTI2025

Redis-Based Streaming Architecture for Accelerator Beam Instrumentation DAQ Systems

The Fermilab Acceleraor Division, Beam Instrumentation Department, is always adopting modern and current software methodologies for complex DAQ architectures. This paper highlights the Redis Adapter (RA) as the key software component enabling high performance, modular communication between digitizers and distributed control systems by leveraging Redis and containerization. The RA provides a unified, efficient interface between Redis based data streams and consumer systems. In the legacy architecture, digitized data flowed through the custom, UDP based Distributed Data Communication Protocol in the middle layer. In the current system, DDCP remains the ingestion path, while the RA serves as the decoupling layer. The proposed system replaces old VME digitizers with a SOM-based digitizer that communicates with Redis using the RA. The RA acts as both a performance-critical bridge and a protocol-agnostic adapter, ensuring compatibility with legacy control frameworks while enabling future scalability and modularity. This restructuring of the middle layer also helps the system achieve high throughput, reduce latency, and simplify the data path. Finally, we will demonstrate how RA is utilized in our two core products to deliver both legacy compatibility and future flexibility.

Joshi, S. [Fermilab]↗
DOE OSTI2022

Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline

Recent advances in (scanning) transmission electron microscopy have enabled a routine generation of large volumes of high-veracity structural data on 2D and 3D materials, naturally offering the challenge of using these as starting inputs for atomistic simulations. In this fashion, the theory will address experimentally emerging structures, as opposed to the full range of theoretically possible atomic configurations. However, this challenge is highly nontrivial due to the extreme disparity between intrinsic timescales accessible to modern simulations and microscopy, as well as latencies of microscopy and simulations per se. Addressing this issue requires as a first step bridging the instrumental data flow and physics-based simulation environment, to enable the selection of regions of interest and exploring them using physical simulations. Here we report the development of the machine learning workflow that directly bridges the instrument data stream into Python-based molecular dynamics and density functional theory environments using pre-trained neural networks to convert imaging data to physical descriptors. Additionally, the pathways to ensure structural stability and compensate for the observational biases universally present in the data are identified in the workflow. This approach is used for a graphene system to reconstruct optimized geometry and simulate temperature-dependent dynamics including adsorption of Cr as an ad-atom and graphene healing effects. However, it is universal and can be used for other material systems.

36 MATERIALS SCIENCE↗
NASA NTRS1985

Extended testing of a general contextual classifier using the massively parallel processor - Preliminary results and test plans

Earlier encouraging test results of a contextual classifier that combines spatial and spectral information employing a general statistical approach are expanded. The earlier results were of limited meaning because they were produced from small (50-by-50 pixel) data sets. An implementation of the contextual classifier on NASA Goddard's Massively Parallel Processor (MPP) is presented; for the first time the MPP makes feasible the testing of the classifier on large data sets (a 12-hour test on a VAX-11/780 minicomputer now takes 5 minutes on the MPP). The MPP is a Single-Instruction, Multiple Data Stream computer, consisting of 16,384 bit serial microprocessors connected in a 128-by-128 mesh array with each element having data transfer connections with its four nearest neighbors so that the MPP is capable of billions of operations per second. Preliminary results are given (with more expected for the conference) and plans are mentioned for extended testing of the contextual classifier on Thematic Mapper data sets.

Tilton, J. C.↗
NASA NTRS2012

The Atacama Cosmology Telescope: Data Characterization and Map Making

We present a description of the data reduction and mapmaking pipeline used for the 2008 observing season of the Atacama Cosmology Telescope (ACT). The data presented here at 148 GHz represent 12% or the 90 TB collected by ACT from 2007 to 2010. In 2008 we observed for 136 days, producing a total of 142h of data (11 TB for the 148 GHz band only), with a daily average of 10.5 h of observation. From these, 108.5 h were devoted to 850 sq deg stripe (11.2 h by 9 deg.1) centered on a declination of -52 deg.7, while 175 h were devoted to a 280 square deg stripe (4.5 h by 4 deg.8) centered at the celestial equator. We discuss sources of statistical and systematic noise, calibration, telescope pointing and data selection. Out of 1260 survey hours and 1024 detectors per array, 816 h and 593 effective detectors remain after data selection for this frequency band, yielding a 38 % survey efficiency. The total sensitivity in 2008, determined from the noise level between 5 Hz and 20 Hz in the time-ordered data stream (TOD), is 32 muK square root of s in CMB units. Atmospheric brightness fluctuations constitute the main contaminant in the data and dominate the detector and noise covariance at low frequencies in the TOD. The maps were made by solving the lease squares problem using the Preconditioned Conjugate Gradient method, incorporating the details of the detector and noise correlations. Cross-correlation with WMAP sky maps as well as analysis from simulations reveal the our maps are unbiased at l > 300. This paper accompanies the public release of the 148 GHz southern stripe maps from 2008. The techniques described here will be applied to future maps and data releases.

Duenner, Rolando↗
NASA NTRS2013

Suomi NPP VIIRS Prelaunch and On-orbit Geometric Calibration and Characterization

The Visible Infrared Imager Radiometer Suite (VIIRS) sensor was launched 28 October 2011 on the Suomi National Polarorbiting Partnership (SNPP) satellite. VIIRS has 22 spectral bands covering the spectrum between 0.412 m and 12.01 m, including 16 moderate resolution bands (M-bands) with a spatial resolution of 750 m at nadir, 5 imaging resolution bands (I-bands) with a spatial resolution of 375 m at nadir, and 1 day-night band (DNB) with a near-constant 750 m spatial resolution throughout the scan. These bands are located in a visible and near infrared (VisNIR) focal plane assembly (FPA), a short- and mid-wave infrared (SWMWIR) FPA and a long-wave infrared (LWIR) FPA. All bands, except the DNB, are co-registered for proper environmental data records (EDRs) retrievals. Observations from VIIRS instrument provide long-term measurements of biogeophysical variables for climate research and polar satellite data stream for the operational communitys use in weather forecasting and disaster relief and other applications. Well Earth-located (geolocated) instrument data is important to retrieving accurate biogeophysical variables. This paper describes prelaunch pointing and alignment measurements, and the two sets of on-orbit correction of geolocation errors, the first of which corrected error from 1,300 m to within 75 m (20 I-band pixel size), and the second of which fine tuned scan angle dependent errors, bringing VIIRS geolocation products to high maturity in one and a half years of the SNPP VIIRS on-orbit operations. Prelaunch calibration and the on-orbit characterization of sensor spatial impulse responses and band-to-band co-registration (BBR) are also described.

Geolocation↗
NASA NTRS2015

Preliminary Findings of Inflight Icing Field Test to Support Icing Remote Sensing Technology Assessment

NASA and the National Center for Atmospheric Research have developed an icing remote sensing technology that has demonstrated skill at detecting and classifying icing hazards in a vertical column above an instrumented ground station. This technology has recently been extended to provide volumetric coverage surrounding an airport. Building on the existing vertical pointing system, the new method for providing volumetric coverage will utilize a vertical pointing cloud radar, a multifrequency microwave radiometer with azimuth and elevation pointing, and a NEXRAD radar. The new terminal area icing remote sensing system processes the data streams from these instruments to derive temperature, liquid water content, and cloud droplet size for each examined point in space. These data are then combined to ultimately provide icing hazard classification along defined approach paths into an airport.

remote sensors↗
NASA NTRS2016

Analysis of 2015 Winter In-Flight Icing Case Studies with Ground-Based Remote Sensing Systems Compared to In-Situ SLW Sondes

National Aeronautics and Space Administration (NASA) and the National Center for Atmospheric Research (NCAR) have developed an icing remote sensing technology that has demonstrated skill at detecting and classifying icing hazards in a vertical column above an instrumented ground station. This technology has recently been extended to provide volumetric coverage surrounding an airport. Building on the existing vertical pointing system, the new method for providing volumetric coverage utilizes a vertical pointing cloud radar, a multi-frequency microwave radiometer with azimuth and elevation pointing, and a NEXRAD radar. The new terminal area icing remote sensing system processes the data streams from these instruments to derive temperature, liquid water content, and cloud droplet size for each examined point in space. These data are then combined to ultimately provide icing hazard classification along defined approach paths into an airport. To date, statistical comparisons of the vertical profiling technology have been made to Pilot Reports and Icing Forecast Products. With the extension into relatively large area coverage and the output of microphysical properties in addition to icing severity, the use of these comparators is not appropriate and a more rigorous assessment is required. NASA conducted a field campaign during the early months of 2015 to develop a database to enable the assessment of the new terminal area icing remote sensing system and further refinement of terminal area icing weather information technologies in general. In addition to the ground-based remote sensors listed earlier, in-situ icing environment measurements by weather balloons were performed to produce a comprehensive comparison database. Balloon data gathered consisted of temperature, humidity, pressure, super-cooled liquid water content, and 3-D position with time. Comparison data plots of weather balloon and remote measurements, weather balloon flight paths, bulk comparisons of integrated liquid water content and icing cloud extent agreement, and terminal-area hazard displays are presented. Discussions of agreement quality and paths for future development are also included.

aviation meteorology↗
NASA NTRS2019

Fuser and Fuser in the Cloud

The ˜Fuser' is the name of the SWIM data integration service that has been built, utilized and evolved on ATD-2 to synchronize trajectory predictions from multiple decision support tools. NASA is researching ways to make this data stream available for broader consumption as an example implementation that would help reduce the burden for consumers while enabling access to data that would help industry prepare and innovate for TFDM (e.g. integrates TTP feed).

Fuser↗