Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,171 records · Page 65
All sourcesNASA NTRSDOE OSTI
NASA NTRS1995

An empirical determination of the polytropic index for the free-streaming solar wind using Helios 1 data

Observations of solar wind proton temperatures indicate that the solar wind is heated as it moves outward toward the orbit of Earth. This heating, which may be the results of electron heat conduction and perhaps MHD waves, has proven difficult to quantify and hence is often neglected in MHD models of the solar wind. An alternate approach to finding explicit heating terms for the MHD energy equation is to use a polytropic approximation. This paper discusses the properties of the polytropic approximation and its application to the solar wind plasma. By using data from the Helios 1 spacecraft, an empirical value for the polytropic index of the free-streaming solar wind is determined. Various corrections to the data are made to account for velocity, nonuniformity in radial sampling, and stream interaction regions. The polytropic index, as derived from proton data, is found to indepedent of speed state, within statistical error, and has an average value of 1.46. If magnetic pressure is included, the polytropic index has an average value of 1.58.

Totten, T. L.↗
DOE OSTI2023

STREAM: A Scalable Federated HPC Telemetry Platform

Obtaining and analyzing high performance computing (HPC) telemetry in real time is a complex task that can impact algo- rithmic performance, operating costs, and ultimately scientific outcomes. If your organization operates multiple HPC systems, filesystems, and clusters, telemetry streams can be synthesized in order to ease operational and analytics burden. In order to collect this telemetry, the Oak Ridge Leadership Computing Facility (OLCF) has deployed STREAM (Streaming Telemetry for Resource Events, Analytics, and Monitoring), which is a distributed and high-performance message bus based on Apache Kafka. STREAM collects center-wide performance information and must interface with many sources, including five HPE deployed supercomputers, each with their own Kafka cluster which is managed by HPCM. OLCF Supercomputers and their attached scratch filesystems currently send more than 300 million messages to over 200 topics producing around 1.3 Terabytes per day of telemetry data to STREAM. This paper describes the architectural principles that enable STREAM to be both resilient and highly performant while supporting multiple upstream Kafka clusters and other data sources. It also discusses the design challenges and decisions faced in adapting our existing system- monitoring infrastructure to support the first Exascale computing platform.

Adamson, Ryan↗
DOE OSTI2019

One SQL to Rule Them All - an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables

Real-time data analysis and management are increasingly critical for today's businesses. SQL is the de facto lingua franca for these endeavors, yet support for robust streaming analysis and management with SQL remains limited. Many approaches restrict semantics to a reduced subset of features and/or require a suite of non-standard constructs. Additionally, use of event timestamps to provide native support for analyzing events according to when they actually occurred is not pervasive, and often comes with important limitations. We present a three-part proposal for integrating robust streaming into SQL, namely: (1) time-varying relations as a foundation for classical tables as well as streaming data, (2) event time semantics, (3) a limited set of optional keyword extensions to control the materialization of time-varying query results. We show how with these minimal additions it is possible to utilize the complete suite of standard SQL semantics to perform robust stream processing. We motivate and illustrate these concepts using examples and describe lessons learned from implementations in Apache Calcite, Apache Flink, and Apache Beam. We conclude with syntax and semantics of a concrete proposal for extensions of the SQL standard and note further areas of exploration.

Begoli, Edmon↗
DOE OSTI2017

Exploiting analytics techniques in CMS computing monitoring

The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS1986

Coding for Efficient Image Transmission

NASA publication second in series on data-coding techniques for noiseless channels. Techniques used even in noisy channels, provided data further processed with Reed-Solomon or other error-correcting code. Techniques discussed in context of transmission of monochrome imagery from Voyager II spacecraft but applicable to other streams of data. Objective of this type coding to "compress" data; that is, to transmit using as few bits as possible by omitting as much as possible of portion of information repeated in subsequent samples (or picture elements).

Rice, R. F.↗
DOE OSTI2020

Performance and usability enhancements for continuous subgraph matching queries on graph-structured data

A query graph, which includes vertices and edges, represents a query on graph-structured data. The query graph is decomposed into query subgraphs. A network analysis tool performs continuous subgraph matching queries to facilitate analysis of computer network traffic, social media events, or other streams of data represented as a dynamic data graph (graph-structured data). This can help identify emerging trends in the data. Some features of the network analysis tool enhance performance by effectively utilizing distributed computing resources (including processing cores and memory at different nodes of a cluster) to speed up the process of updating the dynamic data graph and detecting matches of query subgraphs. Features of a query graph building tool enhance usability by providing intuitive ways to specify query graphs and their subgraphs. Features of a results visualization tool enhance usability by providing an intuitive way to present the results of continuous subgraph matching queries.

Choudhury, Sutanay↗
DOE OSTI2021

via machinae : Searching for stellar streams using unsupervised machine learning

ABSTRACT We develop a new machine learning algorithm, via machinae, to identify cold stellar streams in data from the Gaia telescope. via machinae is based on ANODE, a general method that uses conditional density estimation and sideband interpolation to detect local overdensities in the data in a model agnostic way. By applying ANODE to the positions, proper motions, and photometry of stars observed by Gaia, via machinae obtains a collection of those stars deemed most likely to belong to a stellar stream. We further apply an automated line-finding method based on the Hough transform to search for line-like features in patches of the sky. In this paper, we describe the via machinae algorithm in detail and demonstrate our approach on the prominent stream GD-1. Though some parts of the algorithm are tuned to increase sensitivity to cold streams, the via machinae technique itself does not rely on astrophysical assumptions, such as the potential of the Milky Way or stellar isochrones. This flexibility suggests that it may have further applications in identifying other anomalous structures within the Gaia data set, for example debris flow and globular clusters.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2007

Infrared Imaging System for Studying Brain Function

A proposed special-purpose infrared imaging system would be a compact, portable, less-expensive alternative to functional magnetic resonance imaging (fMRI) systems heretofore used to study brain function. Whereas a typical fMRI system fills a large room, and must be magnetically isolated, this system would fit into a bicycle helmet. The system would include an assembly that would be mounted inside the padding in a modified bicycle helmet or other suitable headgear. The assembly would include newly designed infrared photodetectors and data-acquisition circuits on integrated-circuit chips on low-thermal-conductivity supports in evacuated housings (see figure) arranged in multiple rows and columns that would define image coordinates. Each housing would be spring-loaded against the wearer s head. The chips would be cooled by a small Stirling Engine mounted contiguous to, but thermally isolated from, the portions of the assembly in thermal contact with the wearer s head. Flexible wires or cables for transmitting data from the aforementioned chips would be routed to an integrated, multichannel transmitter and thence through the top of the assembly to a patch antenna on the outside of the helmet. The multiple streams of data from the infrared-detector chips would be sent to a remote site, where they would be processed, by software, into a three-dimensional display of evoked potentials that would represent firing neuronal bundles and thereby indicate locations of neuronal activity associated with mental or physical activity. The 3D images will be analogous to current fMRI images. The data would also be made available, in real-time, for comparison with data in local or internationally accessible relational databases that already exist in universities and research centers. Hence, this system could be used in research on, and for the diagnosis of response from the wearer s brain to physiological, psychological, and environmental changes in real time. The images would also be stored in a relational database for comparison with corresponding responses previously observed in other subjects.

Mintz, Frederick↗
NASA NTRS1998

LITE data processing

The Lidar In-space Technology Experiment (LITE) was a payload onboard the Space Shuttle Discovery as part of the September 1994 STS-64 mission. LITE successfully made both atmospheric and surface measurements at the 355 nm, 532 nm, and 1064 nm wavelengths [Winker et al., 1996]. At mission completion, nearly 45 Gbytes of data had been obtained. As the data were collected they were organized into three data sets. The largest of these data sets contained the single-shot full resolution digitized lidar signals for all three wavelengths. The other two data sets contained the instrument status data block (ISDB) and the quick look science data (QLSD). ISDB information was generated once per second and contained timing parameters and engineering data related to the health, status, and configuration of the instrument. The QLSD are 100shot averages of the 355 nm and 532 nm lidar signals and were generated at ten second intervals. As the LITE data were generated, they were output to low-rate and high-rate telemetry streams. The low-rate data contained the ISDB and QLSD. The lowrate data were transmitted to the ground by the Shuttle S-band system through the Tracking and Data Relay Satellite System (TDRSS). The high-rate data contained the full resolution digitized lidar signals for all three wavelengths, the ISDB, and QLSD. The high-rate data were transmitted in real time by the Ku-band system through TDRSS downlink to the LITE operations center at the Johnson Space Center (JSC). High-rate data were only obtained when the orbiter and TDRSS telemetry link coincided with the real-time operations of LITE. The low-rate data were backed up by a Shuttle recorder during periods when the telemetry link was unavailable, and the entire lowrate data stream was transferred to the ground over the course of the mission. A total of 53 hours of lowrate data and 45.5 hours of high-rate data were acquired. All of the LITE raw telemetry data have been archived at the NASA Langley Research Center (LaRC), and the LITE high-rate data have been processed to the level 1 data product. This paper describes the data processing steps required to convert the LITE raw, high-rate telemetry data to the LITE level 1 data product.

Powell, Kathleen A.↗
DOE OSTI2024

A machine learning decision criterion for reducing scan time for hyperspectral neutron computed tomography systems

We present the first machine learning-based autonomous hyperspectral neutron computed tomography experiment performed at the Spallation Neutron Source. Hyperspectral neutron computed tomography allows the characterization of samples by enabling the reconstruction of crystallographic information and elemental/isotopic composition of objects relevant to materials science. High quality reconstructions using traditional algorithms such as the filtered back projection require a high signal-to-noise ratio across a wide wavelength range combined with a large number of projections. This results in scan times of several days to acquire hundreds of hyperspectral projections, during which end users have minimal feedback. To address these challenges, a golden ratio scanning protocol combined with model-based image reconstruction algorithms have been proposed. This novel approach enables high quality real-time reconstructions from streaming experimental data, thus providing feedback to users, while requiring fewer yet a fixed number of projections compared to the filtered back projection method. In this paper, we propose a novel machine learning criterion that can terminate a streaming neutron tomography scan once sufficient information is obtained based on the current set of measurements. Our decision criterion uses a quality score which combines a reference-free image quality metric computed using a pre-trained deep neural network with a metric that measures differences between consecutive reconstructions. The results show that our method can reduce the measurement time by approximately a factor of five compared to a baseline method based on filtered back projection for the samples we studied while automatically terminating the scans.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Quantifying the Role of Snowmelt in Stream Discharge in an Alaskan Watershed: An Analysis Using a Spatially Distributed Surface Hydrology Model

Here, this study uses a spatially distributed surface hydrology model to investigate the role of snowmelt in stream discharge for the Tanana Flats Basin in interior Alaska. The Parameter ESTimation code is used to calibrate the model with observed stream discharge data. The model was further evaluated using remote sensing-based snow cover product and in situ snowpack water equivalent (SWE) observations. A 36 year (1980–2015) U.S. Geological Survey Precipitation-Runoff Modeling System simulation shows (1) the monthly stream discharge from the Tanana Flats Basin in April decreased by 44%; (2) snow cover area at high altitudes (above 2000 m) decreased in summer, both SWE and snowmelt also decreased significantly, especially in spring; (3) the timings of snowmelt onset and ending shifted by 2 (earlier) and 5 (later) days per decade, respectively; and (4) snowmelt accounts for 40% of the annual stream discharge. This study provides a quantitative tool to investigating hydrological systems considering the impacts of snow dynamics in cold regions. Additionally, this study suggests that future warming will further decrease snow coverage, advance snow melting time, and hereafter change the stream discharge dynamics in the Arctic.

58 GEOSCIENCES↗
DOE OSTI2026

Large-scale real-time signal processing in physics experiments: the ALICE TPC FPGA pipeline

For LHC Run 3, the ALICE Time Projection Chamber was upgraded to operate in continuous readout mode. Interaction rates of up to 50 kHz in Pb-Pb collisions require real-time processing of more than 3 TB s -1 of raw detector data. This requirement is met by a custom FPGA-based processing pipeline that performs the complete front-end data treatment fully in-stream, including common-mode correction, pedestal subtraction, ion-tail filtering, zero suppression, and dense data packing. A central element of the design is a highly parallel common-mode correction algorithm operating directly on the streaming data. It robustly identifies signal-free readout channels on a time-bin basis and applies pad-dependent scaling to compensate for local variations in capacitive coupling in the GEM readout. In combination with pedestal subtraction and ion-tail filtering, this enables accurate baseline restoration under extreme high-occupancy conditions, preventing signal loss while efficiently suppressing noise prior to zero suppression. The pipeline operates continuously at the full detector bandwidth and reduces the raw input rate of approximately 3 TB s -1 to about 900 GBps for Pb-Pb collisions at the target interaction rate. Overall, it represents a large-scale FPGA-based real-time signal-processing implementation for high-energy physics detector readout.

Digital signal processing (DSP)↗
DOE OSTI2024

Public Data Set: Effects of Injected Current Streams on MHD Equilibrium Reconstruction of Local Helicity Injection Plasmas in a Spherical Tokamak

This public data set contains openly-documented, machine readable digital research data corresponding to figures published in J.D. Weberski et al., 'Effects of Injected Current Streams on MHD Equilibrium Reconstruction of Local Helicity Injection Plasmas in a Spherical Tokamak,' Journal of Fusion Energy 43, 72 (2024).

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Validation of non-negative matrix factorization for rapid assessment of large sets of atomic pair distribution function data

The use of the non-negative matrix factorization (NMF) technique is validated for automatically extracting physically relevant components from atomic pair distribution function (PDF) data from time-series data such as in situ experiments. The use of two matrix-factorization techniques, principal component analysis and NMF, on PDF data is compared in the context of a chemical synthesis reaction taking place in a synchrotron beam, applying the approach to synthetic data where the correct composition is known and on measured PDFs from previously published experimental data. The NMF approach yields mathematical components that are very close to the PDFs of the chemical components of the system and a time evolution of the weights that closely follows the ground truth. Lastly, it is discussed how this would appear in a streaming context if the analysis were being carried out at the beamline as the experiment progressed.

36 MATERIALS SCIENCE↗
NASA NTRS2011

Snow Cover, Snowmelt Timing and Stream Power in the Wind River Range, Wyoming

Earlier onset of springtime weather, including earlier snowmelt, has been documented in the western United States over at least the last 50 years. Because the majority (is greater than 70%) of the water supply in the western U.S. comes from snowmelt, analysis of the declining spring snowpack (and shrinking glaciers) has important implications for the management of streamflow. The amount of water in a snowpack influences stream discharge which can also influence erosion and sediment transport by changing stream power, or the rate at which a stream can do work, such as move sediment and erode the stream bed. The focus of this work is the Wind River Range (WRR) in west-central Wyoming. Ten years of Moderate-Resolution Imaging Spectroradiometer (MODIS) snow-cover, cloud-gap-filled (CGF) map products and 30 years of discharge and meteorological station data are studied. Streamflow data from streams in WRR drainage basins show lower annual discharge and earlier snowmelt in the decade of the 2000s than in the previous three decades, though no trend of either lower streamflow or earlier snowmelt was observed within the decade of the 2000s. Results show a statistically-significant trend at the 95% confidence level (or higher) of increasing weekly maximum air temperature (for three out of the five meteorological stations studied) in the decade of the 1970s, and also for the 40-year study period as a whole. The extent of snow-cover (percent of basin covered) derived from the lowest elevation zone (2500-3000 m) of the WRR, using MODIS CGF snow-cover maps, is strongly correlated with maximum monthly discharge on 30 April, where Spearman's Rank correlation, rs,=0.89 for the decade of the 2000s. We also investigated stream power for Bull Lake Creek above Bull Lake; and found a trend (significant at the 90% confidence level) toward reduced stream power from 1970 to 2009. Observed changes in streamflow and stream power may be related to increasing weekly maximum air temperature measured during the 40-year study period, possibly contributing to a reduction in snow cover. In addition, the strong relationship between percent of basin that was snow covered, and maximum monthly streamflow indicates that MODIS snow-cover maps are useful for predicting streamflow, and can be used to improve management of water resources in the drought-prone western United States.

MODIS↗
NASA NTRS2010

The Wettzell System Monitoring Concept and First Realizations

Automated monitoring of operational system parameters for the geodetic space techniques is becoming more important in order to improve the geodetic data and to ensure the safety and stability of automatic and remote-controlled observations. Therefore, the Wettzell group has developed the system monitoring software, SysMon, which is based on a reliable, remotely-controllable hardware/software realization. A multi-layered data logging system based on a fanless, robust industrial PC with an internal database system is used to collect data from several external, serial, bus, or PCI-based sensors. The internal communication is realized with Remote Procedure Calls (RPC) and uses generative programming with the interface software generator idl2rpc.pl developed at Wettzell. Each data monitoring stream can be configured individually via configuration files to define the logging rates or analog-digital-conversion parameters. First realizations are currently installed at the new laser ranging system at Wettzell to address safety issues and at the VLBI station O Higgins as a meteorological data logger. The system monitoring concept should be realized for the Wettzell radio telescope in the near future.

Ettl, Martin↗
NASA NTRS1972

An experimental investigation of a cold jet emitting from a body of revolution into a subsonic free stream

An experimental program was undertaken to determine the pressure distribution induced on aerodynamic bodies by a subsonic cold jet exhausting normal to the body surface and into a subsonic free stream. The investigation was limited to two bodies with single exhaust jets a flat plate at zero angle of attack with respect to the free-stream flow and a cylinder, fitted with a conical nose, with the longitudinal axis alined with the free-stream flow. Experimental data were obtained for free-stream velocity to jet velocity ratios between 0.3 and 0.5. The experimental data are presented in tabular form with appropriate graphs to indicate pressure coefficient contours, pressure coefficient decay, pitching-moment characteristics, and lift characteristics.

Ousterhout, D. S.↗
DOE OSTI2018

Industrial Scale Demonstration of Smart Manufacturing, Achieving Transformational Energy Productivity Gains

While many U.S. manufacturing operations utilize optimization for individual unit processes, smart manufacturing (SM) systems that integrate manufacturing intelligence in real time across an entire production operation are rare in large companies and virtually nonexistent in smaller organizations. One example of an area where SM systems can be applied is in the management of waste heat. A smart system that not only seeks to recover waste heat, but also to use energy more efficiently, is a more cost-effective approach because energy use, waste energy loss and product output are optimized together. The overall objective of this project was to develop an industry-accepted SM Platform that enables smart systems by accelerating and lowering the cost of development and implementation scales for use in diverse sets of manufacturing industry sectors, manufacturing operations and company sizes. Specifically, the project (1) designed and demonstrated the application of a prototype SM Platform for two diverse commercial test beds, (2) demonstrated at one and estimated for the other test bed the potential reduction in waste heat generation, and (3) worked with leading automation vendors to catalyze low-cost commercialization of the technology developed. The Smart Manufacturing Application and Data Platform (SM Platform) developed during this project is an innovative approach that marries IT virtualization technologies with operational data and modeling system requirements to construct a ready-to-go platform for enabling sensor – data – modeling – actuation systems, driven by real-time plant data and performance metrics. This SM Platform allows manufacturing organizations, regardless of their industry or size, to assemble new operational systems at a much lower cost, optimizing process knowledge and improving energy productivity by integrating with existing process control and automation systems. Also developed during this project is a methodology for developing energy productivity metrics that use real-time process information and provide comparisons for potential, practical, and actual performance, as well as examples and the methodology for provisioning of an Energy Dashboard that displays alternatives to optimize energy use within specific business contexts aimed at helping improve energy productivity and reduce emissions in U.S. manufacturing. The SM Platform designed utilizes current manufacturing, operational and IT industry standards. The platform can be used to expedite sensor – data – model – actuation system development and deployment in a cloud-based environment using a novel end-to-end workflow as a service capability that is already integrated with data connection, ingestion, and contextualization services. In addition, the methodology for development of energy productivity metrics and reusable application templates called Toolkits have been defined so that they can be tailored to a specific business situation. The project’s two test beds, one in a hydrogen production plant and the other in a forging, heat treatment, and machining operation, utilize sensor-driven modeling, measurement, and simulation systems. In these two test beds, the implementation of these as a comprehensive system has made it possible for energy productivity to be managed in real-time throughout the plant and enterprise using an energy dashboard. The development pathway to direct machine actuation was articulated. During the first year, the project team developed the most highly instrumented steam-methane reformer (SMR) in the U.S. using infrared cameras and thermocouples. Real-time data are streamed to high fidelity and reduced-order models for analysis. A high fidelity CFD model, parallelized in the SM Platform cloud environment, was demonstrated for operational use. This is the first real-time operational productivity system of a steam methane reformer using both high fidelity and data-driven models together in a real time situation to balance natural gas flow to the reformer tubes. An infrared camera system was developed and shown to outperform thermocouples in coverage, accuracy, reliability, and cost effectiveness for use in a high temperature, harsh environment. A forging, heat-treating, machining line operation test bed includes new instrumentation for data capture analysis, modeling, and simulation and will be integrated into SM Platform workflows so that business performance can be managed in real-time. Composable metrics were developed using the SM Platform to drive improvements in energy productivity, environmental performance, safety, asset management, costs, and overall operations. A major focus of the SM systems developed was optimization of energy productivity at the enterprise level, which resulted in reductions in waste heat generation for the two project test beds. Reductions in the waste heat generated will result in greater energy savings than will waste heat recovery due to inefficiencies in the heat transfer process. For the SMR test bed an efficiency improvement of 1 – 2% with a corresponding waste heat reduction of 5% were demonstrated. It should be noted that the test bed SMR unit had an efficiency of more than 90% as compared to 70 to 80% efficiencies of most SMR units. So implementation of similar SM systems at typical SMR units could yield more than 15 to 25% waste heat reductions. Similarly the forging, heat-treating, machining line operation test bed is a DOE AMO Platinum Level Superior Energy Performance plant with a much higher than typical overall operating efficiency. For this test bed heat integration using recuperators is estimated to result in fuel savings of 15.9% due to waste heat recovery compared to the base case without recuperators with negligible changes in part exit conditions. This results in an 18.9% energy efficiency improvement. The project team estimates that broad multi-sector adoption of the developed technology could reduce waste heat generation by an amount equivalent to 1.3% of total U.S. energy use. This waste heat reduction is associated with 69 million tons reduction in annual carbon dioxide emissions. Other potential benefits include reducing process times, solid and liquid wastes, environmental impacts, and water use. It is estimated that this investment has an anticipated payback period of less than one year in energy-intensive applications. The SM Platform could also reduce the costs of deploying SM systems by 50% in manufacturing operations. Commercialization of the SM Platform will require significant cross-industry collaboration and technology provider market changes. Steps taken toward commercialization included 1) analyzing savings from deployment of the SM Platform and 2) holding SM Platform workshops, webinars, and conferences with targeted individual industry segments, with a focus on small and medium-sized organizations. An upgraded community website has been established as a portal for accessing and distributing information. Most importantly, the SM platform enabling technology developed during this project and its commercialization will continue, evolve and expand under the Clean Energy Smart Manufacturing Innovation Institute (CESMII).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗