Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Leveraging In-Network Computing and Programmable Switches for Streaming Analysis of Scientific Data

With the emergence of programmable network devices that match the performance of fixed function devices, several recent projects have explored in-network computing, where the processing that is traditionally done outside the network is offloaded to the network devices. In-network computing has typically been applied to network functions (e.g., load balancing, NAT, and DNS), caching, data reduction/aggregation, and coordination/consensus functions. In some cases it has been used to accelerate stream-processing tasks that involve small payloads and simple operations. In this work we focus on leveraging in-network computing for stream processing of scientific datasets with large payloads that require complex operations such as floating-point computations and logarithmic functions. We demonstrate in-network computing for a real-world scientific application performing streaming normalization of a 2-D image from a light source experiment. We discuss the challenges we encountered and potential approaches to address them.

Sankaran, Ganesh↗
NASA NTRS2003

Jet Stream Analysis and Forecast Errors Using GADS Aircraft Observations in the DAO, ECMWF, and NCEP Models

We have utilized an extensive set of independent British Airways flight data recording wind vector and temperature observations (the Global Aircraft Data Set [GADS] archive) in three ways: (a) as an independent check of operational analyses; (b) as an analysis observing system experiment (OSE) as if the GADS observations were available in real time; and (c) as the corresponding forecast simulation experiment applicable to future operational forecasts. Using a 31 day sample (0000 UTC 20 December 2000 through 0000 UTC 20 January 2000) from Winter 2000, we conclude that over the data-dense continental U. S. analyzed jet streaks are too weak by -2% to -5%. Over nearby data-sparse regions of Canada, analyzed jet streaks are too weak by -5% to -9%. The second range provides a limit on the accuracy of current jet streak analyses over the portions of the -85% of the earth's surface that are poorly covered by non-satellite observations. The -5% to -9% range is relevant for the pre-third generation satellite (AIRS, IASI, GIFTS) era.

Cardinali, Carla↗
DOE OSTI2020

Empirical Study of Focus-Plus-Context and Aggregation Techniques for the Visualization of Streaming Data

Analysis of streaming data often involves both real-time monitoring of incoming data as well as contextual awareness of data history. A focus-plus-context approach can support both goals, with variable levels of visual aggregation making it possible to provide a high level of detail for incoming and recent data while providing contextual information about recent history. Visual aggregation reduces data resolution in order to show the context of data over large periods of time within a limited display space. With a controlled experiment, we evaluated the effectiveness of different types of aggregation for four types of stream-analysis tasks. Overall, the results show that a focus-plus-context design has little negative impact on the ability to successfully monitor and analyze streaming data, making it possible to show longer periods of time than other approaches. However, visual aggregation can be problematic for trend recognition tasks. This research demonstrates how the effectiveness of the visualization depends on the specifics of the analysis task.

Ragan, Eric↗
DOE OSTI2019

Biomass Electrochemical Reactor for Upgrading Biorefinery Waste to Industrial Chemicals and Hydrogen

The goal of this project was to develop an electrochemical process for depolymerization of biorefinery lignin to industrial chemicals, most notably as replacements in resins. The project team consisted of Ohio University (lead organization), Hexion, Inc. and Lakehead University. There were six distinct project tasks, consisting of Initial Validation, Development of TiO 2 Nanowire-supported Ni-Co Electrocatalysts, Electrochemical Characterization of TiO 2 Nanowire-supported Ni-Co Electrocatalysts, Development of Electrochemical Flow Reactor for Production of Industrial Chemicals and Hydrogen, Product Formulations and Integrated Biorefinery Analysis. The project was divided into two budget periods. After intermediate validation near the end of Budget Period 1, the project was discontinued. This final report is intended to highlight major project objectives and results. Technical metrics were developed between the project team and the DOE validation team during the initial validation period. Critical technical metrics included conversion of lignin to products and product selectivity. Desired products were low molecular weight aromatic compounds in the 150-300 MW range. We targeted this class of compounds because of their potential to replace petroleum derivatives in resin formulations. Electrochemical oxidation of biorefinery lignin can lead to significant lignin depolymerization and conversion, with >40% of the lignin reacted. However, product stream analysis is extremely difficult. We were not able to adequately quantify the concentration of the primary oxidation products, or to determine with any degree of certainty what the product selectivity was. The project team initially proposed analytical techniques such as GC-MS to identify product distributions. However, due to 1) difficulty in extracting products into an organic phase suitable for GC analysis and 2) the wide range of product compounds with similar structures, GC-MS analysis was difficult to perform. The project team did have some success with NMR spectroscopic analysis, but not enough so to adequately identify product selectivity. By UV-vis spectroscopy and statistical analysis, we were able to confirm achieving the technical milestone for lignin conversion, and achieved at least 40% conversion by the electrochemical oxidation technique. However, we were not able to confirm product selectivity using GC-MS analysis, as originally postulated. Despite these difficulties, we were able to incorporate the product stream into a resin, and observed better resin synthesis behavior using the oxidized lignin versus the unoxidized lignin. However, no resin synthesized with oxidized biorefinery lignin met the commercial standards for resin quality. One potential problem with electrochemical oxidation of biomass is the competing oxygen evolution reaction (OER) at the reactor anode. We observed that, especially at higher anode potentials, the OER can consume up to 50% of the energy input to the system, meaning that much of the energy goes toward generation of an essentially worthless product (O 2 ). However, we also observed that the OER can be avoided almost entirely at lower potentials. These uncertainties made assigning a value to the electrochemical reactor product stream difficult. Any product generated electrochemically would have to be valuable enough to offset the additional cost of electricity incurred, including electricity to run the electrochemical reactor and the additional electricity that must be purchased because some of the lignin (which is currently burned in biorefineries to recover energy) is diverted to the electrochemical reactor. It is not clear at this time whether the product stream has sufficient economic value to offset these additional electricity costs. The two most promising areas for this project in terms of demonstrating economic feasibility are product stream analysis and resin synthesis. In particular, product stream analysis has proven to be difficult. With additional time, we would refine analytical techniques to better understand key product stream characteristics, such as aromatic compound content. Evaluating the product streams in resin synthesis would likely provide more information on the economic value of the product stream. If we could determine that value, then we would be able to better evaluate whether the electrochemical conversion approach is feasible. The details of this work are presented in the following final report. The report is organized into individual tasks for clarity, although all tasks were integrated and many occurred simultaneously.

09 BIOMASS FUELS↗
DOE OSTI2020

Experimental nitrogen and phosphorus enrichment stimulates multiple trophic levels of algal and detrital-based food webs: a global meta-analysis from streams and rivers

Anthropogenic increases in nitrogen (N) and phosphorus (P) concentrations can strongly influence the structure and function of ecosystems. Even though lotic ecosystems receive cumulative inputs of nutrients applied to and deposited on land, no comprehensive assessment has quantified nutrient-enrichment effects within streams and rivers. We conducted a meta-analysis of published studies that experimentally increased concentrations of N and/or P in streams and rivers to examine how enrichment alters ecosystem structure (state: primary producer and consumer biomass and abundance) and function (rate: primary production, leaf breakdown rates, metabolism) at multiple trophic levels (primary producer, microbial heterotroph, primary and secondary consumers, and integrated ecosystem). Our synthesis included 184 studies, 885 experiments, and 3497 biotic responses to nutrient enrichment. We documented widespread increases in organismal biomass and abundance (mean response = +48%) and rates of ecosystem processes (+54%) to enrichment across multiple trophic levels, with no large differences in responses among trophic levels or between autotrophic or heterotrophic food-web pathways. Responses to nutrient enrichment varied with the nutrient added (N, P, or both) depending on rate versus state variable and experiment type, and were greater in flume and whole-stream experiments than in experiments using nutrient-diffusing substrata. Generally, nutrient-enrichment effects also increased with water temperature and light, and decreased under elevated ambient concentrations of inorganic N and/or P. Overall, increased concentrations of N and/or P altered multiple food-web pathways and trophic levels in lotic ecosystems. Finally, our results indicate that preservation or restoration of biodiversity and ecosystem functions of streams and rivers requires management of nutrient inputs and consideration of multiple trophic pathways.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Comprehensive Analysis of Streaming and Shutdown Dose Rate Experiments at JET with ORNL Fusion Neutronics Workflows

Current experimental fusion systems and conceptual designs of fusion pilot plants (FPPs) are growing in complexity and size. Several radiation metrics are crucial to the safe operation of fusion machines, including neutron flux streaming through openings and the shutdown dose rate (SDDR). Most current designs of advanced experimental fusion systems—and the most probable candidates for FPPs—are based on the tokamak concept, which is prone to neutron streaming through the myriad openings needed for diagnostic and support systems. SDDR is caused by decay gamma rays from radionuclides that become activated by neutrons during the operation of a fusion system that use deuterium-deuterium (DD), tritium-tritium, or deuterium-tritium plasma. Because computational tools have become essential for determining these radiation metrics, they must be validated against reliable and applicable experimental data. Experiments at the Joint European Torus (JET) provide a unique source of experimental data for validating computational tools and nuclear data used to determine SDDR and neutron fluxes in streaming-dominated geometries. Here, this paper presents the comprehensive analysis of the high-performance DD JET SDDR, and streaming experiments performed using Oak Ridge National Laboratory (ORNL) fusion workflows. The computational results were compared with experimental results that consist of online SDDR measurements with ionization chambers and neutron fluence streaming measurements using thermoluminescent detectors. The ratio of calculated-to-experimental SDDR values ranges from 0.6 to 2.5, and the streaming results range from 0.5 to 8.0. Future work will include analyzing the JET 2021 DTE2 campaign alongside the integration of the Shift Monte Carlo transport code into all ORNL fusion neutronics workflows.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

GraphTango: A Hybrid Representation Format for Efficient Streaming Graph Updates and Analysis

Abstract Streaming graph processing performs batched updates and analytics on a time-evolving graph. The underlying representation format of the graph largely determines the throughputs of these updates and analytics phases. Existing representation formats usually employ variations of hash tables or adjacency lists. However, a recent study showed that the adjacency-list-based approaches perform poorly on heavy-tailed graphs, and the hash table-based approaches suffer on short-tailed graphs. We propose GraphTango, a hybrid representation format that provides excellent update and analytics throughput regardless of the graph’s degree distribution. GraphTango dynamically switches among three different formats based on a vertex’s degree: (i) Low-degree vertices store the edges directly with the neighborhood metadata, confining accesses to a single cache line, (2) Medium-degree vertices use adjacency lists, and (3) High-degree vertices use hash tables as well as adjacency lists. In this case, the adjacency list provides fast traversal during the analytics phase, while the hash table provides constant-time lookups during the update phase. We further optimized the performance by designing an open-addressing-based hash table that fully utilizes every fetched cache line. In addition, we developed a thread-local lock-free memory pool that allows fast growing/shrinking of the adjacency lists and hash tables in a multi-threaded environment. We evaluated GraphTango with the help of the SAGA-Bench framework and compared it with four other representation formats: Stinger, Degree-aware Robin Hood Hashing, and two adjacency list-based formats with different workload balancing scheme. On average, GraphTango provides 4.5x higher insertion throughput, 3.2x higher deletion throughput, and 1.1x higher analytics throughput over the next best format. Furthermore, we integrated GraphTango with the state-of-the-art graph processing frameworks DZiG and RisGraph. Compared to the vanilla DZiG and vanilla RisGraph , [ GraphTango + DZiG ] and [ GraphTango + RisGraph ] reduces the average batch processing time by 2.3x and 1.5x, respectively.

Ahmed, Alif↗
NASA NTRS1974

Optical data processing analysis of stream patterns exhibited on ERTS-1 imagery

The purpose of the investigation described in this paper has been the analysis of large-scale geologic ground patterns in Kansas using ERTS-1 imagery and optical data processing techniques. Optical spatial frequency data provide a repeatable quantitative means of specifying ground pattern characteristics.

Egbert, D.↗
DOE OSTI2024

Data and scripts associated with a manuscript on a meta-analysis synthesizing stream biogeochemical response to wildfires across space and time (v2)

This data package is associated with the publication “Catchment characteristics modulate the influence of wildfires on nitrate and dissolved organic carbon in lotic systems across space and time: A meta-analysis” submitted to Global Biogeochemical Cycles (Cavaiani et al. 2025). This study uses meta-analytical techniques to evaluate the effect of wildfire on in-stream responses in burned and unburned watersheds. The study aims to provide additional insight into the range of responses and net influences that wildfires have on hydro-biogeochemistry across broad spatial scales, burn extents, and the persistence of water-quality change. This study compiles data and metadata from 18 total publications that includes 1) surface water geochemistry data (dissolved organic carbon; nitrate), 2) climate classifications, 3) year of the wildfire, 4) the time lag between when the fire occurred and when the sampling occurred, and 5) study design of the publication. In total, this meta-analysis draws data that spans 8 climate guilds, 3 biomes, 62 watersheds, and 20 unique wildfires. See Sites_meta_data.csv for citations of the papers used in this meta-analysis. All R scripts and the associated data can also be found on GitHub at This data package was originally published in March 2024. It was updated in April 2025 (v2; new and modified files). See the change history section in the readme for more details. This data package contains five primary folders that include the following: (1) inputs; (2) output for analysis; (3) initial plots; (4) R scripts; and (5) GIS data. The data package also contains a data dictionary (dd) that provides column header definitions and a file-level metadata (flmd) file that describes every file. The “inputs” folder contains a list of all publications identified during the formal web search and an indication of whether each publication was included in the final analysis. Additionally, it includes site-level metadata, catchment characteristics, and GIS data for all publications included in the final analysis. The “Output_for_analysis” folder contains all data frames and figures generated from each R script used for additional data analysis. The “initial_plots” folder includes all exploratory figures that will be included in a supplemental and figures that will be submitted with the manuscript for publication. The “R_scripts” folder contains the scripts that perform all the data manipulations, statistical analyses, and plots. The “gis_data” folder includes shape files for each fire included in this meta-analysis. This data package contains the following file types: csv, pdf, jpeg, cpg, dbf, prj, shp, shp.ea.iso.xml, shp.iso.xml, shx.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Phoenix: A Scalable Streaming Hypergraph Analysis Framework

We present Phoenix, a scalable hypergraph analytics framework for data analytics and knowledge discovery that was implemented on the leadership class computing platforms at Oak Ridge National Laboratory (ORNL). Our software framework comprises a distributed implementation of a streaming server architecture which acts as a gateway for various hypergraph generators/external sources to connect. Phoenix has the capability to utilize diverse hypergraph generators, including HyGen, a very large-scale hypergraph generator developed by ORNL. Phoenix incorporates specific algorithms for efficient data representation by exploiting hidden structures of the hypergraphs. Our experimental results demonstrate Phoenix’s scalable and stable performance on massively parallel computing platforms. Phoenix’s superior performance is due to the merging of high-performance computing with data analytic.

Kurte, Kuldeep↗
NASA NTRS1973

Stream network analysis from orbital and suborbital imagery, Colorado River Basin, Texas

The author has identified the following significant results. Orbital SL-2 imagery (earth terrain camera S-190B), received September 5, 1973, was subjected to quantitative network analysis and compared to 7.5 minute topographic mapping (scale: 1/24,000) and U.S.D.A. conventional black and white aerial photography (scale: 1/22,200). Results can only be considered suggestive because detail on the SL-2 imagery was badly obscured by heavy cloud cover. The upper Bee Creek basin was chosen for analysis because it appeared in a relatively cloud-free portion of the orbital imagery. Drainage maps were drawn from the three sources digitized into a computer-compatible format, and analyzed by the WATER system computer program. Even at its small scale (1/172,000) and with bad haze the orbital photo showed much drainage detail. The contour-like character of the Glen Rose Formation's resistant limestone units allowed channel definition. The errors in pattern recognition can be attributed to local areas of dense vegetation and to other areas of very high albedo caused by surficial exposure of caliche. The latter effect caused particular difficulty in the determination of drainage divides.

Baker, V. R.↗