Search NASA⌕ Search

SEARCH · Search NASA

Results for “data flow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Processing Particle Data Flows with SmartNICs

Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freeing the hosts to perform other work. System architects in this space face multiple questions about the best way to leverage SmartNICs as processing elements in data flows. In this paper, we advocate the use of Apache Arrow as a foundation for implementing data-flow tasks on SmartNICs. We report on our experiences adapting a partitioning algorithm for particle data to Apache Arrow and measure the on-card processing performance for the BlueField-2 SmartNIC. Our experiments confirm that the BlueField-2’s (de)compression hardware can have a significant impact on in-transit workflows where data must be unpacked, processed, and repacked.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Blood flow imaging by optimal matching of computational fluid dynamics to 4D-flow data

Three-dimensional, time-resolved blood flow measurement (4D-flow) is a powerful research and clinical tool, but improved resolution and scan times are needed. Therefore, this study aims to (1) present a postprocessing framework for optimization-driven simulation-based flow imaging, called 4D-flow High-resolution Imaging with a priori Knowledge Incorporating the Navier-Stokes equations and the discontinuous Galerkin method (4D-flow HIKING), (2) investigate the framework in synthetic tests, (3) perform phantom validation using laser particle imaging velocimetry, and (4) demonstrate the use of the framework in vivo. An optimizing computational fluid dynamics solver including adjoint-based optimization was developed to fit computational fluid dynamics solutions to 4D-flow data. Synthetic tests were performed in 2D, and phantom validation was performed with pulsatile flow. Reference velocity data were acquired using particle imaging velocimetry, and 4D-flow data were acquired at 1.5 T. In vivo testing was performed on intracranial arteries in a healthy volunteer at 7 T, with 2D flow as the reference. Results Synthetic tests showed low error (0.4%-0.7%). Phantom validation showed improved agreement with laser particle imaging velocimetry compared with input 4D-flow in the horizontal (mean -0.05 vs -1.11 cm/s, P < .001; SD 1.86 vs 4.26 cm/s, P < .001) and vertical directions (mean 0.05 vs -0.04 cm/s, P = .29; SD 1.36 vs 3.95 cm/s, P < .001). In vivo data show a reduction in flow rate error from 14% to 3.5%. Phantom and in vivo results from 4D-flow HIKING show promise for future applications with higher resolution, shorter scan times, and accurate quantification of physiological parameters.

4D-flow MRI↗
DOE OSTI2026

Detector architecture and data flow of the CMS HGCAL Phase-2 Endcap Calorimeter Upgrade

The CMS High Granularity Calorimeter (HGCAL) is the Phase-2 endcap calorimeter upgrade for the High-Luminosity Large Hadron Collider, designed to provide precise spatial, energy, and timing measurements in a high-radiation, high-pileup environment. This work presents an overview of the HGCAL assembly process, detector architecture, and end-to-end data flow. The data flow goes from signal generation in silicon sensors and scintillator tiles through front-end electronics, optical transmission, and back-end data acquisition. The study highlights how the integrated detector, electronics, and readout systems enable efficient event reconstruction and support the physics objectives of the HL-LHC.

Osei Boakye, Prince [Grambling State U.]↗
DOE OSTI2020

A mathematical model of asynchronous data flow in parallel computers *

Abstract We present a simplified model of data flow on processors in a high-performance computing framework involving computations necessitating inter-processor communications. From this ordinary differential model, we take its asymptotic limit, resulting in a model which treats the computer as a continuum of processors and data flow as an Eulerian fluid governed by a conservation law. We derive a Hamilton–Jacobi equation associated with this conservation law for which the existence and uniqueness of solutions can be proven. We then present the results of numerical experiments for both discrete and continuum models; these show a qualitative agreement between the two and the effect of variations in the computing environment’s processing capabilities on the progress of the modelled computation.

Barnard, Richard C.↗
DOE OSTI2020

Pass-efficient methods for compression of high-dimensional turbulent flow data

The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

Data-flow parallelism for high-energy and nuclear physics computing frameworks

The processing tasks of a scientific workflow in high-energy and nuclear physics (HENP) can typically be represented as a directed acyclic graph formed according to the data flow—i.e. the data dependencies among algorithms executed as part of the workflow. With this representation, an HENP computing framework can optimally execute a workflow, exploiting the parallelism inherent among independent tasks. Despite such a natural description of a workflow, most HENP frameworks do not make use of technologies that provide concurrent execution of graph-based tasking structures. In this session, we describe Fermilab efforts to adopt a graph-based technology (specifically Intel’s oneTBB flow graph) for meeting the framework needs of its experiments, notably DUNE. After introducing the physics DUNE intends to explore, we will show that all common processing idioms supported by current HENP frameworks can naturally be supported by oneTBB’s data-flow technology, optimally leveraging the concurrent capabilities of the machine. In addition, we discuss collaborative efforts between Fermilab and the Intel oneTBB development team, who is considering improvements to the flow-graph technology to better support HENP use cases.

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

Improving I/O-aware Workflow Scheduling via Data Flow Characterization and trade-off Analysis

The scientific computing paradigm has transitioned from compute-intensive to I/O-intensive and memory-intensive in the past decade, especially when data-driven science has become common practice. Numerous empirical I/O-aware scheduling optimizations have been developed by incorporating I/O capacity and bandwidth as constraints into scheduling. Unfortunately, there is a lack of data flow (I/O) characterization tool and an understanding of trade-offs between concurrency, locality, and I/O bandwidth. To bridge the gap, this work 1) presents a set of descriptors to characterize, organize, and visualize I/O profiles, including flow size, I/O bandwidth, and operation count, which group data flows by I/O types, tasks, and files; 2) proposes an I/O Roofline model-based trade-off analysis to find the optimal trade-off between flow operational intensity, concurrency, and flow performance. The I/O descriptors generate useful insights into complicated I/O behaviors, suggesting distinct concurrency, storage, and scheduling to be used by types, tasks, and files. The proposed trade-off analysis guides scheduling decisions that generate resource assignment with the best flow parallelism. We evaluate our I/O-aware scheduling methodology on a highly I/O-intensive workflow–1000 Genomes. The experimental results demonstrate speedups of up to 2.4× compared to the state-of-the- art methods.

Guo, Luanzheng [BATTELLE (PACIFIC NW LAB)]↗
DOE OSTI2024

Data-flow parallelism for high-energy and nuclear physics frameworks

The processing tasks of an event-processing workflow in high-energy and nuclear physics (HENP) can typically be represented as a directed acyclic graph formed according to the data flow—i.e. the data dependencies among algorithms executed as part of the workflow. With this representation, an HENP framework can optimally execute a workflow, exploiting the parallelism inherent among independent tasks. Despite such a natural description of a workflow, most HENP frameworks do not make use of technologies that provide concurrent execution of graph-based tasking structures. In this talk, we describe Fermilab efforts to adopt a graph-based technology (specifically Intel’s oneTBB flow graph) for meeting the framework needs of its experiments, notably DUNE. Building on the Meld project as presented at CHEP2023, we demonstrate that all common processing idioms supported by current frameworks can naturally be supported by oneTBB’s data-flow technology, optimally leveraging the concurrent capabilities of the machine. In addition, we discuss collaborative efforts between Fermilab and the Intel oneTBB development team, who is considering improvements to the flow-graph technology to better support HENP use cases.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Qualitative properties of mathematical model for data flow

In this paper, properties of a recently proposed mathematical model for data flow in large-scale asynchronous computer systems are analyzed. In particular, the existence of special weak solutions based on propagating fronts is established. Qualitative properties of these solutions are investigated, both theoretically and numerically.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Modeling Grid Data Flows for Transmission and Distribution Operations: Review, Design, Next Steps

Operational scenarios of the power grids grow multifold to accommodate the diverse needs of both the utilities and end consumers, and the various other stakeholders in-between. To comprehensively model and apply analytics to support objectives and business functions of grid sectors, a reliable approach to characterize and design data flows is crucial. The flows bridge business functions with communications protocols, stakeholders such as the grid actors, and data interfaces comprising different data objects. Additionally, constraints applied to the flow such as cybersecurity, trust, privacy, and ownership among others intersect these entities, requiring the delineation of their interactions under different scenarios. This paper aims to not only highlight relevant research in the space of grid data flows, but also proposes, for the transmission-distribution sector, a novel modeling approach that marries the aforementioned entities: objectives, business functions, data interfaces, communication protocols, data stakeholders, and flow constraints. It elaborates on the design philosophy and the significance of each entity within the model and applies it to an example function of fault location, isolation and service restoration (FLISR). Finally, the next steps to extend the application of this data flow model for other practical operational scenarios are discussed.

Sundararajan, Aditya [ORNL] (ORCID:000000033577854↗
DOE OSTI2020

Efficient Binary Static Code Data Flow Analysis Using Unsupervised Learning

The ever increasing need to ensure that code is reliably, efficiently and safely constructed has fueled the evolution of popular static binary code analysis tools. In identifying potential coding flaws in binaries, tools such as IDA Pro are used to disassemble the binaries into an opcode/assembly language format in support of manual static code analysis. Because of the highly manual and resource intensive nature involved with analyzing large binaries, the probability of overlooking potential coding irregularities and inefficiencies is quite high. In this paper, a light-weight data, unsupervised data flow methodology is described which uses highly-correlated data flow graph (CDFGs) to identify coding irregularities such that analysis time and required computing resources are minimized. Such analysis accuracy and efficiency gains are achieved by using a combination of graph analysis and unsupervised machine learning techniques which allows an analyst to focus on the most statistically significant flow patterns while performing binary static code analysis.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

S AP F LOWER : an automated tool for sap flow data preprocessing, gap-filling, and analysis using deep learning

Sap flow, a critical process in plant water use and ecosystem water cycles, is often measured using thermal dissipation probes (TDP) due to their ease of installation and continuous data collection. However, sap flow data frequently include noise, outliers, and gaps, creating challenges for analysis and requiring substantial manual processing. We developed S AP F LOWER , a tool that automates data preprocessing, model training, gap-filling, sapwood area scaling and modeling, and water use analysis. It integrates autocleaning, machine learning and deep learning models (e.g. random forest, Gaussian process regression, long short-term memory (LSTM), bidirectional LSTM (BiLSTM)), and efficient workflows to process sap flow data. S AP F LOWER can remove over 90% of noisy data while preserving legitimate variations and achieve high accuracy in gap-filling based on user-determined parameters. Random forest, LSTM, and BiLSTM models reduced root mean square error to 10% or less for long-term gaps. Model training and prediction can be performed efficiently within seconds. S AP F LOWER significantly enhances the efficiency and accessibility of TDP data analysis by automating complex tasks, enabling researchers without programming expertise to employ advanced techniques. Future improvements will focus on species-specific corrections for TDP and support for additional measurement methods. S AP F LOWER is openly available on GitHub (https://github.com/JiaxinWang123/SapFlower) and Zenodo (doi: 10.5281/zenodo.13665919).

ecosystem water balance↗
DOE OSTI2023

Bayesian inference of in-medium baryon-baryon scattering cross sections from HADES proton flow data

Within a Bayesian statistical framework using a Gaussian Process emulator for an isospin-dependent Boltzmann-Uehling-Uhlenbeck (IBUU) transport model simulator of heavy-ion reactions at intermediate energies, we infer from the HADES proton flow data the posterior probability distribution functions of in-medium baryon-baryon scattering cross section modification factor X with respect to free-space and the corresponding incompressibility K of nuclear matter as well as their correlation function. In conclusion, the mean value of X is found to be $X$ = $1.32_ {+0.28} ^ {-0.40} $ at 68% confidence level assuming the nuclear incompressibility K will not exceed 400 MeV, providing circumstantial evidence for enhanced baryon-braryon scattering cross sections in hot and dense nuclear matter.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

Blockchain Empowered Provenance Framework for Sensor Identity Management and Data Flow Security in Fossil-based Power Plants

The overall goals of this project are divided into 3 important phases. First, a peer-to-peer SCADA network is to be established where identity profile of sensors from fossil-power plants will be stored in an immutable manner in order to verify data flow integrity in real time. Second, it is aimed to develop a Blockchain-based provenance platform to audit equipment operations along with data and process integrity violations. In this phase, various operational rules will be incorporated in the ledger on behalf of the operating sensor so its abnormal use can be easily detected. In addition, it is aimed to achieve provenance when the rate of transactions is exceedingly high, thus enabling to design a scalable provenance scheme that can work in real time when number of data collecting points or sensor units increase in fossil-based power plants. Finally, a working testbed will be developed to integrate the Blockchain platform with sensor-based fossil power plant comprising of Remote Terminal Units (RTU) and SCADA devices. Using multiple Raspberry-PIs mimicking the role of RTUs that are interfaced with multiple sensors devices will be deployed to serve as a critical component of fossil power plant and the performance metrics such as throughput, latency, bootstrapping time, etc., will be evaluated on the testbed.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2025

Advancing the utility of Data flow accelerators: Kokkos backend for Next Silicon

Next Silicon provides a novel reconfigurable accelerator, which is inherently a high-risk, early exploration effort. Higher readiness level for running NNSA mini-apps and initial mission applications on a data-flow accelerator Enable and demonstrate a new Kokkos back-end that directly drives an underlying dataflow accelerator

Garcia de Gonzalo, Simon Pedro [Sandia National La↗
DOE OSTI2026

Estimating IDP Origins Using ACLED Political Violence Events: Validation with Lebanon IDP Flow Data

A key input in modeling population distribution and flows following conflict events is the inclusion of Internally Displaced Person (IDP) flows between administrative units within a country. These flows are critical for capturing population movement and redistribution driven by current events, particularly conflict. In some cases, IDP destination data are available while origin data is incomplete or unavailable. This creates a gap in understanding where displacement is occurring, limiting the ability to model population redistribution accurately. Without origin data, it is not possible to reallocate population flows or accurately represent where displacement is occurring within the country. This report evaluates whether Armed Conflict Location Event Data (ACLED) political violence event data can be used to estimate IDP origin distributions when direct origin data are unavailable. The approach is validated using historical IDP flow data from Lebanon, where both origins and destinations are observed. Results show that ACLED event distributions strongly correspond to observed IDP-origin patterns, particularly when using cumulative 60-day event windows. The method is most reliable for identifying major origin districts and approximating proportional origin shares. However, it is not intended to reconstruct exact individual displacement flows, but rather to provide a probabilistic spatial allocation of displacement origins.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2021

Modeling Data Flows with Network Calculus in Cyber-Physical Systems: Enabling Feature Analysis for Anomaly Detection Applications

The electric grid is becoming increasingly cyber-physical with the addition of smart technologies, new communication interfaces, and automated grid-support functions. Because of this, it is no longer sufficient to only study the physical system dynamics, but the cyber system must also be monitored as well to examine cyber-physical interactions and effects on the overall system. To address this gap for both operational and security needs, cyber-physical situational awareness is needed to monitor the system to detect any faults or malicious activity. Techniques and models to understand the physical system (the power system operation) exist, but methods to study the cyber system are needed, which can assist in understanding how the network traffic and changes to network conditions affect applications such as data analysis, intrusion detection systems (IDS), and anomaly detection. In this paper, we examine and develop models of data flows in communication networks of cyber-physical systems (CPSs) and explore how network calculus can be utilized to develop those models for CPSs, with a focus on anomaly and intrusion detection. This provides a foundation for methods to examine how changes to behavior in the CPS can be modeled and for investigating cyber effects in CPSs in anomaly detection applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Sap flow data, Jan 2022-Dec 2022, Panama: PNM,BCI, and SLZ

Sap flow was monitored at three Panamanian sites: Parque Natural Metropolitano (PNM), Barro Colorado Island (BCI), and Bosque Protector San Lorenzo (SLZ), using a total of 54 G-20mm sap flow sensors. Raw output from the data logger connected to 54 sap flow sensors are converted from xxx.dat. to xxx.csv. files. The metadata file (E-field log_v1-1_PNM_BCI_SLZ_sapflow_2022.xls) has information on locations where the sensors were installed, and other installation/maintenance details. No data processing or QA/QC was done on the raw datasets except conversion of dat. files to csv. files. Processed datasets have been uploaded separately. Contact mizanur.rahman@pnnl.gov or nate.mcdowell@pnnl.gov if you need to use this dataset for additional information.

54 ENVIRONMENTAL SCIENCES↗