SEARCH · Search NASA

Results for “DATA-FLOW PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

14 records

All sources NASA NTRS DOE OSTI

DOE OSTI2011

Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT)

This report summarizes work carried out by the Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT) Team for the period of January 1, 2011 through June 30, 2011. It discusses highlights, overall progress, period goals, and collaborations and lists papers and presentations. To learn more about our project, please visit our UV-CDAT website (URL: http://uv-cdat.org). This report will be forwarded to the program manager for the Department of Energy (DOE) Office of Biological and Environmental Research (BER), national and international collaborators and stakeholders, and to researchers working on a wide range of other climate model, reanalysis, and observation evaluation activities. The UV-CDAT executive committee consists of Dean N. Williams of Lawrence Livermore National Laboratory (LLNL); Dave Bader and Galen Shipman of Oak Ridge National Laboratory (ORNL); Phil Jones and James Ahrens of Los Alamos National Laboratory (LANL), Claudio Silva of Polytechnic Institute of New York University (NYU-Poly); and Berk Geveci of Kitware, Inc. The UV-CDAT team consists of researchers and scientists with diverse domain knowledge whose home institutions also include the National Aeronautics and Space Administration (NASA) and the University of Utah. All work is accomplished under DOE open-source guidelines and in close collaboration with the project's stakeholders, domain researchers, and scientists. Working directly with BER climate science analysis projects, this consortium will develop and deploy data and computational resources useful to a wide variety of stakeholders, including scientists, policymakers, and the general public. Members of this consortium already collaborate with other institutions and universities in researching data discovery, management, visualization, workflow analysis, and provenance. The UV-CDAT team will address the following high-level visualization requirements: (1) Alternative parallel streaming statistics and analysis pipelines - Data parallelism, Task parallelism, Visualization parallelism; (2) Optimized parallel input/output (I/O); (3) Remote interactive execution; (4) Advanced intercomparison visualization; (5) Data provenance processing and capture; and (6) Interfaces for scientists - Workflow data analysis and visualization construction tools, and Visualization interfaces.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Data-flow parallelism for high-energy and nuclear physics frameworks

The processing tasks of an event-processing workflow in high-energy and nuclear physics (HENP) can typically be represented as a directed acyclic graph formed according to the data flow—i.e. the data dependencies among algorithms executed as part of the workflow. With this representation, an HENP framework can optimally execute a workflow, exploiting the parallelism inherent among independent tasks. Despite such a natural description of a workflow, most HENP frameworks do not make use of technologies that provide concurrent execution of graph-based tasking structures. In this talk, we describe Fermilab efforts to adopt a graph-based technology (specifically Intel’s oneTBB flow graph) for meeting the framework needs of its experiments, notably DUNE. Building on the Meld project as presented at CHEP2023, we demonstrate that all common processing idioms supported by current frameworks can naturally be supported by oneTBB’s data-flow technology, optimally leveraging the concurrent capabilities of the machine. In addition, we discuss collaborative efforts between Fermilab and the Intel oneTBB development team, who is considering improvements to the flow-graph technology to better support HENP use cases.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

Data-flow parallelism for high-energy and nuclear physics computing frameworks

The processing tasks of a scientific workflow in high-energy and nuclear physics (HENP) can typically be represented as a directed acyclic graph formed according to the data flow—i.e. the data dependencies among algorithms executed as part of the workflow. With this representation, an HENP computing framework can optimally execute a workflow, exploiting the parallelism inherent among independent tasks. Despite such a natural description of a workflow, most HENP frameworks do not make use of technologies that provide concurrent execution of graph-based tasking structures. In this session, we describe Fermilab efforts to adopt a graph-based technology (specifically Intel’s oneTBB flow graph) for meeting the framework needs of its experiments, notably DUNE. After introducing the physics DUNE intends to explore, we will show that all common processing idioms supported by current HENP frameworks can naturally be supported by oneTBB’s data-flow technology, optimally leveraging the concurrent capabilities of the machine. In addition, we discuss collaborative efforts between Fermilab and the Intel oneTBB development team, who is considering improvements to the flow-graph technology to better support HENP use cases.

43 PARTICLE ACCELERATORS↗

NASA NTRS1982

A scheme for handling arrays in data-flow systems

An examination of the effects of atomicity (higher resolution) on the performance of array processors (data-flow computers) is presented. Data-flow principles are reviewed, noting the reliance on parallel processing using functional languages to specify sequencing of the operations. Techniques are described for eliminating the necessity of copying whole arrays between processing steps, thereby reducing the number of store cycles. The method involves setting whole columns to specific values rather than individual elements. The individual column values can be processed in parallel, i.e., a locally optimized condition exists. A drawback of the system is the need for more low level arguments, to identify the appropriate processing sequences, and high system complexity.

Gaudiot, J.-L.↗

NASA NTRS2022

The Psyche Planning Software Subsystem: Creating a Robust Toolset for a Discovery-class Mission

Psyche is a Discovery-class mission to the small metal-rich asteroid (16) Psyche, and is slated to launch in 2022. Psyche, like many missions, requires low-cost activity planning and sequence generation that serves as the backbone to overall uplink design. Such tools must be maintainable over long periods of operations, and powerful enough to solve complex issues that deep-space one-off missions encounter. In this paper we introduce cost-effective solutions that leverage inner- and open-source principles to meet a variety of common and novel use cases.The uplink process that was designed to meet these challenges is presented, as well as the data-flow through the high-level architecture of the planning software subsystem. The user-facing planning tools are described, particularly the Science Opportunity Analyzer, the Plan Editor, Psyche’s planning automation in the Blackbird framework, and Psyche Simulation Reports. All these applications are either new or have been substantially revamped to meet Psyche’s concept of operations. In particular, ensuring the entire toolchain can correctly process epoch-relative activities is discussed. Underlying the main applications are a common set of dependencies developed and maintained by a new cross-mission association of planning developers. In this way, Psyche can inherit well-tested functionality which saves effort and ensures its developers can focus on solving domain challenges. Quality control of the applications and libraries is ensured with a code-review and unit-test based novel ‘CM lite’ process. Collaboration with international industry and academia using the open-source modules is already occurring.The planning and scheduling software is designed to maximize operator awareness of the integrated plan at every step of the process and use common interfaces and file formats to easily transfer information. Design choices plus the team’s test-driven development process enables more expansive capabilities compared to the decentralized planning and sequence generation functions typical of Discovery-class orbiters without significant development cost increases. Benefits and drawbacks of Psyche’s approach are discussed, including comparison to other missions and tools where appropriate.

Ramanathan, Keshav↗

NASA NTRS2007

Processing Raw HST Data With Up-to-Date Calibration Data

On-the-Fly Reprocessing (OTFR) is a collection of data-processing routines that work within the context of the Hubble Space Telescope (HST) pipeline data-flow system. The purpose served by OTFR is to generate, on demand, scientifically useful data products from raw HST data stored in an archive. First, on the basis of the requested final data products, OTFR retrieves the corresponding sets of raw data from the archives. Next, OTFR processes the raw data sets to remove artifacts and to establish proper header and other template information. Finally, the calibration routines appropriate to the specific data sets are invoked to produce the requested data products, and the data products are released to an archive distribution system for transmission to the requesting party. OTFR offers two notable advantages: (1) Inasmuch as calibrated data occupy about 8 times as much storage space as do raw data, by obviating storage of calibrated data, OTFR reduces the storage capacity needed by the archive; and (2) the calibration routines can be updated to give requesters the benefit of the most recent calibrations.

Miller, Warren↗

DOE OSTI2022

Processing Particle Data Flows with SmartNICs

Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freeing the hosts to perform other work. System architects in this space face multiple questions about the best way to leverage SmartNICs as processing elements in data flows. In this paper, we advocate the use of Apache Arrow as a foundation for implementing data-flow tasks on SmartNICs. We report on our experiences adapting a partitioning algorithm for particle data to Apache Arrow and measure the on-card processing performance for the BlueField-2 SmartNIC. Our experiments confirm that the BlueField-2’s (de)compression hardware can have a significant impact on in-transit workflows where data must be unpacked, processed, and repacked.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2010

Data-Flow Based Model Analysis

The concept of (meta) modeling combines an intuitive way of formalizing the structure of an application domain with a high expressiveness that makes it suitable for a wide variety of use cases and has therefore become an integral part of many areas in computer science. While the definition of modeling languages through the use of meta models, e.g. in Unified Modeling Language (UML), is a well-understood process, their validation and the extraction of behavioral information is still a challenge. In this paper we present a novel approach for dynamic model analysis along with several fields of application. Examining the propagation of information along the edges and nodes of the model graph allows to extend and simplify the definition of semantic constraints in comparison to the capabilities offered by e.g. the Object Constraint Language. Performing a flow-based analysis also enables the simulation of dynamic behavior, thus providing an "abstract interpretation"-like analysis method for the modeling domain.

Saad, Christian↗

NASA NTRS2010

Software Tool Integrating Data Flow Diagrams and Petri Nets

Data Flow Diagram - Petri Net (DFPN) is a software tool for analyzing other software to be developed. The full name of this program reflects its design, which combines the benefit of data-flow diagrams (which are typically favored by software analysts) with the power and precision of Petri-net models, without requiring specialized Petri-net training. (A Petri net is a particular type of directed graph, a description of which would exceed the scope of this article.) DFPN assists a software analyst in drawing and specifying a data-flow diagram, then translates the diagram into a Petri net, then enables graphical tracing of execution paths through the Petri net for verification, by the end user, of the properties of the software to be developed. In comparison with prior means of verifying the properties of software to be developed, DFPN makes verification by the end user more nearly certain, thereby making it easier to identify and correct misconceptions earlier in the development process, when correction is less expensive. After the verification by the end user, DFPN generates a printable system specification in the form of descriptions of processes and data.

Thronesbery, Carroll↗

DOE OSTI2019

Component based development of computer-aided tools for different applications

Computer-aided methods and tools are used at different stages of chemical process development. However new tools are needed to meet the current and future challenges through innovative and more sustainable solutions that the currently available tools may not be able to provide. This work focuses on component-based software architecture for development of tailor-made computer-aided tools for specific application objectives. The architecture is based on “components”, which can be model-objects that are stored in a library; collection of property models; databases; numerical solvers; design-algorithms (work-flows) etc. Furthermore, the main idea of these components is to incorporate a plug & play framework, where for a specific purpose, after the work-flow and associated data-flow are established, a specific computer-aided tool can be configured through a library of software components. Also, through the use of software components that are based on predictive model-based solution techniques, the final software can be reliably used to provide innovative and more sustainable solutions over a wide range of problems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

NASA NTRS1985

Performance evaluation of a simulated data-flow computer with low-resolution actors

Basic problems related to the exploitation of parallelism in a program include sequencing of the instructions and communication of the data. It is pointed out that the data-flow approach offers an elegant solution to the sequencing problem, since all data dependencies are automatically handled and only instructions with ready input sets are activated. It is shown that a change in the level of subcomputations (actors) affects communications costs. The concept of variable resolution is discussed, and the testbed environment is examined. Attention is given to the architecture of the processing elements, the communication network, and the simulators. A description of the analytical model is also provided. Simulation and results are discussed, taking into account test programs and allocation, the variation of the number of processing elements, the variation of the resolution in directed acyclic graphs, performance in processing loops, and array handling.

Gaudiot, J. L.↗

NASA NTRS2003

Automating the Processing of Earth Observation Data

NASA s vision for Earth science is to build a "sensor web": an adaptive array of heterogeneous satellites and other sensors that will track important events, such as storms, and provide real-time information about the state of the Earth to a wide variety of customers. Achieving this vision will require automation not only in the scheduling of the observations but also in the processing of the resulting data. To address this need, we are developing a planner-based agent to automatically generate and execute data-flow programs to produce the requested data products.

Golden, Keith↗

NASA NTRS2003

Automated Data Processing as an AI Planning Problem

NASA s vision for Earth Science is to build a "sensor web"; an adaptive array of heterogeneous satellites and other sensors that will track important events, such as storms, and provide real-time information about the state of the Earth to a wide variety of customers. Achieving his vision will require automation not only in the scheduling of the observations but also in the processing af tee resulting data. Ta address this need, we have developed a planner-based agent to automatically generate and execute data-flow programs to produce the requested data products. Data processing domains are substantially different from other planning domains that have been explored, and this has led us to substantially different choices in terms of representation and algorithms. We discuss some of these differences and discuss the approach we have adopted.

Golden, Keith↗

NASA NTRS2015

NASA Data Acquisition System Software Development for Rocket Propulsion Test Facilities

Current NASA propulsion test facilities include Stennis Space Center in Mississippi, Marshall Space Flight Center in Alabama, Plum Brook Station in Ohio, and White Sands Test Facility in New Mexico. Within and across these centers, a diverse set of data acquisition systems exist with different hardware and software platforms. The NASA Data Acquisition System (NDAS) is a software suite designed to operate and control many critical aspects of rocket engine testing. The software suite combines real-time data visualization, data recording to a variety formats, short-term and long-term acquisition system calibration capabilities, test stand configuration control, and a variety of data post-processing capabilities. Additionally, data stream conversion functions exist to translate test facility data streams to and from downstream systems, including engine customer systems. The primary design goals for NDAS are flexibility, extensibility, and modularity. Providing a common user interface for a variety of hardware platforms helps drive consistency and error reduction during testing. In addition, with an understanding that test facilities have different requirements and setups, the software is designed to be modular. One engine program may require real-time displays and data recording; others may require more complex data stream conversion, measurement filtering, or test stand configuration management. The NDAS suite allows test facilities to choose which components to use based on their specific needs. The NDAS code is primarily written in LabVIEW, a graphical, data-flow driven language. Although LabVIEW is a general-purpose programming language; large-scale software development in the language is relatively rare compared to more commonly used languages. The NDAS software suite also makes extensive use of a new, advanced development framework called the Actor Framework. The Actor Framework provides a level of code reuse and extensibility that has previously been difficult to achieve using LabVIEW. The

Herbert, Phillip W., Sr.↗