Search NASA⌕ Search

SEARCH · Search NASA

Results for “Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52

Building a new multiphysics workflow in MOOSE: application to tritium migration, trapping and advection in TMAP8

Fusion devices are anticipated to produce and consume several kilograms of tritium per year. This rare fuel resource is both highly mobile and radioactive, making tracking inventories a priority for operation and safety. The fusion safety program at the Idaho National Laboratory has been developing the Tritium Migration and Analysis Program (TMAP), of which the latest version is a MOOSE-based application. TMAP8 is verified against its predecessors and possesses additional multi-dimensional tritium migration modeling capabilities. As we extend its capabilities towards both whole device (in multiple dimensions) and whole plant (with multiple components) simulations, the syntax of inputs must become compact, descriptive, compatible with quality assurance processes, and as error-proof as achievable. The new Physics system developed MOOSE can set up equations and instantiating them on plant components. The system permits the automatic definition of complex discretization with a consistency between object parameters achieved programmatically. The Physics system can currently instantiate the equations for heat conduction and Navier Stokes weakly compressible flow. In MOOSE-terms, it automates the definition of kernels, boundary conditions, and several core and helper materials and fields. As part of this effort, Physics classes were developed for tritium migration, trapping and advection within either a multi-dimensional Navier Stokes fluid dynamics simulation, or a 1D thermal hydraulics piping system. In this presentation, we will showcase the new syntax, its application to several verification and validation cases which were already studied using the classical TMAP8 syntax, and a demonstration of the new coupling capabilities for the migration of tritium into blanket coolant channels and the subsequent advection into the coolant loop.

70 - PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Experiments with Memory-to-Memory Coupling for End-to-End fusion Simulation Workflows

Scientific applications are striving to accurately simulate multiple interacting physical processes that comprise complex phenomena being modeled. Efficient and scalable parallel implementations of these coupled simulations present challenging interaction and coordination requirements, especially when the coupled physical processes are computationally heterogeneous and progress at different speeds. In this paper, we present the design, implementation and evaluation of a memory-to-memory coupling framework for coupled scientific simulations on high-performance parallel computing platforms. The framework is driven by the coupling requirements of the Center for Plasma Edge Simulation, and it provides simple coupling abstractions as well as efficient asynchronous (RDMA-based) memory-to-memory data transport mechanisms that complement existing parallel programming systems and data sharing frameworks. The framework enables flexible coupling behaviors that are asynchronous in time and space, and it supports dynamic coupling between heterogeneous simulation processes without enforcing any synchronization constraints. We evaluate the performance and scalability of the coupling framework using a specific coupling scenario, on the Jaguar Cray XT5 system at Oak Ridge National Laboratory.

99 GENERAL AND MISCELLANEOUS↗

Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.

97 MATHEMATICS AND COMPUTING↗

Darshan for HEP applications

Modern HEP workflows must manage increasingly large and complex data collections. HPC facilities may be employed to help meet these workflows’ growing data processing needs. However, a better understanding of the I/O patterns and underlying bottlenecks of these workflows is necessary to meet the performance expectations of HPC systems.Darshan is a lightweight I/O characterization tool that captures concise views of HPC application I/O behavior. It intercepts application I/O calls at runtime, records file access statistics for each process, and generates log files detailing application I/O access patterns.Typical HEP workflows include event generation, detector simulation, event reconstruction, and subsequent analysis stages. A study of the I/O behavior of the ATLAS simulation and filtering stage, and the CMS simulation workflow using Darshan is presented, including insights into the I/O operations and data access size.

Wang, Rui↗

Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters

Big data is prevalent in HPC computing. Many HPC projects rely on complex workflows to analyze terabytes or petabytes of data. These workflows often require running over thousands of CPU cores and performing simultaneous data accesses, data movements, and computation. It is challenging to analyze the performance involving terabytes or petabytes of workflow data or measurement data of the executions, from complex workflows over a large number of nodes and multiple parallel task executions. To help identify performance bottlenecks or debug the performance issues in large-scale scientific applications and scientific clusters, we have developed a performance analysis framework, using state-ofthe- art open-source big data processing tools. Our tool can ingest system logs and application performance measurements to extract key performance features, and apply the most sophisticated statistical tools and data mining methods on the performance data. It utilizes an efficient data processing engine to allow users to interactively analyze a large amount of different types of logs and measurements. To illustrate the functionality of the big data analysis framework, we conduct case studies on the workflows from an astronomy project known as the Palomar Transient Factory (PTF) and the job logs from the genome analysis scientific cluster. Our study processed many terabytes of system logs and application performance measurements collected on the HPC systems at NERSC. The implementation of our tool is generic enough to be used for analyzing the performance of other HPC systems and Big Data workows.

97 MATHEMATICS AND COMPUTING↗

Adaptive elasticity policies for staging-based in situ visualization

In situ processing aims to alleviate the growing gap between computation and I/O capabilities by performing data processing close to the data source. In situ processing is widely used to process data generated by multiple data sources, including observation data from edge devices or scientific observational facilities and the simulation data generated by scientific computation on a high-performance computing (HPC) platform. For a scientific workflow that is run on an HPC platform and composed of a simulation program and an in situ data analytics or visualization (abbreviated as ana/vis) task, there is an implicit assumption that the computing resources assigned to the workflow keep static during the workflow execution. However, with the converging trend between the HPC and cloud computing platform, running the in situ ana/vis task in an elastic way is promising to decrease its overhead and improve its resource utilization rate. Resource elasticity represents the ability to change resource configurations such as the number of computing nodes/processes during workflow execution. An elastic job may dynamically adjust resource configurations; it may use a few resources at the beginning and more resources toward the end of the job when interesting data appear. However, it is hard to predict a priori how many computing nodes/processes need to be added/removed during the workflow execution to adapt to changing workflow needs. How to efficiently guide elasticity operations, such as growing or shrinking the number of processes used for in situ analysis during workflow execution, is an open-ended research question. In this article, we present adaptive elasticity policies that adopt workflow runtime information collected during workflow execution to predict how to trigger the addition/removal of processes in order to minimize in situ processing overhead. Taking in situ visualization tasks as an example, we integrate the presented elasticity policies into a staging-based elastic workflow and evaluate its efficiency in multiple elasticity scenarios. Compared with the situation without elasticity or with a static elasticity policy that uses a fixed number of processes for each rescaling operation, the adaptive elasticity policy can save overhead in finding a proper resource configuration and improve resource utilization efficiency. Furthermore, one experiment illustrates that the adaptive elasticity policy saves 41% of core-hours compared with the situation without the resource elasticity.

97 MATHEMATICS AND COMPUTING↗

On the significance of density-induced speed of sound variations on US-guided radiotherapy

Purpose: To show the effect of speed of sound (SOS) aberration on ultrasound guided radiotherapy (US-gRT) as a function of implemented workflow. US systems assume that SOS is constant in human soft tissues (at a value of 1540 m/s), while its actual nonuniform distribution produces small but systematic errors of up to a few millimeters in the positions of scanned structures. When a coregistered computerized tomography (CT) scan is available, the US image can be corrected for SOS aberration. Typically, image guided radiotherapy workflows implementing US systems only provide a CT scan at the simulation (SIM) stage. If changes occur in geometry or density distribution between SIM and treatment (TX) stage, SOS aberration can change accordingly, with a final impact on the measured position of structures which is dependent on the workflow adopted. Methods: Four basic scenarios were considered of possible changes between SIM and TX: (1) No changes, (2) only patient position changes (rigid rotation-translation), (3) only US transducer position changes (constrained on patient's surface), and (4) patient tissues thickness changes. Different SOS aberrations may arise from the different scenarios, according to the specific US-gRT workflow used: intermodality (INTER) where TX US scans are compared to SIM CT scans; intramodality (INTRA) where TX US scans are compared to SIM US scans; and INTERc and INTRAc where all US images are corrected for SOS aberration (using density information provided by SIM CT). For an experimental proof of principle, the effect of tissues thickness change was simulated in the different workflows: a dual layered phantom was filled with layers of sunflower oil (SOS 1478 m/s), water (SOS 1482 m/s), and 20% saline solution (SOS 1700 m/s). The phantom was US scanned, the layer thicknesses were increased and the US scans were repeated. The errors resulting from the different workflows were compared. Results: Theoretical considerations show that workflows implementing SOS correction based on SIM-CT scan (INTERc, INTRAc) give null errors in all scenarios except when tissues thickness changes, where an error proportional to the degree of change in SOS maps between SIM and TX ({Delta}SOS) occurs. An uncorrected workflow such as INTER produces in all scenarios a pure SOS error, while uncorrected INTRA produces a null error for rotation-translation of the patient, a {Delta}SOS error for changing tissues thickness and an error proportional to the degree of SOS distribution change along the different lines of view when shifting the transducer. The dual layered phantom demonstrated experimentally that the effect of SOS change between SIM and TX is clinically nonrelevant, being less than the intrinsic resolution of imaging systems, even when a substantial change in thicknesses is applied, provided that a SIM-CT-based SOS aberration correction is applied. Noncorrected workflows produce errors up to 4 mm for INTER and to 3 mm for INTRA in the phantom test. Conclusions: A SOS correction is advantageous for all US-gRT workflows and clinical cases, where the effect of SOS change can be considered a second order effect.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Scientific Data Management (SDM) Center for Enabling Technologies. 2007-2012

Over the past five years, our activities have both established Kepler as a viable scientific workflow environment and demonstrated its value across multiple science applications. We have published numerous peer-reviewed papers on the technologies highlighted in this short paper and have given Kepler tutorials at SC06,SC07,SC08,and SciDAC 2007. Our outreach activities have allowed scientists to learn best practices and better utilize Kepler to address their individual workflow problems. Our contributions to advancing the state-of-the-art in scientific workflows have focused on the following areas. Progress in each of these areas is described in subsequent sections. Workflow development. The development of a deeper understanding of scientific workflows "in the wild" and of the requirements for support tools that allow easy construction of complex scientific workflows; Generic workflow components and templates. The development of generic actors (i.e.workflow components and processes) which can be broadly applied to scientific problems; Provenance collection and analysis. The design of a flexible provenance collection and analysis infrastructure within the workflow environment; and, Workflow reliability and fault tolerance. The improvement of the reliability and fault-tolerance of workflow environments.

97 MATHEMATICS AND COMPUTING↗

FORCE Update 2024

The Framework for Optimization of Resources and Economics (FORCE) tool suite is the U.S. Department of Energy’s Nuclear Integrated Energy Systems (IES) Program flagship tool suite for technoeconomic IES analysis of IES. This tool suite is useful for analysis designed to evaluate and improve the technoeconomics of energy production systems, particularly for systems including nuclear technology. In this report, we document the development activity for the FORCE tool suite to extend its capabilities as performed during fiscal year 2024. In addition to reliability and accessibility, capability is one of the three standards guiding the development of the FORCE tool suite and the software codes that are its constituent parts. Extending the capabilities of the FORCE tool suite allows analysis both within the IES program as well as industry, university, and laboratory partners to perform analysis with more accuracy, insight, and impactful narrative. Four areas of capability development were the focus of activity this year: economic parameter uncertainty quantification, multiresolution analysis, components-to-optimization workflow automation, and statespace construction workflows for real-time optimal control. In economic parameter uncertainty quantification, the ability of HERON to capture risk due to scenarios (weather and energy demand uncertainty) was expanded to also include uncertainties in financial parameters such as capital cost or operation and maintenance costs. By including these sources of uncertainty, which are sometimes very large compared with scenario uncertainty, HERON is better able to capture the risk posed by investment in various IES technology. Because of this, analysts can also consider the reduction in risks that can be realized by choice of some technologies. In multiresolution analysis, development activity extended on work completed previously. In fiscal year 2023, methods for decomposing time series signals, such as demand, solar and wind availability, and price profiles, were analyzed and down-selected to those most effective at splitting signals into different resolutions. These resolutions allow considering the influence of different energy demand and supply behaviors across different time scales. For example, energy demand might be divided into seasonal, weekly, and hourly profiles. In fiscal year 2024, this preliminary work was extended and implemented within the Risk Analysis Virtual Environment (RAVEN) risk and uncertainty analysis platform, which is used throughout the FORCE framework. This development of the “multi-resolution time series analysis” (MR-TSA) module in RAVEN allows training synthetic history generators on complex time series. These synthetic history generators can then be used in HERON for generating scenarios that represent possible market and weather scenarios that can be analyzed on different time scales. We envision completing this work in the future, implementing multiresolution dispatch optimization strategies that can make the most beneficial use of these stratified time histories. In components-to-optimization workflow development, workflows for translating user inputs of components into algorithms for algebraic optimization were selected and implemented. Similar algorithms within the Holistic Energy Resource Optimization Network (HERON) were separated from the main code base of HERON and gathered with the components-to-optimization workflows in the new Dispatch Optimization Variable Engine (DOVE) software library. This modularization allows FORCE users to analyze dispatch optimization and energy system duty cycles independently of HERON, which previously was a burdensome task. Additionally, these dispatch optimization algorithms, set up in an independent library, can now be used across all software applications within FORCE, especially including the real-time optimal control software Optimization of Real-time Capacity Allocation (ORCA). Allowing FORCE software to share dispatch optimization algorithms within a single library allows for improved software maintenance and reliability. In statespace characterization workflow development, alternative workflows for optimizing dispatch with additional technical accuracy was the focus, particularly to improve the real-time optimization decision making in ORCA. Using algorithms and workflows initially developed for the Feasible Actuator Range Modifier (FARM), workflows for determining the statespace representation of IES were identified and demonstrated. The resulting dispatch optimization required a more robust optimization algorithm than that originally used in HERON (and moved to DOVE), which required adding an alternate workflow to DOVE that can more accurately match the behavior of physical systems using a partial differential equation representation. In conclusion, capability developments in the FORCE tool suite in fiscal year 2024 have improved the ability of the FORCE tool suite to perform

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Build and Execute Environment

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.

Guan, Qiang↗

Observing System Simulation Experiment (OSSE) for the HyspIRI Spectrometer Mission

The OSSE software provides an integrated end-to-end environment to simulate an Earth observing system by iteratively running a distributed modeling workflow based on the HyspIRI Mission, including atmospheric radiative transfer, surface albedo effects, detection, and retrieval for agile exploration of the mission design space. The software enables an Observing System Simulation Experiment (OSSE) and can be used for design trade space exploration of science return for proposed instruments by modeling the whole ground truth, sensing, and retrieval chain and to assess retrieval accuracy for a particular instrument and algorithm design. The OSSE in fra struc ture is extensible to future National Research Council (NRC) Decadal Survey concept missions where integrated modeling can improve the fidelity of coupled science and engineering analyses for systematic analysis and science return studies. This software has a distributed architecture that gives it a distinct advantage over other similar efforts. The workflow modeling components are typically legacy computer programs implemented in a variety of programming languages, including MATLAB, Excel, and FORTRAN. Integration of these diverse components is difficult and time-consuming. In order to hide this complexity, each modeling component is wrapped as a Web Service, and each component is able to pass analysis parameterizations, such as reflectance or radiance spectra, on to the next component downstream in the service workflow chain. In this way, the interface to each modeling component becomes uniform and the entire end-to-end workflow can be run using any existing or custom workflow processing engine. The architecture lets users extend workflows as new modeling components become available, chain together the components using any existing or custom workflow processing engine, and distribute them across any Internet-accessible Web Service endpoints. The workflow components can be hosted on any Internet-accessible machine. This has the advantages that the computations can be distributed to make best use of the available computing resources, and each workflow component can be hosted and maintained by their respective domain experts.

Turmon, Michael J.↗

Build and Execute Environment

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.

Guan, Qiang↗