Search NASA⌕ Search

SEARCH · Search NASA

Results for “Workflow Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

An exploration of online-simulation-driven portfolio scheduling in Workflow Management Systems

Workflow Management Systems used to automate the execution of scientific workflow applications on parallel and distributed computing platforms must make scheduling decisions at runtime. A large number of workflow scheduling algorithms have been proposed in the literature, but often these algorithms are evaluated based on simplifying assumptions that may not hold in practice. Furthermore, published algorithm evaluation and/or comparison results are necessarily only for a subset of all possible scenarios, and thus may not include scenarios relevant to particular use-cases. Consequently, it is difficult for Workflow Management Systems (WMSs) developers to decide which scheduling algorithm should be implemented. To obviate this difficulty, one possible approach is to implement a portfolio of scheduling algorithms and select the most effective algorithm at runtime. One method for performing this selection is to run an online simulation for each algorithm in the portfolio. The algorithm that leads to the best performance, in simulation, is selected for future use. The above simulation-driven portfolio scheduling (SDPS) approach has been proposed in a few parallel and distributed computing contexts. The main objective of this work is to evaluate the feasibility and potential merit of SDPS if implemented in WMSs. Here we perform this evaluation using simulated WMS executions, where the simulations are instantiated from real-world platform and workflow configurations. Our main finding is that SDPS is on par with or outperforms an approach in which a single algorithm is used, where this algorithm is the one that performs best on average across all our experimental scenarios. Furthermore, we find that SDPS remains an attractive proposition even in the presence of high levels of simulation error and for simulators with relatively low levels of sophistication. In many of our experimental scenarios we find that mitigating simulation error at runtime can further improve performance. Finally, we show that simulation overhead can be made sufficiently low for SDPS to be feasible in practice.

97 MATHEMATICS AND COMPUTING↗

Job Life Cycle Management Libraries for CMS Workflow Management Projects

Scientific analysis and simulation requires the processing and generation of millions of data samples. These tasks are often comprised of multiple smaller tasks divided over multiple (computing) sites. This paper discusses the Compact Muon Solenoi d (CMS) workflow infrastructure, and specifically the Python based workflow library which is used for so called task lifecycle management. The CMS workflow infrastructure consists of three layers: high level specification of the various tasks based on inp ut/output data sets, life cycle management of task instances derived from the high level specification and execution management. The workflow library is the result of a convergence of three CMS sub projects that respectively deal with scientific analysis, simulation and real time data aggregation from the experiment. This will reduce duplication and hence development and maintenance costs.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Nexus: a modular workflow management system for quantum simulation codes

The management of simulation workflows is a significant task for the individual computational researcher. Automation of the required tasks involved in simulation work can decrease the overall time to solution and reduce sources of human error. A new simulation workflow management system, Nexus, is presented to address these issues. Nexus is capable of automated job management on workstations and resources at several major supercomputing centers. Its modular design allows many quantum simulation codes to be supported within the same framework. Current support includes quantum Monte Carlo calculations with QMCPACK, density functional theory calculations with Quantum Espresso or VASP, and quantum chemical calculations with GAMESS. Users can compose workflows through a transparent, text-based interface, resembling the input file of a typical simulation code. A usage example is provided to illustrate the process.

97 MATHEMATICS AND COMPUTING↗

SWARM: Reimagining scientific workflow management systems in a distributed world

Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. In conclusion, this paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

Swarm intelligence↗

Holistic Approaches to HPC Power and Workflow Management

Constraints on power consumption are having pervasive effects on high-performance computing (HPC) systems, the facilities in which they are housed, and the application codes themselves. Whether these constraints are driven by physical limits on available power within a facility, are due to utility demand response, result from opportunities to reduce utility costs, or are imposed by sponsoring agencies, power management must now be added to the traditional goals of application and algorithm correctness, scalability, and performance. The specifics of these power constraints are likely to vary from site to site, and optimal strategies for managing power will vary. In this paper, we present the results of implementing several strategies for managing HPC system power and quantify the impact of these power reductions on application performance. These can be added to the suite of extant options that can be used to manage power and workflow on HPC systems.

97 MATHEMATICS AND COMPUTING↗

Holistic Approaches to HPC Power & Workflow Management: Preprint

Constraints on power consumption are having pervasive effects on high-performance computing (HPC) systems, the facilities in which they are housed, and the application codes themselves. Whether these constraints are driven by physical limits on available power within a facility, are due to utility demand response, result from opportunities to reduce utility costs, or are imposed by sponsoring agencies, power management must now be added to the traditional goals of application and algorithm correctness, scalability, and performance. The specifics of these power constraints are likely to vary from site to site, and optimal strategies for managing power will vary. In this paper, we present the results of implementing several strategies for managing HPC system power and quantify the impact of these power reductions on application performance. These can be added to the suite of extant options that can be used to manage power and workflow on HPC systems.

97 MATHEMATICS AND COMPUTING↗

Accelerating Science Impact through Big Data Workflow Management and Supercomputing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. ATLAS, one of the largest collaborations ever assembled in the the history of science, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. To manage the workflow for all data processing on hundreds of data centers the PanDA (Production and Distributed Analysis)Workload Management System is used. An ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF), is realizing within BigPanDA and megaPanDA projects. These projects are now exploring how PanDA might be used for managing computing jobs that run on supercomputers including OLCF’s Titan and NRC-KI HPC2. The main idea is to reuse, as much as possible, existing components of the PanDA system that are already deployed on the LHC Grid for analysis of physics data. The next generation of PanDA will allow many data-intensive sciences employing a variety of computing platforms to benefit from ATLAS experience and proven tools in highly scalable processing.

97 MATHEMATICS AND COMPUTING↗

Big PanDa Workflow Management on Titan for High Energy and Nuclear Physics and for Future Extreme Scale Scientific Application

Over a three year period, from 2016-2019, this project demonstrated the scientific benefits of integrating the Titan supercomputer at Oak Ridge Leadership Computing Facility into traditional high throughput grid based distributed computing systems managed by PanDA, the workflow management system used for the execution of all distributed computing applications by the ATLAS experiment at the Large Hadron Collider. PanDA manages millions of batch jobs daily at hundreds of clusters worldwide on request by thousands of physicist users, and processes more than an exabyte of data annually using grid middleware. High levels of operational use of Titan was sustained by PanDA in order to meet the physics goals of ATLAS. The success of this project led to the use of other supercomputers worldwide by ATLAS, and to the adoption of PanDA by other experiments and other scientists. Multiple innovative operational and computer science research goals were achieved supporting the use of supercomputers for scientific domains with large scale distributed data and distributed processing needs.

97 MATHEMATICS AND COMPUTING↗

The Evolution of the Pegasus Workflow Management Software

Since 2001 the Pegasus Workflow Management System has evolved into a robust and scalable system that automates the execution of a number of complex applications running on a variety of heterogeneous, distributed high-throughput, and high-performance computing environments. Pegasus was built on the principle of separation between the workflow description and workflow execution, providing the ability to port and adapt the workflow based on the target execution environment. Through its user-driven research and development it has adapted to the needs of a number of scientific communities, utilizing and developing novel algorithms and software engineering solutions. This paper describes the evolution of Pegasus over time and provides motivations behind the design decisions. The paper concludes with selected lessons learned.

97 MATHEMATICS AND COMPUTING↗

Risk Assessment and Management Workflow—An Example of the Southwest Regional Partnership

This paper summarizes the risk assessment and management workflow developed and applied to the Southwest Regional Partnership on Carbon Sequestration (SWP) Phase III Demonstration Project. The risk assessment and management workflow consists of six primary tasks, including management planning, identification, qualitative analysis, quantitative analysis, response planning, and monitoring. Within the workflow, the SWP assembled and iteratively updated a risk registry that identifies risks for all major activities of the project. Risk elements were ranked with respect to the potential impact to the project and the likelihood of occurrence. Both qualitative and quantitative risk analyses were performed. To graphically depict the interactions among risk elements and help building risk scenarios, process influence diagrams were used to represent the interactions. The SWP employed quantitative methods of risk analysis including Response Surface Method (RSM), Polynomial Chaos Expansion (PCE), and the National Risk Assessment Partnership (NRAP) toolset. The SWP also developed risk response planning and performed risk control and monitoring to prevent the risks from affecting the project and ensure the effectiveness of risk management. As part of risk control and monitoring, existing and new risks have been tracked and the response plan was subsequently evaluated. Findings and lessons learned from the SWP’s risk assessment and management efforts will provide valuable information for other commercial geological CO2 storage projects.

Lee, Si-Yong↗

A characterization of workflow management systems for extreme-scale applications

We present that the automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today’s computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. Finally, the paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

97 MATHEMATICS AND COMPUTING↗

Performance Understanding and Analysis for Exascale Data Management Workflows

The goal of the Exascale Data Management Workflows project is to create tools to support techniques for in-situ data analysis and data management. Online data management co-running with simulations accelerates the scientific processes being carried out, provides rapid and timely scientific insights, and can help avoid unnecessary and scientifically invalid simulation computations, particularly when combined with or able to utilize data from experimental instrument and/or from past simulation runs. The University of Oregon team is contributing to the performance instrumentation and analysis aspect of this project. Our focus has been on efficient performance monitoring of data management at scale. The general approach of the MONA project is depicted in Figure 1 below. The figure shows performance monitoring applied to an I/O workflow associated with a scientific simulation, with online measurements captured from this workflow informing methods for workflow reconfiguration and dynamic adjustment. The goal is to maintain suitable levels of Quality of Service for workflow execution, by understanding the under lying causes of workflow performance and behavior. One outcome is workflow performance models able to characterize realistic workflows. Another outcome is performance ’mini-apps’ implementing such workflow behavior. Out of scope for this project are advanced methods for online workflow control.

97 MATHEMATICS AND COMPUTING↗

The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate $\mathcal{O}$(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Just-In-Time Workflow Management for DUNE

The poster describes the justIN workflow management system funded by the UK for DUNE and now used for all DUNE centrally managed data processing and simulation

McNab, Andrew [CERN]↗

The ATLAS Workflow Management System Evolution in the LHC Run3 and towards the High-Luminosity LHC era

The ATLAS experiment has 18+ years of experience using workload management systems to deploy and develop workflows to process and to simulate data on the distributed computing infrastructure. Simulation, processing and analysis of LHC experiment data require the coordinated work of heterogeneous computing resources. In particular, the ATLAS experiment utilizes the resources of 250 computing centers worldwide, the power of supercomputing centres, and national, academic and commercial cloud computing resources. In this contribution, we present new techniques for cost-effectively improving efficiency introduced in workflow management system software. The evolution from a mesh framework to new types of computing facilities such as cloud and HPCs is described, as well as new types of production and analysis workflows.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Scientific Process Automation and Workflow Management

We introduce and describe scientific workflows, i.e., executable descriptions of automatable scientific processes such as computational science simulations and data analyses. Scientific workflows are often expressed in terms of tasks and their (data ow) dependencies. This chapter first provides an overview of the characteristic features of scientific workflows and outlines their life cycle. A detailed case study highlights workflow challenges and solutions in simulation management. We then provide a brief overview of how some concrete systems support the various phases of the workflow life cycle, i.e., design, resource management, execution, and provenance management. We conclude with a discussion on community-based workflow sharing.

97 MATHEMATICS AND COMPUTING↗

Automatic Imagery Data Analysis for Proactive Computer-Based Workflow Management during Nuclear Power Plant Outages

This report is being submitted for the task “Final Report” of DOE NEUP Project 15-8121 “Automatic Imagery Data Analysis for Proactive Computer-Based Workflow Management during Nuclear Power Plant Outages.” Typical nuclear power plant (NPP) outages always involve thousands of maintenance and refueling activities and a large number of workers in limited workspaces, while having tight schedules and zero-tolerance for accidents. During an outage, thousands of workers will be working in various workspaces across the NPP. High outage costs and expensive delays (approximately 1.5 million dollars of loss per day of delay) in NPP maintenance demand tight outage schedules. In packed workspaces, an automatic system that monitors human behaviors in real-time and provides insights about current and pending schedule deviations from the plan is critical for ensuring: 1) effective collaboration among workers and worker teams from different trades; 2) less waste of time and resources due to the lack of situational awareness; and 3) proactive outage project control. The overall goal of this project is to test the hypothesis that real-time imagery-based object tracking and spatial analysis, as well as human behavior modeling of outage participants, will significantly improve the efficiency of outage control while lowering the rates of accidents and incidents. Three objectives of this project are: 1) Establish real-time object tracking and spatiotemporal analysis methods that automatically assess the productivity of field activities and detect anomalous spatiotemporal relationships among activities that cause inefficiencies and risks; 2) Establish realtime human tracking and human factor modeling methods for automatically diagnosing unexpected actions of and interactions between outage participants, those which cause inefficient collaborations between Advanced Outage Control Center (AOCC), satellite outage centers, NPP workers, and maintenance service providers; and 3) Test the proposed automated object tracking, human behavior modeling, and spatiotemporal analysis methods in outage control case studies in order to characterize the effectiveness of automated imagery-data-driven methods in proactively improving the efficiency and safety of workflows in outage coordination and risk management. Recent studies of detailed human behavior monitoring on construction sites have examined the potential of applying advanced computer vision algorithms in detecting and tracking anomalous workers (i.e., workers who do not wear hard hats or safety vests) for ensuring job site safety. Some studies of human factor studies revealed the importance of modeling detailed interactions between individuals within and across teams in better understanding the impact of human in proactive project control. Other studies in the construction domain have developed computational simulation frameworks that formalize detailed spatiotemporal interactions between tasks in order to simulate the impacts of individual tasks on the performance of workflows. While integrated analyses that combine human factor assessment, as well as image processing and simulation, are in demand for effective decision-making, limited studies have examined the potential of such integrated analyses in NPP outage control. This research project has examined an automatic outage monitoring and control system that integrates human factor analysis, computer vision techniques, and simulation methods in order to enable engineers to better understand the interactions between humans, resources, and workflows during outage processes. The project aims at providing NPP maintenance agencies insights into more efficient use of limited resources in extending the life of a nuclear plant, as well as reducing waste while ensuring sufficient generation of electricity. This study is significant for the safety of nuclear plants, sustainable electricity generation for livable communities, and cost savings for maintaining electricity infrastructures in the United States.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗