Search NASA⌕ Search

SEARCH · Search NASA

Results for “Workflow Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Advances and Enhancements in the Fabric for Frontier Experiments Project at Fermilab

The FabrIc for Frontier Experiments (FIFE) project within the Fermilab Scientific Computing Division is charged with integrating offline computing components into a common computing stack for the non-LHC Fermilab experiments, supporting experiment offline computing, and consulting on new, novel workflows. We will discuss the general FIFE onboarding strategy, the upgrades and enhancements in the FIFE toolset, and plans for the coming year. These enhancements include: expansion of opportunistic computing resources (including GPU and high-performance computing resources) available to experiments; assistance with commissioning computing resources at European sites for individual experiments; StashCache repositories for experiments; enhanced job monitoring tools; and a custom workflow management service. Additionally we have completed the first phase of a Federated Identity Management system to make it easier for FIFE users to access Fermilab computing resources.

Herner, Kenneth↗

Economic Assessment of Nuclear Hybrid Energy Systems: Nuclear-Renewable-Water Integration in Arizona

One of the goals of the Nuclear-Renewable Hybrid Energy Systems (N-R HES) modeling and simulation (M&S) project is to assess the economic viability of N-R HES in a market that contains Variable Renewable Energy (VRE) sources like solar. The N-R HES M&S project is a multi-laboratory effort, led by Idaho National Laboratory (INL) and supported by Argonne National Laboratory (ANL) and Oak Ridge National Laboratory (ORNL). Development of the N-R HES software framework was initiated at INL in 2016 [1] and has reached some level of maturity such that it can be applied to more than simple demonstration cases, e.g. real industry problems. Nevertheless, more capabilities are constantly added to accommodate the special needs of challenging, real-life problems. The N-R HES framework is built on the Risk Analysis Virtual ENvironment (RAVEN) code [2], which it uses as a driver and workflow manager for all calculations. There are four main cornerstones of the N-R HES simulation framework: 1) The generation of stochastic synthetic time series. 2) A set of probabilistic analysis and optimization algorithms available in RAVEN. 3) A set of models for representing the physical behavior of N-R HES. These can be reduced-order models or high-fidelity Modelica [3] models. 4) The RAVEN plug-in called CashFlow [4] that maps physical performance into economic performance. Within this framework, a broad spectrum of questions related to N-R HES can be addressed. One of the challenges currently of high interest is the increasing penetration of VRE altering the profile of the net demand (demand after removing all non-curtailable renewable energy sources, e.g. rooftop solar). The N-R HES software framework is capable of analyzing the potential feasibility of mitigating the resultant volatility in the net electricity demand. One possible solution to manage net demand volatility is adding stabilizing loads to the grid. These loads can be external industrial processes that are able to ramp production up and down quickly, or for regulated markets, adding variable internal loads that will effectively increase the baseload of the power plant and therefore reduce the risk of internal demand that is lower than the capacity of the power plant. The latter is a possible solution currently being considered by Arizona Public Service (APS), the operating owner of the Palo Verde (PV) Nuclear Power Plant (NPP).

97 MATHEMATICS AND COMPUTING↗

Workflow Submit Nodes as a Service on Leadership Class Systems

DOE scientists, today, have access to high performance computing (HPC) facilities with very powerful systems that enable them to execute their computations faster, more efficiently, and at greater scales than ever before. To further their knowledge and produce new discoveries, scientists rely on workflows - sometimes very complex - that provide them with an easy way to automate, reproduce and verify their computations. However, historically, creating workflow submission environments in large HPC facilities has been cumbersome, requires expertise and many man-hours of effort due to the peculiarities, policies, and the restrictions that these systems present. In this paper we discuss the approach a large DOE facility (OLCF) is taking in order to provide containers as a service to its users. This capability is used to create Pegasus workflow management system submit nodes as a service (WSaaS) at the Oak Ridge Leadership Computing Facilities (OLCF), targeting the Summit supercomputer. This deployment builds upon the Kubernetes/Openshift cluster (Slate) that exists within OLCF’s DMZ and its automation triggers. Additionally, we evaluate our approach’s overhead and effort to deploy the solution as compared to previous solutions, such as setting up a Pegasus submission environment on OLCF’s login nodes or submitting jobs remotely via the rvGAHP.

Papadimitriou, George↗

Vectorization of CMSSW offline software

The CMS experiment has been utilizing vectorization, or SIMD, in parts of its data processing applications for over a decade. On x86 platforms the vectorization level is still SSE3. In the past attempts to use wider vector instruction sets such as AVX or AVX-512 have, in practice, not resulted in improvements in the overall event processing throughput, because the CPUs scale down their frequency when processing AVX instructions. In addition, a notable part of the global pool of CMS’ resources has been old systems either not supporting AVX, or where the CPU frequency downscaling impacts all cores of the CPU. CMS has nevertheless continued to vectorize more of its application code, and in this work we review profiling methods we have found effective to find out pieces of code that would benefit from vectorization, and techniques to transform those codes such that the GCC compiler is able to auto-vectorize those codes. The build system used for CMSSW, Scram, has also been enha nced to be able to build code for multiple CPU microarchitectures such that the shared libraries of desired microarchitecture level can be loaded based on the CPU of the system. This multi-microarchitecture setup is invisible to the workflow management system, which makes its deployment straightforward. We describe in detail how this multi-microarchitecture build is set up, and measure the impact of using wider vector units than SSE3 on the event processing throughput of CMS applications such as simulation and reconstruction on recent x86 CPUs.

Gartung, Patrick↗

Interactive Visualization of Multi-Group Cross Sections on High-Fidelity Spatial Meshes

Core simulation tools utilizing Monte Carlo transport methods represent the 'Holy Grail' for nuclear reactor analysis. However, deterministic transport methods pose a number of advantages to Monte Carlo from a computational perspective, which seems likely to make these methods more viable in the near to intermediate term. Deterministic methods almost universally discretize the energy domain in a few to hundreds of energy groups. One of the key challenges for deterministic transport-based methods to be viable is accurate, reactor agnostic multi-group cross section generation. This paper describes tools for data processing and visualization of multigroup cross sections generated on high-fidelity spatial meshes from the OpenMC Monte Carlo code. Next generation deterministic transport reactor core modeling tools will require multi-group cross section (MGXS) generation on high-fidelity spatial meshes. High-fidelity MGXS are necessary for the accurate simulation of neutron physics in nuclear reactors with a high degree of spectral heterogeneity - e.g., large localized neutron flux gradients throughout a reactor core. Although many local (e.g., pin cell) approximations have been developed to model the scalar flux to compute spectral-averaged MGXS, these techniques cannot account for complex intra-pin and intra-assembly spatial self-shielding effects. Monte Carlo presents the most accurate method for reactor agnostic multi-group cross section generation since it does not require the use of local approximations to the flux. Monte Carlo has been proposed as a highly desirable framework for reactor agnostic cross section generation for many years. Monte Carlo methods can effectively embed all of the relevant spectral effects into the MGXS through the random particle tracking and interaction sampling processes. Previous work has applied MCNP and Serpent to compute few-group constants for diffusion calculations, and OpenMC to compute multi-group cross sections for transport-based methods. The primary difficulty with using Monte Carlo for MGXS generation is the tremendous computational resources required to adequately converge multi-group cross sections in each spatial region and energy group. Recent development of the OpenMC Monte Carlo code for high-performance computing platforms has made multigroup cross section generation from Monte Carlo more tractable in the foreseeable future. The recent release of OpenMC v0.7.1 includes a new Python-based module for MGXS generation. The MGXS generation module leverages many of the algorithms and features for 'big data' analysis of large tally datasets on complex spatial meshes. This paper describes the data management workflow and visualization tools which have been developed to explore the plethora of MGXS data generated with OpenMC. It is the authors' hope that the development of the visualization tool described in this paper will ultimately enable the design of machine learning algorithms to accelerate the convergence rate of high-fidelity multi-group cross section generated with Monte Carlo. (authors)

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments

The annual Workshop on Workflows in Support of Large-Scale Science (WORKS) is a premier venue for the scientific workflow community to present the latest advances in research and development on the many facets of scientific workflows throughout their life-cycle. The Lightning Talks at WORKS focus on describing a novel tool, scientific workflow, or concept, which are work-in-progress and address emerging technologies and frameworks to foster discussion in the community. This paper summarizes the lightning talks at the 2023 edition of WORKS, covering five topics: leveraging large language models to build and execute workflows; developing a common workflow scheduler interface; scaling uncertainty workflow applications on exascale computing systems; evaluating a transcriptomics workflow for cloud vs. HPC systems; and best practices in migrating legacy workflows to workflow management systems.

Titov, Mikhail↗

Enabling Interconnected Science Workflows through an Adapter Approach

The INTERSECT Software framework project aims to create an open federated library that connects, coordinates, and controls systems in the scientific domain. It features the Adapter, a flexible and extensible interface inspired by the Adapter design pattern in object-oriented programming. By utilizing Adapters, the INTERSECT SDK enables effective communication and coordination within a diverse ecosystem of systems. This adaptability facilitates the execution of complex operations within the framework, promoting collaboration and efficient workflow management in scientific research. Additionally, the generalizability of Adapters and their patterns enhances their utility in other scientific software projects and challenges.

McGaha, Jesse↗

Autonomic Management of Application Workflows on Hybrid Computing Infrastructure

In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.

Kim, Hyunjoo↗

Building a Cyberinfrastructure for Environmental System Science: Modeling Frameworks, Data Management, and Scientific Workflows, Workshop Report, Potomac, Maryland, April 30-May 1, 2015

The Environmental System Science (ESS) activity within the Department of Energy’s (DOE) Office of Biological and Environmental Research (BER) seeks to advance a robust, predictive understanding of terrestrial environments, extending from bedrock to the top of the vegetative canopy and from molecular to global scales, through an iterative cycle of model-driven experimentation and observation dubbed MODEX. Considerable progress has been made toward achieving this overarching goal, but widely recognized is the fragmentation across projects and disciplines of the relevant modeling and simulation capabilities, observational and experimental data, analysis algorithms, and workflow tools. This fragmentation creates significant challenges for studying impacts and feedbacks in these complex multiscale systems. These challenges are further exacerbated by ongoing disruptive changes in high-performance computational architectures and the exponential growth in the types and volume of data that render obsolete the conventional approaches to software development and data management. Overcoming these challenges will require the development of a BER Climate and Environmental Sciences Division (CESD)–wide enabling cyberinfrastructure to support data management, cross-domain modeling, data analysis, and collaborative research. To explore the potential for working groups to initiate and guide a more integrated and community-based cyberinfrastructure, BER held the ESS Workshop on Model-Data Integration: Modeling Frameworks, Data Management, and Scientific Workflows on April 30-May 1, 2015, following the ESS Principal Investigator Meeting in Potomac, Maryland. Participants included model developers, software engineers, and data management specialists from eight national laboratories, which represented a wide range of projects and programs from CESD as well as cross-cutting projects from DOE’s Office of Advanced Scientific Computing Research (ASCR; both BER and ASCR are operated from within DOE’s Office of Science). A series of plenary talks provided background information and clarified the workshop’s three objectives: (1) develop requirements for this community-based cyberinfrastructure to ensure enhanced scientific productivity of the community as a whole; (2) identify challenges associated with developing this new cyberinfrastructure using a phased approach guided by project-driven use cases; and, given these requirements and challenges, (3) chart a path forward for ESS working groups to lead the phased development of the new cyberinfrastructure.

54 ENVIRONMENTAL SCIENCES↗

Towards a Standard Process Management Infrastructure for Workflows Using Python

Orchestrating the execution of ensembles of processes lies at the core of scientific workflow engines on large scale parallel platforms. This is usually handled using platform-specific command line tools, with limited process management control and potential strain on system resources. The PMIx standard provides a uniform interface to system resources. The low level C implementation of PMIx has hampered its use in workflow engines, leading to the development of Python binding that has yet to gain traction. In this paper, we present our work to harden the PMIx Python client, demonstrating its usability using a prototype Python driver to orchestrate the execution of an ensemble of processes. We present experimental results using the prototype on the Summit supercomputer at Oak Ridge National Laboratory. This work lays the foundation for wider adoption of PMIx for workflow engines, and encourages wider support of more PMIx functionality in vendor provided system software stacks.

Elwasif, Wael↗

Data management and processing workflow for the Materials Physics and Engineering group beamlines at the Advanced Photon Source

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is particularly important for large user facilities like the Advanced Photon Source. In this article, the deployment of the APS Data Management System (DM) at the 1-ID and 6-BM beamlines of the APS is described. These two beamlines support a wide range of experimental techniques and generate data at relatively high rates, making them ideal candidates to illustrate the deployment and customization of the DM system and its tools. Using several usage examples at these beamlines, various capabilities of the DM system are described.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

The future of self-driving laboratories: from human in the loop interactive AI to gamification

Recent developments in artificial intelligence (AI) and machine learning (ML), implemented through self-driving laboratories (SDLs), are rapidly creating unprecedented opportunities for the accelerated discovery and optimization of materials. This paper provides a joint analysis of SDLs from both academic and industry perspectives, highlighting the importance of integrating human intelligence in these systems. It discusses the necessity of careful planning in SDL design across physical, data, and workflow dimensions, including instrumental setup, experimental workflow, data management, and human–SDL interaction. The significance of integrating human input within SDLs, especially as the focus shifts from individual tools and tasks to the creation and management of complex workflows, is emphasized. The paper stresses on the crucial role of reward function design in developing forward-looking workflows and examines the interplay between hardware evolution, ML application across chemical processes, and the influence of reward systems in research. Ultimately, the article advocates for a future where SDLs blend human intuition in hypothesis formulation with AI's precision, speed, and data-handling capabilities.

97 MATHEMATICS AND COMPUTING↗

Automated generation of scientific workflow generators with WfChef

Scientific workflow applications have gained significant importance, and their automated and efficient execution on large-scale computing platforms has been the subject of extensive research and development. For these efforts to be successful, a solid experimental methodology is needed to evaluate workflow algorithms and systems. A foundation for this methodology is the availability of realistic workflow instances. Although public repositories provide workflow instances for a few scientific applications, these are limited in scope, and workflow instances are not available for all application scales of interest. To address this limitation, previous work has developed generators of synthetic workflow instances of arbitrary scales. Despite being popular, the implementation of these generators is a manual and labor-intensive process that requires expert application knowledge. As a result, these generators only target a handful of applications, even though there are hundreds of workflow applications in production. Here, we introduce WfChef , a fully automated framework for constructing a synthetic workflow generator for any scientific application. Based on an input set of workflow instances for a particular application, WfChef automatically produces a synthetic workflow generator. To measure the realism of the generated workflows, we define and evaluate several metrics. Using these metrics, we compare the realism of the workflows generated by WfChef generators to that of the workflows generated by the previously available, hand-crafted generators. We find that WfChef generators not only require zero development effort (because they are automatically produced), but also generate workflows that are more realistic than those generated by hand-crafted generators.

97 MATHEMATICS AND COMPUTING↗