Search NASA⌕ Search

SEARCH · Search NASA

Results for “computational workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Advancing subsurface analysis: Integrating computer vision and deep learning for the near real-time interpretation of borehole image logs in the Illinois Basin-Decatur Project

The accurate quantification and mapping of subsurface natural fracture systems using borehole imaging logs are critical for the success of CO 2 sequestration in geologic formations, optimization of engineered geothermal systems, and hydrocarbon production enhancement. However, traditional interpretation processes suffer from time-consuming procedures and human bias. To address these challenges and expedite fracture analysis, we investigated the application of integrated computer vision and DL workflows to automate image log analysis. Specifically, the design of our workflow was crafted to swiftly detect fractures and baffles by using actual electrical resistivity of borehole wall from microresistivity imaging device alongside their binary representation. This novel approach significantly reduces computational time while providing invaluable insights. By incorporating conventional logging and microseismic data, we present a regional subsurface natural fracture mapping technique. Through the minimization of human bias in image log analysis, our automated workflow achieves reduced fracture interpretation time and costs while ensuring robust and reproducible results. We demonstrated the efficacy of our approach by applying the workflow to the Illinois Basin-Decatur Project site. The automated workflow successfully identified major fractured zones, multiple baffles, and an interbedded layer with a high resolution of 0.01 ft or 0.12 in. (0.3 cm) and can be upscaled to any desired resolution. Validation through microseismic and image log interpretations allows for accurate and near-real-time mapping of fractures and baffles, significantly enhancing CO 2 pressure forecasting and postinjection site care. Our approach stands out due to its robustness, consistency, and reduced computational cost compared with alternative feature extraction technologies. It presents exciting possibilities for advancing CO 2 sequestration and engineered geothermal efforts by offering comprehensive and efficient fracture mapping solutions. This technology can contribute significantly to the optimization of CO 2 sequestration projects, facilitating sustainable environmental practices, and combating climate change.

Geochemistry & Geophysics↗

$\mathrm{RADICAL}$-Pilot and $\mathrm{PMIx}$/$\mathrm{PRRTE}$: Executing Heterogeneous Workloads at Large Scale on Partitioned $\mathrm{HPC}$ Resources

Execution of heterogeneous workflows on high-performance computing (HPC) platforms present unprecedented resource management and execution coordination challenges for runtime systems. Task heterogeneity increases the complexity of resource and execution management, limiting the scalability and efficiency of workflow execution. Re-source partitioning and distribution of tasks execution over portioned re-sources promises to address those problems but we lack an experimental evaluation of its performance at scale. Here this paper provides a performance evaluation of the Process Management Interface for Exascale (PMIx) and its reference implementation PRRTE on the leadership-class HPC plat-form Summit, when integrated into a pilot-based runtime system called RADICAL-Pilot. We partition resources across multiple PRRTE Distributed Virtual Machine (DVM) environments, responsible for launching tasks via the PMIx interface. We experimentally measure the work-load execution performance in terms of task scheduling/launching rate and distribution of DVM task placement times, DVM startup and termination overheads on the Summit leadership-class HPC platform. Integrated solution with PMIx/PRRTE enables using an abstracted, standardized set of interfaces for orchestrating the launch process, dynamic process management and monitoring capabilities. It extends scaling capabilities allowing to overcome a limitation of other launching mechanisms (e.g., JSM/LSF). Explored different DVM setup configurations provide insights on DVM performance and a layout to leverage it. Our experimental results show that heterogeneous workload of 65,500 tasks on 2048 nodes, and partitioned across 32 DVMs, runs steady with resource utilization not lower than 52%. While having less concurrently executed tasks resource utilization is able to reach up to 85%, based on results of heterogeneous workload of 8200 tasks on 256 nodes and 2 DVMs.

97 MATHEMATICS AND COMPUTING↗

Science Capsule: Towards Sharing and Reproducibility of Scientific Workflows

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Ghoshal, Devarshi↗

Optimization of Water-Alternating-CO2 Injection Field Operations Using a Machine-Learning-Assisted Workflow

Summary This paper will present a robust workflow to address multiobjective optimization (MOO) of carbon dioxide (CO2)-enhanced oil recovery (EOR)-sequestration projects with a large number of operational control parameters. Farnsworth unit (FWU) field, a mature oil reservoir undergoing CO2 alternating water injection (CO2-WAG) EOR, will be used as a field case to validate the proposed optimization protocol. The expected outcome of this work would be a repository of Pareto-optimal solutions of multiple objective functions, including oil recovery, carbon storage volume, and project economics. FWU’s numerical model is used to demonstrate the proposed optimization workflow. Because using MOO requires computationally intensive procedures, machine-learning-based proxies are introduced to substitute for the high-fidelity model, thus reducing the total computation overhead. The vector machine regression combined with the Gaussian kernel (Gaussian-SVR) is used to construct proxies. An iterative self-adjusting process prepares the training knowledge base to develop robust proxies and minimizes computational time. The proxies’ hyperparameters will be optimally designed using Bayesian optimization to achieve better generalization performance. Trained proxies will be coupled with multiobjective particle swarm Optimization (MOPSO) protocol to construct the Pareto-front solution repository. The outcomes of this workflow will be a repository containing Pareto-optimal solutions of multiple objectives considered in the CO2-WAG project. The proposed optimization workflow will be compared with another established methodology using a multilayer neural network (MLNN) to validate its feasibility in handling MOO with a large number of parameters to control. Optimization parameters used include operational variables that might be used to control the CO2-WAG process, such as the duration of the water/gas injection period, producer bottomhole pressure (BHP) control, and water injection rate of each well included in the numerical model. It is proved that the workflow coupling Gaussian-SVR proxies and the iterative self-adjusting protocol is more computationally efficient. The MOO process is made more rapid by squeezing the size of the required training knowledge base while maintaining the high accuracy of the optimized results. The outcomes of the optimization study show promising results in successfully establishing the solution repository considering multiple objective functions. Results are also verified by validating the Pareto fronts with simulation results using obtained optimized control parameters. The outcome from this work could provide field operators an opportunity to design a CO2-WAG project using as many inputs as possible from the reservoir models. The proposed work introduces a novel concept that couples Gaussian-SVR proxies with a self-adjusting protocol to increase the computational efficiency of the proposed workflow and to guarantee the high accuracy of the obtained optimized results. More importantly, the workflow can optimize a large number of control parameters used in a complex CO2-WAG process, which greatly extends its utility in solving large-scale MOO problems in various projects with similar desired outcomes.

Energy & Fuels↗

Performance Characterization and Provenance of Distributed Task-based Workflows on HPC Platforms

Understanding performance and provenance of task-based workflows poses significant challenges, particularly in distributed configurations where resources are shared by multiple applications. Task-based workflow management systems further complicate performance predictability because of their dynamicity that subtly alters task execution order from run to run. In this paper we propose a layered characterization framework for performance and task provenance for Dask.distributed workflows running on high-performance computing (HPC) platforms. It collects data from jobs, the workflow management system, and the operating system to aid in understanding the performance of these workflows. Our approach encompasses three main contributions: first, an extension of Dask.distributed to capture high-fidelity task provenance using Mochi data services; second, the adaptation of the established HPC I/O characterization tool Darshan to gather high-fidelity I/O data, thereby enhancing the granularity of our analysis; and third, a framework to combine and process the collected data and provide helpful insights into performance characterization and reproducibility, alongside our lessons learned.

Dask↗

Integration and Commissioning of a Prototype Federated Cloud for Scientific Workflows

The Fermilab Grid and Cloud Computing Department and the KISTI Global Science experimental Data hub Center propose a joint project. The goals are to enable scientific workflows of stakeholders to run on multiple cloud resources by use of (a) Virtual Infrastructure Automation and Provisioning, (b) Interoperability and Federat ion of Cloud Resources , and (c) High-Throughput Fabric Virtualization. This is a matching fund project in which Fermilab and KISTI will contribute equal resources .

97 MATHEMATICS AND COMPUTING↗

Accelerating Scientific Computing in the Post-Moore’s Era

Novel uses of graphical processing units for accelerated computation revolutionized the field of high-performance scientific computing by providing specialized workflows tailored to algorithmic requirements. As the era of Moore’s law draws to a close, many new non–von Neumann processors are emerging as potential computational accelerators, including those based on the principles of neuromorphic computing, tensor algebra, and quantum information. While development of these new processors is continuing to mature, the potential impact on accelerated computing is anticipated to be profound. We discuss how different processing models can advance computing in key scientific paradigms: machine learning and constraint satisfaction. Significantly, each of these new processor types utilizes a fundamentally different model of computation, and this raises questions about how to best use such processors in the design and implementation of applications. While many processors are being developed with a specific domain target, the ubiquity of spin-glass models and neural networks provides an avenue for multi-functional applications. Furthermore, this also hints at the infrastructure needed to integrate next-generation processing units into future high-performance computing systems.

97 MATHEMATICS AND COMPUTING↗

Accelerated, scalable and reproducible AI-driven gravitational wave detection

The development of reusable artificial intelligence (AI) models for wider use and rigorous validation by the community promises to unlock new opportunities in multi-messenger astrophysics. Here we develop a workflow that connects the Data and Learning Hub for Science, a repository for publishing AI models, with the Hardware-Accelerated Learning (HAL) cluster, using funcX as a universal distributed computing service. Using this workflow, an ensemble of four openly available AI models can be run on HAL to process an entire month's worth (August 2017) of advanced Laser Interferometer Gravitational-Wave Observatory data in just seven minutes, identifying all four binary black hole mergers previously identified in this dataset and reporting no misclassifications. This approach combines advances in AI, distributed computing and scientific data infrastructure to open new pathways to conduct reproducible, accelerated, data-driven discovery. By combining a repository for artificial intelligence models and a supercomputing cluster, an entire month's worth of advanced LIGO data is analysed in just 7 min, finding all binary black hole mergers previously identified in this dataset and reporting no misclassifications.

79 ASTRONOMY AND ASTROPHYSICS↗

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

The computing resource needs of LHC experiments are expected to continue growing significantly during the Run 3 and into the HL-LHC era. The landscape of available resources will also evolve, as High Performance Computing (HPC) and Cloud resources will provide a comparable, or even dominant, fraction of the total compute capacity. The future years present a challenge for the experiments’ resource provisioning models, both in terms of scalability and increasing complexity. The CMS Submission Infrastructure (SI) provisions computing resources for CMS workflows. This infrastructure is built on a set of federated HTCondor pools, currently aggregating 400k CPU cores distributed worldwide and supporting the simultaneous execution of over 200k computing tasks. Incorporating HPC resources into CMS computing represents firstly an integration challenge, as HPC centers are much more diverse compared to Grid sites. Secondly, evolving the present SI, dimensioned to harness the current CMS computing capacity, to reach the resource scales required for the HLLHC phase, while maintaining global flexibility and efficiency, will represent an additional challenge for the SI. To preventively address future potential scalability limits, the SI team regularly runs tests to explore the maximum reach of our infrastructure. In this note, the integration of HPC resources into CMS offline computing is summarized, the potential concerns for the SI derived from the increased scale of operations are described, and the most recent results of scalability test on the CMS SI are reported.

Pérez-Calero Yzquierdo, Antonio↗

A High Performance Computational Framework for Dynamic Security Assessment under Uncertainty

Dynamic security assessment (DSA) is a critical function to evaluate power grids’ capability to survive the transition caused by a set of disturbances to an acceptable steady-state condition. Its computational burden is heavy. With the challenges brought by renewable energy and new smart grid technologies, DSA under uncertainty has to be considered to study the impact of forecast errors on DSA simulation, which further increases the computational burden. To address this challenge, this paper presents a high performance computational framework to support DSA simulation user uncertainty. The computational framework aims to provide a seamless workflow that links data from high performance computing, statistical analysis, to visualization so that a problem can be easily expressed in a way compatible with different functions. It also enables software compatibility such that application development can be more efficient. Case study results of the ESCA60 system and a western U.S. size system show the advantages and efficiency of the computational framework.

dynamic security assessment, high performance comp↗

Echo: Data and Analysis Management [Slides]

Echo software capabilities are described. Echo provides a comprehensive suite of tools for data wrangling, management, and analysis. It is an object-oriented approach to data analysis designed for robust and scalable execution of complex analysis workflows written in MATLAB.

97 MATHEMATICS AND COMPUTING↗

Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS

In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.

97 MATHEMATICS AND COMPUTING↗

Empowering Scientific Innovation Through An Integrated Research Infrastructure: The Role of the Advanced Computing Ecosystem

As the landscape of computational science evolves, the Department of Energy (DOE) is reimagining the roles of its large-scale computing facilities to meet emerging research challenges. The Integrated Research Infrastructure (IRI) program aims to transform how experiments are designed, conducted, and shared, with significant impacts on all stakeholders. In response, the Oak Ridge Leadership Computing Facility (OLCF) has established the Advanced Computing Ecosystem (ACE), a strategic framework to prepare its hardware, software, and experimental capabilities for the IRI era. ACE focuses on integrating novel compute environments, orchestrating advanced workflows, and developing foundational technologies, ensuring a seamless transition to IRI while accelerating scientific discovery. This paper outlines ACE's role in advancing OLCF's mission and its impact on the future of computational science.

Widener, Patrick↗

Improv Dynamic Workflows

A workflow is a series of dependent computations which are executed to yield an experimental result, much like sheet music describes a musical performance. Dynamic workflows are like improvisational jazz, in which the musicians create unique, situation-driven, collaborative variations on themes. Dynamic workflows can potentially yield better results, save computation, and/or save human decision-making effort versus pre-specified or manual workflows, as they need not run the exhaustive set of studies which might be required in a pre-specified set, and they make their own experimental design decisions at run time. If you have an existing Maestro study where you would like to save computation and/or human time, Improv is for you. Improv provides core capabilities for creating a simple, hierarchical structure over Maestro studies, allowing you to connect ``experimental'' Maestro strudies with decision-making code to select parameters and run corresponding studies.

Goforth, John↗

The role of machine learning in scientific workflows

Machine learning (ML) is being applied in a number of everyday contexts from image recognition, to natural language processing, to autonomous vehicles, to product recommendation. In the science realm, ML is being used for medical diagnosis, new materials development, smart agriculture, DNA classification, and many others. In this article, we describe the opportunities of using ML in the area of scientific workflow management. Scientific workflows are key to today’s computational science, enabling the definition and execution of complex applications in heterogeneous and often distributed environments. We describe the challenges of composing and executing scientific workflows and identify opportunities for applying ML techniques to meet these challenges by enhancing the current workflow management system capabilities. We foresee that as the ML field progresses, the automation provided by workflow management systems will greatly increase and result in significant improvements in scientific productivity.

97 MATHEMATICS AND COMPUTING↗

Unstructured Grid Development for the Space Launch System Liftoff and Transition Lineloads Computational Analysis

Production of aerodynamic lineloads databases for the Space Launch System (SLS) vehicle at Liftoff and Transition (LOT) conditions has required the development of a Computational Fluid Dynamics (CFD) workflow capable of producing high-quality solutions for this unique phase of flight. Aerodynamic considerations included a wide range of flow angles (from 0°up to 90°total angle of attack), resulting leeside separation, and interaction effects between the three bodies of the integrated SLS vehicle, as well as the nearby launch tower. Computational mesh development for similar problems at the NASA Langley Research Center (such as for the Constellation/Ares launch vehicles) has primarily relied on in-house tools such as VGRID/POSTGRID, with grids designed for NASA-developed and maintained flow solvers such as USM3D and FUN3D. The workflow for such problems has evolved over the development of the various SLS configurations to incorporate new tools such as the Heldenpatch/Heldenmesh grid generator (Helden Aerospace) and CREATE-AV Kestrel (US Department of Defense) flow solver. This paper describes efforts to benchmark a grid generation approach for LOT problems using Heldenpatch/Heldenmesh and Kestrel, verified against prior best practices from VGRID/POSTGRID. Parameters studied include surface grid density, first-layer viscous cell height, and volume grid growth rate parameters. Resulting solutions are compared based on total force and moment values, sectional line loads, and surface pressures, all validated against existing wind tunnel aerodynamic databases where available for the SLS Block 1B Cargo configuration.

Space launch systems↗

Frontiers in computing for artificial intelligence

An emerging diversity of computational platforms offers many different approaches to adopting the paradigm of artificial intelligence to the study of electron-ion collisions. Here we review several leading candidates in this computational frontier and their workflows for experimental applications of artificial intelligence that may impact the future Electron-Ion Collider. We discuss the motivation for exploring novel methods to solve artificial intelligence and machine learning problems including with customized devices, quantum simulation, and heterogeneous computing systems. Furthermore, these technologies offer promising approaches to address some of the leading concerns of future computing that may impact the Electron-Ion Collider but they will require further development and testing in order to support future planning efforts.

detector design and construction technologies and ↗

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing (HPC) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS) that used a Virtual Data Space (VDS) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗