Search NASA⌕ Search

SEARCH · Search NASA

Results for “computational workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24

Streaming Data in HPC Workflows Using ADIOS

The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗

Deploying Machine Learning Workflows into HPC environment

Outline: Workflows Overview; Common Workflow Language (CWL), Example of a CWL; BEE Overview; Machine Learning Components; Machine Learning Scientific Workflow using CWL → A new test case for BEE; Discussion: Benefits and Caveats of current ML workflow; Conclusion.

97 MATHEMATICS AND COMPUTING↗

Predictive Modeling and Diagnostic Monitoring of Extreme Science Workflows (Final Report)

This proposal addresses a critical issue of performance prediction identified in the report from the ASCR \Computational Modeling of Big Networks (COMBINE)" workshop: "end-to-end performance is not predictable due to a variety of factors. Even when some performance forecasts or predictions can be made, they often cannot explain the reasons why some predictions fail." We will develop new analytical models to predict the end-to-end performance of scientific workflows on DOE computing infrastructures, and use simulations and experimentation to validate and refine these models, as well as to pinpoint the sources of model inaccuracy. We will also use these models to help diagnose application and infrastructure problems, and to adapt the system based on this diagnosis. This section provides background in the areas relevant to the proposed work. RPI’s specific tasks within the Panorama project are as follows: (1) Develop Aspen-Simulation interface for Workflow Model Driven Simulation. (2) Validate manual performance models of two target workflow scenarios with empirical measurement and simulation. (3) Extend ROSS-Aspen API to simulate workflow descriptions when required. (4) Validate Aspen performance models of two target workflow scenarios with automatic performance model empirical measurement and simulation. (5) Design and implement final system to automatically generate Aspen performance models from workflow descriptions (including methods to compensate for limitations of Aspen analytical models). (6) Validate improved Aspen performance models with target workflow on production infrastructure. To date, all the project milestones assigned to us where reached within the best of our abilities over the course of the project performance period. Below describes the key outcome from our collaborative research in a system named, Durango .

97 MATHEMATICS AND COMPUTING↗

Ad Hoc File Systems for High-Performance Computing

Storage backends of parallel compute clusters are still based mostly on magnetic disks, while newer and faster storage technologies such as flash-based SSDs or non-volatile random access memory (NVRAM) are deployed within compute nodes. Including these new storage technologies into scientific workflows is unfortunately today a mostly manual task, and most scientists therefore do not take advantage of the faster storage media. One approach to systematically include nodelocal SSDs or NVRAMs into scientific workflows is to deploy ad hoc file systems over a set of compute nodes, which serve as temporary storage systems for single applications or longer-running campaigns. This paper presents results from the Dagstuhl Seminar 17202 “Challenges and Opportunities of User-Level File Systems for HPC” and discusses application scenarios as well as design strategies for ad hoc file systems using node-local storage media. The discussion includes open research questions, such as how to couple ad hoc file systems with the batch scheduling environment and how to schedule stage-in and stage-out processes of data between the storage backend and the ad hoc file systems. Also presented are strategies to build ad hoc file systems by using reusable components for networking and how to improve storage device compatibility. Various interfaces and semantics are presented, for example those used by the three ad hoc file systems BeeOND, GekkoFS, and BurstFS. Their presentation covers a range from file systems running in production to cutting-edge research focusing on reaching the performance limits of the underlying devices.

97 MATHEMATICS AND COMPUTING↗

Arjun Shankar: Advancing National Missions through Smart Data

Computer scientist Arjun Shankar is director of the Compute and Data Environment for Science (CADES), ORNL’s multidisciplinary big data computing center. CADES offers computing, networking and data analytics to facilitate workflows for both ORNL and external research projects.

97 MATHEMATICS AND COMPUTING↗

Efficiently predicting pressure-composition-temperature diagrams to discover low-stability metal hydrides

Quantitatively accurate computational predictions of metal hydride thermodynamics are challenging but critical for alloy performance optimization across a multitude of technological domains, including hydrogen storage, compression, purification, and getters. Recent machine learning approaches have demonstrated great success in this area, but can potentially suffer from several shortcomings since they rely on imbalanced experimental training data and can have poor out-of-distribution (ood) test performance. Here, in this study, we circumvent such pitfalls by developing a computationally efficient, first principles-based workflow for direct prediction of metal hydride phase equilibrium, i.e., the pressure-composition-temperature (PCT) diagram. We then demonstrate its utility on predicting low stability hydrides derived from compositionally complex C14 Laves phase AB2 alloys. Specifically, we computationally predict and then experimentally validate an AB 2 alloy series (z < 0.6 for Ti 2−z Zr z CrMnFeNi) with ideal hydriding thermodynamics for a two-stage metal hydride-based compressor for pressurizing boil off from liquefied hydrogen. Importantly, this study lays the groundwork for accurate and efficient discovery/optimization of ood, low-stability hydrides for which purely data-driven approaches lack sufficient accuracy.

08 HYDROGEN↗

Creating Continuous Integration Infrastructure for Software Development on U.S. Department of Energy High-Performance Computing Systems

The Exascale Computing Project (ECP) software deployment effort developed and advanced DevOps capabilities. One goal was to enable robust continuous integration (CI) workflows that span the protected high performance computing (HPC) environments found within many of the Department of Energy’s (DOE) national laboratories. This article highlights several challenges encountered with enabling automation, such as charging models for CI jobs, and meeting individualized security requirements that revolve around strongly associating running code with a human identity. Here, it also describes how the Jacamar CI tool evolved to meet latter requirements and became a key aspect of the solutions currently offered. Derived from this experience, we offer a conceptual framework for understanding current and future CI challenges at DOE facilities and offer suggestions for long-term solutions.

97 MATHEMATICS AND COMPUTING↗

Managing Dynamic Workflows in BEE

BEE is a powerful tool for: Managing and visualizing scientific workflows; Simplifying workflow execution on HPC and cloud platforms. BEE supports much of the CWL specification. Did not support execution of complex ”scattering” workflows. By introducing the PseudoTask: Can generate tasks to run on variable number of inputs; BEE is another step closer to supporting the entire CWL specification; BEE can now support parallelized workflows with scattering tasks.

97 MATHEMATICS AND COMPUTING↗

Towards a better understanding of on and off target effects of the lymphocyte-specific kinase LCK for the development of novel and safer pharmaceuticals

In this work we have developed a multi-tiered computational platform to study protein-drug interactions. At the beginning of the workflow more efficient and less accurate methods are used to enable large libraries of proteins in many conformations and massive chemical libraries to be screened. At each subsequent step in the workflow a subset of input data is investigated with increased accuracy and more computationally expensive methods. We demonstrate the developed workflow with the investigation of the lymphocyte-specific kinase LCK, which is implicated as a drug target in many cancers and also known to have toxic effects when unintentionally targeted. Several LCK states and conformations are investigated using molecular docking and generalized Born and surface area continuum solvation (MM/GBSA). Different variations in the drug screening process provide unique results that may elucidate the biological mechanisms underlying the drug interactions.

97 MATHEMATICS AND COMPUTING↗

Pynta-An Automated Workflow for Calculation of Surface and Gas-Surface Kinetics

Many important industrial processes rely on heterogeneous catalytic systems. However, given all possible catalysts and conditions of interest, it is impractical to optimize most systems experimentally. Automatically generated microkinetic models can be used to efficiently consider many catalysts and conditions. However, these microkinetic models require accurate estimation of many thermochemical and kinetic parameters. Manually calculating these parameters is tedious and error prone, involving many interconnected computations. Here, we present Pynta, a workflow software for automating the calculation of surface and gas–surface reactions. Pynta takes the reactants, products, and atom maps for the reactions of interest, generates sets of initial guesses for all species and saddle points, runs all optimizations, frequency, and IRC calculations, and computes the associated thermochemistry and rate coefficients. It is able to consider all unique adsorption configurations for both adsorbates and saddle points, allowing it to handle high index surfaces and bidentate species. Pynta implements a new saddle point guess generation method called harmonically forced saddle point searching (HFSP). HFSP defines harmonic potentials based on the optimized adsorbate geometries and which bonds are breaking and forming that allow initial placements to be optimized using the GFN1-xTB semiempirical method to create reliable saddle point guesses. This method is reaction class agnostic and fast, allowing Pynta to consider all possible adsorbate site placements efficiently. We demonstrate Pynta on 11 diverse reactions involving monodenate, bidentate, and gas-phase species, many distinct reaction classes, and both a low and a high index facet of Cu. Our results suggest that it is very important to consider reactions between adsorbates adsorbed in all unique configurations for interadsorbate group transfers and reactions on high index surfaces.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

The ATLAS Workflow Management System Evolution in the LHC Run3 and towards the High-Luminosity LHC era

The ATLAS experiment has 18+ years of experience using workload management systems to deploy and develop workflows to process and to simulate data on the distributed computing infrastructure. Simulation, processing and analysis of LHC experiment data require the coordinated work of heterogeneous computing resources. In particular, the ATLAS experiment utilizes the resources of 250 computing centers worldwide, the power of supercomputing centres, and national, academic and commercial cloud computing resources. In this contribution, we present new techniques for cost-effectively improving efficiency introduced in workflow management system software. The evolution from a mesh framework to new types of computing facilities such as cloud and HPCs is described, as well as new types of production and analysis workflows.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Custom Accessors: Enabling Scalable Data Ingestion, (Re-)Organization, and Analysis on Distributed Systems

The emerging class of high velocity and high volume data analytic workflows comprise interwoven data ingestion, organization, and processing stages, with ingestion and organization steps often contributing comparable or even higher computational costs than actual processing steps. Since complex workflows consist of a variety of phases that view and use data differently, being able to construct efficient, scalable, distributed data structures (arrays, vectors, sets, maps, and multi-maps) is essential and requires custom methods to extend and shrink containers, analyze and position data, and, maintain globallyconsistent meta-data. In this paper, we propose a novel datastructure access paradigm based on the concept of Accessors. At a high level, accessors are customizable callable objects that can modify the behavior of insert, read, update, and delete operations for distributed containers while preserving atomicity guarantees. Accessors provide a very clean and natural way to implement a variety of programming patterns, e.g., conditional insertion/deletion and cascading computations, which would be otherwise hard (or even impossible) to express in parallel and distributed settings without using locks. We demonstrate the practicality and usefulness of our approach with two representative use cases and study the performance of these applications on a distributed High-Performance Computing system. Our analysis highlights that our proposed abstraction allows for an effective overlapping and concurrent execution of different workflow steps (e.g., data ingestion and analysis), which in a conventional analytics pipeline would execute sequentially, contributing cumulatively to the overall latency.

Castellana, Vito G. [BATTELLE (PACIFIC NW LAB)] (O↗

Kepler + MeDICi - Service-Oriented Scientific Workflow Applications

Scientific applications are often structured as workflows that execute a series of interdependent, distributed software modules to analyze large data sets. The order of execution of the tasks in a workflow is commonly controlled by complex scripts, which over time become difficult to maintain and evolve. In this paper, we describe how we have integrated the Kepler scientific workflow platform with the MeDICi Integration Framework, which has been specifically designed to provide a standards-based, lightweight and flexible integration platform. The MeDICi technology provides a scalable, component-based architecture that efficiently handles integration with heterogeneous, distributed software systems. This paper describes the MeDICi Integration Framework and the mechanisms we used to integrate MeDICi components with Kepler workflow actors. We illustrate this solution with a workflow application for an atmospheric sciences application. The resulting solution promotes a strong separation of concerns, simplifying the Kepler workflow description and promoting the creation of a reusable collection of components available for other workflow applications in this domain.

99 GENERAL AND MISCELLANEOUS↗

Development of Time Lapse VSP Integration Workflow: A Case Study at Farnsworth CO2-EOR Project

Abstract This study aims to develop a 4D Vertical Seismic Profile (VSP) integration workflow to improve the prediction of subsurface stress changes. The selected study site is a 5-spot pattern within the ongoing CO2-EOR operations at the Farnsworth Field Unit FWU in Ochiltree County, Texas. The specific pattern has undergone extensive geological and geomechanical characterization through the acquisition of 3D seismic data, geophysical well logs, and core. This workflow constrains a numerical hydromechanical model by applying a penalty function formed between "modeled" versus "observed" time-lapse compressional and shear seismic velocity changes. Analyses of geophysical logs and ultra-sonic measurements on core exhibit measurable sensitivities to changes in both fluid saturation and mean effective stress. These data are used to develop a site-specific rock physics model and stress-velocity relationship, which inform the numerical models used to generate the "modeled" portion of the penalty function. The "observed" portion of the penalty function is provided by a novel elastic full-waveform inversion of the available 3D baseline and three monitor surveys to produce high-quality estimates of time-lapse compressional and shear seismic velocity changes. The modeling workflow accounts sequentially for fluid substitution and stress impacts. Hydrodynamic and geomechanical properties of the 3D coupled numerical model are estimated through geostatistical integration of well log and core data with 3D seismic inversion products. Changes in seismic velocities due to fluid substitution are computed using the Biot-Gassmann workflow and site-specific rock physics. Stress impacts on time-lapse seismic velocity changes are modeled from the effective stress output of the hydromechanical model and are initially based on the velocity versus effective stress relationship extracted from core mechanical testing. Based on the principle of superposition of seismic wavefields, seismic velocity changes attributed to fluid substitution and that due to changes in mean effective stress are treated as linearly additive. The modeled results are upscaled using Backus averaging to reconcile scale discrepancies between the modeled and measured datasets to formulate the penalty function. This manuscript presents the forward modeling process and concludes that for the base case, the seismic velocity changes due to mean effective stress dominates over the seismic velocity changes attributed to fluid substitution because of the extensive range of the pressure perturbations. Successful minimization of this penalty function calibrates the coupled hydrodynamic geomechanical numerical model and affirms the suitability of acoustic time-lapse measurements such as 4D-VSP for geomechanical calibration.

02 PETROLEUM↗

Performance Analysis and Optimization for Scientific Data Workloads

Scientific data generated at experimental and observational facilities are increasingly being processed on large-scale compute systems. Most of the experimental data analysis workflows are not designed or implemented to run on large scale environments and take full advantage of HPC compute and storage resources. These applications are unlike the traditional tightly-coupled scientific applications and hence face significant performance and scalability challenges as the volume of data increases exponentially. In this paper, we conduct a performance and scalability analysis for experimental analysis applications and workflows operating on data from light sources. Our analysis detects and quantifies I/O performance, scalability and runtime bottlenecks for three data analysis applications that run on NERSC resources. Based on our analysis we propose and implement a set of optimizations that lead to reducing the amount of time spent on I/O operations by almost 90%.

97 MATHEMATICS AND COMPUTING↗

Integrated Research Infrastructure Architecture Blueprint Activity (Final Report 2023)

The complexity of scientific pursuits is increasing rapidly with aspects that require dynamic integration of experiment, observation, theory, modeling, simulation, visualization, machine learning (ML), artificial intelligence (AI), and analysis. Research projects across the Department of Energy (DOE) are increasingly data and compute intensive. Innovative research teams are accelerating the pace of discovery by using high-performance computational and data tools in their research workflows and leveraging multiple research infrastructures. Additionally, several recent high-level U.S. government reports underscore the necessity of a new advanced computing ecosystem for international competitiveness and national security. International competitors are moving forward with major research infrastructure integration efforts that seek to capture a competitive advantage in the global innovation race. Owing to its unparalleled constellation of world-class experimental and observational facilities and high-performance and extreme-scale computational, data, and networking infrastructure, DOE is positioned to be a global leader in this new era of integrated science. However, this new integration paradigm will demand continuing evolution to ensure the U.S. remains a global leader in research and innovation. The DOE Office of Science (SC) has seized on the strategic importance of integration and has adopted a vision for Integrated Research Infrastructure (IRI): To empower researchers to meld DOE’s world-class research tools, infrastructure, and user facilities seamlessly and securely in novel ways to radically accelerate discovery and innovation. To respond to the evolving computational requirements of research and the competitive international innovation landscape, experimental facilities could be connected with high performance computing resources for near real-time analysis, and resources should be provided for merging enormous and diverse data for AI/ML techniques and analysis.

97 MATHEMATICS AND COMPUTING↗

Nexus: a modular workflow management system for quantum simulation codes

The management of simulation workflows is a significant task for the individual computational researcher. Automation of the required tasks involved in simulation work can decrease the overall time to solution and reduce sources of human error. A new simulation workflow management system, Nexus, is presented to address these issues. Nexus is capable of automated job management on workstations and resources at several major supercomputing centers. Its modular design allows many quantum simulation codes to be supported within the same framework. Current support includes quantum Monte Carlo calculations with QMCPACK, density functional theory calculations with Quantum Espresso or VASP, and quantum chemical calculations with GAMESS. Users can compose workflows through a transparent, text-based interface, resembling the input file of a typical simulation code. A usage example is provided to illustrate the process.

97 MATHEMATICS AND COMPUTING↗