Search NASA⌕ Search

SEARCH · Search NASA

Results for “computational workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

Computing with a Chemical Reservoir

Contemporary computation is expensive, with large language models and artificial intelligence becoming more common in daily life. However, high-performance computing is reaching the limits in speed and energy expenditure, and domain science requires ever-increasing computational capacity, with simulations and data analysis pipelines ever-growing in complexity. As we progress towards post-exascale computation, with the associated high energy costs, new methods of energy-conscious computation are required. Novel analog and hybrid digital-analog systems can overcome these challenges, and chemical reactions offer a promising avenue. Computers based on chemistry can provide compact desktop devices with immense computational power. These devices are readily scalable by considering greater reaction systems or vessels, meeting the high-performance requirements for scientific workflows. In this article, we present ChemComp, a compilation pipeline for the conversion of ordinary differential equations into implementable chemical reactions. We then demonstrate the solving capabilities of ChemComp by emulating a potential chemical reservoir device. We leverage the multi-layer intermediate representation (MLIR) compiler framework to implement an expressive chemical reaction abstraction and propose a path for chemical reaction networks (CRNs) to represent mathematical problems effectively. Combined, we demonstrate a potential workflow that can harness chemistry’s computing power to create energy-efficient, high-performance computation systems for contemporary computing needs.

artificial intelligence↗

DICER: Data Intensive Computing Environment and Runtime for Evaluating Unprecedented Scale of Geospatial-Temporal Human Mobility Data

With the significant increase in sources and volume of human mobility data through commercial data vendors as well as microsimulation of cities, the scale of geospatial-temporal data to analyze and assess for mobility characterization has grown to the level of Big Data. There are mobility related commercial organizations deploying scalable computing, but often the system architecture, workflow, and intermediate processing components are not fully disclosed in relevant scope. Current research literature has a notable lack of studies demonstrating architectures and workflows for human mobility analytics that are implemented on a TeraByte scale of geospatial-temporal data. In this context, this paper presents a hyperscale-level system solution named DICER (Data Intensive Computing Environment and Runtime) for processing and analytics of geospatial-temporal data at big data scale. Although the cluster computing architecture of DICER with Apache Spark job running on Kubernetes cluster is not new, there are innovations in the workflow, hierarchical processing logic, and a wide range of intermediate preprocessing and mobility metrics calculation. We have performed case studies to validate the effectiveness of DICER system solution by performing detailed analytics and assessment of human mobility microsimulation output at three different scopes and scale, including a usecase with 16.97 TeraByte and 259.2 Billion rows of data. In addition, we have presented another case study of utilizing DICER to perform the same mobility processing and comparative analytics on large-scale commercially available geospatial-temporal data. All these case studies validate the efficiency and usefulness of DICER in computing population mobility characteristics from geospatial-temporal trajectory data at an unprecedented scale (not only just data volume, but also combination of: number of user entities, temporal frequency, spatial resolution, data duration).

De, Debraj↗

SU-E-T-419: Workflow and FMEA in a New Proton Therapy (PT) Facility

Purpose: Workflow is an important component in the operational planning of a new proton facility. By integrating the concept of failure mode and effect analysis (FMEA) and traditional QA requirements, a workflow for a proton therapy treatment course is set up. This workflow serves as the blue print for the planning of computer hardware/software requirements and network flow. A slight modification of the workflow generates a process map(PM) for FMEA and the planning of QA program in PT. Methods: A flowchart is first developed outlining the sequence of processes involved in a PT treatment course. Each process consists of a number of sub-processes to encompass a broad scope of treatment and QA procedures. For each subprocess, the personnel involved, the equipment needed and the computer hardware/software as well as network requirements are defined by a team of clinical staff, administrators and IT personnel. Results: Eleven intermediate processes with a total of 70 sub-processes involved in a PT treatment course are identified. The number of sub-processes varies, ranging from 2-12. The sub-processes within each process are used for the operational planning. For example, in the CT-Sim process, there are 12 sub-processes: three involve data entry/retrieval from a record-and-verify system, two controlled by the CT computer, two require department/hospital network, and the other five are setup procedures. IT then decides the number of computers needed and the software and network requirement. By removing the traditional QA procedures from the workflow, a PM is generated for FMEA analysis to design a QA program for PT. Conclusion: Significant efforts are involved in the development of the workflow in a PT treatment course. Our hybrid model of combining FMEA and traditional QA program serves a duo purpose of efficient operational planning and designing of a QA program in PT.

07 ISOTOPE AND RADIATION SOURCES↗

Large-Scale Compute-Intensive Analysis via a Combined In-situ and Co-scheduling Workflow Approach

Large-scale simulations can produce tens of terabytes of data per analysis cycle, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in situ and co-scheduling approaches for handling Petabyte-size outputs. An initial in situ step is used to reduce the amount of data to be analyzed, and to separate out the data-intensive tasks handled off-line. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.

Big data↗

Workflow Submit Nodes as a Service on Leadership Class Systems

DOE scientists, today, have access to high performance computing (HPC) facilities with very powerful systems that enable them to execute their computations faster, more efficiently, and at greater scales than ever before. To further their knowledge and produce new discoveries, scientists rely on workflows - sometimes very complex - that provide them with an easy way to automate, reproduce and verify their computations. However, historically, creating workflow submission environments in large HPC facilities has been cumbersome, requires expertise and many man-hours of effort due to the peculiarities, policies, and the restrictions that these systems present. In this paper we discuss the approach a large DOE facility (OLCF) is taking in order to provide containers as a service to its users. This capability is used to create Pegasus workflow management system submit nodes as a service (WSaaS) at the Oak Ridge Leadership Computing Facilities (OLCF), targeting the Summit supercomputer. This deployment builds upon the Kubernetes/Openshift cluster (Slate) that exists within OLCF’s DMZ and its automation triggers. Additionally, we evaluate our approach’s overhead and effort to deploy the solution as compared to previous solutions, such as setting up a Pegasus submission environment on OLCF’s login nodes or submitting jobs remotely via the rvGAHP.

Papadimitriou, George↗

Exploiting analytics techniques in CMS computing monitoring

The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Scalable Multi-Facility Workflows for Artificial Intelligence Applications in Climate Research

Earth observation satellites and earth system models are sources of vast, multi-modal datasets that are invaluable for advancing climate and environmental research. However, their scale and complexity pose significant challenges for processing and analysis. In this paper we discuss our experiences in developing and using a scientific research application using an automated multi-facility workflow that orchestrates data collection, preprocessing, artificial intelligence (AI) inferencing, and data movement across diverse computational resources, leveraging the Advanced Computing Ecosystem Testbed at the Oak Ridge Leadership Computing Facility (OLCF). We demonstrate that our workflow can be seamlessly integrated and orchestrated across research facilities managed by different federal agencies, thus allowing users to extract new scientific insights from climate datasets. The experimental results indicate that the multi-facility workflow significantly reduces processing time, enhances scalability, and maintains high efficiency across varying workloads. Notably, our workflow processes 12,000 high-resolution satellite images in just 44 seconds using 80 workers distributed across 10 nodes on the OLCF systems. Such high throughput is essential for dynamic tokenization and sharding of petascale satellite data for distributed AI model training and inferencing at scale across thousands of GPUs.

Kurihana, Takuya [ORNL] (ORCID:0000000156698565)↗

Surrogate models for development of unconventional shale reservoirs by an integrated numerical approach of hydraulic fracturing, flow and geomechanics, and machine learning

We develop well-completion surrogate models by taking an integrated workflow of hydraulic fracturing, flow, geomechanics, and machine learning simulation. There are three steps in the proposed workflow. First, history-matching processes are conducted with the field data including pumping and production data for characterization. Second, full-physics simulation is performed with various parameters of the field development (e.g., cluster spacing, clusters per stage, pumping rates and times, amount of proppant, and well spacing) to generate multiple simulation results by changing the parameters of the completion design with well-known hydraulic fracturing, reservoir, geomechanics simulators to calculate fracture geometry, reservoir depressurization, induced stress changes. The workflow is demonstrated over a field in the Southern Midland Basin. Here, we take two completion scenarios: a single well case followed by a multi-well case. Finally, a Long Short-Term Memory (LSTM) machine learning algorithm is employed to create surrogate models that can replicate the full-physics simulation results. Furthermore, results show that the trained models applied in the single well and multi-well cases for a particular geological system can provide good accuracy close to those provided by full-physics simulations. Specifically, the site-specific surrogate models can predict fracture parameters (length, height, and surface area) and cumulative production accurately with computational efficiency, suggesting our proposed workflow can be used as a pragmatic tool for expediting the well completion optimization process.

Geomechanics↗

3D modeling of deep borehole electromagnetic measurements with energized casing source for fracture mapping at the Utah Frontier Observatory for Research in Geothermal Energy

Here, we present a 3D numerical modelling analysis evaluating the deployment of a borehole electromagnetic measurement tool to detect and image a stimulated zone at the Utah Frontier Observatory for Research in Geothermal Energy geothermal site. As the depth to the geothermal reservoir is several kilometres and the size of the stimulated zone is limited to several 100 m, surface-based controlled-source electromagnetic measurements lack the sensitivity for detecting changes in electrical resistivity caused by the stimulation. To overcome the limitation, the study evaluates the feasibility of using a three-component borehole magnetic receiver system at the Frontier Observatory for Research in Geothermal Energy site. To provide sufficient currents inside and around the enhanced geothermal reservoir, we use an injection well as an energized casing source. To efficiently simulate energizing the injection well in a realistic 3D resistivity model, we introduce a novel modelling workflow that leverages the strengths of both 3D cylindrical-mesh-based electromagnetic modelling code and 3D tetrahedral-mesh-based electromagnetic modelling code. The former is particularly well-suited for modelling hollow cylindrical objects like casings, whereas the latter excels at representing more complex 3D geological structures. In this workflow, our initial step involves computing current densities along a vertical steel-cased well using a 3D cylindrical electromagnetic modelling code. Subsequently, we distribute a series of equivalent current sources along the well's trajectory within a complex 3D resistivity model. We then discretize this model using a tetrahedral mesh and simulate the borehole electromagnetic responses excited by the casing source using a 3D finite-element electromagnetic code. This multi-step approach enables us to simulate 3D casing source electromagnetic responses within a complex 3D resistivity model, without the need for explicit discretization of the well using an excessive number of fine cells. We discuss the applicability and limitations of this proposed workflow within an electromagnetic modelling scenario where an energized well is deviated, such as at the Frontier Observatory for Research in Geothermal Energy site. Using the workflow, we demonstrate that the combined use of the energized casing source and the borehole electromagnetic receiver system offer measurable magnetic field amplitudes and sensitivity to the deep localized stimulated zone. The measurements can also distinguish between parallel-fracture anisotropic reservoirs and isotropic cases, providing valuable insights into the fracture system of the stimulated zone. Besides the magnetic field measurements, vertical electric field measurements in the open well sections are also highly sensitive to the stimulated zone and can be used as additional data for detecting and imaging the target. We can also acquire additional multiple-source data by grounding the surface electrode at various locations and repeating borehole electromagnetic measurements. This approach can increase the number of monitoring data by several factors, providing a more comprehensive dataset for analysing the deep-localized stimulated zone. The numerical analysis indicates that it is feasible to use the combination of the energized casing and downhole electromagnetic measurements in monitoring localized stimulated zone at large depths.

58 GEOSCIENCES↗

STAR Data Production Workflow on HPC: Lessons Learned & Best Practices

The Solenoidal Tracker at RHIC (STAR) is a multi-national supported experiment located at Brookhaven National Lab. The raw physics data captured from the detector is on the order of tens of PBytes per data acquisition campaign, which makes STAR fit well within the definition of a big data science experiment. The production of the data has typically run on standard nodes or on standard Grid computing environments. All embedding simulations (complex workflow mixing real and simulated events) have been run on standard Linux resources at the National Energy Research Scientific Computing Center (NERSC) aka PDSF. However, HPC resources such as Cori have become available for STAR's data production as well as embedding, and STAR has been the very first experiment to show feasibility of running a sustainable data production campaign on this computing resource. The use of Docker containers with Shifter is required to run on HPC @ NERSC - this approach encapsulates the environment in which a standard STAR workflow runs. From the deployment of a tailored Scientific Linux environment (requiring many of its own libraries and special configurations required to run) to the deployment of third-party software and the STAR specific software stack, it has become impractical to rely on a set of containers containing each specific software release. To this extent, solutions based on the CERN VM File System (CVMFS) for the deployment of software and services have been employed in HENP, but one needs to make careful scalability considerations when using a resource like Cori, such as not allowing all software to be deployed in containers or bare node. Additionally, CVMFS clients are not compatible on Cori nodes and one needs to rely on an indirect NFS/DVS mount scheme. In our contribution, we will discuss our strategies from the past and our current solution based on CVMFS. Furthermore, running on HPC is not a simple task as each aspect of the workflow must be enabled to scale, run efficiently, and the workflow needs to fit within the boundaries of the provided queue system (SLURM in this case). Lastly, we will also discuss what we have learned so far about what is the best method for grouping jobs to maximize a single 48 core HPC node within a specific time frame and maximize our workflow efficiency.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.

Computer Science↗

ATLAS Data Analysis using a Parallel Workflow on Distributed Cloud-based Services with GPUs

A new type of parallel workflow is developed for the ATLAS experiment at the Large Hadron Collider, that makes use of distributed computing combined with a cloud-based infrastructure. This has been developed for a specific type of analysis using ATLAS data, one popularly referred to as Simulation-Based Inference (SBI). The JAX library is used for the parts of the workflow to compute gradients as well as accelerate program execution using just-in-time compilation, which becomes essential in a full SBI analysis and can also offer significant speed-ups in more traditional types of analysis.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Orchestration of materials science workflows for heterogeneous resources at large scale

In the era of big data, materials science workflows need to handle large-scale data distribution, storage, and computation. Any of these areas can become a performance bottleneck. We present a framework for analyzing internal material structures (e.g., cracks) to mitigate these bottlenecks. We demonstrate the effectiveness of our framework for a workflow performing synchrotron X-ray computed tomography reconstruction and segmentation of a silica-based structure. Our framework provides a cloud-based, cutting-edge solution to challenges such as growing intermediate and output data and heavy resource demands during image reconstruction and segmentation. Specifically, our framework efficiently manages data storage, scaling up compute resources on the cloud. The multi-layer software structure of our framework includes three layers. A top layer uses Jupyter notebooks and serves as the user interface. A middle layer uses Ansible for resource deployment and managing the execution environment. A low layer is dedicated to resource management and provides resource management and job scheduling on heterogeneous nodes (i.e., GPU and CPU). At the core of this layer, Kubernetes supports resource management, and Dask enables large-scale job scheduling for heterogeneous resources. The broader impact of our work is four-fold: through our framework, we hide the complexity of the cloud’s software stack to the user who otherwise is required to have expertise in cloud technologies; we manage job scheduling efficiently and in a scalable manner; we enable resource elasticity and workflow orchestration at a large scale; and we facilitate moving the study of nonporous structures, which has wide applications in engineering and scientific fields, to the cloud. While we demonstrate the capability of our framework for a specific materials science application, it can be adapted for other applications and domains because of its modular, multi-layer architecture.

97 MATHEMATICS AND COMPUTING↗

Integrating HPC, AI, and Workflows for Scientific Data Analysis: Report from Dagstuhl Seminar 23352

The Dagstuhl Seminar 23352, titled “Integrating HPC, AI, and Workflows for Scientific Data Analysis,” held from August 27 to September 1, 2023, was a significant event focusing on the synergy between High-Performance Computing (HPC), Artificial Intelligence (AI), and scientific workflow technologies. The seminar recognized that modern Big Data analysis in science rests on three pillars: workflow technologies for reproducibility and steering, AI and Machine Learning (ML) for versatile analysis, and HPC for handling large data sets. These elements, while crucial, have traditionally been researched separately, leading to gaps in their integration. The seminar aimed to bridge these gaps, acknowledging the challenges and opportunities at the intersection of these technologies. The event highlighted the complex interplay between HPC, workflows, and ML, noting how ML has increasingly been integrated into scientific workflows, thereby enhancing resource demands and bringing new requirements to HPC architectures, like support for GPUs and iterative computations. The seminar also addressed the challenges in adapting HPC for large-scale ML tasks, including in areas like deep learning, and the need for workflow systems to evolve to leverage ML in data analysis fully. Moreover, the seminar explored how ML could optimize scientific workflow systems and HPC operations, such as through improved scheduling and fault tolerance. A key focus was on identifying prestigious use cases of ML in HPC and understanding their unique, unmet requirements. The stochastic nature of ML and its impact on the reproducibility of data analysis on HPC systems was also a topic of discussion.

97 MATHEMATICS AND COMPUTING↗

Enabling discovery data science through cross-facility workflows

Experimental and observational instruments for scientific research (such as light sources, genome sequencers, accelerators, telescopes and electron microscopes) increasingly require High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. Next-generation instruments are being deployed with higher resolutions and faster data capture rates, creating a big data crunch that cannot be handled by modest institutional computing resources. Often these big data analysis pipelines also require near real-time computing and have higher resilience requirements than the simulation and modeling workloads more traditionally seen at HPC centers. While some facilities have enabled workflows to run at a single HPC facility, there is a growing need to integrate capabilities across HPC facilities to enable cross-facility workflows, either to provide resilience to an experiment, increase analysis throughput capabilities, or to better match a workflow to a particular architecture. In this paper we describe the barriers to executing complex data analysis workflows across HPC facilities and propose an architectural design pattern for enabling scientific discovery using cross-facility workflows that includes orchestration services, application programming interfaces (APIs), data access and co-scheduling.

Antypas, Katerina B.↗

Scaling the SciDAC QuantOm Workflow

As part of the Scientific Discovery through Advanced Computing (SciDAC) program, the Quantum Chromodynamics Nuclear Tomography (QuantOM) project aims to analyze data from Deep Inelastic Scattering (DIS) experiments conducted at Jefferson Lab and the upcoming Electron Ion Collider. The DIS data analysis is performed on an event-level by combining the input from theoretical and experimental nuclear physics into a single, composable workflow. The optimization itself (I.e. fitting the experimental data with theoretical predictions) is carried out by a machine / deep learning algorithm. The size of the acquired DIS data as well as the complexity of the workflow itself require that the analysis is performed across multiple GPUs on high performance computing systems, such as Polaris at Argonne National Laboratory. This presentation discusses the novelties and challenges that came along with parallelizing this workflow. Recent results are compared to common distributed training techniques.

Lersch, Daniel↗

Automation of User Analysis Workflow in CMS

CMS has a distributed computing model, based on a hierarchy of tiered regional computing centres. However, the end physicist is not interested in the details of the computing model nor the complexity of the underlying infrastructure, but only to a ccess and use efficiently and easily the remote services. The CMS Remote Analysis Builder (CRAB) is the official CMS tool that allows the access to the distributed data in a transparent way. We present the current development direction, which is focused o n improving the interface presented to the user and adding intelligence to CRAB such that it can be used to automate more and more the work done on behalf of user. We also present the status of deployment of the CRAB system and the lessons learnt in deplo ying this tool to the CMS collaboration.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Multipoint Aerostructural Optimization of Wind Turbine Rotors Using a Coupled Blade‐Resolved Aerostructural Solver

Physics‐based design optimization workflows thread the needle between computational cost limitations and simulation complexity, often compromising between modeling detail and the range of operating design conditions. Multipoint aerostructural optimization of wind turbine rotors has so far been confined to low‐fidelity analyses or to high‐fidelity studies with simplified structural models, leaving the most complex design trade‐offs unexplored. We close this gap by performing the first tightly coupled gradient‐based multipoint aerostructural rotor optimization using 3D aerodynamic and structural solvers with discrete coupled adjoints. The optimizer simultaneously varies blade planform, airfoil shapes, and structural thickness through more than 270 design variables, minimizing a weighted combination of rotor mass and power across multiple wind speeds. Applied to a modified DTU 10‐MW benchmark under conservative structural and aerodynamic constraints, our multipoint optimization reduces rotor mass by up to 36% and increases power by 12%–15% across the main operating conditions; biasing the objective toward power yields power gains up to 18% and a 17% mass reduction. For a nominal wind distribution, 3‐point rotor designs accounting for low RPM and high thrust conditions capture dominant trade‐offs and outperform single‐point designs. Adding two off‐design points changes individual‐condition power by less than 3% but leaves the weighted average within 0.5%, and the mass‐power bias has a stronger effect on the final design than the operating‐point weighting itself. Our framework extends naturally to richer load cases and site‐specific wind distributions, providing a basis for high‐fidelity multipoint design earlier in industrial workflows.

17 WIND ENERGY↗