Beyond APPs - Workflows of APPs
Explore the source record for details and available documents.
97 MATHEMATICS AND COMPUTING↗
SEARCH · Search NASA
Results for “computational workflow”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The Evolution of the Pegasus Workflow Management Software
Since 2001 the Pegasus Workflow Management System has evolved into a robust and scalable system that automates the execution of a number of complex applications running on a variety of heterogeneous, distributed high-throughput, and high-performance computing environments. Pegasus was built on the principle of separation between the workflow description and workflow execution, providing the ability to port and adapt the workflow based on the target execution environment. Through its user-driven research and development it has adapted to the needs of a number of scientific communities, utilizing and developing novel algorithms and software engineering solutions. This paper describes the evolution of Pegasus over time and provides motivations behind the design decisions. The paper concludes with selected lessons learned.
97 MATHEMATICS AND COMPUTING↗
Scientific Data Management Integrated Software Infrastructure Center (SDM/ISIC): Scientific Process Automation (SPA) (Final Report)
The KEPLER project has made a significant impact and established a sustained leadership in the field of scientific workflows. The open source KEPLER project started in 2003 when members of the SciDAC-SDM Center/SPA Team, sponsored by this DOE-funded project (DE-FC02-01ER25486) and members of the SEEK project (NSF/ITR awards DBI-0225674 and DBI-0533368) decided to collaborate and jointly develop a scientific workflow system based on the open source PTOLEMY II system (Ludäscher was a co-PI on both projects). In the first project phase, between 2001 and 2003, the SDSC team (Altintas, Ludäscher) worked closely with a domain scientist (Matt Coleman, LLNL) and created early versions of a Promoter Identification Workflow (PIW). Towards the end of that period, the open source PTOLEMY II system was adopted by SciDAC-SDM and SEEK as the basis for a general scientific workflow system and problem-solving environment to design and execute scientific workflows, giving rise to Kepler. Once started through SciDAC-SDM and SEEK, the KEPLER leadership team was able to attract further support from funding agencies including from DOE (as part of SciDAC-2), NSF, NIH, and the Gordon and Betty Moore Foundation. Important advances to KEPLER were made by the SciDAC/SDM-SPA team during the report period (2001–2007), in particular by the SDSC team (and after Ludäscher's move) the UC Davis team, including (i) research and development of the workflow infrastructure, (ii) library (actor) development, and (iii) development of concrete scientific workflows, in collaboration with scientists. Years later, the KEPLER system was listed as one of the prominent “Big Data” research outcomes of DOE in the White House Big Data Factsheet, announced as part of the White House Big Data press release.
97 MATHEMATICS AND COMPUTING↗
High-throughput ab initio reaction mechanism exploration in the cloud with automated multi-reference validation
Quantum chemical calculations on atomistic systems have evolved into a standard approach to studying molecular matter. These calculations often involve a significant amount of manual input and expertise, although most of this effort could be automated, which would alleviate the need for expertise in software and hardware accessibility. Here, we present the AutoRXN workflow, an automated workflow for exploratory high-throughput electronic structure calculations of molecular systems, in which (i) density functional theory methods are exploited to deliver minimum and transition-state structures and corresponding energies and properties, (ii) coupled cluster calculations are then launched for optimized structures to provide more accurate energy and property estimates, and (iii) multi-reference diagnostics are evaluated to back check the coupled cluster results and subject them to automated multi-configurational calculations for potential multi-configurational cases. All calculations are carried out in a cloud environment and support massive computational campaigns. Key features of all components of the AutoRXN workflow are autonomy, stability, and minimum operator interference. We highlight the AutoRXN workflow with the example of an autonomous reaction mechanism exploration of the mode of action of a homogeneous catalyst for the asymmetric reduction of ketones.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
A New Workflow of X-ray CT Image Processing and Data Analysis of Structural Features in Rock Using Open-Source Software
X-ray computed tomography (CT) images of rock specimens often contain artifacts which must be corrected before scientific analyses are performed. Here, we present a new workflow of automated image processing to utilize poor-quality X-ray CT scan images. The workflow runs on the open-source image analysis software and efficiently separates desired features from low-contrast scanned images. The new workflow is a two-step technique using contrast enhancement and automated feature segmentation to generate noise-free binary images. The results of binary images using the proposed workflow and using a conventional thresholding technique are analyzed to show the quality of the proposed method. The paper also presents a workflow of estimating the structural geometries of features in two and three dimensions. The results of the structural feature analyses and computational time were compared between the open-source (ImageJ) and commercial image analysis software (Bruker Computed Tomography Analyzer). The commercial software was more computationally efficient, but the task-specific macros in open-source software enabled the user-desired automation in image processing and data extraction of desired structural features of comparable quality.
47 OTHER INSTRUMENTATION↗
Novel Proposals for FAIR, Automated, Recommendable, and Robust Workflows
Lightning talks of the Workflows in Support of Large-Scale Science (WORKS) workshop are a venue where the workflow community (researchers, developers, and users) can discuss work in progress, emerging technologies and frameworks, and training and education materials. This paper summarizes the WORKS 2022 lightning talks, which cover five broad topics: data integrity of scientific workflows; a machine learning-based recommendation system; a Python toolkit for running dynamic ensembles of simulations; a cross-platform, high-performance computing utility for processing shell commands; and a meta(data) framework for reproducing hybrid workflows.
Abhinit, Ishan↗
Towards Cross-Facility Workflows Orchestration through Distributed Automation
Modern science relies on end-to-end workflows that incorporate experimental instruments and utilize edge, cloud, or high-performance computing and storage resources. These components are geographically dispersed across various user facilities and interconnected through high-speed networks. In this paper, we present Zambeze, an automated distributed framework designed to facilitate this new class of cross-facility workflows. Utilizing swarm intelligence principles, Zambeze orchestrates science campaigns by managing distributed autonomous agents. These agents can offer a suite of services, including computing, storage, and data management. We demonstrate the feasibility of Zambeze through a real-world application involving electron microscopy, enhanced with Artificial Intelligence capabilities.
Skluzacek, Tyler↗
Conquering Data Chaos: Research Data Management with Kubernetes
Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.
collaborative environment↗
In Situ Machine Learning for Intelligent Data Capture on Exascale Platforms. Final Report
In many dynamic systems, interesting events occur locally in time and space. Examples of such systems include ignition events in combustion simulations, material fractures in mechanics simulations, and extreme weather events in climate simulations. Due to memory constraints and data I/O costs, current simulation workflows save data at regularly spaced time-steps, at a fixed rate determined before the start of the simulation. Often this mode of operation results in missed events of interest, necessitating a simulation restart from before an event occurred with more frequent data saves. This data saving workflow is grossly inefficient and is already a bottleneck in the computing process. We propose to develop machine learning algorithms that can detect when interesting dynamical events are occurring, triggering data saves. These machine learning algorithms will perform in situ anomaly detection to flag regions with different dynamical properties than those previously recorded. The adaptive data saves would be local in time and space to match the event of interest, thereby enabling a much more efficient workflow that will reduce data I/O costs and data storage memory requirements. The algorithms will be tested on two applications: auto-ignition simulations and climate simulations. A critical component of this project will be developing machine learning algorithms that can be deployed efficiently in situ on HPC platforms with out-of-the-box functionality. The development of in situ machine learning methods to detect anomalous events would enable a more efficient and effective workflow, in which all the relevant data are saved in a single simulation run, without re-starts or scientist intervention.
42 ENGINEERING↗
Partnership For Edge Physics Simulation
In this effort, we will extend our prior work as part of CPES (i.e., DART and DataSpaces) to support in-situ tight coupling between application codes that exploits data locality and core-level parallelism to maximize on-chip data exchange and reuse. This will be accomplished by mapping coupled simulations so that the data exchanges are more localized within the nodes. Coupled simulation workflows can more effectively utilize the resources available on emerging HEC platforms if they can be mapped and executed to exploit data locality as well as the communication patterns between application components. Scheduling and running such workflows requires an extended framework that should provide a unified hybrid abstraction to enable coordination and data sharing across computation tasks that run on the heterogeneous multi-core-based systems, and develop a data-locality based dynamic tasks scheduling approach to increase on-chip or intra-node data exchanges and in-situ execution. This effort will extend our prior work as part of CPES (i.e., DART and DataSpaces), which provided a simple virtual shared-space abstraction hosted at the staging nodes, to support application coordination, data sharing and active data processing services. Moreover, it will transparently manage the low-level operations associated with the inter-application data exchange, such as data redistributions, and will enable running coupled simulation workflow on multi-cores computing platforms.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum
Modern scientific workflows require hybrid infrastructures combining numerous decentralized resources on the IoT/Edge interconnected to Cloud/HPC systems (aka the Computing Continuum) to enable their optimized execution. Understanding and optimizing the performance of such complex Edge-to-Cloud workflows is challenging. Capturing the provenance of key performance indicators, with their related data and processes, may assist in understanding and optimizing workflow executions. However, the capture overhead can be prohibitive, particularly in resource-constrained devices, such as the ones on the IoT/Edge.To address this challenge, based on a performance analysis of existing systems, we propose ProvLight, a tool to enable efficient provenance capture on the IoT/Edge. We leverage simplified data models, data compression and grouping, and lightweight transmission protocols to reduce overheads. We further integrate ProvLight into the E2Clab framework to enable workflow provenance capture across the Edge-to-Cloud Continuum. This integration makes E2Clab a promising platform for the performance optimization of applications through reproducible experiments.We validate ProvLight at a large scale with synthetic workloads on 64 real-life IoT/Edge devices in the FIT IoT LAB testbed. Evaluations show that ProvLight outperforms state-of-the-art systems like ProvLake and DfAnalyzer in resource-constrained devices. ProvLight is 26—37x faster to capture and transmit provenance data; uses 5—7x less CPU; 2x less memory; transmits 2x less data; and consumes 2—2.5x less energy. ProvLight [1] and E2Clab [2] are available as open-source tools.
Rosendo, Daniel↗
SOMA: Observability, monitoring, and in situ analytics for exascale applications
With the rise of exascale systems and large, data-centric workflows, the need to observe and analyze high performance computing (HPC) applications during their execution is becoming increasingly important. HPC applications are typically not designed with online monitoring in mind, therefore, the observability challenge lies in being able to access and analyze interesting events with low overhead while seamlessly integrating such capabilities into existing and new applications. We explore how our service-based observation, monitoring, and analytics (SOMA) approach to collecting and aggregating both application-specific diagnostic data and performance data addresses these needs. Furthermore, we present our SOMA framework and demonstrate its viability with LULESH, a hydrodynamics proxy application. Then we focus on Astaroth, a multi-GPU library for stencil computations, highlighting the integration of the TAU and APEX performance tools and SOMA for application and performance data monitoring.
97 MATHEMATICS AND COMPUTING↗
Interval From Imaging to Treatment Delivery in the Radiation Surgery Age: How Long Is Too Long?
Purpose: The purpose of this study was to evaluate workflow and patient outcomes related to frameless stereotactic radiation surgery (SRS) for brain metastases. Methods and Materials: We reviewed all treatment demographics, clinical outcomes, and workflow timing, including time from magnetic resonance imaging (MRI), computed tomography (CT) simulation, insurance authorization, and consultation to the start of SRS for brain metastases. Results: A total of 82 patients with 151 brain metastases treated with SRS were evaluated. The median times from consultation, insurance authorization, CT simulation, and MRI for treatment planning were 15, 7, 6, and 11 days to SRS. Local freedom from progression (LFFP) was lower in metastases with MRI ≥14 days before treatment (P=.0003, log rank). The 6- and 12-month LFFP rate were 95% and 75% for metastasis with interval of <14 days from MRI to treatment compared to 56% and 34% for metastases with MRI ≥14 days before treatment. On multivariate analysis, LFFP remained significantly lower for lesions with MRI ≥14 days at SRS (P=.002, Cox proportional hazards; hazard ratio: 3.4, 95% confidence interval: 1.6-7.3). Conclusions: Delay from MRI to SRS treatment delivery for brain metastases appears to reduce local control. Future studies should monitor the timing from imaging acquisition to treatment delivery. Our experience suggests that the time from MRI to treatment should be <14 days.
62 RADIOLOGY AND NUCLEAR MEDICINE↗
MR-OPERA: A Multicenter/Multivendor Validation of Magnetic Resonance Imaging–Only Prostate Treatment Planning Using Synthetic Computed Tomography Images
Purpose: To validate the dosimetric accuracy and clinical robustness of a commercially available software for magnetic resonance (MR) to synthetic computed tomography (sCT) conversion, in an MR imaging–only workflow for 170 prostate cancer patients. Methods and Materials: The 4 participating centers had MriPlanner (Spectronic Medical), an atlas-based sCT generation software, installed as a cloud-based service. A T2-weighted MR sequence, covering the body contour, was added to the clinical protocol. The MR images were sent from the MR scanner workstation to the MriPlanner platform. The sCT was automatically returned to the treatment planning system. Four MR scanners and 2 magnetic field strengths were included in the study. For each patient, a CT-treatment plan was created and approved according to clinical practice. The sCT was rigidly registered to the CT, and the clinical treatment plan was recalculated on the sCT. The dose distributions from the CT plan and the sCT plan were compared according to a set of dose-volume histogram parameters and gamma evaluation. Treatment techniques included volumetric modulated arc therapy, intensity modulated radiation therapy, and conventional treatment using 2 treatment planning systems and different dose calculation algorithms. Results: The overall (multicenter/multivendor) mean dose differences between sCT and CT dose distributions were below 0.3% for all evaluated organs and targets. Gamma evaluation showed a mean pass rate of 99.12% (0.63%, 1 SD) in the complete body volume and 99.97% (0.13%, 1 SD) in the planning target volume using a 2%/2-mm global gamma criteria. Conclusions: Results of the study show that the sCT conversion method can be used clinically, with minimal differences between sCT and CT dose distributions for target and relevant organs at risk. The small differences seen are consistent between centers, indicating that an MR imaging–only workflow using MriPlanner is robust for a variety of field strengths, vendors, and treatment techniques.
62 RADIOLOGY AND NUCLEAR MEDICINE↗
STEPs-SOL, a Peptoid Force Field Parameterization to Include Solvent Effects
As peptoids (N-substituted glycines) continue to gain popularity as a class of biomimetic polymers, the importance and demand for accurate force fields in molecular simulations also grow. Building on the vacuum-optimized Systematic and Extensible Force Field for Peptoids (STEPs) force field, here we present STEPs-SOL, a novel peptoid force field parametrization that effectively incorporates solvent effects to enhance the accuracy of peptoid simulations. The development of STEPs-SOL is based on the need for precise electrostatic modeling achieved through solvent-specific partial charge optimization. Here, our systematic approach significantly improves agreement with experimental measurements, reducing the mean absolute error in cis/trans ratio predictions (ΔG c/t ) by an average of 38% across multiple peptoid residues and solvent environments. This improved parametrization addresses computational challenges associated with nonbonded energies while maintaining a workflow that relies on high-level quantum mechanical data rather than depending solely on limited experimental equilibrium properties. By evaluating the effects of conformational bias in restrained electrostatic potential (RESP) charge generation and examining their impact on peptoid conformations in various solvents, we enhance our understanding of peptoid structural dynamics while providing a more accurate modeling framework.
force field↗
Defect Genome of Cubic Perovskites for Fuel Cell Applications
Heterogeneities such as point defects, inherent to material systems, can profoundly influence material functionalities critical for numerous energy applications. This influence in principle can be identified and quantified through development of large defect data sets which we call the defect genome, employing high-throughput ab initio calculations. However, high-throughput screening of material models with point defects dramatically increases the computational complexity and chemical search space, creating major impediments toward developing a defect genome. In this paper, we overcome these impediments by employing computationally tractable ab initio models driven by highly scalable workflows, to study formation and interaction of various point defects (e.g., O vacancies, H interstitials, and Y substitutional dopant), in over 80 cubic perovskites, for potential proton-conducting ceramic fuel cell (PCFC) applications. The resulting defect data sets identify several promising perovskite compounds that can exhibit high proton conductivity. Furthermore, the data sets also enable us to identify and explain, insightful and novel correlations among defect energies, material identities, and defect-induced local structural distortions. Finally, such defect data sets and resultant correlations are necessary to build statistical machine learning models, which are required to accelerate discovery of new materials.
30 DIRECT ENERGY CONVERSION↗
PaR-PaR Laboratory Automation Platform
Explore the source record for details and available documents.
Gregory, Linshiz↗
Where Is the Provenance? Ethical Replicability and Reproducibility in GIScience and Its Critical Applications
As replicability and reproducibility (R&R) crises develop within emerging convergent inquiry, ethical use of provenance information is central to the establishment and preservation of trust in critical applications of GIScience and geospatial technologies. Today large volumes of geospatial data are generated at high velocity from satellite sensors and unmanned aircraft systems, citizen sensors, geolocation-based data services, global navigation satellite systems, and so on. The extensive use of these data for applications such as disaster and humanitarian response raises the issue of R&R from competing perspectives of location privacy and geospatial data quality. Although geospatial data can be integrated and linked with contextual information to identify individuals’ movements, steps taken to ensure privacy can complicate the multiuser development of high-quality geospatial workflows. Provenance information as digital records of historical (retrospective) and potential future (prospective) geospatial processes is often overlooked, misunderstood, or inadequately addressed. We explore the relationship between provenance information, location privacy, and geospatial data quality in the context of R&R with a focus on disaster analytics. Here, we argue that in the era of big data and deep learning, GIScientists and associated institutions bear greater responsibility both for geospatial workflow quality and for location privacy. Given vastly heterogenous computational landscapes, we provide practical recommendations for ethically driven provenance and R&R research and development within the GIScience community and beyond.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗