Elastic Workflows with PMIx [Slides]
Scientific workflows increasing in complexity. Elastic workflows promise improved application and system performance.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Scientific workflows increasing in complexity. Elastic workflows promise improved application and system performance.
In order to address the need for published data, considerable effort has gone into formalizing the process of data publication. From funding agencies to publishers, data publication has rapidly become a requirement. Digital Object Identifiers (DOI) and data citations have enhanced the integration and availability of data. The challenge facing data publishers now is to deal with the increased number of publishable data products and most importantly the difficulties of publishing diverse data products into an online archive. The Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC), a NASA-funded data center, faces these challenges as it deals with data products created by individual investigators. This paper summarizes the challenges of curating data and provides a summary of a workflow solution that ORNL DAAC researcher and technical staffs have created to deal with publication of the diverse data products. Finally, the workflow solution presented here is generic and can be applied to data from any scientific domain and data located at any data center.
We investigated the opportunity to transition from paper and PDF procedures to more interactive workflows. This transition has been considered because evidence shows that these more interactive digital workflows increase safety and efficiency, while making complex procedures easier for the user to follow and track. The culmination of the project is a report and presentation for the lab deputy director at the end of the fiscal year, in which our team will recommend a path forward based on the findings from literature, employee interviews, and other information gathering activities.
Summary Scientific breakthroughs in biomolecular methods and improvements in hardware technology have shifted from a long‐running simulation to a large set of shorter simulations running simultaneously, called an ensemble. In an ensemble, simulations are usually coupled with analyses of data produced by the simulations. In situ methods can be used to analyze large volumes of data generated by scientific simulations at runtime (i.e., simulations and analyses are performed concurrently). In this work, we study the execution of ensemble‐based simulations paired with in situ analyses using in‐memory staging methods. Using an ensemble of molecular dynamics in situ workflows with multiple simulations and analyses, we first show that collecting traditional metrics such as makespan, instructions per cycle, memory usage, or cache miss ratio is not sufficient to characterize complex behaviors of ensembles. We propose a method to evaluate the performance of ensembles of workflows that captures multiple resource usage aspects: resource efficiency, resource allocation, and resource provisioning. Experimental results demonstrate that the proposed method can effectively distinguish the performance of different component placements in an ensemble with up to 32 ensemble members. By evaluating different co‐location scenarios, our proposed performance indicators demonstrate benefits of co‐locating simulation and coupled analyses within a compute node.
Abstract To‐date, only two UIC Class VI permits have been issued by the US Environmental Protection Agency. We illustrate a four‐phase workflow to first identify regional storage resources and then down‐select sites to yield permit‐ready locations that can accept and store large volumes of CO 2 . Specific permit requirements should guide objectives and define deliverables of respective workflow phases. In the first phase we used available regional data and screened structure and injection zones to locate resources that match CO 2 volumes planned to be captured. Available data were also used to assess presence and depth of usable groundwater, the key resource being protected via permitting. We then used advanced, closed‐form, analytical solutions (EASiTool) to estimate CO 2 injectivity into each hydrologically connected injection compartment. In the second phase we acquired and conditioned additional wireline logs and leased available seismic datasets. We interpreted the depositional systems from wireline well‐log character and mapped sandbody geometry to interpolate injection and confining‐zone distribution. Using available data, we mapped faults and locations of freshwater and overpressure (or other capacity‐limiting geologic parameters) in more detail. In the third phase, we used the augmented geologic data to develop a static model for the selected area, extracted the areas of highest interest, and generated and ran dynamic (flow) models. In a fourth phase, we reduced major uncertainties identified in earlier phases. Our case study indicates that to complete preparation of a permit application requires (1) improved lithologic characterization information (thicknesses and horizontal and vertical connectivity) and (2) better definition of poorly defined local faults. © 2023 The Authors. Greenhouse Gases: Science and Technology published by Society of Chemical Industry and John Wiley & Sons Ltd.
ynchrotron light sources are routinely used to perform imaging experiments. In this paper, we review the relevant computational stages, identify bottlenecks, and highlight future opportunities to streamline data acquisition for experimental microscopy workflows. We demonstrate our preliminary exploration with an end-to-end scientific workflow on Summit based on micro-computed tomography data. Computational elements include: 1) reconstruction of volumetric image data; 2) denoising with deep neural networks; and 3) non-local means based segmentation and quantitative analysis.
The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.
Abstract Future machine learning strategies for materials process optimization will likely replace human capital-intensive artisan research with autonomous and/or accelerated approaches. Such automation enables accelerated multimodal characterization that simultaneously minimizes human errors, lowers costs, enhances statistical sampling, and allows scientists to allocate their time to critical thinking instead of repetitive manual tasks. Previous acceleration efforts to synthesize and evaluate materials have often employed elaborate robotic self-driving laboratories or used specialized strategies that are difficult to generalize. Herein we describe an implemented workflow for accelerating the multimodal characterization of a combinatorial set of 915 electroplated Ni and Ni–Fe thin films resulting in a data cube with over 160,000 individual data files. Our acceleration strategies do not require manufacturing-scale resources and are thus amenable to typical materials research facilities in academic, government, or commercial laboratories. The workflow demonstrated the acceleration of six characterization modalities: optical microscopy, laser profilometry, X-ray diffraction, X-ray fluorescence, nanoindentation, and tribological (friction and wear) testing, each with speedup factors ranging from 13–46x. In addition, automated data upload to a repository using FAIR data principles was accelerated by 64x.
The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate $\mathcal{O}$(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.
This work proposes a finite element (FE) analysis workflow to simulate directed energy deposition (DED) additive manufacturing at a macroscopic length scale (i.e. part length scale) and to predict thermal conditions during manufacturing, as well as distortions, strength and residual stresses at the completion of manufacturing. The proposed analysis method incorporates a multi-step FE workflow to elucidate the thermal and mechanical responses in laser engineered net shaping (LENS) manufacturing. For each time step, a thermal element activation scheme captures the material deposition process. Then, activated elements and their associated geometry are analyzed first thermally for heat flow due to radiation, convection, and conduction, and then mechanically for the resulting stresses, displacements, and material property evolution. Finally, simulations agree with experimentally measured in situ thermal measurements for simple cylindrical build geometries, as well as general trends of local hardness distribution and plastic strain accumulation (represented by relative distribution of geometrically necessary dislocations).
Continuous carbon fiber prepreg slit and cut into rectangular platelets has proven to be a useful material for net shape molding of semi-structural and structural components in the aerospace and automotive industries. Furthermore, to assist the designer in use of these prepreg platelet molding compounds, sometimes called carbon fiber sheet molding compounds, simulation tools are required that can predict the as-manufactured fiber orientation state which has a significant impact on the resulting performance. Herein, an analysis workflow for design-enabling predictions is demonstrated for a double dome geometry with two different initial charge configurations. Here, the workflow is validated through comparison with experimental short shots, orientation state, and stiffness trends. Significantly, to complete the validation, a method is proposed for determining the confidence bounds on measured orientation state enhancing the results of optical microscopy which can only produce a small sample of platelet orientations.
Accurate RF (Radio Frequency) simulations of fusion systems like ITER require the definition of high-fidelity analysis geometries that include detailed antenna, reactor wall, and physics regions. Here, this paper will describe a workflow for the execution of adaptive high-performance simulations of RF fusion systems. In this workflow, the simulation input consists of a CAD model attributed with the needed analysis attributes. The analysis mesh is automatically generated and the analysis steps are executed using the time-harmonic Maxwell's equations solved using high-order Nédélec finite elements. A patch recovery-based error estimator is used to drive a conforming mesh adaptation procedure.
Carbon dioxide enhanced oil recovery (CO 2 -EOR) projects not only extract residual oil but also sequestrate CO 2 in the depleted reservoirs. Here, this study develops a machine-learning-based workflow to co-optimize the hydrocarbon recovery, CO 2 sequestration volume and project net present value (NPV) simultaneously. Considering the trade-off relationships among the objective functions, support vector regression with Gaussian kernel (Gaussian- SVR) proxies are coupled with multi-objective particle swarm optimization (PSO) protocol and generate Pareto optimal solutions. Taking advantage of the high computational efficacy of the proxy model, economic uncertainties introduced by tax credits, capital costs and oil price are investigated by this study. The results indicate that the tax incentive policy (Section 45Q) plays a vital role in enhancing the economic returns of CO 2 -EOR projects, especially under the depression of crude oil market. The proposed workflow has been successfully implemented to optimize a water alternative CO 2 (CO 2 -WAG) injection project in a depleted oil sand in the US. The optimization results yield an incremental oil production of 15.8 MM STB and 1.37 MM metric tons of CO 2 storage in a 20-year development strategy, with the highest project NPV to be 205.6 MM US dollars.
Purpose: Early and accurate assessment of radiation injury by radiation-responsive biomarkers is critical for triage and early intervention. Biofluids such as urine and serum are convenient for such analysis. Recent research has also suggested that exosomes are a reliable source of biomarkers in disease progression. In the present study, we analyzed total urine proteome and exosomes isolated from urine or serum for potential biomarkers of acute and persistent radiation injury in mice exposed to lethal whole body irradiation (WBI). Methods and Materials: For feasibility studies, the mice were irradiated at 10.4 Gy WBI, and urine and serum samples were collected 24 and 72 hours after irradiation. Exosomes were isolated and analyzed using liquid chromatography mass spectrometry/mass spectrometry-based workflow for radiation exposure signatures. A data dependent acquisition and SWATH-MS combined workflow approach was used to identify significantly exosome biomarkers indicative of acute or persistent radiation-induced responses. For the validation studies, mice were exposed to 3, 6, 8, or 10 Gy WBI, and samples were analyzed for comparison. Results: A comparison between total urine proteomics and urine exosome proteomics demonstrated that exosome proteomic analysis was superior in identifying radiation signatures. Feasibility studies identified 23 biomarkers from urine and 24 biomarkers from serum exosomes after WBI. Urinary exosome signatures identified different physiological parameters than the ones obtained in serum exosomes. Exosome signatures from urine indicated injury to the liver, gastrointestinal, and genitourinary tracts. In contrast, serum showed vascular injuries and acute inflammation in response to radiation. Selected urinary exosomal biomarkers also showed changes at lower radiation doses in validation studies. Conclusions: Exosome proteomics revealed radiation- and time-dependent protein signatures after WBI. A total of 47 differentially secreted proteins were identified in urinary and serum exosomes. Together, these data showed the feasibility of defining biomarkers that could elucidate tissue-associated and systemic response caused by high-dose ionizing radiation. This is the first report using an exosome proteomics approach to identify radiation signatures.
Written language and conventional data structures for representing scientific procedures suffer from low process detail, often fail to accurately represent protocols, and lack universality. New strategies for the handling of experimental data are needed to provide viable process information for both humans and machines. In this work, we present the universal workflow language (UWL) and interface (UWLi). UWL is a findable, accessible, interoperable, and reusable (FAIR)-compatible, graph-based data architecture that can capture arbitrary scientific procedures through workflow representation, and UWLi is an accompanying software package for building, manipulating, and interpreting UWL entries. The UWL format was found to be highly effective in identifying deficiencies in the reported process details of high-impact, peer-reviewed scientific journals, and in simulated scenarios, the graph format was shown to be more effective than conventional methods in predictively modeling the outcome of diverse scientific protocols. Implementation of UWL could enable more accurate scientific communication and more impactful process datasets.
In particle accelerators, RF cavities are used to accelerate charged particle beams to designed high energy for physical applications. In a typical accelerator design, the optimization of RF cavities and the optimization of beam dynamics are carried out in separate studies. For a more general and unrestricted accelerator design, a coupled optimization of the RF cavities and the beam parameters is required. For this coupled optimization problem, we have developed an integrated electromagnetics and beam dynamics workflow management system. Within this system, the geometries for a set of cavity components are first adjusted; the field modes are then computed with an electromagnetics program, and imported into a beam dynamics program for beam dynamics simulation. This workflow is encapsulated into a parallel multi-objective optimizer to achieve the integrated accelerator design optimization. A multi fidelity strategy is developed to improve the speed of the optimizer. Furthermore, this integrated global optimization capability is illustrated using a photoinjector design example and yields an improved design.
The Second Target Station (STS) at the US Department of Energy’s Oak Ridge National Laboratory is designed to become the world’s highest peak-brightness spallation source of cold neutrons. Successful completion of the STS, which is currently in the preliminary design phase, will provide transformative new capabilities to examine novel materials for future technologies. At STS, neutrons will be generated by spallation reactions in a solid tungsten target. They will be moderated and thermalized in two cold (20 K) para-hydrogen moderators. Careful optimization of these moderators is essential to the project’s success. To find optimal moderator designs, an advanced optimization workflow integrates high-fidelity neutronics calculations using the Monte Carlo N-Particle (MCNP) transport code MCNP6.2 with state-of-the-art optimization algorithms in the Dakota optimization toolkit. For each design iteration, a parametrized solid CAD geometry is generated in Creo and automatically converted into an unstructured mesh geometry by Attila 4MC for the neutronics calculation with MCNP. Iterations repeat until optimal designs are found. Herein this paper presents the results of a sensitivity and optimization study for the cylindrical and tube moderators. Both moderators can be optimized for maximum peak brightness, maximum time-integrated brightness, or any combination between these extremes. Maximum peak brightness is achieved by using smaller optimal dimensions of the moderators, whereas maximum time-integrated brightness is achieved by using larger dimensions. A Pareto front details the designs that optimally balance both brightness metrics. The Pareto front can be found in only 40–110 iterations with 4–5 design parameters when using the efficient global and Pareto-set optimization algorithms in Dakota. Additionally, important engineering constraints can be taken into account, such as the coupling between the cylindrical moderator radius and aluminum vessel wall thicknesses required to ensure structural integrity of the vessels. This interaction has a significant impact on the resulting optimal designs. Our new, highly efficient, fully automated optimization workflow will be used to optimize additional STS components in the future and can be adopted for design and optimization studies at other experimental neutron and accelerator facilities.
We report three-dimensional (3D) X-ray micro-computed tomography (μCT) has been widely used in petroleum engineering because it can provide detailed pore structural information for a reservoir rock, which can be imported into a pore-scale numerical model to simulate the transport and distribution of multiple fluids in the pore space. The partial volume blurring (PVB) problem is a major challenge in segmenting raw μCT images of rock samples, which impacts boundaries and small targets near the resolution limit. We developed a deep-learning (DL)-based workflow for accurate and fast partial volume segmentation. The DL model's performance depends primarily on the training data quality and model architecture. This study employed the entropy-based-masking indicator kriging (IK-EBM) to segment 3D Berea sandstone images as training datasets. The comparison between IK-EBM and manual segmentation using a 3D synthetic sphere pack, which had a known ground truth, showed that IK-EBM had higher accuracy on partial volume segmentation. We then trained and tested the UNet++ model, a state-of-the-art supervised encoder-decoder model, for binary (i.e., void and solid) and four-class segmentation. We compared the UNet++ with the commonly used U-Net and wide U-Net models and showed that the UNet++ had the best performance in terms of pixel-wise and physics-based evaluation metrics. Specifically, boundary-scaled accuracy demonstrated that the UNet++ architecture outperformed the regular U-Net architecture in the segmentation of pixels near boundaries and small targets, which were subjected to the PVB effect. Feature map visualization illustrated that the UNet++ bridged the semantic gaps between the feature maps extracted at different depths of the network, thereby enabling faster convergence and more accurate extraction of fine-scale features. The developed workflow significantly enhances the performance of supervised encoder-decoder models in partial volume segmentation, which has extensive applications in fundamental studies of subsurface energy, water, and environmental systems.