SEARCH · Search NASA

Results for “distributed workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

All sources NASA NTRS DOE OSTI

DOE OSTI2017

High Performance Interactive System Dynamics Visualization

This presentation describes a system dynamics simulation (SD) framework that supports an end-to-end analysis workflow that is optimized for deployment on ESIF facilities(Peregrine and the Insight Center). It includes (I) parallel and distributed simulation of SD models, (ii) real-time 3D visualization of running simulations, and (iii) comprehensive database-oriented persistence of simulation metadata, inputs, and outputs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Density estimation via measure transport: Outlook for applications in the biological sciences

Abstract One among several advantages of measure transport methods is that they allow or a unified framework for processing and analysis of data distributed according to a wide class of probability measures. Within this context, we present results from computational studies aimed at assessing the potential of measure transport techniques, specifically, the use of triangular transport maps, as part of a workflow intended to support research in the biological sciences. Scenarios characterized by the availability of limited amount of sample data, which are common in domains such as radiation biology, are of particular interest. We find that when estimating a distribution density function given limited amount of sample data, adaptive transport maps are advantageous. In particular, statistics gathered from computing series of adaptive transport maps, trained on a series of randomly chosen subsets of the set of available data samples, leads to uncovering information hidden in the data. As a result, in the radiation biology application considered here, this approach provides a tool for generating hypotheses about gene relationships and their dynamics under radiation exposure.

gene expression data↗

DOE OSTI2019

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗

DOE OSTI2021

A Systematic Interpretation of Subsurface Proppant Concentration from Drilling Mud Returns: Case Study from Hydraulic Fracturing Test Site (HFTS-2) in Delaware Basin

The aim of this study is generation and validation of a proppant log using analysis of drilling mud returns for child wells. Proppant log provides qualitative as well as quantitative insights into spatial distribution of proppant sand particles from prior stimulation of parent wells. While the basic methodology was developed and formalized during analysis of material collected from through fracture cores at Hydraulic Fracturing Test Site in Midland Basin (HFTS – 1), the test wells at HFTS – 2 in the neighboring Delaware Basin allowed the opportunity to validate the workflow on actual mud return samples from subsurface. As a child well is being drilled, periodic mud return samples are collected at the rig site and preserved for analysis. The workflow involves systematic cleaning of the samples including various steps such as washing, drying and segregation of samples into relevant size fractions of interest (< Mesh 20) based on specifications of pumped sand during stimulation of the parent well. Clean samples are imaged using high resolution transparency scanning. Scan images are then systematically analyzed for particles of interest using computer vision techniques. Sample counts are further validated using elemental analysis of smaller sub-samples at various depths of interest. This step is necessary to isolate proppant versus other naturally occurring minerals such as sulphates and carbonates which show similar optical properties. We successfully correlated proppant distribution against the existing parent well and validated propped versus relatively un-propped zones for a child well at the test site. The advantage of testing the proppant log concept at the HFTS – 2 site is the plethora of additional diagnostic data that is available to validate our primary observations. We can correlate spatial proppant distribution against variability in stimulation response based on independent observations such as image logs, microseismic attributes as well as DAS response, all of which tend to corroborate one another. One of our significant successes was being able to describe varying degrees of impact of the parent well along the lateral length of a stimulated child well. Our workflow represents a systematic and one-of-a-kind interpretation of spatial proppant distribution while drilling child wells. This provides unique opportunities to better understand the current state of the Downloaded from http://onepetro.org/URTECONF/proceedings-pdf/21URTC/2-21URTC/D021S031R003/2477415/urtec-2021-5189-ms.pdf/1 by Carol Worster on 28 February 2022 URTeC 5189 2 reservoir being targeted including zones which are likely more drained relative to others and how the planned completion of the child well can be improved. Lastly, this log can be useful is validating optimal well spacing in relatively new fields under development.

58 GEOSCIENCES↗

DOE OSTI2024

New Virtual Test Bed Capabilities: Virtual DOME Model and New Updates to Repository

The Department of Energy (DOE) Office of Nuclear Energy National Reactor Innovation Center accelerates the deployment of novel reactor concepts by establishing both physical and virtual spaces for building and testing various components, systems, and complete pilot plants. The Virtual Test Bed represents the virtual arm of the National Reactor Innovation Center and is a joint effort with the DOE Nuclear Energy Advanced Modeling and Simulation Program. The Virtual Test Bed mission is to accelerate the deployment of advanced reactors by facilitating the adoption of cutting-edge DOE advanced modeling and simulation tools to design, evaluate, and license reactors. This is primarily achieved by storing example challenge problems in an externally available repository and by developing models to fill the M&S gaps needed for potential demonstrators. Activities conducted this fiscal year focused on developing of a Demonstration of Microreactor Experiments shield model to help accelerate the confirmatory analysis required for the reactor demonstration. This model and workflow will allow developers to leverage advanced modeling and simulation tools to ensure their reactor demonstration concept will meet dose requirements and that the surrounding shield will stay within concrete temperature limits during steady-state and transient operation conditions. An initial model has been developed to evaluate the temperature distribution in the concrete shield during steady-state operation, including neutron and gamma heating effects. Various modeling strategies have been examined to understand their applicability and limitations with different reactor designs to make the workflow as reactor-agnostic as possible and computationally effective to maximize its usability. In addition to describing the Demonstration of Microreactor Experiments shield model and associated results, this report summarizes other accomplishments regarding repository maintenance and improvement and new external models hosted on the repository.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2025

Efficiently predicting pressure-composition-temperature diagrams to discover low-stability metal hydrides

Quantitatively accurate computational predictions of metal hydride thermodynamics are challenging but critical for alloy performance optimization across a multitude of technological domains, including hydrogen storage, compression, purification, and getters. Recent machine learning approaches have demonstrated great success in this area, but can potentially suffer from several shortcomings since they rely on imbalanced experimental training data and can have poor out-of-distribution (ood) test performance. Here, in this study, we circumvent such pitfalls by developing a computationally efficient, first principles-based workflow for direct prediction of metal hydride phase equilibrium, i.e., the pressure-composition-temperature (PCT) diagram. We then demonstrate its utility on predicting low stability hydrides derived from compositionally complex C14 Laves phase AB2 alloys. Specifically, we computationally predict and then experimentally validate an AB 2 alloy series (z < 0.6 for Ti 2−z Zr z CrMnFeNi) with ideal hydriding thermodynamics for a two-stage metal hydride-based compressor for pressurizing boil off from liquefied hydrogen. Importantly, this study lays the groundwork for accurate and efficient discovery/optimization of ood, low-stability hydrides for which purely data-driven approaches lack sufficient accuracy.

08 HYDROGEN↗

DOE OSTI2023

MILK : a Python scripting interface to MAUD for automation of Rietveld analysis

Modern diffraction experiments ( e.g. in situ parametric studies) present scientists with many diffraction patterns to analyze. Interactive analyses via graphical user interfaces tend to slow down obtaining quantitative results such as lattice parameters and phase fractions. Furthermore, Rietveld refinement strategies ( i.e. the parameter turn-on-off sequences) tend to be instrument specific or even specific to a given dataset, such that selection of strategies can become a bottleneck for efficient data analysis. Managing multi-histogram datasets such as from multi-bank neutron diffractometers or caked 2D synchrotron data presents additional challenges due to the large number of histogram-specific parameters. To overcome these challenges in the Rietveld software Material Analysis Using Diffraction ( MAUD ), the MAUD Interface Language Kit ( MILK ) is developed along with an updated text batch interface for MAUD . The open-source software MILK is computer-platform independent and is packaged as a Python library that interfaces with MAUD . Using MILK , model selection ( e.g. various texture or peak-broadening models), Rietveld parameter manipulation and distributed parallel batch computing can be performed through a high-level Python interface. A high-level interface enables analysis workflows to be easily programmed, shared and applied to large datasets, and external tools to be integrated with MAUD . Through modification to the MAUD batch interface, plot and data exports have been improved. The resulting hierarchical folders from Rietveld refinements with MILK are compatible with Cinema: Debye–Scherrer , a tool for visualizing and inspecting the results of multi-parameter analyses of large quantities of diffraction data. In this manuscript, the combined Python scripting and visualization capability of MILK is demonstrated with a quantitative texture and phase analysis of data collected at the HIPPO neutron diffractometer.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Development of Prototypical District-Scale Models: Preprint

The U.S. has set the climate goal to achieve net-zero greenhouse gas emissions by 2050. District-scale solutions, which include scale-specific opportunities for energy and emissions savings, can be investigated and implemented to help accelerate decarbonization and progress toward this goal. However, there is currently a lack of district-scale models of buildings and community energy systems that can be used to evaluate potential district-scale technologies and strategies across a range of representative community types. This initial work aims to define and develop prototype district models that can be adapted to support the planning, design, and operation of buildings and energy systems in districts considering the complexity and interactions of diverse building loads, weather impacts, distributed energy resources (e.g., PV, EV, electric and thermal energy storage), electric and thermal grid systems, and pricing signals. An overall workflow for developing these prototype district models is established. Stakeholders and potential users of the prototype district models provided technical feedback. The specifications of the selected high priority districts were defined and documented in a scorecard format. An example prototype district model was implemented with the URBANoptTM platform workflows. A case study was performed to demonstrate the model application.

district↗

NASA NTRS2006

LDCM Grid Prototype (LGP)

The LGP successfully demonstrated that grid technology could be used to create a collaboration among research scientists, their science development machines, and distributed data to create a science production system in a nationally distributed environment. Grid technology provides a low cost and effective method of enabling production of science products by the science community. To demonstrate this, the LGP partnered with NASA GSFC scientists and used their existing science algorithms to generate virtual Landsat-like data products using distributed data resources. LGP created 48 output composite scenes with 4 input scenes each for a total of 192 scienes processed in parallel. The demonstration took 12 hours, which beat the requirement by almost 50 percent, well within the LDCM requirement to process 250 scenes per day. The LGP project also showed the successful use of workflow tools to automate the processing. Investing in this technology has led to funding for a ROSES ACCESS proposal. The proposal intends to enable an expert science user to produce products from a number of similar distributed instrument data sets using the Land Cover Change Community-based Processing and Analysis System (LC-ComPS) Toolbox. The LC-ComPS Toolbox is a collection of science algorithms that enable the generation of data with ground resolution on the order of Landsat-class instruments.

Weinstein, Beth↗

DOE OSTI2024

Online Detection of Power Grid Anomalies via Federated Learning

Data from sensors is critical for advanced applica- tions that support efficient, reliable, and resilient electric grid operations. Historically, data from phasor measurement units (PMU) has been utilized to develop a wide variety of wide area control and protection applications suitable for power grid control centers. However, until now, most of these could not be deployed for automated operations due to a set of data corruption challenges and uncertainty in the incoming data pipeline. In this paper, we address the problem of detecting different variety of anomalies that are evident in different high- speed power grid measurements. The paper discusses a workflow for handling problems with data acquisition and highlights some of the key findings suitable for anomaly detection in a centralized and distributed environment. The effectiveness of the proposed method was demonstrated with results utilizing realistic PMU datasets

Shinkle, Matthew W.↗

DOE OSTI2024

A Microservices Architecture Toolkit for Interconnected Science Ecosystems

Microservices architecture is a promising approach for developing reusable scientific workflow capabilities for inte- grating diverse resources, such as experimental and observational instruments and advanced computational and data management systems, across many distributed organizations and facilities. In this paper, we describe how the INTERSECT Open Architec- ture leverages federated systems of microservices to construct interconnected science ecosystems, review how the INTERSECT software development kit eases microservice capability develop- ment, and demonstrate the use of such capabilities for deploying an example multi-facility INTERSECT ecosystem.

Brim, Michael↗

DOE OSTI2019

LIVVkit 2.1: automated and extensible ice sheet model validation

A collection of scientific analyses, metrics, and visualizations for robustvalidation of ice sheet models is presented using the Land Ice Verificationand Validation toolkit (LIVVkit), version 2.1, and the LIVVkit Extensionsrepository (LEX), version 0.1. This software collection targets stand-aloneice sheet or coupled Earth system models, and handles datasets and analysesthat require high-performance computing and storage. LIVVkit aims to enableefficient and fully reproducible workflows for postprocessing, analysis, andvisualization of observational and model-derived datasets in a shareableformat, whereby all data, methodologies, and output are distributed to usersfor evaluation. Extending from the initial LIVVkit software framework, wedemonstrate Greenland ice sheet simulation validation metrics using thecoupled Community Earth System Model (CESM) as well as an idealizedstand-alone high-resolution Community Ice Sheet Model, version 2 (CISM2),coupled to the Albany/FELIX velocity solver (CISM-Albany or CISM-A). As oneexample of the capability, LIVVkit analyzes the degree to which modelscapture the surface mass balance (SMB) and identifies potential sources ofbias, using recently available in situ and remotely sensed data ascomparison. Related fields within atmosphere and land surface models, e.g.,surface temperature, radiation, and cloud cover, are also diagnosed. Appliedto the CESM1.0, LIVVkit identifies a positive SMB bias that is focusedlargely around Greenland's southwest region that is due to insufficientablation.

58 GEOSCIENCES↗

DOE OSTI2022

Deep-learning-based workflow for boundary and small target segmentation in digital rock images using UNet++ and IK-EBM

We report three-dimensional (3D) X-ray micro-computed tomography (μCT) has been widely used in petroleum engineering because it can provide detailed pore structural information for a reservoir rock, which can be imported into a pore-scale numerical model to simulate the transport and distribution of multiple fluids in the pore space. The partial volume blurring (PVB) problem is a major challenge in segmenting raw μCT images of rock samples, which impacts boundaries and small targets near the resolution limit. We developed a deep-learning (DL)-based workflow for accurate and fast partial volume segmentation. The DL model's performance depends primarily on the training data quality and model architecture. This study employed the entropy-based-masking indicator kriging (IK-EBM) to segment 3D Berea sandstone images as training datasets. The comparison between IK-EBM and manual segmentation using a 3D synthetic sphere pack, which had a known ground truth, showed that IK-EBM had higher accuracy on partial volume segmentation. We then trained and tested the UNet++ model, a state-of-the-art supervised encoder-decoder model, for binary (i.e., void and solid) and four-class segmentation. We compared the UNet++ with the commonly used U-Net and wide U-Net models and showed that the UNet++ had the best performance in terms of pixel-wise and physics-based evaluation metrics. Specifically, boundary-scaled accuracy demonstrated that the UNet++ architecture outperformed the regular U-Net architecture in the segmentation of pixels near boundaries and small targets, which were subjected to the PVB effect. Feature map visualization illustrated that the UNet++ bridged the semantic gaps between the feature maps extracted at different depths of the network, thereby enabling faster convergence and more accurate extraction of fine-scale features. The developed workflow significantly enhances the performance of supervised encoder-decoder models in partial volume segmentation, which has extensive applications in fundamental studies of subsurface energy, water, and environmental systems.

02 PETROLEUM↗

DOE OSTI2026

Comparison of automated chemical-guided segmentation and human annotation of soil organic matter in X-ray microcomputed tomography imaging in contrasted soil types

Soil organic matter (OM) formation and persistence is strongly influenced by the spatial distribution of organic substrates and microscale soil heterogeneity by dictating OM accessibility to microorganisms. However, traditional size and/or density fractionation techniques disrupt aggregate architecture, eliminating spatial information needed to fully understand intra-aggregate OM distribution. To quantify three-dimensional OM spatial distribution and automate segmentation in X-ray microcomputed tomography (µCT) imaging without human annotation bias, we developed an iodine gas vapor (I2) based staining workflow that eliminates labor-intensive manual annotation while maintaining segmentation accuracy, using aggregates from four taxonomically diverse soils (Xerofluvent, Haploxeroll Sphagnofibrist, Palehumult) with an 8-fold range of soil organic carbon. Human annotation of 10 µCT slices by the experienced and inexperienced annotators resulted in variations up to 3% in the Dice similarity coefficient (DSC), reflecting a degree of inherent subjectivity of manual labeling. Such inconsistencies are expected to compound as the number of manually annotated slices increases. Dual-energy µCT imaging at 33.1 keV (below the iodine (I) K-edge) and 33.2 keV (above the I K-edge) was used to resolve aggregate microstructure following I2 staining. The automated image subtraction pipeline identified OM regions by the I Kedge induced brightness increases, achieving DSC values of 0.58–0.83 relative to an experienced annotator. Sensitivity analyses revealed that the reconstruction alpha value—optimized via the open-source tool TomocuPy—and the 3D registration slice count were the primary determinants of accuracy, providing a novel benchmark for dual-energy soil imaging. The pipeline without GPU acceleration achieved 9.6 to 43.2 times faster than manual annotation. Using GPU-accelerated image post-processing and affine transformation matrices, the pipeline successfully segmented OM elements for large-scale datasets (3232×3232 pixel, 2048 slices) within ~5200 s from raw file acquisition to segmented output. The high-throughput approach enables the quantification of OM spatial distribution across diverse and heterogeneous soil.

Soil microbial biomass↗

DOE OSTI2026

3D seismic imaging of a fracture damage zone controlling reservoir compartmentalization at the Raft River EGS using multi-azimuth walkaway VSP

Accurate imaging of steeply dipping fracture zones in crystalline enhanced geothermal systems (EGS) is critical for constraining permeability architecture and guiding stimulation design. However, such structures remain poorly resolved by conventional surface seismic methods. We present a fully three-dimensional (3D) elastic-waveform inversion-migration workflow applied to multi-azimuth walkaway vertical seismic profiling (VSP) data acquired in a deviated borehole at the Raft River EGS. The workflow integrates first-arrival traveltime tomography, multi-scale elastic waveform inversion (EWI), and elastic least-squares reverse-time migration (ELSRTM) to recover high-resolution compressional-and shear-wave velocity models and to image structural discontinuities in the crystalline basement. The results reveal a laterally continuous low-velocity anomaly, with shear-wave velocity reductions of 25-30%, consistent with fractureinduced mechanical weakening. Two steeply dipping discontinuities bound a 50-80 m wide fracture damage zone. Independent constraints from microseismic clustering and geochemical compartmentalization corroborate the geometry and structural significance of this feature. Synthetic modeling further confirms that structures of this scale are resolvable (~30 m). These findings indicate that the Narrows structure is a distributed fracture damage zone rather than a discrete fault plane. By resolving fracture-zone geometry at the tens-of-meters scale using a single borehole, this workflow provides a practical and transferable approach for improving structural characterization, reducing uncertainty in permeability architecture, and supporting reservoir modeling and stimulation design in fractured crystalline EGS reservoirs.

58 GEOSCIENCES↗

DOE OSTI2024

GWAS supported by computer vision identifies large numbers of candidate regulators of in planta regeneration in Populus trichocarpa

Plant regeneration is an important dimension of plant propagation and a key step in the production of transgenic plants. However, regeneration capacity varies widely among genotypes and species, the molecular basis of which is largely unknown. Association mapping methods such as genome-wide association studies (GWAS) have long demonstrated abilities to help uncover the genetic basis of trait variation in plants; however, the performance of these methods depends on the accuracy and scale of phenotyping. To enable a large-scale GWAS of in planta callus and shoot regeneration in the model tree Populus, we developed a phenomics workflow involving semantic segmentation to quantify regenerating plant tissues over time. We found that the resulting statistics were of highly non-normal distributions, and thus employed transformations or permutations to avoid violating assumptions of linear models used in GWAS. We report over 200 statistically supported quantitative trait loci (QTLs), with genes encompassing or near to top QTLs including regulators of cell adhesion, stress signaling, and hormone signaling pathways, as well as other diverse functions. Our results encourage models of hormonal signaling during plant regeneration to consider keystone roles of stress-related signaling (e.g. involving jasmonates and salicylic acid), in addition to the auxin and cytokinin pathways commonly considered. The putative regulatory genes and biological processes we identified provide new insights into the biological complexity of plant regeneration, and may serve as new reagents for improving regeneration and transformation of recalcitrant genotypes and species.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Toward an Autonomous Workflow for Single Crystal Neutron Diffraction

The operation of the neutron facility relies heavily on beamline scientists. Some experiments can take one or two days with experts making decisions along the way. Leveraging the computing power of HPC platforms and AI advances in image analyses, here we demonstrate an autonomous workflow for the single-crystal neutron diffraction experiments. The workflow consists of three components: an inference service that provides real-time AI segmentation on the image stream from the experiments conducted at the neutron facility, a continuous integration service that launches distributed training jobs on Summit to update the AI model on newly collected images, and a frontend web service to display the AI tagged images to the expert. Ultimately, the feedback can be directly fed to the equipment at the edge in deciding the next-step experiment without requiring an expert in the loop. With the analyses of the requirements and benchmarks of the performance for each component, this effort serves as the first step toward an autonomous workflow for real-time experiment steering at ORNL neutron facilities.

Yin, Junqi↗

NASA NTRS2019

An Integrated Data Analytics Platform

An Integrated Science Data Analytics Platform is an environment that enables the confluence of resources for scientific investigation. It harmonizes data, tools and computational resources which subsequently enable the research community to focus on the investigation rather than spending time on security, data preparation, management, etc. OceanWorks is a NASA technology integration project to establish a cloud-based Integrated Ocean Science Data Analytics Platform at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) for big ocean science. It focuses on advancement and maturity by bringing together several NASA open-source, big data projects for parallel analytics, anomaly detection, in-situ to satellite data matchup, quality-screened data subsetting, search relevancy, and data discovery. Our communities are relying on data distributed through data centers such as the PO.DAAC, COAPS, NCAR, and many others to conduct their research. In typical investigations, scientists would engage in: search for data, evaluate the relevance of that data, download it, and then apply algorithms to identify trends. Such workflow cannot scale if the research involves a massive amount of data or multi-variate measurements. NASA’s Surface Water and Ocean Topography (SWOT) mission is expected to produce massive amount of observational data during its 3-year nominal mission. Collections like SWOT challenges all existing Earth Science data archival, distribution and analysis paradigms. In this paper, we will discuss how OceanWorks enhances the analysis of physical ocean data where the computation is done on an elastic cloud platform next to the archive to deliver fast, web-accessible services for working with oceanographic measurements.

Yang, Chaowei↗