Search NASA⌕ Search

SEARCH · Search NASA

Results for “Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25

Workflow-Based Software Development Environment

The Software Developer's Assistant (SDA) helps software teams more efficiently and accurately conduct or execute software processes associated with NASA mission-critical software. SDA is a process enactment platform that guides software teams through project-specific standards, processes, and procedures. Software projects are decomposed into all of their required process steps or tasks, and each task is assigned to project personnel. SDA orchestrates the performance of work required to complete all process tasks in the correct sequence. The software then notifies team members when they may begin work on their assigned tasks and provides the tools, instructions, reference materials, and supportive artifacts that allow users to compliantly perform the work. A combination of technology components captures and enacts any software process use to support the software lifecycle. It creates an adaptive workflow environment that can be modified as needed. SDA achieves software process automation through a Business Process Management (BPM) approach to managing the software lifecycle for mission-critical projects. It contains five main parts: TieFlow (workflow engine), Business Rules (rules to alter process flow), Common Repository (storage for project artifacts, versions, history, schedules, etc.), SOA (interface to allow internal, GFE, or COTS tools integration), and the Web Portal Interface (collaborative web environment

Izygon, Michel E.↗

EARTHDATA PUB: A Data Publication Workflow Solution for NASA’s EOSDIS

Each NASA Distributed Active Archive Center (DAAC) faces the challenge of dealing with an increasingly diverse number of publishable data products from diverse data producers. Data producers, on the other hand, may experience pain points when interacting with the EOSDIS for the first time or when publishing different data at different DAACs. As a result, there has been a growing need to develop a common software framework that serves as a common interface for data producers, rigorously defines the data publication procedure for DAAC staff, facilitates the management of various data publication processes, and tracks the progress of data publication. This software should also account for the different configurations at different DAACs. Currently, two primary data publication workflow and tracking tools exist in operation at the EOSDIS: Semi-Automated ingest System (SAuS) and Data Publication workflow Portal (DAPPeR). However, neither tool is cloud-ready. Automated data processing could be managed by Cumulus, an EOSDIS cloud-based data ingest, archive and management system. However, Cumulus does not support manual tasks or on-premise implementations. We propose to develop the Earthdata Publication Minimum Viable Product (Earthdata Pub MVP) -- a cloud-hosted solution that works with both cloud and on-premise systems and implements the communications and exchange requirements generated by the Earthdata Pub information architecture team.

Rice, Justin L.↗

Development of Computational Materials Workflows for Additively Manufactured Metallic Materials to Enable Accelerated Prediction of Fatigue Performance

The maturation of computational materials approaches for fatigue performance prediction in a qualification and certification process is stifled by the ability to validate complex, microstructure-based simulations. Such a validation strategy bears immediate challenges including generating accurate virtual microstructures, efficiently solving physics-based mechanical simulations over relevant spatial and temporal scales, and acquiring high-fidelity calibration and validation data at the appropriate length scale. This presentation will overview these common challenges and present a case study to demonstrate a computational materials workflow for additively manufactured metallic materials. In this study, process-specific defects are characterized using segmented X-Ray micro-computed tomography measurements and overlaid on virtual microstructures. Accelerated crystal plasticity-based fatigue simulations are performed to demonstrate cyclic evolution and localization of mechanical fields in the vicinity of defects in response to their precise spatial configuration. An example of how this computational materials workflow may support next-generation qualification is discussed.

computational materials↗

End-to-end interoperability and workflows from building architecture design to one or more simulations

An end-to-end interoperability and workflows from building architecture design to one or more simulations, in one aspect, may comprise establishing a BIM enablement platform architecture. A data model defines data entities and entity relationships for enabling the interoperability and workflows. A data definition language may be implemented that defines and creates a table schema of a database associated with the data model. Data management services and/or application programming interfaces may be implemented for interacting with the data model. Web services may also be provided for interacting with the data model via the Web. A user interface may be implemented that communicates with users and uses the BIM enablement platform architecture, the data model, the data definition language, data management services and application programming interfaces to provide functions to the users to perform work related to building information management.

97 MATHEMATICS AND COMPUTING↗

End-to-end interoperability and workflows from building architecture design to one or more simulations

An end-to-end interoperability and workflows from building architecture design to one or more simulations, in one aspect, may comprise establishing a BIM enablement platform architecture. A data model defines data entities and entity relationships for enabling the interoperability and workflows. A data definition language may be implemented that defines and creates a table schema of a database associated with the data model. Data management services and/or application programming interfaces may be implemented for interacting with the data model. Web services may also be provided for interacting with the data model via the Web. A user interface may be implemented that communicates with users and uses the BIM enablement platform architecture, the data model, the data definition language, data management services and application programming interfaces to provide functions to the users to perform work related to building information management.

97 MATHEMATICS AND COMPUTING↗

Production processing and workflow management software evaluation in the DUNE collaboration

The Deep Underground Neutrino Experiment (DUNE) will be theworld’s foremost neutrino detector when it begins taking data in the mid-2020s.Two prototype detectors, collectively known as ProtoDUNE, have begun tak-ing data at CERN and have accumulated over 3 PB of raw and reconstructeddata since September 2018. Particle interaction within liquid argon time projec-tion chambers are challenging to reconstruct, and the collaboration has set upa dedicated Production Processing group to perform centralized reconstructionof the large ProtoDUNE datasets as well as to generate large-scale Monte Carlosimulation. Part of the production infrastructure includes workflow manage-ment software and monitoring tools that are necessary to eciently submit andmonitor the large and diverse set of jobs needed to meet the experiment’s goals.We will give a brief overview of DUNE and ProtoDUNE, describe the varioustypes of jobs within the Production Processing group’s purview, and discuss thesoftware and workflow management strategies are currently in place to meetexisting demand. We will conclude with a description of our requirements in aworkflow management software solution and our planned evaluation process.

Herner, Kenneth R.↗

Chimbuko: A Workflow-Level Scalable Performance Trace Analysis Tool

ABSTRACT Due to the sheer volume of data it is typically impractical to analyze the detailed performance of an HPC application running at-scale. While conventional small-scale benchmarking and scaling studies are often sufficient for simple applications, many modern workflow-based applications couple multiple elements with competing resource demands and complex inter-communication patterns for which performance cannot easily be studied in isolation and at small scale. This work discusses Chimbuko, a performance analysis framework that provides real-time, in situ anomaly detection. By focusing specifically on performance anomalies and their origin (aka provenance), data volumes are dramatically reduced without losing necessary details. To the best of our knowledge, Chimbuko is the first online, distributed, and scalable workflow-level performance trace analysis framework. We demonstrate the tool's usefulness on Oak Ridge National Laboratory's Summit system.

97 MATHEMATICS AND COMPUTING↗

2023 Real-time Optimization Workflow Status Update

Nuclear integrated energy systems are composed of a diverse set of energy generation sources and exist in dynamic and competitive electricity markets. With the inclusion of thermal energy storage, nuclear power systems can store heat for future use through various processes, such as water desalination or hydrogen generation. This heat storage can be managed in such a way as to economically optimize its usage. By combining real-time price data from the day-ahead and real-time markets with predictive and intelligent models, the charging and discharging of the thermal energy storage may be determined and optimized. This research details an approach through models and systems for the real-time optimization (RTO) of capacity allocation. Virtual models of the energy system and its physics phenomenon and component interactions provide intelligence to verification and prediction of operations. An optimization framework can use data generated from both a set of physical assets as well as the virtual models to predict future performance and create optimization and control workflows. Data warehouse technologies can be used to combine data across models, optimization workflows, market price data, and sensor data to intuitively store various types of data and provide integrations to physical control systems as well as user visualizations. Put together, these components can create a system for the RTO of nuclear integrated energy systems. Various virtual models and bench-scale physical demonstrations have been successfully performed and verified using this system. Larger scale testbeds that include thermal energy storage systems have been identified as future opportunities. A gap analysis that details the steps necessary to reach a physical demonstration at this scale is provided, along with conclusions on the current effort and future work.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.

Santos Souza, Renan↗

GeoThermalCloud for EGS – An Open-source, User-friendly, Scalable AI Workflow for Modeling Enhanced Geothermal Systems

Enhanced Geothermal Systems (EGS) offer a vast potential to expand the use of geothermal energy. Heat is extracted from this engineered system by injecting relatively cold water into subsurface fractures, which are in contact with hot dry rock, and brought back to surface through production wells. Creating EGS requires improving the natural permeability of hot crystalline rocks. In this short conference paper, we present a reproducible workflow for modeling EGS. Our workflow called the GeoThermalCloud (GTC) for EGS, leverages recent advances in machine learning, deep learning, and high-performance computing. This GTC framework is currently being made open-source, user-friendly, and reproducible through python scripts as well as Google Colab/Jupyter Notebooks. This GTC for EGS modeling scripts are made available at https://github.com/SmartTensors/GeoThermalCloud.jl/tree/master/EGS and will constantly be updated to cater for geothermal community. Current GTC framework provides scripts to train deep learning (DL) models for techno-economics and data worth analysis. The Geothermal Design Tool (https://github.com/GeoDesignTool/GeoDT.git), a fast and simplified multi-physics solver, is used to develop a database for training DL models. This short paper provides details on the scripts to curate, process, and train DL models. The scripts can easily be modified to train on databases generated by other popular open-source simulators such as PFLOTRAN, STOMP, TOUGH, and GEOSX or commercial software such as ResFrac and COMSOL.

15 GEOTHERMAL ENERGY↗

nmrrr : A Reproducible Workflow for Binning and Visualizing NMR Spectra From Environmental Samples

Nuclear magnetic resonance (NMR) spectroscopy is a useful tool for detection and identification of molecular structural information, with increasing applications in environmental sciences. NMR instrument outputs are however heterogeneous and require extensive post-processing, creating barriers to their use and application by non-specialists. Here, we report on a new open-source R package, nmrrr, that processes and visualizes spectral data obtained from one-dimensional solution-state and solid-state NMR experiments; the package also performs relevant calculations commonly applied in natural organic matter communities, such as computing the relative abundance of various functional groups. We document the package's installation, dependencies, and functions; and provide a standard workflow for processing NMR data. This package is currently available on CRAN and GitHub, and community contributions are welcome.

54 ENVIRONMENTAL SCIENCES↗

Holistic Approaches to HPC Power and Workflow Management

Constraints on power consumption are having pervasive effects on high-performance computing (HPC) systems, the facilities in which they are housed, and the application codes themselves. Whether these constraints are driven by physical limits on available power within a facility, are due to utility demand response, result from opportunities to reduce utility costs, or are imposed by sponsoring agencies, power management must now be added to the traditional goals of application and algorithm correctness, scalability, and performance. The specifics of these power constraints are likely to vary from site to site, and optimal strategies for managing power will vary. In this paper, we present the results of implementing several strategies for managing HPC system power and quantify the impact of these power reductions on application performance. These can be added to the suite of extant options that can be used to manage power and workflow on HPC systems.

97 MATHEMATICS AND COMPUTING↗

Portable, heterogeneous ensemble workflows at scale using libEnsemble

libEnsemble is a Python-based toolkit for running dynamic ensembles, developed as part of the DOE Exascale Computing Project. The toolkit utilizes a unique generator–simulator–allocator paradigm, where generators produce input for simulators, simulators evaluate those inputs, and allocators decide whether and when a simulator or generator should be called. The generator steers the ensemble based on simulation results. Generators may, for example, apply methods for numerical optimization, machine learning, or statistical calibration. libEnsemble communicates between a manager and workers. Flexibility is provided through multiple manager–worker communication substrates each of which has different benefits. These include Python’s multiprocessing, mpi4py, and TCP. Multisite ensembles are supported using Balsam or Globus Compute. We overview the unique characteristics of libEnsemble as well as current and potential interoperability with other packages in the workflow ecosystem. We highlight libEnsemble’s dynamic resource features: libEnsemble can detect system resources, such as available nodes, cores, and GPUs, and assign these in a portable way. These features allow users to specify the number of processors and GPUs required for each simulation; and resources will be automatically assigned on a wide range of systems, including Frontier, Aurora, and Perlmutter. Such ensembles can include multiple simulation types, some using GPUs and others using only CPUs, sharing nodes for maximum efficiency. We also describe the benefits of libEnsemble’s generator–simulator coupling, which easily exposes to the user the ability to cancel, and portably kill, running simulations based on models that are updated with intermediate simulation output. We demonstrate libEnsemble’s capabilities, scalability, and scientific impact via a Gaussian process surrogate training problem for the longitudinal density profile at the exit of a plasma accelerator stage. In conclusion, the study uses gpCAM for the surrogate model and employs either Wake-T or WarpX simulations, highlighting efficient use of resources that can easily extend to exascale.

Dynamic ensembles↗

Input data to electricity_entity_boundaries workflow

The EIA 861 electricity entity information dataset which acts as the input data to the electricity_entity_boundaries workflow (https://github.com/IMMM-SFA/electricity_entity_boundaries).

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Automated workflow for engineering services

This paper describes a prototype of an automated workflow system, developed by the Jet Propulsion Laboratory, that provides navigation and telecommunication services.

automated workflow web services database legacy ap↗

Large-Scale Compute-Intensive Analysis via a Combined In-situ and Co-scheduling Workflow Approach

Large-scale simulations can produce tens of terabytes of data per analysis cycle, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in situ and co-scheduling approaches for handling Petabyte-size outputs. An initial in situ step is used to reduce the amount of data to be analyzed, and to separate out the data-intensive tasks handled off-line. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.

Big data↗

Holistic Approaches to HPC Power & Workflow Management: Preprint

Constraints on power consumption are having pervasive effects on high-performance computing (HPC) systems, the facilities in which they are housed, and the application codes themselves. Whether these constraints are driven by physical limits on available power within a facility, are due to utility demand response, result from opportunities to reduce utility costs, or are imposed by sponsoring agencies, power management must now be added to the traditional goals of application and algorithm correctness, scalability, and performance. The specifics of these power constraints are likely to vary from site to site, and optimal strategies for managing power will vary. In this paper, we present the results of implementing several strategies for managing HPC system power and quantify the impact of these power reductions on application performance. These can be added to the suite of extant options that can be used to manage power and workflow on HPC systems.

97 MATHEMATICS AND COMPUTING↗

A Processing and Analytics System for Microscopy Data Workflows: The Pycroscopy Ecosystem of Packages

Major advancements in fields as diverse as biology and quantum computing have relied on a multitude of microscopy techniques. Despite the considerable proliferation of these instruments, significant bottlenecks remain in terms of processing, analysis, storage, and retrieval of the acquired datasets. Aside from lack of file standards, individual domain-specific analysis packages are often disjoint from the underlying datasets, and thus keeping track of analysis and processing steps remains tedious for the end-user, hampering reproducibility. Here, in this study, the pycroscopy ecosystem of packages is introduced, an open-source python-based ecosystem underpinned by a common data model. The data model, termed the N-dimensional spectral imaging data format, is realized in pycroscopy's sidpy package. This package is built on top of dask arrays, thus leveraging dask array attributes, but expanding them to accelerate microscopy relevant analysis and visualization. Several examples of the use of the pycroscopy ecosystem to create workflows for data ingestion and analysis of scanning transmission electron microscopy (STEM) and scanning probe microscopy data are shown. Adoption of such standardized routines will be critical to usher in the next generation of autonomous instruments where processing, computation, and meta-data storage will be critical to overall experimental operations.

97 MATHEMATICS AND COMPUTING↗