SEARCH · Search NASA

Results for “computational workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41

All sources NASA NTRS DOE OSTI

DOE OSTI2022

Regularizing Invertible Neural Networks for Airfoil Design Through Dimension Reduction

This work examines how dimension reduction can improve the performance of invertible neural networks (INN) for airfoil design. Design workflows are typically expensive, relying on many evaluations of high fidelity computational fluid dynamics (CFD) models. Furthermore, the inverse design problem is typically ill-posed. That is, multiple valid solutions exist that satisfy the design criteria. Regularization can reduce this inverse design space and simplify the problem. We study the use of subspace-based input dimension reduction to act as a regularizer for the INN model and improve the recovery of new airfoil shapes with desired performance characteristics. We find that the dimension reduction identifies two dominant modes, relating to airfoil thickness and camber, that optimally determine the airfoil's aerodynamics. We demonstrate the capability of the proposed INN model to generate 100 airfoils that satisfy the specific aerodynamic and structural characteristics.

airfoil design↗

NASA NTRS2019

Evolving HPC and Application Design Toward a Coupled Data Assimilation System at NASA Suitable for Emerging Exascale Platforms

The prediction capabilities of global models have continuously evolved from the traditional medium-range global weather prediction application to span scales in support of hourly prediction of convective scale storms to seasonal Earth system prediction. This evolution has increased the demands on the system infrastructure design and workflow to achieve the required performance on modern high-performance computing (HPC) platforms. The planned evolution of the Goddard Earth Observing System (GEOS) modeling and assimilation system will stress the capabilities of conventional HPC overwhelming the available compute cycles at the NASA Center for Climate Simulation (NCCS) at the NASA Goddard Space Flight Center in the coming 5-10 years. This has led to the re-design of key elements of the assimilation and modeling systems to achieve significant gains in performance on anticipated Exacale platforms. The transition of the assimilation system to the Joint Effort for Data assimilation Integration (JEDI) framework has positioned GEOS to exploit new efficient algorithms for data assimilation (DA) in a fully-coupled Earth system context. The suitability of the GEOS model to leverage a domain specific language (DSL) approach and artificial intelligence (AI) is being explored to accelerate computational performance and data exchange efficiency of the coupled Earth system model. The storage and processing of large data volumes produced by these advance systems is being redesigned with a data-centric cloud-based approach. We will highlight the recent efforts in these areas and emphasize the demand for further development and re-design to achieve the science objectives in support of NASA's Earth system modeling and assimilation missions.

Putman, Bill↗

DOE OSTI2021

Scoreboard

Emerging HPC machines have given rise to enhanced compute power that far outstrips the machine's ability to save large scale results for post-processing. To combat this, in situ data analysis techniques are slowly being adopted. With in situ data management favoring workflows composed of multiple simulations and analyses connected in transit on heterogeneous machines, scientists and engineers need a tool that enables them to create data extracts, visualizations, and interactively monitor and steer their simulations. Scoreboard Phase II is a next generation analysis software that supports composite in transit workflows on heterogeneous architectures and restores interactivity to in situ data analysis through simulation monitoring and computational steering. Scoreboard provides a simulation dashboard with graphs of metrics over time, controls for setting custom simulation steering parameters, controls for managing the set of data extracts being produced in the simulation, as well as the ability to explore data extracts, all from a web browser. Realizing the vision outlined in this project required research into making a system that integrates end to end from simulations all the way to the user. In situ tools generally suffer from complexity and excessive software dependencies. Scoreboard, by contrast, is easy to build and integrate into simulation codes and it provides first class FORTRAN support. The Scoreboard library is capable of in situ and in transit data analysis that can produce data extracts commonly needed for Computational Fluid Dynamics (CFD) analysis. Simulations can transparently stage data in transit to a Scoreboard Endpoint program, which can accept their data and produce the requested data extracts. This lets simulations return to their work while the Endpoint works on the analysis. Efficiently staging the data at scale was a topic of this research. Scoreboard provides the means to let the user manage data extracts and monitor/steer many simulations from a web browser. This area of the research focused on discovery of in transit network components to expose and control their steering parameters within an interactive browser-based user interface that includes: system topology, gathered metrics, notifications, dynamically-generated steering controls, and exploration of visualization data products.

Whitlock, BradJoseph [Intelligent Light] (00000001↗

DOE OSTI2018

Xi-cam : a versatile interface for data visualization and analysis

Xi-cam is an extensible platform for data management, analysis and visualization. Xi-cam aims to provide a flexible and extensible approach to synchrotron data treatment as a solution to rising demands for high-volume/high-throughput processing pipelines. The core of Xi-cam is an extensible plugin-based graphical user interface platform which provides users with an interactive interface to processing algorithms. Plugins are available for SAXS/WAXS/GISAXS/GIWAXS, tomography and NEXAFS data. With Xi-cam 's `advanced' mode, data processing steps are designed as a graph-based workflow, which can be executed live, locally or remotely. Remote execution utilizes high-performance computing or de-localized resources, allowing for the effective reduction of high-throughput data. Xi-cam 's plugin-based architecture targets cross-facility and cross-technique collaborative development, in support of multi-modal analysis. Xi-cam is open-source and cross-platform, and available for download on GitHub.

Pandolfi, Ronald J. (ORCID:0000000308248548)↗

DOE OSTI2025

Creating Apptainer Workflows with Docker-Compose-like Utilities

Creating Apptainer Workflows with Docker-Compose-like Utilities In this presentation, I will explore the utilization of a tool called process-compose, inspired by docker-compose, to create Apptainer-based services. This approach allows for easy deployment and management of fully containerized applications on High Performance Computing (HPC) systems without requiring elevated privileges. Benefits to the Ecosystem: By incorporating process-compose and Apptainer, I aim to address several key challenges in the HPC ecosystem: Simplified Workflow Management: Process-compose provides a user-friendly interface for defining and managing complex containerized application services, reducing the setup time and lowering the barrier to entry for new users. Enhanced Portability: Apptainer ensures that containerized applications can run consistently across different HPC environments, promoting greater portability and reducing compatibility issues. Process-compose is also a single binary that does not need to be installed by admin level users. Community Driven Solutions: This approach aligns with the goals of the High Performance Software Foundation (HPSF) to advance community-driven solutions. By sharing our experiences and insights, I hope to foster collaboration and innovation within the HPC community. Increased Productivity: The combination of process-compose and Apptainer streamlines the serve deployment process, allowing researchers and developers to focus more on their scientific work rather than the intricacies of system or service administration. Through this presentation, attendees will gain valuable insights into the practical implementation of containerized workflows on HPC systems, learn about the benefits of using process-compose and Apptainer, and understand how these tools can contribute to a more efficient HPC ecosystem.

97 - MATHEMATICS AND COMPUTING↗

DOE OSTI2019

“Teamwork Makes the Dream Work”: Tribal Competition Evolutionary Search as a Surrogate for Free-Energy-Based Structural Predictions

Crystal structure prediction has been a grand challenge in material science owing to the large configurational space that one must explore. Evolutionary (genetic) algorithms coupled with first principles calculations are commonly used in crystal structure prediction to sample the ground and metastable states of materials based on configurational energies. However, crystal structure predictions at finite temperature (T), pressure (P), and composition (X) require a free-energy-based search that is often computationally expensive and tedious. Here in this paper, we introduce a new machine-learning workflow for structure prediction that is based on a concept inspired by the evolution of human tribes in primitive society. Our tribal genetic algorithm (GA) combines configurational sampling with evolutionary optimization to accurately predict entropically stabilized phases at finite (T, P, X), at a computational cost that is an order of magnitude smaller than that required for a free-energy-based search. In a departure from standard GA techniques, the populations of individuals are divided into multiple tribes based on a bond-order fingerprint, and genetic operations are modified to ensure that cluster configurations are sampled adequately to capture entropic contributions. Team competition introduced into the evolutionary process allows winning teams (representing a better set of individuals) to expand their sizes; this translates into a more expanded search of the phase space allowing us to explore solutions near possible global minimum. Each team explores a specific section of the structural phase space and avoids bias on solutions arising from the use of individual populations in a purely energy-based search. We demonstrate the efficacy of our approach by performing the structural prediction of a representative two-dimensional two-body system as well as Lennard-Jones clusters over a range of temperatures up to its melting point. Our approach outperforms the standard GA approaches and enables structural search under “real nonambient conditions” on both bulk systems and finite-sized clusters.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Characterizing Quantum Classifier Utility in Natural Language Processing Workflows

Quantum Natural Language Processing (QNLP) develops natural language processing (NLP) models for deployment on quantum computers. We explore feature and data prototype selection techniques to address challenges posed by encoding high dimensional features. Our study builds quantum circuit classifiers that includes classical feature pre-processing, quantum embedding and quantum model training. The quantum models are built on 4 or 6 qubits and the quantum neural network (QNN) uses the established bricklayer design. We compare the dependence of model performance (in terms of accuracy and F1 scores) on feature length, embedding gates and parameterized unitary design. We compare the performance of quantum machine learning models to classical convolution neural network model (CNN) on binary and multi-class classification tasks using two datasets of synthetic features and labels. The first is the ECP-CANDLE P3B3 dataset a corpus of synthetically generated cancer pathology reports. The second dataset is extracted from well-known benchmark dataset (MADELON) - features are generated with a combination of informative, repeated and uninformative features. Both datasets are used for binary classification and multi-class classification with 3 classes. We observe robust, accurate performance from all models on the binary classification tasks, but multiclass classification is a challenge for the quantum models-there is a notable decrease in accuracy when using 3 classes. Overall the performance is comparable in terms of recall and accuracy between QNNs and CNNs, even with large datasets. These results provide a point of comparison between quantum and classical models on real-world datasets.

Hamilton, Kathleen↗

DOE OSTI2017

Process Integration and Optimization of ICME Carbon Fiber Composites for Vehicle Lightweighting: A Preliminary Development

Process integration and optimization is the key enabler of the Integrated Computational Materials Engineering (ICME) of carbon fiber composites. In this paper, automated workflows are developed for two types of composites: Sheet Molding Compounds (SMC) short fiber composites, and multi-layer unidirectional (UD) composites. For SMC, the proposed workflow integrates material processing simulation, microstructure representation volume element (RVE) models, material property prediction and structure preformation simulation to enable multiscale, multidisciplinary analysis and design. Processing parameters, microstructure parameters and vehicle subframe geometry parameters are defined as the design variables; the stiffness and weight of the structure are defined as the responses. For multi-layer UD structure, this work focuses on the discussion of different design representation methods and their impacts on the optimization performance. Challenges in ICME process integration and optimization are also summarized and highlighted. Two case studies are conducted to demonstrate the integrated process and its application in optimization.

36 MATERIALS SCIENCE↗

NASA NTRS2015

The Open Data Repositorys Data Publisher

Data management and data publication are becoming increasingly important components of researcher's workflows. The complexity of managing data, publishing data online, and archiving data has not decreased significantly even as computing access and power has greatly increased. The Open Data Repository's Data Publisher software strives to make data archiving, management, and publication a standard part of a researcher's workflow using simple, web-based tools and commodity server hardware. The publication engine allows for uploading, searching, and display of data with graphing capabilities and downloadable files. Access is controlled through a robust permissions system that can control publication at the field level and can be granted to the general public or protected so that only registered users at various permission levels receive access. Data Publisher also allows researchers to subscribe to meta-data standards through a plugin system, embargo data publication at their discretion, and collaborate with other researchers through various levels of data sharing. As the software matures, semantic data standards will be implemented to facilitate machine reading of data and each database will provide a REST application programming interface for programmatic access. Additionally, a citation system will allow snapshots of any data set to be archived and cited for publication while the data itself can remain living and continuously evolve beyond the snapshot date. The software runs on a traditional LAMP (Linux, Apache, MySQL, PHP) server and is available on GitHub (http://github.com/opendatarepository) under a GPLv2 open source license. The goal of the Open Data Repository is to lower the cost and training barrier to entry so that any researcher can easily publish their data and ensure it is archived for posterity.

Astrobiology data↗

NASA NTRSDate not supplied

Automation of the ICME Workflow Incorporating Material Digital Twins at Different Length Scales Within a Robust Information Management System

Recent successes in Integrated Computational Materials Engineering (ICME) have demonstrated the potential in designing fit-for-purpose materials for a given application in a cost and time efficient manner. However, the material design process must contain a level of automation in the material decision process, implementing some optimization algorithms, to truly enable the full benefits of ICME, particularly when considering materials at multiple length/time scales. In this work, we will demonstrate how the GRC ICME schema and Python framework automates a workflow that captures, analyzes, maintains, and disseminates the digital footprint in the context of tailoring resin material at the nanoscale of a woven composite Y-joint at the macroscale for an Aurora D8 double bubble fuselage. This digital footprint incorporates the interaction of both structural digital twins and material twins at various length scales.

Brandon L. Hearley↗

DOE OSTI2025

Parallelizing autotuning for HPC applications: Unveiling the potential of the speculation strategy in Bayesian optimization

In the exascale computing era, tuning High-Performance Computing (HPC) applications has become a significant computational challenge. Although Bayesian optimization (BO) has emerged as a promising tool for HPC performance tuning, the BO workflow is inherently sequential (i.e., one function evaluation at a time) and cannot leverage the huge amount of parallel resources present in modern supercomputers, resulting in a considerable underutilization of their computational capabilities. This paper explores the trade-off between search quality and parallelism in BO, investigating a diverse set of methods. Building upon both previous approaches from the literature and novel methodologies introduced in this work, our study provides a deep analysis to accelerate BO performance tuning. By examining a set of synthetic functions and practical HPC applications, our exploration analyzes the interaction among various BO methods for parallelization, the quantity of parallel resources, the runtime distribution of target HPC applications, and the costs associated with different search orchestration mechanisms that have been overlooked in previous studies. Compared to sequential BO, our novel methodology achieves comparable quality while demonstrating robust scalability in search time as the amount of parallel resources increases; it also outperforms a state-of-the-art tuner, which supports parallelization, achieving up to 3.67x faster search time. We provide high-value insights for practitioners seeking to leverage the power of parallel computing for efficient HPC application tuning. Additionally, to further assist researchers in accelerating the performance tuning of their HPC applications, we provide an extension of an existing open-source tuning framework that incorporates our methods.

Bayesian optimization↗

DOE OSTI2020

Computational Optimization of A 3D Printed Collimator

This contribution describes the computational methodology behind an optimization procedure for a scattered beam collimator. The workflow includes producing a file that can be manufactured via additive methods. A conical collimator, optimized for neutron diffraction experiments in a high pressure clamp cell, is presented as an example. In such a case the scattering from the sample is much smaller than that of the pressure cell. Monte Carlo Ray tracing in MCViNE was used to model scattering from a Si powder sample and the cell. A collimator was inserted into the simulation and the number and size of channels were optimized to maximize the rejection of the parasitic signal coming from the complex sample environment. Constraints, provided by the additive manufacturing process as well as a specific neutron diffractometer, were also included in the optimization. The source code and the tutorials are available in c3dp (Islam (2019)).

74 ATOMIC AND MOLECULAR PHYSICS↗

DOE OSTI2019

A Co-Design Study Of Fusion Whole Device Modeling Using Code Coupling

Complex workflows consisting of multiple simulation and analysis codes running concurrently through in-memory coupling is becoming popular due to inherent advantages in online management of large-scale data, resilience, and the code development process. However, orchestrating such a multi-application workflow to efficiently utilize resources on a heterogeneous architecture is challenging. In this paper, we present our results with running the Fusion Whole Device Modeling benchmark workflow on Summit, a pre-exascale supercomputer at Oak Ridge National Laboratory. We explore various resource distribution and process placement mechanisms, including sharing compute nodes between processes from separate applications. We show that fine-grained process placement can have a significant impact towards efficient utilization of the compute power of a node on Summit, and conclude that sophisticated tools for performing co-design studies of multi-application workflows can play an important role towards efficient orchestration of such workflows.

Choi, Jong Youl↗

DOE OSTI2023

Scaling Automatic Vector Data Alignment to Satellite Imagery

Given the tremendous volume of accessible Earth Observation (EO) data, there is a need to develop scalable Geospatial Artificial Intelligence (GeoAI) solutions for time-sensitive applications. Scalability in this context refers to rapidly processing large-scale EO data using high performance computing resources. Accurate mapping of the built environment from remote sensing (RS) imagery has been one of the crucial components in GeoAI workflows for a wide spectrum of humanitarian applications. Derived vector data of built environment is often leveraged for disaster preparedness and response activities. However, factors such as differences in ortho-rectification, atmospheric conditions and human error, results in spatial misalignment between vector data and the timely available RS imagery. Model training for downstream tasks such as object detection, change analysis, etc., is negatively impacted due to such spatial misalignment. Although there has been progress towards automatic alignment of vector data, the lack of scalability remains an open research challenge. This paper proposes to leverage parallel computing to optimize an automatic vector data alignment workflow. It further employs CPU-level multi-core parallelism for improving the performance of the workflow for scalable built environment mapping. We report observations and discuss findings from the preliminary experiments performed on the Summit Supercomputer.

Potnis, Abhishek↗

DOE OSTI2012

Monitoring techniques and alarm procedures for CMS services and sites in WLCG

The CMS offline computing system is composed of roughly 80 sites (including most experienced T3s) and a number of central services to distribute, process and analyze data worldwide. A high level of stability and reliability is required from the underlying infrastructure and services, partially covered by local or automated monitoring and alarming systems such as Lemon and SLS, the former collects metrics from sensors installed on computing nodes and triggers alarms when values are out of range, the latter measures the quality of service and warns managers when service is affected. CMS has established computing shift procedures with personnel operating worldwide from remote Computing Centers, under the supervision of the Computing Run Coordinator at CERN. This dedicated 24/7 computing shift personnel is contributing to detect and react timely on any unexpected error and hence ensure that CMS workflows are carried out efficiently and in a sustained manner. Synergy among all the involved actors is exploited to ensure the 24/7 monitoring, alarming and troubleshooting of the CMS computing sites and services. We review the deployment of the monitoring and alarming procedures, and report on the experience gained throughout the first two years of LHC operation. We describe the efficiency of the communication tools employed, the coherent monitoring framework, the proactive alarming systems and the proficient troubleshooting procedures that helped the CMS Computing facilities and infrastructure to operate at high reliability levels.

Molina-Perez, J.↗

DOE OSTI2019

Understanding data motion in the modern HPC data center

The utilization and performance of storage, compute, and network resources within HPC data centers have been studied extensively, but much less work has gone toward characterizing how these resources are used in conjunction to solve larger scientific challenges. To address this gap, we present our work in characterizing workloads and workflows at a data-center-wide level by examining all data transfers that occurred between storage, compute, and the external network at the National Energy Research Scientific Computing Center over a three-month period in 2019. Using a simple abstract representation of data transfers, we analyze over 100 million transfer logs from Darshan, HPSS user interfaces, and Globus to quantify the load on data paths between compute, storage, and the wide-area network based on transfer direction, user, transfer tool, source, destination, and time. We show that parallel I/O from user jobs, while undeniably important, is only one of several major I/O workloads that occurs throughout the execution of scientific workflows. We also show that this approach can be used to connect anomalous data traffic to specific users and file access patterns, and we construct time-resolved user transfer traces to demonstrate that one can systematically identify coupled data motion for individual workflows.

Lockwood, GK↗

DOE OSTI2020

Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

Godoy, William↗

DOE OSTI2021

DART-PFLOTRAN: An ensemble-based data assimilation system for estimating subsurface flow and transport model parameters

Ensemble-based Data Assimilation (EDA), based on the Monte Carlo approach, has been effectively applied to estimate model parameters through inverse modeling in subsurface flow and transport problems. However, implementation of EDA approach involves a complicated workflow that include setting up and executing ensemble forward model simulations, processing observations and model simulation results for parameter updates, and repeat for sequential or iterative EDA. To facilitate the management of such workflow and lower the barriers for adopting EDA-based parameter estimation in subsurface science, we develop a generic software frame-work linking the Data Assimilation Research Testbed (DART) with a massively parallel subsurface FLOw and TRANsport code PFLOTRAN. The new DART-PFLOTRAN leverages both the core data assimilation engines in DART and the computational power afforded by PFLOTRAN. In addition to the standard smoother and filtering options, DART-PFLOTRAN enables an iterative EDA workflow based on the Ensemble Smoother for Multiple Data Assimilation method (ES-MDA) to improve estimation accuracy for nonlinear forward problems. Here, we verify the implementation of ES-MDA in DART-PFLOTRAN using two synthetic cases designed to estimate static permeability and dynamic exchange fluxes across the riverbed, respectively, from continuous temperature measurements made across a depth profile. One-dimensional hydro-thermal simulations are performed in both cases to relate temperature responses with the parameters of interest. In the case of estimating dynamic parameters, we demonstrate the flexibility of DART-PFLOTRAN in automating sequential ES-MDA workflow, which will significantly reduce the time researchers spend on managing complex workflows in similar applications. Both studies yield accurate estimations of the parameters compared to their synthetic truth, while ES-MDA leads to more accurate estimation when a high level of nonlinearity exist between observed responses and unknown parameters. With a code base in Python and Fortran, DART-PFLOTRAN paves the way for applications in large-scale subsurface inverse modeling by automating the complex workflow of sequential ES-MDA that can be executed on various computing platforms.

97 MATHEMATICS AND COMPUTING↗