Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,189 records · Page 66
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

BAWLD-CH 4 : a comprehensive dataset of methane fluxes from boreal and arctic ecosystems

Methane (CH 4 ) emissions from the boreal and arctic region are globally significant and highly sensitive to climate change. There is currently a wide range in estimates of high-latitude annual CH 4 fluxes, where estimates based on land cover inventories and empirical CH 4 flux data or process models (bottom-up approaches) generally are greater than atmospheric inversions (top-down approaches). A limitation of bottom-up approaches has been the lack of harmonization between inventories of site-level CH 4 flux data and the land cover classes present in high-latitude spatial datasets. Here we present a comprehensive dataset of small-scale, surface CH 4 flux data from 540 terrestrial sites (wetland and non-wetland) and 1247 aquatic sites (lakes and ponds), compiled from 189 studies. The Boreal–Arctic Wetland and Lake Methane Dataset (BAWLD-CH 4 ) was constructed in parallel with a compatible land cover dataset, sharing the same land cover classes to enable refined bottom-up assessments. BAWLD-CH 4 includes information on site-level CH 4 fluxes but also on study design (measurement method, timing, and frequency) and site characteristics (vegetation, climate, hydrology, soil, and sediment types, permafrost conditions, lake size and depth, and our determination of land cover class). The different land cover classes had distinct CH 4 fluxes, resulting from definitions that were either based on or co-varied with key environmental controls. Fluxes of CH 4 from terrestrial ecosystems were primarily influenced by water table position, soil temperature, and vegetation composition, while CH 4 fluxes from aquatic ecosystems were primarily influenced by water temperature, lake size, and lake genesis. Models could explain more of the between-site variability in CH 4 fluxes for terrestrial than aquatic ecosystems, likely due to both less precise assessments of lake CH 4 fluxes and fewer consistently reported lake site characteristics. Analysis of BAWLD-CH 4 identified both land cover classes and regions within the boreal and arctic domain, where future studies should be focused, alongside methodological approaches. Overall, BAWLD-CH 4 provides a comprehensive dataset of CH 4 emissions from high-latitude ecosystems that are useful for identifying research opportunities, for comparison against new field data, and model parameterization or validation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

ISAVS: Interactive Scalable Analysis and Visualization System

Modern science is inundated with ever increasing data sizes as computational capabilities and image acquisition techniques continue to improve. For example, simulations are tackling ever larger domains with higher fidelity, and high-throughput microscopy techniques generate larger data that are fundamental to gather biologically and medically relevant insights. As the image sizes exceed memory, and even sometimes local disk space, each step in a scientific workflow is impacted. Current software solutions enable data exploration with limited interactivity for visualization and analytic tasks. Furthermore analysis on HPC systems often require complex hand-written parallel implementations of algorithms that suffer from poor portability and maintainability. We present a software infrastructure that simplifies end-to-end visualization and analysis of massive data. First, a hierarchical streaming data access layer enables interactive exploration of remote data, with fast data fetching to test analytics on subsets of the data. Second, a library simplifies the process of developing new analytics algorithms, allowing users to rapidly prototype new approaches and deploy them in an HPC setting. Third, a scalable runtime system automates mapping analysis algorithms to whatever computational hardware is available, reducing the complexity of developing scaling algorithms. Here, we demonstrate the usability and performance of our system using a use case from neuroscience: filtering, registration, and visualization of tera-scale microscopy data. We evaluate the performance of our system using a leadership-class supercomputer, Shaheen II.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Computational Algorithms for Unit Commitment with AC Power Flows (Final Report)

Security-constrained unit commitment (SCUC) is a key component in power system operations. When AC power flow constraints are considered in the SCUC model (AC-SCUC), the problem becomes extremely difficult due to its discrete and non-convex nature, as described in “Grid Optimization Competition Challenge 3 Problem Formulation (GOCC)”. There are four main challenges: (i) Discrete decisions regarding unit online/offline status and start-up/shut-down procedures for every single unit. The number of discrete decision variables increases considerably when a system integrates multiple generators; (ii) Configuration-based combined-cycle formulations, and multi-commodity models that include ramping products, spin/non-spin products, and regulation up/down products. The combined-cycle units introduce additional discrete decision variables and auxiliary service products further complicate the model by connecting multi-commodity products’ continuous and discrete variables; (iii) SCUC models with AC power flow constraints are far more complex due to massive bilinear terms in the large-scale nonlinear power balance equations. The nonlinear power balance equations are further complicated by the discrete step control variables of shunts; (iv) N − 1 contingency analysis. The size of the model increases linearly with the number of contingencies considered, greatly increasing the size of the optimization model. Accordingly, there is an emergent need to develop a robust algorithm capable of deriving a high-quality solution in a short time and passing through contingency tests simultaneously. In this project, we explore innovative techniques to address this challenging problem by integrating advanced polyhedral theory, approximation methods, relaxation strategies, decomposition techniques, and parallel computing. Each technique approaches the problem from a different perspective, leveraging its specific strengths to tackle distinct challenges. Each individual method has demonstrated its effectiveness in the PI’s previous research. Their integration is expected to significantly reduce the computational time required to solve the proposed complex problem. Successful completion of this project has the potential to transform the industry by enhancing optimization solvers capable of handling large-scale day-ahead energy market clearing models within strict time constraints, while incorporating AC power flow constraints. This advancement will lead to reduced overall generation costs and, consequently, increased social welfare.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2017

Parallel Fokker–Planck-DSMC algorithm for rarefied gas flow simulation in complex domains at all Knudsen numbers

A major challenge for the conventional Direct Simulation Monte Carlo (DSMC) technique lies in the fact that its computational cost becomes prohibitive in the near continuum regime, where the Knudsen number (Kn)—characterizing the degree of rarefaction—becomes small. In contrast, the Fokker–Planck (FP) based particle Monte Carlo scheme allows for computationally efficient simulations of rarefied gas flows in the low and intermediate Kn regime. The Fokker–Planck collision operator—instead of performing binary collisions employed by the DSMC method—integrates continuous stochastic processes for the phase space evolution in time. This allows for time step and grid cell sizes larger than the respective collisional scales required by DSMC. Dynamically switching between the FP and the DSMC collision operators in each computational cell is the basis of the combined FP-DSMC method, which has been proven successful in simulating flows covering the whole Kn range. Until recently, this algorithm had only been applied to two-dimensional test cases. In this contribution, we present the first general purpose implementation of the combined FP-DSMC method. Utilizing both shared- and distributed-memory parallelization, this implementation provides the capability for simulations involving many particles and complex geometries by exploiting state of the art computer cluster technologies.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2017

What does fault tolerant Deep Learning need from MPI?

Deep Learning (DL) algorithms have become the {\em de facto} Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive -- even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults -- requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: {\em What is needed from MPI for designing fault tolerant DL implementations?} In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by extending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet neural network topology demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.

Amatya, Vinay C.↗
DOE OSTI2020

How fast can one resize a distributed file system?

Efficient resource utilization becomes a major concern as large-scale distributed computing infrastructures keep growing in size. Malleability, the possibility for resource managers to dynamically increase or decrease the amount of resources allocated to a job, is a promising way to save energy and costs. However, state-of-the-art parallel and distributed storage systems have not been designed with malleability in mind. The reason is mainly the supposedly high cost of data transfers required by resizing operations. Nevertheless, as network and storage technologies evolve, old assumptions about potential bottlenecks can be revisited. In this study, we evaluate the viability of malleability as a design principle for a distributed storage system. We specifically model the minimal duration of the commission and decommission operations. To show how our models can be used in practice, we evaluate the performance of these operations in HDFS, a relevant state-of-the-art distributed file system. We show that the existing decommission mechanism of HDFS is good when the network is the bottleneck, but can be accelerated by up to a factor 3 when storage is the limiting factor. We also show that the commission in HDFS can be substantially accelerated. With the highlights provided by our model, we suggest improvements to speed both operations in HDFS. We discuss how the proposed models can be generalized for distributed file systems with different assumptions and what perspectives are open for the design of efficient malleable distributed file systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

End-to-end I/O portfolio for the summit supercomputing ecosystem

The I/O subsystem for the Summit supercomputer, No. 1 on the Top500 list, and its ecosystem of analysis platforms is composed of two distinct layers, namely the in-system layer and the center-wide parallel file system layer (PFS), Spider 3. The in-system layer uses node-local SSDs and provides 26.7 TB/s for reads, 9.7 TB/s for writes, and 4.6 billion IOPS to Summit. The Spider 3 PFS layer uses IBM's Spectrum Scale™ and provides 2.5 TB/s and 2.6 million IOPS to Summit and other systems. While deploying them as two distinct layers was operationally efficient, it also presented usability challenges in terms of multiple mount points and lack of transparency in data movement. To address these challenges, we have developed novel end-to-end I/O solutions for the concerted use of the two storage layers. We present the I/O subsystem architecture, the end-to-end I/O solution space, their design considerations and our deployment experience.

Oral, Sarp↗
DOE OSTI2020

ddcMD.os

ddcMD is a general purpose molecular dynamics (MD) code that supports MPI parallelism. MD codes are used for simulation of particles systems and capture all the many-body effects of the underlying particle potential that defines the physical system. Though MD can be used to model systems from the subatomic to astrological length scales ddcMD is mainly focused on the atomic scale length scale, In this release of ddcMD the support will be mainly for systems using the coarse-grain Martini potential, a particle potential for biological systems.

Glosli, JamesN↗
DOE OSTI2019

Asynchronous Navier-Stokes Solver on 3D Unstructured Grids for the Exascale Era

This project has developed multiple fluid dynamics solvers for complex 3D flows using fully asynchronous distributed-memory task-parallel algorithms on top of the Charm++ runtime system. The algorithms solve the Euler or Navier-Stokes equations of compressible flows using unstructured tetrahedron meshes with optional solution-adaptive mesh-, and polynomial-order refinement. We have demonstrated excellent strong scaling up 50K compute cores and the benefits of Charm++’s automatic load balancing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

ECP ST Capability Assessment Report (CAR) for VTK-m (FY20)

The ECP/VTK-m project is providing the core capabilities to perform scientific visualization on Exascale architectures. The ECP/VTK-m project fills the critical feature gap of performing visualization and analysis on processors like graphics-based processors. The results of this project will be delivered in tools like ParaView, Vislt, and Ascent as well as in stand-alone form. Moreover, these projects are depending on this ECP effort to be able to make effective use of ECP architectures. One of the biggest recent changes in high-performance computing is the increasing use of accelerators. Accelerators contain processing cores that independently are inferior to a core in a typical CPU, but these cores are replicated and grouped such that their aggregate execution provides a very high computation rate at a much lower power. Current and future CPU processors also require much more explicit parallelism. Each successive version of the hardware packs more cores into each processor, and technologies like hyper threading and vector operations require even more parallel processing to leverage each core's full potential. VTK-m is a toolkit of scientific visualization algorithms for emerging processor architectures. VTK-m supports the fine-grained concurrency for data analysis and visualization algorithms required to drive extreme scale computing by providing abstract models for data and execution that can be applied to a variety of algorithms across many different processor architectures. The ECP/VTK-m project is building up the VTK-m codebase with the necessary visualization algorithm implementations that run across the varied hardware platforms to be leveraged at the Exascale. We will be working with other ECP projects, such as ALPINE, to integrate the new VTK-m code into production software to enable visualization on our HPC systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Developing the Science Basis for Understanding Polymer Encapsulant Degradation Mechanisms: DuraMAT 2.0 Final Project Report

Polymeric encapsulants are essential materials in photovoltaic modules, protecting sensitive electronics from the environment while providing mechanical integrity to the multilayered assembly. However, these polymeric materials are susceptible to degradation processes driven by the ingress of environmental species, ultraviolet radiation, thermal stresses, and mechanical loading. In this study, we employ a combined atomistic simulation and accelerated aging experimental approach to study the molecular-scale mechanisms of encapsulant degradation. Classical molecular dynamics simulations quantify the diffusion of environmental and degradation species through the polymer matrix, producing composition-specific diffusion coefficients. Reactive simulations characterize activation energy barriers and reaction rate constants for key chemical pathways. In parallel, thermal-desorption analyses coupled with mass spectrometry monitor the emergence and concentration profiles of degradation products under controlled stressor conditions. By integrating simulation and experiment, we establish quantitative correlations between polymer composition, species diffusivity, and chemical reactivity. We anticipate that these relations and quantitative values could serve as high-fidelity inputs to reaction-diffusion models, enabling physics-informed lifetime predictions and guiding the design of more durable encapsulant materials for solar energy applications.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Dayflow: CONUS Daily Streamflow Reanalysis, Version 1 (V1)

Dayflow V1 is a historical streamflow reanalysis dataset reconstructed for a 36-year period (1980-2015). The dataset provides both daily and monthly scale streamflow information at about 2.7 million NHDPlusV2 stream reaches in the conterminous US (CONUS). Dayflow is the result of a nationally scalable modeling framework that integrates the simulated runoff from the Variable Infiltration Capacity (VIC) model with the Routing Application for Parallel computatIon of Discharge (RAPID) routing model. Two types of streamflow products, simulated streamflow with or without assimilation of historic US Geological Survey (USGS) streamflow observations, are provided in Dayflow V1. A comprehensive evaluation at 7,526 USGS National Water Information System (NWIS) gauges is performed for both types of streamflow products. The resulting key evaluation metrics are also included in the Dayflow V1 Dataset.

13 HYDRO ENERGY↗
DOE OSTI2017

Logamediate Inflation in f ( T ) Teleparallel Gravity

We study logamediate inflation in the context of f ( T ) teleparallel gravity. f ( T )-gravity is a generalization of the teleparallel gravity which is formulated on the Weitzenbock spacetime, characterized by the vanishing curvature tensor (absolute parallelism) and the non-vanishing torsion tensor. We consider an f ( T )-gravity model which is sourced by a canonical scalar field. Assuming a power-law f ( T ) function in the action, we investigate an inflationary universe with a logamediate scale factor. Our results show that, although logamediate inflation is completely ruled out by observational data in the standard inflationary scenario based on Einstein gravity, it can be compatible with the 68% confidence limit joint region of Planck 2015 TT,TE,EE+lowP data in the framework of f ( T )-gravity.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Towards multiscale modeling of ocean surface turbulent mixing using coupled MPAS-Ocean v6.3 and PALM v5.0

A multiscale modeling approach for studying the ocean surface turbulent mixing is explored by coupling an ocean general circulation model (GCM) MPAS-Ocean with the Parallelized Large Eddy Simulation Model (PALM). The coupling approach is similar to the superparameterization approach that has been used to represent the effects of deep convection in atmospheric GCMs. However, the focus of this multiscale modeling approach is on the small-scale turbulent mixing and their interactions with the larger-scale processes in the ocean, so that a more flexible coupling strategy is used. To reduce the computational cost, a customized version of PALM is ported on the general-purpose graphics processing unit (GPU) with OpenACC, achieving 10–16 times overall speedup as compared to running on a single CPU. Even with the GPU-acceleration technique, a superparameterization-like approach to represent the ocean surface turbulent mixing in GCMs using embedded high fidelity and three-dimensional large eddy simulations (LESs) over the global ocean is still computationally intensive and infeasible for long simulations. However, running PALM regionally on selected MPAS-Ocean grid cells is shown to be a promising approach moving forward. The flexible coupling between MPAS-Ocean and PALM allows further exploration of the interactions between the ocean surface turbulent mixing and larger-scale processes, as well as future development and improvement of ocean surface turbulent mixing parameterizations for GCMs.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗
DOE OSTI2026

Multi-task Parallelism for Robust Pre-training of Graph Foundation Models on Multi-source, Multi-fidelity Atomistic Modeling Data

Graph foundation models using graph neural networks promise sustainable, efficient atomistic modeling. To tackle challenges of processing multi-source, multi-fidelity data during pre-training, recent studies employ multi-task learning, in which shared message passing layers initially process input atomistic structures regardless of source, then route them to multiple decoding heads that predict data-specific outputs. This approach stabilizes pre-training and enhances a model’s transferability to unexplored chemical regions. Preliminary results on approximately four million structures are encouraging, yet questions remain about generalizability to larger, more diverse datasets and scalability on supercomputers. We propose a multi-task parallelism method that distributes each head across computing resources with GPU acceleration. Implemented in the open-source HydraGNN architecture, our method was trained on over 24 million structures from five datasets and tested on the Perlmutter, Aurora, and Frontier supercomputers, demonstrating efficient scaling on all three highly heterogeneous super-computing architectures.

Lupo Pasini, Massimiliano [ORNL] (ORCID:0000000249↗
DOE OSTI2019

Significance of Finish Cooling Temperature to Microstructure and Property Relationship of Low-Carbon V-N-Cr Microalloyed High-Strength Steel

Herein, three final cooling temperatures were employed to achieve high strength and toughness communication in the V-N-Cr microalloyed steels. This study shows that V-N-Cr microalloyed steel primarily consisted of ferrite, granular bainite, acicular ferrite, and a small quantity of M/A phase at the final cooling temperature of 600 °C, while the polygonal ferrite, lamellar bainite, granular bainite, and a small fraction of acicular ferrite were obtained at 450 °C. The width of lamellar bainite was ~ 200-400 nm, and acicular ferrite was composed of fine non-parallel ferrite platelet structures with high dislocations densities. As the final cooling temperature decreased further, the microstructure included granular bainite and lath bainite at 350 °C. The nanoscale precipitates of V-N-Cr microalloyed steel were divided into two types, large-scale precipitates near the grain boundary that precipitated at high temperatures and small precipitates of diameter 5-10 nm at low temperatures, which significantly contributed toward strength. The hot rolled low-carbon V-N-Cr microalloy steel exhibited optimal mechanical properties at the final rolling temperature of 830 °C, as well the final cooling temperature was 450 °C. The highest yield strength and tensile strength of V-N-Cr microalloyed steel were 835 MPa and 989 MPa, respectively. The outstanding impact toughness of 161 J tested at − 60 °C was received. The transformation strengthening and precipitation hardening played a crucial role in affecting the thermo-mechanical properties.

36 MATERIALS SCIENCE↗