Search NASA⌕ Search

SEARCH · Search NASA

Results for “Exascale computer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Biomass Feedstock Flow Modeling at Exascale

Variable biomass feedstock handling in hoppers and screw conveyors can cause upstream failures and disrupt downstream conversion operations. Understanding how particle size, shape, and moisture content affect feedstock flowability is important for efficient biorefinery operation. We present the use of our open-source discrete element method (DEM) solver for predicting flows in large scale hoppers and assess its performance on state-of-the-art high performance computing platforms that use heterogenous computing architectures.

bioenergy↗
DOE OSTI2025

Development and evaluation of a new 4DEnVar-based weakly coupled ocean data assimilation system in E3SMv2

The development, implementation, and evaluation of a new weakly coupled ocean data assimilation (WCODA) system for the fully coupled Energy Exascale Earth System Model version 2 (E3SMv2) utilizing the four-dimensional ensemble variational (4DEnVar) method are presented in this study. The 4DEnVar method, based on the dimension-reduced projection four-dimensional variational (DRP-4DVar) approach, replaces the adjoint model with the ensemble technique, thereby reducing computational demands. Monthly mean ocean temperature and salinity data from the EN4.2.1 reanalysis are integrated into the ocean component of E3SMv2 from 1950 to 2021 with the goal of providing realistic initial conditions for decadal predictions and predictability studies. The performance of the WCODA system is assessed using various metrics, including the reduction rate of the cost function, root mean square error (RMSE) differences, correlation differences, and model biases. Results indicate that the WCODA system effectively assimilates the reanalysis data into the climate model, consistently achieving negative reduction rates of the cost function and notable improvements in RMSE and correlation across various ocean layers and regions. Significant enhancements are observed in the upper ocean layers across the majority of global ocean regions, particularly in the north Atlantic, north Pacific, and Indian Ocean. Model biases in sea surface temperature and salinity are also substantially reduced. For sea surface temperature, cold biases in the north Pacific and north Atlantic are diminished by about 1–2 °C, and warm biases in the Southern Ocean are corrected by approximately 1.5–2.5 °C. In terms of salinity, improvements are observed with bias reductions of about 0.5–1 psu in the north Atlantic and north Pacific and up to 1.5 psu in parts of the Southern Ocean. The ultimate goal of the WCODA system is to advance the predictive capabilities of E3SM for subseasonal to decadal climate predictions, thereby supporting research on strategic energy-sector policies and planning.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Resilience and fault tolerance in high-performance computing for numerical weather and climate prediction

Progress in numerical weather and climate prediction accuracy greatly depends on the growth of the available computing power. As the number of cores in top computing facilities pushes into the millions, increased average frequency of hardware and software failures forces users to review their algorithms and systems in order to protect simulations from breakdown. This report surveys hardware, application-level and algorithm-level resilience approaches of particular relevance to time-critical numerical weather and climate prediction systems. A selection of applicable existing strategies is analysed, featuring interpolation-restart and compressed checkpointing for the numerical schemes, in-memory checkpointing, user-level failure mitigation and backup-based methods for the systems. Numerical examples showcase the performance of the techniques in addressing faults, with particular emphasis on iterative solvers for linear systems, a staple of atmospheric fluid flow solvers. The potential impact of these strategies is discussed in relation to current development of numerical weather prediction algorithms and systems towards the exascale. Trade-offs between performance, efficiency and effectiveness of resiliency strategies are analysed and some recommendations outlined for future developments.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Visualizing an Exascale Data Center Digital Twin: Considerations, Challenges and Opportunities

Digital twins are an excellent tool to model, visualize, and simulate complex systems, to understand and optimize their operation. In this work, we present the technical challenges of real-time visualization of a digital twin of the Frontier supercomputer.We show the initial prototype and current state of the twin and highlight technical design challenges of visualizing such a large High Performance Computing (HPC) system. The goal is to understand the use of augmented reality as a primary way to extract information and collaborate on digital twins of complex systems. This leverages the spatio-temporal aspect of a 3D representation of a digital twin, with the ability to view historical and real-time telemetry, triggering simulations of a system state and viewing the results, which can be augmented via dashboards for details. Finally, we discuss considerations and opportunities for augmented reality of digital twins of large-scale, parallel computers.

Maiterth, Matthias↗
DOE OSTI2020

Collaborative Research: Improved Efficiency and Coupling of the Radiation Code in the ACME Earth System Model. Final Report

This final report details all work performed on the project by both project partners. This project provided support to properly couple RTE+RRTMGP, a high-performance broadband radiation code, within DOE’s Energy Exascale Earth System Model (E3SM). RTE+RRTMGP is a successor to the RRTMG radiation code, which has been widely accepted for its speed and accuracy by the global modeling community, and has been in use in the NCAR CESM for many years and was implemented in the initial version of E3SM. However, the computational cost of RRTMG remains high relative to other components in part due to its complexity and to its inefficient use of modern optimization strategies, issues that were rectified by the development of RTE+RRTMGP. Many of the accomplishment in this project necessitated significant collaboration with the E3SM development team. One focus of the project was to enhance the code’s optimization on the limited number of emerging computing systems on which the model is expected be used, including Many Integrated Core (MIC) architectures and Graphics Processing Unit (GPU) hardware. We also developed additional capabilities for RTE+RRTMGP that E3SM scientists identified as important for the planned applications of the model. The result of our project was optimization of a key physical component (radiative transfer calculations) of E3SM, directly supporting E3SM’s overarching global modeling objectives. More broadly, this project provided overall advancements in the use of radiative transfer calculations in atmospheric modeling and simulation, particularly for climate.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2021: Oak Ridge Leadership Computing Facility

Oak Ridge National Laboratory’s (ORNL’s) Leadership Computing Facility (OLCF) continues to surpass its operational target goals of supporting users; delivering fast, reliable computational ecosystems; creating innovative solutions for high-performance computing (HPC) needs; contributing to the community to build the next generation HPC workforce, and managing risks, safety, and security associated with operating some of the most powerful computers in the world. The results can be seen in the cutting-edge science conducted by users and the praise from the research community. Calendar year (CY) 2021 saw continued excellence in research supported by the OLCF’s leadership-class computing resources, including Summit (the nation’s most powerful supercomputer), the global scratch file system Alpine, the Scalable Protected Infrastructure (SPI), the Exploratory Visualization Environment for Research in Science and Technology (EVEREST), and the archival mass-storage resource High-Performance Storage System (HPSS). While maintaining access and exceptional user support for Summit, the OLCF continued to make progress on the installation and deployment of Frontier, which will be the nation’s first exascale system when it comes online at the start of CY 2023. Users have already begun running and optimizing scientific codes on Crusher, the OLCF test and development system equipped with Frontier’s architecture. Throughout the year, the OLCF maintained a strong culture of operational excellence, including risk management, workplace safety, and cybersecurity. The OLCF’s rigorous risk management strategy anticipated and mitigated risks, and at this time there are no high-priority operational risks. Similarly, ORNL and the OLCF were committed to operating under the US Department of Energy’s (DOE’s) safety regulations that ensure a safe workplace. Technical staff tracked and monitored existing threats and vulnerabilities within the OLCF while continually developing tools and practices to enhance operations without increasing the facility’s risk. CY 2021 was filled with outstanding results and accomplishments, including a very high rating from users on overall satisfaction for the eighth consecutive year; a tremendous number of node hours delivered to 1,671 researchers on Summit; and the successful delivery of the allocation split of roughly 60%, 20%, and 20% of core-hours offered for the Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), and Director’s Discretionary (DD) programs, respectively (Section 2). COVID-19 research remained a focus in 2021, and the ALCC and DD programs allocated over 1 million Summit hours to the COVID-19 High Performance Computing Consortium. These accomplishments, coupled with the high utilization rates (i.e., overall and capability usage), represent the fulfillment of the promise of leadership class machines: efficient facilitation of leadership-class computational applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

An evaluation of the E3SMv1 Arctic ocean and sea-ice regionally refined model

The Energy Exascale Earth System Model (E3SM) is a state-of-the-science Earth system model (ESM) with the ability to focus horizontal resolution of its multiple components in specific areas. Regionally refined global ESMs are motivated by the need to explicitly resolve, rather than parameterize, relevant physics within the regions of refined resolution, while offering significant computational cost savings relative to the respective cost of configurations with high-resolution (HR) everywhere on the globe. In this paper, we document results from the first Arctic regionally refined E3SM configuration for the ocean and sea-ice components (E3SM-Arctic-OSI), while employing data-based atmosphere, land, and hydrology components. Our aim is an improved representation of the Arctic coupled ocean and sea-ice state, its variability and trends, and the exchanges of mass and property fluxes between the Arctic and the sub-Arctic. We find that E3SM-Arctic-OSI increases the realism of simulated Arctic ocean and sea-ice conditions compared to a similar low-resolution E3SM simulation without the Arctic regional refinement in ocean and sea-ice components (E3SM-LR-OSI). In particular, exchanges through the main Arctic gateways are greatly improved with respect to E3SM-LR-OSI. Other aspects, such as the Arctic freshwater content variability and sea-ice trends, are also satisfactorily simulated. Yet, other features, such as the upper-ocean stratification and the sea-ice thickness distribution, need further improvements, involving either more advanced parameterizations, model tuning, or additional grid refinements. Overall, E3SM-Arctic-OSI offers an improved representation of the Arctic system relative to E3SM-LR-OSI, at a fraction (15 %) of the computational cost of comparable global high-resolution configurations, while permitting exchanges with the lower-latitude oceans that cannot be directly accounted for in Arctic regional models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Performance Analysis of PIConGPU: Particle-in-Cell on GPUs using NVIDIA’s NSight Systems and NSight Compute

PIConGPU, Particle In Cell on GPUs, is an open source simulations framework for plasma and laser-plasma physics used to develop advanced particle accelerators for radiation therapy of cancer, high energy physics and photon science. While PIConGPU has been optimized for at least 5 years to run well on NVIDIA GPU-based clusters, there has been limited exploration by the development team of potential scalability bottlenecks using recently updated and new tools including NVIDIA’s NVProf tool and the brand-new NVIDIA NSight Suite (Systems and Compute) tools. PIConGPU is a highly optimized application that runs production jobs at scale on a system Oak Ridge Leadership Facility’s (OLCF) Summit supercomputer (using the full machine at 4600 nodes; at 98% of GPU utilization on all ~28000 NVIDIA Volta GPUs). PIConGPU has been selected as one of the the eight applications for OLCF’s coveted Center for Accelerated Application Readiness (CAAR) program aimed at the facility’s Frontier supercomputer (OLCF’s first exascale system to launch in 2021), to partner with our vendors (primary vendors: AMD and Cray/HPE) ensuring that Frontier will be able to perform large-scale science when it opens to users in 2022. To this effect, performance engineers on the PIConGPU team wanted to dive deep into the application to understand at the finest granularity, which portions of the code could be further optimized to exploit the hardware on Summit at it’s maximum potential and also to elucidate which key kernels should be tracked and optimized for the CAAR effort to port this code to Frontier. Any bottlenecks that are observed via performance profiling on Summit are likely to also impact scalability on the Frontier-dev system and the Frontier Early Access (EA) system. Additionally, the engineers wanted to take a closer look at the newest NVIDIA profiling tools which allows us to identify the most useful features on these tools and will provide an opportunity to compare it to new AMD and Cray’s performance analysis tool releases and provide feedback to our vendor partners on what features are most important and mission critical for CAAR efforts. The primary goal of this report is to focus on the evaluation of PIConGPU’s most time-intensive kernels using NVProf and NSight Suite. Three kernels, Current Deposition (also known as Compute Current), Particle Push (Move and Mark), and Shift Particles are known to be some of the most time-consuming kernels in PIConGPU. The Current xi Deposition kernel and Particle Push kernel both set up the particle attributes for running any physics simulation with PIConGPU, so it is crucial to improve the performance of these two kernels. In this report, we measure single GPU metrics for the three kernels, offer high level takeaways from the conducted analysis, and compare the profiling data from NSight Compute to that of NVProf. This analysis was performed using a grid size of 240 x 272 x 224, and 10 time steps with the Mid-November Figure of Merit (FOM) run setup. The Traveling Wave Electron Acceleration (TWEAC) science case used in this run is a representative science case for PIConGPU. This execution can also be used for baseline analysis on AMD MI50/ MI60 systems. As of the time of writing, the PIConGPU application has limited use for features of NSight Systems, so this report will mainly focus on insights garnered from NSight Compute. For this analysis, we run the “full” metric set available in NSight Compute version 2020.1.2 and use NSight Systems version 2020.3.1 to generate the application timeline.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Enabling Multireference Calculations on Multimetallic Systems with Graphic Processing Units

Modeling multimetallic systems efficiently enables faster prediction of desirable chemical properties and the design of new materials. This work describes an initial implementation for performing multireference wave function method localized active-space self-consistent field (LASSCF) calculations through the use of multiple graphics processing units (GPUs) to accelerate time-to-solution. Density fitting is leveraged to reduce memory requirements, and we demonstrate the ability to fully utilize multi-GPU compute nodes. Performance improvements of 5–10x in total application runtime were observed in LASSCF calculations for multimetallic catalyst systems up to 1200 AOs and an active space of (22e,40o) using up to four NVIDIA A100 GPUs. Furthermore, written with performance portability in mind, a comparable performance is also observed in early runs on the Aurora exascale system using Intel Max Series GPUs.

Algorithms↗
DOE OSTI2025

High-Resolution Modeling of the Gulf of Mexico using E3SM

Coastal ocean modeling is a high priority in the DOE‘s Energy Exascale Earth System Model (E3SM). The goal is to accurately predict the risk of damage to coastal resources and infrastructure due to a changing climate in the coming decades. North American coastal communities are areas of particular interest, as this fits under the topic of US national security and planning assessments in a changing climate. LANL Institutional Computing time for the Tier 1 allocation ”Coastal Ocean and Sea Ice Modeling” have been used for development and testing of numerical methods needed for E3SM coastal applications.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

MPAS-Seaice (v1.0.0): sea-ice dynamics on unstructured Voronoi meshes

Abstract. We present MPAS-Seaice, a sea-ice model which uses the Model for Prediction Across Scales (MPAS) framework and spherical centroidal Voronoi tessellation (SCVT) unstructured meshes. As well as SCVT meshes, MPAS-Seaice can run on the traditional quadrilateral grids used by sea-ice models such as CICE. The MPAS-Seaice velocity solver uses the elastic–viscous–plastic (EVP) rheology and the variational discretization of the internal stress divergence operator used by CICE, but adapted for the polygonal cells of MPAS meshes, or alternatively an integral (“finite-volume”) formulation of the stress divergence operator. An incremental remapping advection scheme is used for mass and tracer transport. We validate these formulations with idealized test cases, both planar and on the sphere. The variational scheme displays lower errors than the finite-volume formulation for the strain rate operator but higher errors for the stress divergence operator. The variational stress divergence operator displays increased errors around the pentagonal cells of a quasi-uniform mesh, which is ameliorated with an alternate formulation for the operator. MPAS-Seaice shares the sophisticated column physics and biogeochemistry of CICE and when used with quadrilateral meshes can reproduce the results of CICE. We have used global simulations with realistic forcing to validate MPAS-Seaice against similar simulations with CICE and against observations. We find very similar results compared to CICE, with differences explained by minor differences in implementation such as with interpolation between the primary and dual meshes at coastlines. We have assessed the computational performance of the model, which, because it is unstructured, runs with 70 % of the throughput of CICE for a comparison quadrilateral simulation. The SCVT meshes used by MPAS-Seaice allow removal of equatorial model cells and flexibility in domain decomposition, improving model performance. MPAS-Seaice is the current sea-ice component of the Energy Exascale Earth System Model (E3SM).

58 GEOSCIENCES↗
DOE OSTI2025

Large-Message All-to-All Communication at Frontier Scale

Near the full scale of exascale supercomputers, latency can dominate the cost of all-to-all communication even for very large message sizes. We describe GPU-aware all-to-all implementations designed to reduce latency for large message sizes at extreme scales, and we present their performance using 65536 tasks (8192 nodes) on the Frontier supercomputer at the Oak Ridge Leadership Computing Facility. Two implementations perform best for different ranges of message size, and all outperform the vendor-provided MPI_Alltoall. Our results show promising options for improving implementations of MPI_Alltoall_init.

White, Trey [ORNL] (ORCID:000900052186075X)↗
DOE OSTI2024

Assessing the sensitivity of aerosol mass budget and effective radiative forcing to horizontal grid spacing in E3SMv1 using a regional refinement approach

Abstract. Atmospheric aerosols have important impacts on air quality and the Earth–atmospheric energy balance. However, as computing power is limited, Earth system models generally use coarse spatial grids and parameterize finer-scale atmospheric processes. These parameterizations and the simulation of atmospheric aerosols are often sensitive to model horizontal resolutions. Understanding the sensitivities is necessary for the development of Earth system models at higher resolutions with the deployment of more powerful supercomputers. Using the Energy Exascale Earth System Model (E3SM) version 1, this study investigates the impact of horizontal grid spacing on the simulated aerosol mass budget, aerosol–cloud interactions, and the effective radiative forcing of anthropogenic aerosols (ERFaer) over the contiguous United States. We examine the resolution sensitivity by comparing the nudged simulation results for 2016 from the low-resolution model (LR) and the regional refinement model (RRM). As expected, the simulated emissions of natural dust, sea salt, and marine organic matter are substantially higher in the RRM than in the LR. In addition, RRM simulates stronger aqueous-phase production of sulfate through the enhanced oxidation of sulfur dioxide by hydrogen peroxide due to increased cloud liquid water content. In contrast, the gas-phase chemical production of sulfate is slightly suppressed. The RRM resolves more large-scale precipitation and produces less convective precipitation than the LR, leading to increased (decreased) aerosol wet scavenging by large-scale (convective) precipitation. Regarding aerosol effects on clouds, RRM produces larger temporal variabilities in the large-scale liquid cloud fractions than LR, resulting in increased microphysical cloud processing of aerosols (more interstitial aerosols are converted to cloud-borne aerosols via aerosol activation) in RRM. Water vapor condensation is also enhanced in RRM compared to LR. Consequently, the RRM simulation produces more cloud droplets, a larger cloud droplet radius, a higher liquid water path, and a larger cloud optical depth than the LR simulation. A comparison of the present-day and pre-industrial simulations indicates that, for this contiguous United States domain, the higher-resolution increases ERFaer at the top of the model by about 12 %, which is mainly attributed to the strengthened indirect effect associated with aerosol–cloud interactions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Evaluation of Autoconversion Representation in E3SMv2 Using an Ensemble of Large-Eddy Simulations of Low-Level Warm Clouds

In numerical atmospheric models that treat cloud and rain droplet populations as separate condensate categories, precipitation initiation in warm clouds is often represented by an autoconversion rate (Au), which is the rate of formation of new rain droplets through the collisions of cloud droplets. Being a function of the cloud droplet size distribution (DSD), the local Au is commonly parameterized as a function of DSD moments: cloud droplet number (n c ) and mass (q c ) concentrations. When applied in a large-scale model, the grid-mean Au must also include a correction, or enhancement factor, to account for the horizontal variability of the cloud properties across the model grid. In this study, we evaluate the Au representation in the Energy Exascale Earth System Model version 2 (E3SMv2) climate model using large-eddy simulations (LES), which explicitly resolve cloud droplet spectra, and therefore the local Au, as well as its spatial variability. The analysis of an ensemble of warm low-level cloud cases shows that the E3SMv2 formulation represents the Au reasonably well compared to the horizontally averaged explicitly computed rate from LES. The agreement, however, comes from a combination of an underestimated E3SM-tuned local Au rate and an overestimated subgrid cloud variability enhancement factor. The latter bias is traced to neglecting the horizontal variability of n c and its co-variability with q c in parameterizing the grid-mean Au.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Initial full core SMR simulations with NekRS

This document describes the completion of a recent milestone of the ExaSMR program concerning "Full core simulations". The ExaSMR project is developing tools for coupling Monte Carlo (MC) radiation transport solvers to a computational fluid dynamics (CFD) solver. Work is based on the Shift MC, OpenMC MC, and Nek5000/NekRS CFD codes. As part of this milestone, a novel set of pin-resolved CFD full-core simulations have been performed for the first time. These simulations represent a significant increase in capability in what is now possible with CFD on pre-Exascale systems. The simulations have been performed with the spectral element solvers NekRS on the supercomputer Summit. Multiple simulations campaigns have been conducted: (1) LES simulations in bare bundle, (2) RANS simulations in bare bundles, (3) RANS simulations with momentum sources and (4) Conjugate heat transfer calculations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Performance Portability Evaluation of Fluid-Structure Interaction Simulations on Heterogeneous Platforms

The rapid proliferation of heterogeneous programming languages and multi-vendor hardware has underscored the critical need to evaluate the performance portability of scientific applications. In this work, we present the systematic porting and optimization of a massively parallel fluid-structure interaction code across multiple heterogeneous programming frameworks for deployment on leadership-class supercomputers from major vendors. Our analysis focuses on at-scale performance for simulations involving hundreds of millions of deformable cells, executed on a combination of CPUs and GPUs spanning thousands of nodes on exascale machines. We benchmark the performance of each implementation, highlighting the trade-offs inherent in adopting diverse programming models. Key insights regarding the portability of CUDA on multi-vendor platforms, the superior multi-core CPU performance from SYCL, and architectural considerations on performance optimization are distilled from our experience, offering guidance to other users of high performance computing based on our findings.

Martin, Aristotle [Duke University]↗
DOE OSTI2022

A weighted state redistribution algorithm for embedded boundary grids

State redistribution is an algorithm that stabilizes cut cells for embedded boundary grid methods. This work extends the earlier algorithm in several important ways. First, state redistribution is extended to three spatial dimensions. Second, we discuss several algorithmic changes and improvements motivated by the more complicated cut cell geometries that can occur in higher dimensions. In particular, we introduce a weighted version with less dissipation in an easily generalizable framework. Third, we demonstrate that state redistribution can also stabilize a solution update that includes both advective and diffusive contributions. Notably, the stabilization algorithm is shown to be effective for incompressible as well as compressible reacting flows. Finally, we discuss the implementation of the algorithm for several exascale-ready simulation codes based on AMReX, demonstrating ease of use in combination with domain decomposition, hybrid parallelism and complex physics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

PIPER: Performance Insight for Programmers and Exascale Runtimes (Final Technical Report)

This project concentrated on the development of novel performance tools and analysis techniques for large scale parallel systems and applications, eventually targeting exascale platforms. Activities conducted included work to novel root cause detection approaches and an auto-tuning framework for parallel systems. The work has led to several software solutions, which are available as open source.

97 MATHEMATICS AND COMPUTING↗