Post Exascale Software in the ASCR Facilities Ecosystem
Explore the source record for details and available documents.
Engineering topics
Publications and source records attributed to Roth, Philip.
Explore the source record for details and available documents.
The advent of Exascale computing invites an assessment of existing best practices for developing application readiness on the world's largest supercomputers. This work details observations from the last four years in preparing scientific applications to run on the Oak Ridge Leadership Computing Facility's (OLCF) Frontier system. This paper addresses a range of topics in software including programmability, tuning, and portability considerations that are key to moving applications from existing systems to future installations. A set of representative workloads provides case studies for general system and software testing. We evaluate the use of early access systems for development across several generations of hardware. Finally, we discuss how best practices were identified and disseminated to the community through a wide range of activities including user-guides and trainings. We conclude with recommendations for ensuring application readiness on future leadership computing systems.
Cloud platforms offer a variety of benefits that are very appealing for a large scale HPC facility with a diverse and dynamic user base and workload set. At the same time, there is cause for concern about transitioning to the cloud. Incorporating cloud resources into existing HPC facilities or even fully transitioning to a cloud deployment poses significant challenges at the technical, organizational, and economic levels. Regardless, based on current trends it is highly likely that cloud platforms will become an integral component of many HPC centers in some form. To gain a better understanding of both the limitations and capabilities of current cloud infrastructures we evaluated the public offerings of the three leading cloud platforms (Amazon Web Services, Microsoft Azure, and Google Cloud Platform) using a selection of representative application workloads from our facility. Our findings show that while current HPC offerings are still nascent, significant progress is being made to address the present shortcomings. At the same time, significant challenges and questions remain about whether HPC cloud offerings will be able to deliver the full range of expected benefits.
The roofline model provides a concise overview of the maximum performance capabilities of a given computer system through a combination of peak memory bandwidth and compute performance rates. The increasing complexity of scheduling and cache in recent GPUs, however, has introduced complicated performance variability that is not captured by arithmetic intensity alone. This work examines the effect of problem size and GPU launch configurations on roofline performance for V100, A100, MI100, and MI250X graphics processing units. We introduce an extended roofline model that takes problem size into account, and find that strong scaling on GPUs can be characterized by saturation problem sizes as additional key metrics. Saturation problem sizes break up a plot of GPU performance vs. problem size into three distinct performance regimes– size-limited, cache-bound, and DRAM-bound. With our extended roofline model, we are able to provide a robust view of these performance regimes across recent GPU architectures.
The Summit system at Oak Ridge National Lab-oratory (ORNL) has been the world's top AI for science su-percomputer for several years, ranked world's fastest computer at its 2018 launch and currently top system in the US and #2 on the TOP5OO list. Summit's purposeful design to handle both conventional modeling and simulation science and emerging AI workloads has made it a leading destination for AI-powered computational science. We report here on AI for science usage on Summit near the midpoint of its lifespan. We review AI usage across the many science projects that have used Summit. We then examine in detail a set of applications scaling AI to full system as well as projects implementing AI-coordinated science discovery workflows on Summit. Finally, we offer some observations regarding the future of advancing scientific knowledge and understanding via AI, especially in the context of leadership-class scientific computing.
Abstract Fission gas release within uranium dioxide nuclear fuel occurs as gas atoms diffuse through grains and arrive at grain boundary (GB) bubbles; these GB bubbles grow and interconnect with grain edge bubbles; and grain edge tunnels grow and connect to free surfaces. In this study, a hybrid multi-scale/multi-physics simulation approach is presented to investigate these mechanisms of fission gas release at the mesoscale. In this approach, fission gas production, diffusion, clustering to form intragranular bubbles, and re-solution within grains are included using spatially resolved cluster dynamics in the Xolotl code. GB migration and intergranular bubble growth and coalescence are included using the phase field method in the MARMOT code. This hybrid model couples Xolotl to MARMOT using the MultiApp and Transfer systems in the MOOSE framework, with Xolotl passing the arrival rate of gas atoms at GBs and intergranular bubble surfaces to MARMOT and MARMOT passing evolved GBs and bubble surface positions to Xolotl. The coupled approach performs well on the two-dimensional simulations performed in this work, producing similar results to the standard phase field model when Xolotl does not include fission gas clustering or re-solution. The hybrid model performs well computationally, with a negligible cost of coupling Xolotl and MARMOT and good parallel scalability. The hybrid model predicts that intragranular fission gas clustering and bubble formation results in up to 70% of the fission gas being trapped within grains, causing the increase in the intergranular bubble fraction to slow by a factor of six. Re-solution has a small impact on the fission gas behavior at 1800 K but it has a much larger impact at 1000 K, resulting in a twenty-times increase in the concentration of single gas atoms within grains. Due to the low diffusion rate, this increase in mobile gas atoms only results in a small acceleration in the growth of the intergranular bubble fraction. Finally, the hybrid model accounts for migrating GBs sweeping up gas atoms. This results in faster intergranular bubble growth with smaller initial grain sizes, since the additional GB migration results in more immobile gas clusters reaching GBs.
Divertor design and choice of plasma-facing materials (PFM) will be essential to the success of next-generation fusion reactors as they operate under more powerful scenarios. Understanding and controlling interactions between the plasma and PFM is essential to making these choices. Within these plasma–material interactions and especially in tungsten (W), the interplay between the most abundant plasma species (hydrogen isotopes and helium, He) with the wall material alters fuel retention. However, this interplay is yet to be sufficiently understood to confidently project fuel retention levels to future fusion devices. The paper presents a series of integrated simulations of fusion plasmas and their interaction with tungsten. Specifically, this study assesses the impact of He plasma pre-exposure on hydrogenic species retention during 100 s of burning plasma operations (BPO) in ITER. Multiple pre-exposure scenarios are considered, including sub-surface damage resulting from exposures in the linear device PISCES and from early ITER He-operation. The predictions from these consecutive He-BPO exposures show that fuel content and spatial distribution in the material are largely determined by the He-induced damage, as manifest in: (i) changes in surface temperature expected during BPO have little effect on fuel retention in the presence of He-induced damage; (ii) gas content stabilizes quickly in substrates pre-exposed in PISCES, at levels set by the concentration of pre-existing vacancies, while it continues to increase in substrates initially pristine or pre-exposed to ITER He plasmas; (iii) the presence of He and He–V clusters in the near-surface region locally increases hydrogenic retention, but decreases its permeation; this results in hydrogenic species that remain closer to the surface in pre-damaged substrates, while the bulk content is higher for initially pristine cases. In summary, the interaction and binding of D and T with the pre-existing He–V clusters modifies retention and permeation of hydrogen species during ITER BPO.
Custom spatial subdivision of N-dimensional lattice region in GPU memory. An extension of the method of a hyper-octree is used to allow for arbitrary subdivision ratios in each dimension. In addition, the decision to refine (or select) a given sub-region is based on a user-provided callback, allowing for arbitrary refinement domains to be expressed either geometrically or programmatically.
The increasing complexity of modern scientific simulations has given rise to the notion of re-usability as a means to reduce development time and effort. One form of re-usability involves hierarchical modelling, where the code and artifacts that model a physical phenomenon are used as the building blocks for more complex coupled physical systems. This reusability mode allows for improvements in constituents sub-models to impact the overall fidelity of the entire system with minimal effort. This benefit, however, hinges upon the ability to provide a fairly loose coupling across model boundaries, where changes in any sub-model do not result in wholesale changes to other sub-models, or in the structure and details of the code for the entire physical system. In this paper, we present the design, implementation, and case studies for incorporating sub-workflows as building blocks in a framework for loosely coupled high performance simulations. We outline the issues involved in providing lightweight customization points for full workflows that allows their use unchanged in a nested-workflow setting, while providing each sub-workflow with a separate execution context that ensures non-interference with other parts of the simulation. We present several use cases that demonstrate the successful use of the proposed design and its flexibility in accommodating different customization and scaling requirements.
Characterization of a parallel application’s communication patterns can be useful for performance analysis, debugging, and system design. However, obtaining and interpreting a characterization can be difficult. AChax implements an approach that uses search and a library of known communication patterns to automatically characterize communication patterns. Our approach has some limitations that reduce its effectiveness for the patterns and pattern combinations used by some real-world applications. By viewing AChax’s pattern recognition problem as an image recognition problem, it may be possible to use deep learning to address these limitations. In this position paper, we present our current ideas regarding the benefits and challenges of integrating deep learning into AChax and our conclusion that a hybrid approach combining deep learning classification, regression, and the existing AChax approach may be the best long-term solution to the problem of parameterizing recognized communication patterns.
An understanding of a parallel application’s communication behavior is useful for a range of activities including debugging and optimization, job scheduling, target system selection, and system design. Because it can be challenging to understand communication behavior, especially for those who lack expertise or who are not familiar with the application, I and two colleagues recently developed an automated, search-based approach for recognizing and parameterizing application communication behavior using a library of common communication patterns. This initial approach was effective for characterizing the behavior of many workloads, but I identified some combinations of communication patterns for which the method was inefficient or would fail. In this paper, I discuss one such troublesome pattern combination and propose modifications to the recognition method to handle it. Specifically, I propose an alternative approach that uses communication graphs instead of traditional communication matrices to improve recognition accuracy for collective communication operations, and that uses a non-greedy recognition technique to avoid search space dead-ends that trap the original greedy recognition approach. My modified approach uses aggressive search space pruning and heuristics to control the potential for state explosion caused by its non-greedy pattern recognition method. I demonstrate the improved recognition accuracy and pruning efficacy of the modified approach using several synthetic and real-world communication pattern combinations.
Explore the source record for details and available documents.
In this project we created a community tool infrastructure for program development tools targeting Petascale class machines and beyond. This includes tools for performance analysis, debugging, and correctness tools, as well as tuning and optimization frameworks. The developed infrastructure provides a comprehensive and extensible set of individual tool building components. We started with the basic elements necessary across all tools in such an infrastructure followed by a set of generic core modules that allow a comprehensive performance analysis at scale. Further, we developed a methodology and workflow that allows others to add or replace modules, to integrate parts into their own tools, or to customize existing solutions. In order to form the core modules, we built on the existing Open|SpeedShop infrastructure and decomposed it into individual modules that match the necessary tool components. At the same time, we addressed the challenges found in performance tools for petascale systems in each module. When assembled, this instantiation of community tool infrastructure provides an enhanced version of Open|SpeedShop, which, while completely different in its architecture, provides scalable performance analysis for petascale applications through a familiar interface. This project also built upon and enhances capabilities and reusability of project partner components as specified in the original project proposal. The overall project team’s work over the project funding cycle was focused on several areas of research, which are described in the following sections. The reminder of this report also highlights related work as well as preliminary work that supported the project. In addition to the project partners funded by the Office of Science under this grant, the project team included several collaborators who contribute to the overall design of the envisioned tool infrastructure. In particular, the project team worked closely with the other two DOE NNSA laboratories Los Alamos and Sandia leveraging co-funding for Krell by ASC’s Common Computing Environment (CCE) program as laid out in the original proposal. The ASC CCE co-funding, coordinated through LLNL, was for 50% of the total project funding, with the ASC CCE portion of the funding going entirely to Krell, while the ASCR funding itself was split between Krell and the funded partners. This report covers the entire project from both funding sources. Additionally, the team leveraged the expertise of software engineering researchers from Carnegie Mellon University, who specialize in software framework design, in order to achieve a broadly acceptable component framework. The Component Based Tool Framework (CBTF) software has been released to the community. Information related to the project and the released software can be found on the CBTF wiki page at: http://sourceforge.net/p/cbtf/wiki/Home.
Petascale computing infrastructures for scientific discovery make petascale demands on information storage capacity, performance, concurrency, reliability, availability, and manageability.The Petascale Data Storage Institute focuses on the data storage problems found in petascale scientific computing environments, with special attention to community issues such as interoperability, community buy-in, and shared tools.The Petascale Data Storage Institute is a collaboration between researchers at Carnegie Mellon University, National Energy Research Scientific Computing Center, Pacific Northwest National Laboratory, Oak Ridge National Laboratory, Sandia National Laboratory, Los Alamos National Laboratory, University of Michigan, and the University of California at Santa Cruz.
In this project, we cast distributed resource access as operations on files in a global name space and developed a common, scalable solution for group operations on distributed processes and files. The resulting solution enables tool and middleware developers to quickly create new scalable software or easily improve the scalability of existing software. The cornerstone of the project was the design of a new programming idiom called group file operations that eliminates iterative behavior when a single process must apply the same set of file operations to a group of related files. To demonstrate our novel and scalable ideas for group file operations and global name space composition, we developed a group file system called TBON-FS that leverages a tree-based overlay network (TBON), specifically MRNet, for logarithmic communication and distributed data aggregation. We also developed proc++, a new synthetic file system co-designed for use in scalable group file operations. Over the course of the project, we evaluated the utility and performance of group file operations, global name space composition, TBON-FS, and proc++ in three case studies. The first study focused on the ease in using group file operations and TBON-FS to quickly develop several new scalable tools for distributed system administration and monitoring. The second study evaluated the integration of group file operation and TBON-FS within the Ganglia Distributed Monitoring System to improve its scalability for clusters. The final study involved the integration of group file operations, TBON-FS, and proc++ within TotalView, the widely-used parallel debugger. For this project, the work of the Oak Ridge National Laboratory (ORNL) team occurred primarily in two directions: bringing the MRNet tree-based overlay network (TBON) implementation to the Cray XT platform, and investigating techniques for predicting the performance of MRNet topologies on such systems. Rogue Wave Software (RWS), formerly TotalView Technologies Inc., worked with the University ofWisconsin (UW) team on the design and prototyping of a scalable version of the TotalView debugger. RWS assisted UW with their "proc++" design effort. RWS assisted UW with strategy for integrating proc++ into TotalView.