Datacenter Explorer
An Unreal Engine plugin for visualizing and interacting with hierarchical datacenter infrastructure through JSON configurations.
Greenwood, Scott↗
Engineering topics
Maiterth, Matthias
Publications and source records attributed to Maiterth, Matthias.
A Digital Twin Framework for Liquid-cooled Supercomputers as Demonstrated at Exascale
We present ExaDigiT, an open-source framework for developing comprehensive digital twins of liquid-cooled supercomputers. It integrates three main modules: (1) a resource allocator and power simulator, (2) a transient thermo-fluidic cooling model, and (3) an augmented reality model of the supercomputer and central energy plant. The framework enables the study of "what-if" scenarios, system optimizations, and virtual prototyping of future systems. Using Frontier as a case study, we demonstrate the framework's capabilities by replaying six months of system telemetry for systematic verification and validation. Such a comprehensive analysis of a liquid-cooled exascale supercomputer is the first of its kind. ExaDigiT elucidates complex transient cooling system dynamics, runs synthetic or real workloads, and predicts energy losses due to rectification and voltage conversion. Throughout our paper, we present lessons learned to benefit HPC practitioners developing similar digital twins. We envision the digital twin will be a key enabler for sustainable, energy-efficient supercomputing.
Brewer, Wes↗
Towards Sustainable Post-Exascale Leadership Computing
As computing systems approach the limits of traditional silicon technology, the diminishing returns in performance per watt present a significant barrier to sustaining growth in HPC. From a large-scale scientific supercomputing facility point of view, we propose a multifaceted strategy toward specialized hardware and architectures that are optimized for energy efficiency in specific applications. We also emphasize the need for integrating energy-aware practices across all levels of HPC, from system design and software development to operational policies. We discuss strategic opportunities such as the adoption of application-specific accelerators, the development of energy-efficient algorithms, and the implementation of data-driven operational analytics. Our goal is to develop a comprehensive roadmap ensuring that future leadership systems at OLCF can meet scientific demands while operating within stringent energy budgets, thereby supporting sustainable computing growth.
Shin, Woong↗
Exploring the Frontiers of Energy Efficiency using Power Management at System Scale
In the face of surging power demands for exascale HPC systems, this work tackles the critical challenge of understanding the impact of software-driven power management techniques like Dynamic Voltage and Frequency Scaling (DVFS) and Power Capping. These techniques have been actively developed over the past few decades. By combining insights from GPU benchmarking to understand application power profiles, we present a telemetry data-driven approach for deriving energy savings projections. This approach has been demonstrably applied to the Frontier supercomputer at scale. Our findings based on three months of telemetry data indicate that, for certain resource-constrained jobs, significant energy savings (up to 8.5%) can be achieved without compromising performance. This translates to a substantial cost reduction, equivalent to 1438 MWh of energy saved. The key contribution of this work lies in the methodology for establishing an upper limit for these best-case scenarios and its successful application. This work enables HPC professionals to optimize the power-performance trade-off within constrained power budgets, not only for the exascale era but also beyond.
Karimi, Ahmad Maroof↗
Visualizing an Exascale Data Center Digital Twin: Considerations, Challenges and Opportunities
Digital twins are an excellent tool to model, visualize, and simulate complex systems, to understand and optimize their operation. In this work, we present the technical challenges of real-time visualization of a digital twin of the Frontier supercomputer.We show the initial prototype and current state of the twin and highlight technical design challenges of visualizing such a large High Performance Computing (HPC) system. The goal is to understand the use of augmented reality as a primary way to extract information and collaborate on digital twins of complex systems. This leverages the spatio-temporal aspect of a 3D representation of a digital twin, with the ability to view historical and real-time telemetry, triggering simulations of a system state and viewing the results, which can be augmented via dashboards for details. Finally, we discuss considerations and opportunities for augmented reality of digital twins of large-scale, parallel computers.
Maiterth, Matthias↗
Dynamic Modeling of Power Conversion Stages for an Exascale Supercomputer
In this paper a power conversion and energy consumption model for an exascale supercomputer is investigated. Power consumption, energy loss and efficiency are derived for the 27.2 MW liquid-cooled, centralized, High Performance Computing (HPC) power system, which is supplied directly from the 480 V three-phase mains. Two energy conversion stages are analyzed, measured and modeled. The model is developed in order to be adapted and implemented in a digital twin platform utilizing a Resource Allocator and Power Simulator (RAPS) module. RAPS enables estimation of potential energy savings in the direct AC power supply architecture via both conventional rectifier load sharing (commonly used in HPC systems), as well as smart rectifier load sharing. Moreover, besides the direct AC supply architecture analysis, the full direct DC supply architecture with with 1 kV DC bus were also studied. Comparison of 10 hour time frame operation of the system, with direct 480 V AC voltage supply with conventional and smart load sharing and medium dc voltage supply were done. For the direct AC supply architecture, with conventional and smart load sharing the predicted power loss was approximately 840 kW and 820 kW, respectively and the predicted total system efficiency was 92.87% and 93.05%, respectively. For the direct DC supply architecture with the 1000 V DC supply bus power loss was approximately 340 kW and the predicted total system efficiency was 97.02%.
Wojda, Rafal↗
Digital Twins for Data Centers
Fueled by an unprecedented adoption of AI (Artificial Intelligence), data centers are becoming the largest growing consumers of energy. Digital Twins provide living digital models of physical systems that enable data-driven analysis and application of AI to better manage selective aspects of the data center and drive efficiency for sustainability. Digital twins have emerged as a way to create virtual prototypes of physical artifacts, which may be used in a variety of contexts. Physical artifacts include airplanes, factories, or even static objects, such as bridges or dams. Digital twin helps monitor changes and assist in predicting planned or unplanned behaviors of physical objects. In this paper, we discuss digital twins for data centers.
97 MATHEMATICS AND COMPUTING↗
Optimizing Distributed Training on Frontier for Large Language Models
Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Loss scaling studies have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of 38.38%, 36.14%, and 31.96%, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved 100% weak scaling efficiency on 1024 and 3072 Mi250X GPUs, respectively. We also achieved strong scaling efficiencies of 89% and 87% for these two models. We trained these models only tens of iterations instead of training till completion.
Yin, Junqi↗
Towards the Development of an Exascale Network Digital Twin
Explore the source record for details and available documents.
Holmen, John↗
OLCF Frontier Supercomputer 2023-04-29 HPL Power Data used for Top500/Green500 Submission
OLCF Frontier Supercomputer 2023-04-29 HPL Power Data used for Top500/Green500 Submission This dataset provides the scripts used to derive the data for submission at [https://www.top500.org/lists/top500/2023/06/](https://www.top500.org/lists/top500/2023/06/) plus the primary data is included in the dataset. Additional information is available in the README.md.
97 MATHEMATICS AND COMPUTING↗