Search NASA⌕ Search

SEARCH · Search NASA

Results for “Gemini Interconnect”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

16 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2012

A uGNI-Based Asynchronous Message-driven Runtime System for Cray Supercomputers with Gemini Interconnect

Gemini as the network for new Cray XE/XT systems features low latency, high bandwidth and strong scalability. Its hardware support for remote direct memory access enables efficient implementation of the global address space programming languages. Although the Generic Network Interface (GNI) is designed to support message-passing applications, it is still challenging to attain good performance for applications written in alternative programming models, such as the message-driven programming model. In our earlier work we showed that CHARM++, an object-oriented message-driven programming model, scales up to the full Jaguar Cray machine. In this paper, we describe a general and light-weight asynchronous Low-level RunTime System (LRTS) for CHARM+, and its implementation on the uGNI software stack for Cray XE systems. Several techniques are presented to exploit the uGNI capability by reducing memory copy and registration overhead, taking advantage of persistent communication, and improving intra-node communication. Our micro-benchmark results demonstrate that the uGNI-based runtime system outperforms the MPI-based implementation by up to 50% in terms of message latency. For communication intensive applications such as N-Queens, this implementation scales up to 15,360 cores of a Cray XE6 machine and is 70% faster than an MPI-based implementation. In molecular dynamics application NAMD, the performance is also considerably improved by as high as 18%.

Asynchronous message-passing↗
DOE OSTI2012

A uGNI-based Asynchronous Message-driven Runtime System for Cray Supercomputers with Gemini Interconnect

Gemini, the network for the new Cray XE/XK systems, features low latency, high bandwidth and strong scalability. Its hardware support for remote direct memory access enables efficient implementation of the global address space programming languages. Although the user Generic Network Interface (uGNI) provides a low-level interface for Gemini with support to the message-passing programming model (MPI), it remains challenging to port alternative programming models with scalable performance. CHARM++ is an object-oriented message-driven programming model. Its applications have been shown to scale up to the full Jaguar Cray XT machine. In this paper, we present an implementation of this programming model on uGNI for the Cray XE/XK systems. Several techniques are presented to exploit the uGNI capabilites by reducing memory copy and registration overhead, taking advantage of the persistent communication, and improving intra-node communication. Our microbenchmark results demonstrate that the uGNI-based runtime system outperforms the MPI-based implementation by up to 50% in terms of message latency. For communication intensive applications such as N-Queens, this implementation scales up to 15, 360 cores of a Cray XE6 machine and is 70% faster than the MPI-based implementation. In molecular dynamics application NAMD, the performance is also considerably improved by as much as 18%.

Computer Science↗
DOE OSTI2012

Designing Scalable PGAS Communication Subsystems on Cray Gemini Interconnect

The Cray Gemini Interconnect has been recently introduced as a next generation network architecture for building multi-petaflop supercomputers. Cray XE6 systems including LANL Cielo, NERSC Hopper, ORNL Titan and proposed NCSA BlueWaters leverage the Gemini Interconnect as their primary Interconnection network. At the same time, programming models such as the Message Passing Interface (MPI) and Partitioned Global Address Space (PGAS) models such as Unified Parallel C (UPC) and Co-Array Fortran (CAF) have become available on these systems. Global Arrays is a popular PGAS model used in a variety of application domains including hydrodynamics, chemistry and visualization. Global Arrays uses Aggregate Re- mote Memory Copy Interface (ARMCI) as the communication runtime system for Remote Memory Access communication. This paper presents a design, implementation and performance evaluation of scalable and high performance communication subsystems on Cray Gemini Interconnect using ARMCI. The design space is explored and time-space complexities of commu- nication protocols for one-sided communication primitives such as contiguous and uniformly non-contiguous datatypes, atomic memory operations (AMOs) and memory synchronization is presented. An implementation of the proposed design (referred as ARMCI-Gemini) demonstrates the efficacy on communication primitives, application kernels such as LU decomposition and full applications such as Smooth Particle Hydrodynamics (SPH) application.

ARMCI↗
DOE OSTI2014

Fan-In Communications On A Cray Gemini Interconnect

Using the Cray Gemini interconnect as our platform, we present a study of an important class of communication operations the fan-in communication pattern. By its nature, fan-in communications form hot spots that present significant challenges for any interconnect fabric and communication software stack. Yet despite the inherent challenges, these communication patterns are common in both applications (which often perform reductions and other collective operations that include fan-in communication such as barriers) and system software (where they assume an important role within parallel file systems and other components requiring high-bandwidth or low-latency I/O). Our study determines the effectiveness of differing clientserver fan-in strategies. We describe fan-in performance in terms of aggregate bandwidth in the presence of varying degrees of congestion, as well as several other key attributes. Comparison numbers are presented for the Cray Aries interconnect. Finally, we provide recommended communication strategies based on our findings.

Jones, Terry R↗
DOE OSTI2014

An Evaluation of One-Sided and Two-Sided Communication Paradigms on Relaxed-Ordering Interconnect

The Cray Gemini interconnect hardware provides multiple transfer mechanisms and out-of-order message delivery to improve communication throughput. In this paper we quantify the performance of one-sided and two-sided communication paradigms with respect to: 1) the optimal available hardware transfer mechanism, 2) message ordering constraints, 3) per node and per core message concurrency. In addition to using Cray native communication APIs, we use UPC and MPI micro-benchmarks to capture one- and two-sided semantics respectively. Our results indicate that relaxing the message delivery order can improve performance up to 4.6x when compared with strict ordering. When hardware allows it, high-level one-sided programming models can already take advantage of message reordering. Enforcing the ordering semantics of two-sided communication comes with a performance penalty. Furthermore, we argue that exposing out-of-order delivery at the application level is required for the next-generation programming models. Any ordering constraints in the language specifications reduce communication performance for small messages and increase the number of active cores required for peak throughput.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Understanding the Impact of Interconnect Failures on System Operation

Hardware failures are inevitable on large high performance computing systems. Faults or performance degradations in the high-speed network can reduce the entire system s performance. Since the introduction of the Gemini interconnect, Cray systems have become resilient to many networking faults that were fatal in their previous generation systems. These new network reliability and resiliency features have enabled higher uptimes on Cray systems by allowing them to continue running with reduced network performance. Oak Ridge National Laboratory has developed a set of user-level diagnostics that stresses the high-speed network and searches for components that are not performing as expected. Nearest-neighbor bandwidth tests check every network chip and network link in the system. Additionally, performance counters stored on the network ASIC s memory mapped registers (MMRs) are used to better understand the state of the network. Applications have also been characterized under various suboptimal network conditions to better understand what impact network problems have on user codes.

Cray↗
DOE OSTI2012

Optimizing fine-grained communication in a biomolecular simulation application on Cray XK6

Achieving good scaling for fine-grained communication intensive applications on modern supercomputers remains challenging. In our previous work, we have shown that such an application -- NAMD -- scales well on the full Jaguar XT5 without long-range interactions; Yet, with them, the speedup falters beyond 64K cores. Although the new Gemini interconnect on Cray XK6 has improved network performance, the challenges remain, and are likely to remain for other such networks as well. We analyze communication bottlenecks in NAMD and its CHARM++ runtime, using the Projections performance analysis tool. Based on the analysis, we optimize the runtime, built on the uGNI library for Gemini. We present several techniques to improve the fine-grained communication. Consequently, the performance of running 92224-atom Apoa1 with GPUs on TitanDev is improved by 36%. For 100-million-atom STMV, we improve upon the prior Jaguar XT5 result of 26 ms/step to 13 ms/step using 298,992 cores on Jaguar XK6.

Sun, Yanhua↗
DOE OSTI2012

Optimizing Fine-grained Communication in a Biomolecular Simulation Application on Cray XK6

Achieving good scaling for fine-grained communication intensive applications on modern supercomputers remains challenging. In our previous work, we have shown that such an application NAMD scales well on the full Jaguar XT5 without long-range interactions; Yet, with them, the speedup falters beyond 64K cores. Although the new Gemini interconnect on Cray XK6 has improved network performance, the challenges remain, and are likely to remain for other such networks as well. We analyze communication bottlenecks in NAMD and its CHARM++ runtime, using the Projections performance analysis tool. Based on the analysis, we optimize the runtime, built on the uGNI library for Gemini. We present several techniques to improve the fine-grained communication. Consequently, the performance of running 92224-atom Apoa1 with GPUs on TitanDev is improved by 36%. For 100-million-atom STMV, we improve upon the prior Jaguar XT5 result of 26 ms/step to 13 ms/step using 298,992 cores on Jaguar XK6.

Sun, Yanhua↗
DOE OSTI2012

Exploiting Atomic Operations for Barrier on Cray XE/XK Systems

Barrier is a collective operation used by many scientific applications and parallel libraries for synchronization. Typically, a Barrier operation is implemented by exchanging a short data message that requires demultiplexing, thereby adding undesired latency to the operation. In this work, we reduce the latency of Barrier operations for Cray XE/XK systems by leveraging the atomic operations provided by the Gemini interconnect, tailoring algorithms to utilize these capabilities, and utilizing a hierarchical design to arrive at an efficient implementation. Our micro-benchmark evaluation shows that for a 4,096 process Barrier operation, the atomic-operations-based Barrier outperforms the data exchange Barrier by 52% and the native Barrier by 111%.

Atomic Operations↗
DOE OSTI2014

A performance comparison of current HPC systems: Blue Gene/Q, Cray XE6 and InfiniBand systems

We present here a performance analysis of three of current architectures that have become commonplace in the High Performance Computing world. Blue Gene/Q is the third generation of systems from IBM that use modestly performing cores but at large-scale in order to achieve high performance. The XE6 is the latest in a long line of Cray systems that use a 3-D topology but the first to use its Gemini interconnection network. InfiniBand provides the flexibility of using compute nodes from many vendors that can be connected in many possible topologies. The performance characteristics of each vary vastly, and the way in which nodes are allocated in each type of system can significantly impact on achieved performance. In this work we compare these three systems using a combination of micro-benchmarks and a set of production applications. In addition we also examine the differences in performance variability observed on each system and quantify the lost performance using a combination of both empirical measurements and performance models. Our results show that significant performance can be lost in normal production operation of the Cray XE6 and InfiniBand Clusters in comparison to Blue Gene/Q.

Application analysis↗
DOE OSTI2013

Comparing the Performance of Blue Gene/Q with Leading Cray XE6 and InfiniBand Systems

Abstract—Three types of systems dominate the current High Performance Computing landscape: the Cray XE6, the IBM Blue Gene, and commodity clusters using InfiniBand. These systems have quite different characteristics making the choice for a particular deployment difficult. The XE6 uses Cray’s proprietary Gemini 3-D torus interconnect with two nodes at each network endpoint. The latest IBM Blue Gene/Q uses a single socket integrating processor and communication in a 5-D torus network. InfiniBand provides the flexibility of using nodes from many vendors connected in many possible topologies. The performance characteristics of each vary vastly along with their utilization model. In this work we compare the performance of these three systems using a combination of micro-benchmarks and a set of production applications. In particular we discuss the causes of variability in performance across the systems and also quantify where performance is lost using a combination of measurements and models. Our results show that significant performance can be lost in normal production operation of the Cray XT6 and InfiniBand Clusters in comparison to Blue Gene/Q.

-High Performance Computing, Performance Evaluatio↗
DOE OSTI2021

Study of interconnect errors, network congestion, and applications characteristics for throttle prediction on a large scale HPC system

Today’s High Performance Computing (HPC) systems contain thousand of nodes which work together to provide performance in the order of petaflops. The performance of these systems depends on various components like processors, memory, and interconnect. Among all, interconnect plays a major role as it glues together all the hardware components in an HPC system. A slow interconnect can impact a scientific application running on multiple processes severely as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks a study that explores different interconnect errors, congestion events and applications characteristics on a large-scale HPC system. In our previous work, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors, and congestion events. In this work, we first show how congestion events can impact application performance. We then investigate application characteristics interaction with interconnect errors and network congestion to predict applications encountering congestion with more than 90% accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

OpenSHMEM Reference Implementation using UCCS-uGNI Transport Layer. In: PGAS '14 Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models, Article No. 18

OpenSHMEM is a library interface implementation and specification that enables the implementation of the Partitioned Global Address Space (PGAS) model. It exports modern RDMA network functionality and communication semantics to applications very efficiently. There are many closed source implementations of OpenSHMEM for modern RDMA interconnects such as InfiniBand and Cray's Gemini and Aries. Given the important role that Cray systems play in HPC, in this paper, we present an open source implementation of OpenSHMEM for Cray XE/XK/XC systems. To implement OpenSHMEM, we use the uGNI interface. uGNI is a generic interface that is designed for multiple programming models. The interface fits well the goal of UCCS. Having OpenSHMEM with UCCS-uGNI allows usage of the same implementation over multiple interconnects. This also translates into many advantages that come with common code such as resource sharing, increasing productivity because of less code maintenance, etc. Preliminary results show that OpenSHMEM-UCCS performs comparable to state-of-the-art Cray SHMEM for Put, Get, and AMO operations.

Janjusic, Tomislav↗
DOE OSTI2023

Architecture for Co-Simulation of Transportation and Distribution Systems with Electric Vehicle Charging at Scale in the San Francisco Bay Area

This work describes the Grid-Enhanced, Mobility-Integrated Network Infrastructures for Extreme Fast Charging (GEMINI) architecture for the co-simulation of distribution and transportation systems to evaluate EV charging impacts on electric distribution systems of a large metropolitan area and the surrounding rural regions with high fidelity. The current co-simulation is applied to Oakland and Alameda, California, and in future work will be extended to the full San Francisco Bay Area. It uses the HELICS co-simulation framework to enable parallel instances of vetted grid and transportation software programs to interact at every model timestep, allowing high-fidelity simulations at a large scale. This enables not only the impacts of electrified transportation systems across a larger interconnected collection of distribution feeders to be evaluated, but also the feedbacks between the two systems, such as through control systems, to be captured and compared. The findings are that with moderate passenger EV adoption rates, inverter controls combined with some distribution system hardware upgrades can maintain grid voltages within ANSI C.84 range A limits of 0.95 to 1.05 p.u. without smart charging. However, EV charging control may be required for higher levels of charging or to reduce grid upgrades, and this will be explored in future work.

24 POWER TRANSMISSION AND DISTRIBUTION↗