Search NASA⌕ Search

SEARCH · Search NASA

Results for “interconnected network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Securing Distributed Energy Resource Integration

The penetration of distributed energy resources (DER) is growing at much higher rates than predicted 20 years ago. Far from being used only in residential settings, DER are now installed on distribution and transmission circuits. In this position, they do not have the same properties as traditional generators and are more flexible in many cases. The growing penetration and range of uses for DER motivate the need to reliably and safely integrate them into the grid. Operators must be able to rely on them not only for normal operation, but also during abnormal conditions like black starts or adverse cyber scenarios. To that end, we study the communications, device interfaces, and potential consequences of DER operation under abnormal and adversarial conditions. The weaknesses of communications networks are studied based on the industrial protocols used, and the benefits of security features are examined. The device interfaces are found to be vulnerable to attack based on the requirements in the IEEE-1547 standard for DER interconnection and interoperability, which is expected to be adopted in the next ten years. In addition to exploring the requirements of the standard, we show that these vulnerabilities and others do exist and can be used maliciously in a modern storage system DER. Consequences of these vulnerabilities range from exacerbated grid instability, to simultaneous loss of large portions of DER penetration, to physical damage to inverters or DER themselves and other sensitive equipment. We tie these outcomes to specific attacker actions in an effort to give operators a better threat intelligence view that allows them to prioritize mitigations. Finally, we discuss mitigations that could prevent many of the adversarial scenarios described. Some solutions can be added to existing infrastructure, while others may require longer term planning for grid modernization with consideration for security.

25 ENERGY STORAGE↗

Frontier (HPE Cray EX) Exascale Supercomputer at the Oak Ridge Leadership Computing Facility

Frontier is the HPE Cray EX exascale supercomputer deployed and operated by the Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National Laboratory (ORNL). Frontier is designed for large-scale modeling, simulation, and AI workloads and is built from HPE Cray EX system architecture with AMD CPUs and AMD Instinct GPU accelerators connected by the HPE Slingshot interconnect. System composition (representative production configuration): Frontier is composed of approximately 74 cabinets with 128 compute nodes per cabinet (~9,400 compute nodes total). Each compute node contains one 64-core AMD EPYC CPU and four AMD Instinct MI250X GPUs. Nodes are connected using HPE Slingshot (Slingshot-200 class) networking with multiple NIC ports per node providing high injection bandwidth. Frontier is connected to the Orion parallel file system (multi-tier Lustre) providing a large, center-wide high-performance storage namespace. Operational context: Frontier entered public prominence as the first system to reach No. 1 on the TOP500 list in May 2022 (HPL benchmark), establishing the first widely recognized exascale-era performance milestone. The system supports DOE Office of Science mission workloads and enables leadership-class computational science and AI for open science users.

AMD EPYC↗

Evaluation of PV Module Packaging Strategies of Monofacial and Bifacial PERC Using Degradation Pathway Network Modeling

As the PV industry is rapidly expanding, it is important to thoroughly investigate the long-term impact of packaging strategies on the performance of PV modules. In this study, the variants in sets differ on the basis of manufacturer (A/B), encapsulant (EVA/POE), rear encapsulant (UV-cutoff/opaque/transparent), module architecture (GB/DG) and cell type (monofacial/bifacial). The minimodules were exposed for 2520 hours in modified damp heat, with or without full spectrum light. Every 504 hours, stepwise electrical characterization techniques were employed to track changes in minimodules. Degradation pathway modeling using network structural equation modeling was employed to study pairwise relationships between variables and service lifetime prediction in minimodules. Through this study, differences in quality control are identified in minimodules made by different manufacturers. Minimodules with UV-cutoff rear encapsulant show relatively better stability, whereas the ones with opaque rear encapsulant show greater power loss. In addition, GB having UV-cutoff rear encapsulation and GB with POE having opaque rear encapsulation were identified to be stable as they lack a best model fit. Here, the primary power loss mechanism in degrading variants is interconnect corrosion.

14 SOLAR ENERGY↗

Real-time Implementation of Grid Code Compliant Grid Edge Energy Management System

Integrated distributed energy resources (DER) in a distribution system need to follow grid codes to avoid violations that result in DER/circuit segment disconnection. To comply with grid code requirements at the grid edge level, network constrained grid edge energy management system (EMS) can be deployed. The objective of grid edge EMS is to provide economic solution for active and reactive power DER setpoints at each dispatch interval and ensure voltage regulation to support secure interconnection of the grid edge segment to the distribution system with multiple inverter based DER units. In this work, real-time simulation of grid code compliant grid edge EMS is deployed in a realistic feeder circuit segment. For real-time simulation, communication between the grid edge EMS and DERs is done exploiting IEC 61850-7-420. It enables interoperability among different DERs and grid edge EMS. No prior art has deployed IEC 61850-7-420 GOOSE communication protocol for grid edge EMS. Conversion of IEC 61850 GOOSE messages to Modbus communication protocol is also performed to communicate with grid edge EMS in commodity-off the shelf embedded boards in this work. The real-time simulation in OPAL-RT real-time digital simulator shows the out-performance of grid edge EMS by reducing the voltage violation in the distribution circuit.

Energy management system↗

Machine Learning-Based PV Reserve Determination Strategy for Frequency Control on the WECC System

Frequency control from photovoltaic (PV) power plants has great potential to address the frequency response challenge of the power system with high penetrations of renewable generation. Using model-based approaches to determine the optimal PV headroom reserve, however, requires significant online computation and is intractable for an interconnection level system. This paper proposes a machine learning based strategy, that is suitable for real-time operation, to determine the optimal PV reserve for frequency control. The proposed machine learning algorithm is trained and tested on 1,987 offline simulations of a 60% renewable penetration Western Electricity Coordinating Council (WECC) system. Furthermore, the proposed reserve determination strategy is applied on a realistic 1-day operation profile of the WECC system and demonstrates a savings of more than 40% PV headroom compared to a conservative approach. It is evident that the proposed strategy can efficiently and effectively determine the optimal PV frequency control reserve for realistic interconnection systems.

frequency control↗

Network Constraints Consideration for Grid-Edge Energy Management System

Increased deployment of distributed energy resources (DER) in distribution system is bringing need for enhanced grid intelligence, control, and flexibility. This is significant at the edge of the grid where DERs, loads or microgrids are located. Integrated DERs in a distribution system need to follow grid codes to avoid violations that results in DER disconnection. To comply with grid code requirements (e.g. IEEE 1547-2018) at the grid edge level, network constrained grid edge energy management system (GEEMS) is proposed in this paper. The objective of GEEMS is to provide economic solution for active and reactive power dispatch set-points at each interval and ensure voltage regulation to support secure interconnection of the grid edge segment to the distribution system. To evaluate the proposed GEEMS framework, four DERs are included into IEEE 13 bus system. GEEMS outperforms the existing economic dispatch-based energy management system by reducing the voltage violation.

Electric vehicle depot charging station↗

Understanding Regional Inertia Dynamics in CAISO from Real Grid Disturbances

The shift from synchronous generators to inverter-based resources has caused power system inertia to be unevenly distributed across power grids. As a result, certain grid regions are more vulnerable to high rate-of-change of frequency (RoCoF) during disturbances. This paper presents a measurement-based framework for estimating grid inertia in CAISO (California Independent System Operator) region using real disturbance-driven frequency data from the Frequency Monitoring Network (FNET/GridEye). By analyzing confirmed disturbances from 2013 to 2024, we identify trends in regional inertia and frequency dynamics, highlighting their relationship with renewable generation and the evolving duck curve. Regional RoCoF values were up to six times higher than interconnection-wide values, coinciding with declining inertia. Recent recovery in inertia is attributed to the increased deployment of battery energy storage systems with synthetic inertia capabilities. These findings underscore the importance of regional inertia monitoring, strategic resource planning, and adaptive operational practices to ensure grid reliability amid growing renewable integration.

Dulal, Saurav [University of Tennessee, Knoxville ↗

Power grid frequency prediction using spatiotemporal modeling

Understanding power system dynamics is essential for interarea oscillation analysis and the detection of grid instabilities. The FNET/GridEye is a GPS-synchronized wide-area frequency measurement network that provides an accurate picture of the normal real-time operational condition of the power system-dynamics, giving rise to new and intricate spatiotemporal patterns of power loads. We propose to model FNET/GridEye grid frequency data from the U.S. Eastern Interconnection with a spatiotemporal statistical model. We predict the frequency data at locations without observations, a critical need during disruption events where measurement data are inaccessible. Spatial information is accounted for either as neighboring measurements in the form of covariates or with a spatiotemporal correlation model captured by a latent Gaussian field. Finally, the proposed method is useful in estimating power system dynamic response from limited phasor measurements and holds promise for predicting instability that may lead to undesirable effects such as cascading outages.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Membrane synthesis via in-situ pore formation in silica gels through dynamic miscibility with soybean oil

The dynamic immiscibility between vegetable oil and organic silica precursors during sol-gel synthesis was used to tailor the microstructure of gels made with tetraethoxysilane (TEOS) or tetraethoxysilane+methyltrimethoxysilane (MTMS). Vegetable oil was investigated as an environmentally friendly and cheaper alternative to many other porogens. Oil concentrations of 9.09 v/v% and 33.33 v/v% were studied. For both TEOS and TEOS+MTMS gels, the miscibility of the oil and precursor decreases as the silica gel network forms, leading to oil-rich regions within the silica matrix. Gels containing MTMS maintained partial miscibility with the oil during gel formation due to the non-hydrolysable methyl groups on the MTMS. The pore structure of MTMS gels appeared more interconnected compared to the isolated pores observed in gels made with only TEOS. To create membranes with hierarchical pore structures, solvent exchanges were performed on gels or dried gels (i.e., xerogels) were calcined. The effect of the methyl group was also observed in the results of the time sweep rheology, as the addition of oil to the TEOS+MTMS gels caused an increase in the time to form the spanning gel. The trend was the opposite for oil additions to TEOS gels, which only contained polar groups. Xerogel membranes made with TEOS+MTMS had higher specific surface areas (SSAs) than those made with TEOS. For oil concentrations of 33.33 v/v%, xerogels made from TEOS+MTMS had an average SSA of 1068 + 113 m2 g-1, an order of magnitude higher than the average SSA of 103 + 6 m2 g-1 for xerogels made with TEOS.

sol-gel, dynamic immiscibility, oil-alkoxide inter↗

Leveraging public AI tools to explore systems biology resources in mathematical modeling

Predictive mathematical modeling is an essential part of systems biology and is interconnected with information management. Systems biology information is often stored in specialized formats to facilitate data storage and analysis. These formats are not designed for easy human readability and thus require specialized software to visualize and interpret results. Therefore, comprehending modeling and underlying networks and pathways is contingent on mastering systems biology tools, which is particularly challenging for users with no or little background in data science or system biology. To address this challenge, we investigated the usage of public Artificial Intelligence (AI) tools in exploring systems biology resources in mathematical modeling. We tested public AI’s understanding of mathematics in models, related systems biology data, and the complexity of model structures. Our approach can enhance the accessibility of systems biology for non-system biologists and help them understand systems biology without a deep learning curve.

59 BASIC BIOLOGICAL SCIENCES↗

Graph-Based Representations and Applications to Process Simulation

Rapid and robust convergence of a process flowsheet is critical to enable large-scale simulations that address core scientific questions related to process design, optimization, and sustainability. However, due to the highly coupled and nonlinear nature of chemical processes, efficiently solving a flowsheet remains a challenge. In this work, we show that graph representations of the underlying physical phenomena in unit operations may help identify potential avenues to systematically reformulate the network of equations and enable more robust topology-based convergence of flowsheets. To this end, we developed graph abstractions of the governing equations of vapor-liquid and liquid-liquid equilibrium separation equipment. These graph abstractions consist of a mesh of interconnected variable nodes and equation nodes that are systematically generated through PhenomeNode, a new open-source library in Python developed in this study. We show that partitioning the graph into separate mass, energy, and equilibrium subgraphs can help decouple nonlinearities and guide decomposition algorithms. By employing the graph abstraction on an industrial separation process for separating glacial acetic acid from water, we implemented a new block decomposition scheme in BioSTEAM and demonstrated that this can accelerate convergence over a traditional sequential modular approach.

Distillation↗

Adaptive Protection and Validated Models to Enable Deployment of High Penetrations of Solar PV (PV-MOD)

The availability and validation of various PV models in commercial tools differ, with some models not yet thoroughly validated for advanced inverter functionalities and reliable performance under weak system conditions. Many existing models do not fully incorporate new inverter control functions, which can affect system stability. The increasing deployment of solar PV and other inverter-based resources (IBRs), including distributed energy resources (DERs), is influencing the reliable operation of protection schemes in distribution systems and microgrids. Emerging adaptive protection schemes (APS) offer new opportunities for protecting these systems during varying configurations and DER operating conditions, though their demonstration and validation remain limited. Adaptive protection schemes face similar challenges, as they are typically designed for specific configurations. There is a growing need for tools and methodologies to streamline the deployment of adaptive protection for safe and reliable DER integration. The project main objective was to develop and validate high-fidelity generic models of solar PV facilities for stability, protection, EMT, and QSTS analyses. This objective was achieved, and these models can now be integrated into commercial software tools, enabling utilities, vendors, and developers to study high-penetration PV systems more confidently. The project also demonstrated advanced applications of these models, including the design and deployment of adaptive protection schemes in high-penetration field applications and microgrids, supporting grid safety and reliability. Several milestones were reached by the end of the project. A sophisticated inverter test plan was developed, and inverters representative of the North American marketplace were selected. EPRI and NREL tested various inverters, conforming to IEEE standards. Improvements were made to existing generic models of IBR units, IBR plants, and aggregated feeders for various analyses. The first generic electromagnetic transient (EMT) model for a solar PV plant was developed, conforming to IEEE Std 2800™-2022 and validated against laboratory measurements of a 2.2 MVA large-scale battery energy storage system (BESS) inverter. That model was then used to produce reference responses illustrating examples of validated and verified IBR plant models that pass or fail tests for technical minimum capability and performance as specified in the IEEE standard. The developed, tested, and validated generic models can be used for transmission planning, stability assessments, expansion planning, and evaluating potential future IBR interconnection requirements. They can also support interconnection screens and conformity assessments of IBR plants, including solar PV. The project significantly contributed to the ongoing standardization and model-based representation and verification of IBR responses. The project further addressed challenges of common distribution protection schemes with increasing deployment of DER by developing, validating, and demonstrating adaptive protection schemes (APS) that can improve the reliable and safe integration of DER into distribution systems. New APS were designed using improved DER models for three common distribution systems: a radial feeder, a meshed network, and a microgrid. Modeling and hardware-in-the-loop (HIL) testing of the APS were conducted, successfully showing their effectiveness and selectivity. Proof-of-concept field demonstration was achieved for two APS, i.e., one on a radial feeder and another one in a microgrid. Field demonstration could not be achieved for the APS on a meshed network, primarily due apprehension of one utility partner and also due to limited access to the protective algorithms in the network protectors. Guidelines developed from the lessons learned in the project lay out the general process followed in the design, installation, and commissioning of APS for various distribution systems. Distribution utility partners’ apprehension about field demonstration of the new APS were addressed—with varying success—by taking a stepped risk-management approach of modeling of a wide range of sensitivities first, performing in-depth proof-of-concept testing in the laboratory including HIL next, and finally deliberately implementing and commissioning the actual protection equipment and algorithms into parts of—or in parallel operation to—the three real distribution systems. Future work should include pilot projects that further show the acceptable performance of the developed APS before these schemes be rolled out more widely. Inclusion of both utility and original equipment manufacturers (OEMs) in future projects could increase chances of successful field demonstration. Despite challenges in achieving the field demonstration goal of the project for all three APS, the research significantly contributed to the innovation of adaptive protection solutions for scalable and reliable DER integration into distribution systems. This project significantly enhances the understanding of the impact of using appropriate inverter models on distribution and transmission (T&D) systems. By addressing the limitations of existing generic models, the project introduces high-fidelity models for stability, protection, electromagnetic transient (EMT), and quasi-static time series (QSTS) analyses. These models, integrated into commercial software tools, enable utilities, vendors, and developers to confidently study high-penetration PV systems. The project also demonstrates advanced applications, including adaptive protection schemes (APS) for distribution systems and microgrids, ensuring grid safety and reliability. The technical effectiveness and economic feasibility of the methods are evident through the development and validation of sophisticated inverter test plans and the selection of representative inverters. Testing by EPRI and NREL on retail, commercial, and utility-scale inverters, conforming to IEEE standards, underscores the robustness of the models. Improvements to existing generic models for various analyses further enhance their validity and applicability. The project also identifies gaps in common distribution protection schemes and designed new APS using improved DER models, demonstrating their effectiveness through modeling and hardware-in-the-loop (HIL) testing. The project’s benefits to the public are manifold. By advancing the standardization and model-based representation of IBR response, it supports transmission planning, stability assessments, and future IBR interconnection requirements. The generic models can facilitate better communication between transmission planners and developers, supporting expected IBR plant capability and performance. Additionally, the development of APS for radial feeders, meshed networks, and microgrids supports the integration of distributed energy resources (DERs) into distribution systems, enhancing grid reliability and safety. The project’s emphasis on thorough testing and simplicity in design ensures practical and scalable solutions for DER integration.

14 SOLAR ENERGY↗

Through the lens of bioenergy crops: advances, bottlenecks, and promises of plant engineering

Advances in engineering of bioenergy crops were driven over the past years by adapting technological breakthroughs and accelerating conventional applications but also exposed intriguing challenges. New tools revealed rich interconnectivity in the exponentially growing and dynamic 'big' omics data' of metabolomes, transcriptomes, and genomes at previously inaccessible magnitude (global, cross-species, meta-) and resolution (single cell). Insights enabled fresh hypotheses and stimulated disciplines such as functional genomics with discovery of broad regulatory networks and their determinants, that is, DNA parts, including promoters, regulatory elements, and transcription factors. Their rational design, assembly into increasingly complex blueprints, and installation into diverse chassis is an existing frontier that may benefit from emerging technologies to address bottlenecks. Interweaving nature-inspired to fully synthetic parts has already allowed building of fine-tuned regulatory circuits, or new-to-nature metabolic routes insulated from the biological context of the chassis species. Similarly, developments and the evolving need for unifying principles in plant transformation and species-agnostic technologies highlight future opportunities for engineering the next generation of bioenergy plants.

60 APPLIED LIFE SCIENCES↗

Coupled Investigation of Fracture Permeability Impact on Reservoir Stress and Seismic Slip Behavior (Final Technical Report)

Enhanced Geothermal Systems (EGS) produce clean energy by circulating fluid through hot rock deep underground and bringing that heat to the surface to generate electricity. For this process to work reliably, fluids must be able to move efficiently through networks of natural or engineered fractures in the rock. Enhancing and maintaining subsurface permeability over time is essential for sustainable energy production. However, fluid injection changes the underground temperature, pressure, rock stress, and chemistry, which can alter permeability and sometimes trigger earthquakes. Predicting these interconnected processes remains a key challenge. To address this, we combined high-temperature laboratory experiments with high-fidelity simulations to better understand how fractures in geothermal reservoirs evolve over time. Our experiments measured how fractures respond to stress, slip, slip rate, and chemical reactions under geothermal conditions. These data were integrated into coupled thermal-hydrological-mechanical-chemical and earthquake (THMC+E) models tailored to the Utah FORGE site. The validated modeling framework improves predictions of reservoir performance and seismic response and helps guide operational decisions. This work reduces technical risk and strengthens the scientific foundation needed to make geothermal energy a reliable and scalable clean energy resource.

15 GEOTHERMAL ENERGY↗

FORCE Development Status Update: Vertical Integration and Benchmarking of System Dynamics

Recent efforts to establish effective models for grid energy analysis, especially given the increase in variable renewable energy (VRE) sources and the economic challenges faced by traditional nuclear energy, have generated new technological considerations. One effort to improve the economic viability of nuclear power involves investigating integrated energy systems (IES) which include secondary energy systems that introduce flexibility and secondary market possibilities to existing and perceived future nuclear energy generation technologies. To analyze the technical and economic potential of IES, the Framework for Optimization of Resources and Economics (FORCE) tool suite was developed through a collaboration among national laboratories. Within the FORCE tool suite, the Holistic Energy Resource Optimization Network (HERON) provides algorithms for analyzing the long-term viability of potential IES technologies, while HYBRID provides algorithms and models to achieve high-resolution analysis of coupling physics over a short time period. Continued maturing of the FORCE tool suite requires further interconnections between the various tools in the suite in order to ensure consistent analysis. Analyses performed by applying HYBRID to transient process modeling should be easily harvestable as inputs to HERON analyses. The first item in this status update is a demonstration of an automated data pipeline for loading data from HYBRID into HERON analyses. Application of HYBRID results to HERON, as part of using the FORCE tool suite, relies on robust modeling assumptions for the various models included in HYBRID. The second result of this status update is the benchmarking and validation of cost and operational data, with a particular focus on natural gas energy generators. These generators are benchmarked with a focus on contrasting them with proposed thermal energy storage (TES) technologies.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

FORCE Regression Testing

Via programs including the Light Water Reactor Sustainability and Integrated Energy Systems, the U.S. Department of Energy has invested in the Framework for Optimization of ResourCes and Economics (FORCE) software framework (Idaho National Laboratory 2024a) for the technical and economic analysis of nuclear-integrated energy systems (IES). Nuclear IES expand the use of nuclear from traditional baseload electricity generation to a flexible and adaptive source of combined heat and power. Nuclear heat can be used in the production of a variety of energy currencies such as hydrogen and ammonia as well as other heat applications including water desalination and district heating. FORCE is designed with the intent to provide interconnected analysis tools that enable the accurate technical and economic assessment of specific nuclear IES configurations for individual energy markets. FORCE consists of three main analysis pathways: HYBRID (Idaho National Laboratory 2024b), which contains high-resolution physical models for IES; Holistic Energy Resource Optimization Network (HERON) (Idaho National Laboratory 2024c), which analyzes IES long-term economic viability; and Optimization of Real-time Capacity Allocation (ORCA) (Idaho National Laboratory 2024d), designed for real-time control of IES via digital twins and optimal decision making, including autonomous and remote operation research. Development of the FORCE ecosystem is guided by three pillars: capability, which assures that the computational requirements of IES analysis are met by the software tools; reliability, which provides for consistent code performance and expected behaviors; and accessibility, which lowers the barrier to entry for using the software and accelerates analysis by users beyond the FORCE primary developers. Reliability of the FORCE ecosystem is established according to the American Nuclear Society?s Nuclear Quality Assurance (NQA-1) program [American Society of Mechanical Engineers 1982], with specific levels of software quality assurance (SQA) within NQA-1 applied to each software tool in FORCE. As the tools within FORCE have matured, some integration algorithms to accurately connect the software tools for holistic analysis have been developed and deployed within the FORCE software repository. In accordance with NQA-1 standards, regression tests are required to guarantee the software performs consistently even when new capabilities are added to the software. In this report, we document the deployment of both unit tests, which test the consistent behavior of small pieces of the FORCE code base, as well as integration tests, which test the consistent performance of full use cases for the FORCE integration algorithms. We further document the encapsulation of these tests within a test harness, which collectively checks for each successful test completion on demand. Finally, we document the automation of the test harness using GitHub Actions [GitHub 2024], which require all tests succeed before any new capability or other changes can be added to the FORCE integration software

97 MATHEMATICS AND COMPUTING↗

P38 heterogeneous multi-tiled system with support for message queues (MoSAIC) v0.1

The proposed system is written in the hardware description language (HDL) verilog targeting an FPGA board. It is intended as a testbed to explore architecture tradeoffs in multi-tiled heterogeneous architectures. Although we target FPGAs, the system can be implemented as a monolithic SoC or a package comprised of many chiplets that are interconnected in the same package using a NoC. The proposed NoC is lightweight and follows an axi-lite interface. The endpoints of the NoC are a heterogeneous mix of "tiles" as endpoints that are general purpose processors, fixed function accelerators, and programmable accelerators. We assume that the network interfaces for the NoC endpoints are all addressable in a global name-space in that they represent an address range (for memory addresses) or a range of unique identifiers that are associated with each individual tile. This makes the functionality abstract from the standpoint of the NoC design details. Message queues offer a direct inter-processor interface between peer general purpose cores and diverse accelerators that comprise an SoC. Although they share the same NoC infrastructure for inter-tile communication within an SoC or SiP, the hardware message queues bypass the memory hierarchy and thus do not pollute the memory state or invoke the cache coherence mechanism.

Gonzalez, LouisaPatricia↗

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF’s SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink’s topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for a multi-GPU application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as an application’s overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

Performance Evaluation, GPU, Interconnect, NUMA, P↗