SEARCH · Search NASA

Results for “computer performance”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,729 records · Page 96

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

A Study of Parallel Scalability and Dynamic Workload Balancing in GlennICE

The Glenn Icing Computational Environment (GlennICE) is a computational tool designed to calculate ice growth on complex three-dimensional geometries. It utilizes user-supplied computational fluid dynamics solutions for the geometry of interest. Key developments include advancements in convergence of collection efficiency, trajectory optimization, and refinement methodology. These improvements have significantly enhanced GlennICE’s efficiency for practical engineering applications. A recent study focused on benchmarking GlennICE’s scalability in a parallel environment using static scheduling. Findings indicated a potential twofold increase in efficiency through workload balance enhancements. This paper presents an analysis of the solver’s new workload balancing improvements, incorporating shared memory and dynamic scheduling routines. Results demonstrate a highly efficient and consistent algorithm across high-performance computing clusters.

Computational Icing↗

NASA NTRS2000

Supercomputing Aspects for Simulating Incompressible Flow

The primary objective of this research is to support the design of liquid rocket systems for the Advanced Space Transportation System. Since the space launch systems in the near future are likely to rely on liquid rocket engines, increasing the efficiency and reliability of the engine components is an important task. One of the major problems in the liquid rocket engine is to understand fluid dynamics of fuel and oxidizer flows from the fuel tank to plume. Understanding the flow through the entire turbo-pump geometry through numerical simulation will be of significant value toward design. One of the milestones of this effort is to develop, apply and demonstrate the capability and accuracy of 3D CFD methods as efficient design analysis tools on high performance computer platforms. The development of the Message Passage Interface (MPI) and Multi Level Parallel (MLP) versions of the INS3D code is currently underway. The serial version of INS3D code is a multidimensional incompressible Navier-Stokes solver based on overset grid technology, INS3D-MPI is based on the explicit massage-passing interface across processors and is primarily suited for distributed memory systems. INS3D-MLP is based on multi-level parallel method and is suitable for distributed-shared memory systems. For the entire turbo-pump simulations, moving boundary capability and efficient time-accurate integration methods are built in the flow solver, To handle the geometric complexity and moving boundary problems, an overset grid scheme is incorporated with the solver so that new connectivity data will be obtained at each time step. The Chimera overlapped grid scheme allows subdomains move relative to each other, and provides a great flexibility when the boundary movement creates large displacements. Two numerical procedures, one based on artificial compressibility method and the other pressure projection method, are outlined for obtaining time-accurate solutions of the incompressible Navier-Stokes equations. The performance of the two methods is compared by obtaining unsteady solutions for the evolution of twin vortices behind a flat plate. Calculated results are compared with experimental and other numerical results. For an unsteady flow, which requires small physical time step, the pressure projection method was found to be computationally efficient since it does not require any subiteration procedure. It was observed that the artificial compressibility method requires a fast convergence scheme at each physical time step in order to satisfy the incompressibility condition. This was obtained by using a GMRES-ILU(0) solver in present computations. When a line-relaxation scheme was used, the time accuracy was degraded and time-accurate computations became very expensive.

Kwak, Dochan↗

DOE OSTI2020

Evaluation of Low Dose Silicon Carbide Temperature Monitors

Thermocouples are generally used to provide real-time temperature indications in instrumented tests performed at materials and test reactors. Melt wires or paint spots are often included in such tests as an independent technique of detecting peak temperatures incurred during irradiation. In addition, less expensive static capsule tests, which have no leads attached for real-time data transmission, often rely on melt wires and paint spots as a post-irradiation technique for peak temperature indication. Unfortunately, these techniques are limited in that they can only detect whether a single temperature is or is not exceeded. Silicon carbide (SiC) monitors are advantageous because a single monitor can be used to determine the peak temperature reached within a relatively broad range (200 – 800°C). Although the use of SiC monitors was proposed more than five decades ago, the ultimate performance limits of this technique are not fully understood. The Nuclear Science User Facilities (NSUF) is the United States Department of Energy Office of Nuclear Energy's only designated nuclear energy user facility. Its mission is to provide nuclear energy researchers access to world-class capabilities and to facilitate the advancement of nuclear science and technology. This mission is supported by providing access to state-of-the-art experimental irradiation testing, post irradiation examination facilities, and high performance computing capabilities as well as technical and scientific assistance for the design and execution of projects. As part of an NSUF project, low dose silicon carbide monitors were irradiated in the Belgian Reactor 2 and were then evaluated both at the SCK•CEN and at Idaho National Laboratory's High Temperature Test Laboratory to determine their peak temperature achieved during irradiation. The technical significance of this work was that the monitors were irradiated to a dose that was significantly less than recommended in published literature. This paper will discuss the evaluation process, the irradiation test, and the performance of the low dose silicon carbide temperature monitors.

36 MATERIALS SCIENCE↗

NASA NTRS1994

Dynamic file-access characteristics of a production parallel scientific workload

Multiprocessors have permitted astounding increases in computational performance, but many cannot meet the intense I/O requirements of some scientific applications. An important component of any solution to this I/O bottleneck is a parallel file system that can provide high-bandwidth access to tremendous amounts of data in parallel to hundreds or thousands of processors. Most successful systems are based on a solid understanding of the expected workload, but thus far there have been no comprehensive workload characterizations of multiprocessor file systems. This paper presents the results of a three week tracing study in which all file-related activity on a massively parallel computer was recorded. Our instrumentation differs from previous efforts in that it collects information about every I/O request and about the mix of jobs running in a production environment. We also present the results of a trace-driven caching simulation and recommendations for designers of multiprocessor file systems.

Kotz, David↗

DOE OSTI2025

Creating Apptainer Workflows with Docker-Compose-like Utilities

Creating Apptainer Workflows with Docker-Compose-like Utilities In this presentation, I will explore the utilization of a tool called process-compose, inspired by docker-compose, to create Apptainer-based services. This approach allows for easy deployment and management of fully containerized applications on High Performance Computing (HPC) systems without requiring elevated privileges. Benefits to the Ecosystem: By incorporating process-compose and Apptainer, I aim to address several key challenges in the HPC ecosystem: Simplified Workflow Management: Process-compose provides a user-friendly interface for defining and managing complex containerized application services, reducing the setup time and lowering the barrier to entry for new users. Enhanced Portability: Apptainer ensures that containerized applications can run consistently across different HPC environments, promoting greater portability and reducing compatibility issues. Process-compose is also a single binary that does not need to be installed by admin level users. Community Driven Solutions: This approach aligns with the goals of the High Performance Software Foundation (HPSF) to advance community-driven solutions. By sharing our experiences and insights, I hope to foster collaboration and innovation within the HPC community. Increased Productivity: The combination of process-compose and Apptainer streamlines the serve deployment process, allowing researchers and developers to focus more on their scientific work rather than the intricacies of system or service administration. Through this presentation, attendees will gain valuable insights into the practical implementation of containerized workflows on HPC systems, learn about the benefits of using process-compose and Apptainer, and understand how these tools can contribute to a more efficient HPC ecosystem.

97 - MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offload] and OpenACC), (3) Python (e.g., numpy, Numba, cuPy, and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by the GPT-based OpenAI Codex available in Visual Studio Code as of April 2023 to generate a vast amount of implementations given simple + + prompt variants. To quantify and compare the results, we propose a proficiency metric around the initial 10 suggestions given for each prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with the adoption and maturity of programming models. For example, OpenMP and CUDA score really high, whereas HIP is still lacking. We found that prompts from either a targeted language such as Fortran or the more general purpose Python can benefit from adding code keywords, while Julia prompts perform acceptably well for its mature programming models (e.g., Threads and CUDA.jl). We expect for these benchmarks to provide a point of reference for each programming model's community. Overall, understanding the convergence of large language models, AI, and HPC is crucial due to its rapidly evolving nature and how it is redefining human-computer interactions.

Godoy, William↗

DOE OSTI2024

Site-Projected Thermal Conductivity: Application to Defects, Interfaces, and Homogeneously Disordered Materials

With the rapid advance of high-performance computing and electronic technologies, understanding thermal conductivity in materials has become increasingly important. This study presents a novel method: the site-projected thermal conductivity that quantitatively estimates the local (atomic) contribution to heat transport, leveraging the Green–Kubo thermal transport equations. The effectiveness of this approach on disordered and amorphous graphene, amorphous silicon, and grain boundaries in silicon–germanium alloys is demonstrated. Amorphous graphene reveals a percolation behavior for thermal transport. The results highlight the potential of the method to provide new insights into the thermal behavior of materials, offering a promising avenue for materials design and performance optimization.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Advanced Simulation and Computing: ASC FY24 Implementation Plan

The DOE National Nuclear Security Administration (NNSA) Stockpile Stewardship Program (SSP) is an integrated technical program for maintaining the safety, security, and reliability of the U.S. nuclear stockpile. The SSP incorporates nuclear test data, computational modeling and simulation, and experimental facilities to advance understanding of nuclear weapons. The suite of data analyzed comes from activities including previous nuclear tests, stockpile surveillance, experimental research, and development and engineering programs. This integrated national program requires the continued use of experimental facilities and the computational capabilities to support the SSP missions. These component parts, in addition to an appropriately scaled production capability, enable NNSA to support stockpile requirements. The ultimate goal of the SSP, and thus of the Advanced Simulation and Computing (ASC) program, is to ensure that the U.S. maintains a safe, secure, and effective strategic deterrent. The ASC program is a cornerstone of the SSP, providing simulation capabilities and computational resources to support the annual stockpile assessment and certification process, study advanced nuclear weapons design and manufacturing processes, analyze accident scenarios and weapons aging, and provide the tools to enable stockpile Life Extension Programs (LEPs) and the resolution of Significant Finding Investigations (SFIs). This work requires a balance of resources, including technical staff, hardware, simulation software, and computer science solutions. The ASC program focuses on increasing the predictive capabilities in a three-dimensional (3D) simulation environment while maintaining support to the SSP. The Program continues to improve its unique tools for understanding and solving progressively more difficult stockpile problems (sufficient resolution, dimensionality, and scientific details), and quantifying critical margins and uncertainties. Resolving each issue requires increasingly difficult analyses because the aging process has progressively moved the stockpile further from the original test base. While the focus remains on the U.S. nuclear weapons program, where possible, the Program also enables the use of high-performance computing (HPC) and simulation tools to address broader national security needs, such as foreign nuclear weapon assessments and nuclear counterterrorism. The 2022 Nuclear Posture Review (NPR) calls for NNSA to “deliver a modern, adaptive nuclear security enterprise based on an integrated strategy for risk management, production-based resilience, science and technology innovation, and workforce initiatives.” Furthermore, “NNSA will establish a Science and Technology Innovation Initiative to accelerate the integration of science and technology (S&T) throughout its activities.” Executing this strategy necessitates the continued emphasis on developing and sustaining high-quality scientific and engineering staff, as well as supporting computational and experimental capabilities. These components constitute the foundation of the nuclear weapons program. The continued success of the SSP and LEPs is predicated upon the ability to credibly certify the stockpile, without a return to underground nuclear tests (UGTs). Shortly after the nuclear test moratorium entered into force in 1992, the Accelerated Strategic Computing Initiative (ASCI) was established to provide an extensive simulation capability to underpin stockpile certification. While computing and simulation have always been essential to the success of the nuclear weapons program, the program goal of ASCI was to execute NNSA’s vision of using these tools in support of the stockpile stewardship mission. The ASCI program was essential to the successful demonstration of the SSP, providing critical nuclear weapons simulation and modeling capabilities. ASCI officially evolved into the ASC program in fiscal year (FY) 2005, but the mission remains essentially the same: provide the simulation and computational capabilities that underpin the ability to maintain a safe, secure, effective nuclear weapon stockpile, without returning to underground nuclear testing. The capabilities that the ASC program provides at the national laboratories play a vital role in the nuclear security enterprise and are necessary for fulfilling the stockpile stewardship and life extension requirements outlined for NNSA. The Program develops modern simulation tools that provide insights into stockpile aging issues, provide the computational and simulation tools that enable designers and analysts to certify the current stockpile and life-extended nuclear weapons, and inform the decision-making process when any modifications in nuclear warheads or the associated manufacturing processes are deemed necessary. Furthermore, ASC is enhancing the predictive simulation capabilities that are essential to evaluate weapons effects, design experiments, and ensure test readiness. The ASC program continues to improve its unique tools to solve stockpile problems— with a focus on sufficient resolution, dimensionality, and scientific detail—to enable Quantification of Margins and Uncertainties (QMU) and to resolve the increasingly difficult analyses needed for stockpile stewardship. The needs of the Stockpile Management and Production Modernization programs (formerly Directed Stockpile Work) also drive the requirements for simulation and computational resources. These requirements include planned LEPs, stockpile support activities, and mitigation efforts against the potential for technical surprise. All of the weapons within the current stockpile are in some stage of the life extension process. The simulation and computational capabilities are crucial for successful execution of these life extensions and for ensuring NNSA can certify these life-extended weapons without conducting a UGT.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

On Defense Strategies for Recursive System of Systems Using Aggregated Correlations

We consider a class of Recursive System of Systems (RSoS), wherein systems are recursively defined and the basic systems at finest level are composed of discrete cyber and physical components. This formulation captures the models of systems that are adaptively refined to account for their varied structure, such as sites of a heterogenous distributed computing infrastructure. The components can be disrupted by cyber of physical means, and can also be suitably reinforced to survive the attacks. We characterize the disruptions at each level of recursion using aggregate failure correlation functions that specify the conditional failure probability of RSoS given the failure of an individual system at that level. At finest levels, the survival probabilities of basic systems satisfy simple product-form, first order differential conditions using the multiplier functions, which generalize conditions based on contest success functions and statistical independence of component survival probabilities. We formulate the problem of ensuring the performance of RSoS as a game between an attacker and a provider, each with a utility function composed of a survival probability term and a cost term, both expressed in terms of the number of basic system components attacked and reinforced. We derive sensitivity functions at Nash Equilibrium that highlight the dependence of survival probabilities of systems on cost terms, correlation functions, and their partial derivatives. We apply these results to a simplified model of distributed high-performance computing infrastructures.

Rao, Nageswara↗

NASA NTRS2000

Effect of scanline orientation on ventricular flow propagation: assessment using high frame-rate color Doppler echocardiography

Color M-mode echocardiography has recently been utilized to describe diastolic flow propagation velocity (Vp) in the left ventricle. While increasing temporal resolution from 15 to 200 Hz, this M-mode technique requires the user to select a single scanline, potentially limiting quantification of Vp due to the complex three-dimensional inflow pattern. We previously performed computational fluid dynamics simulations to demonstrate the insignificance of the scanline orientation, however geometric complexity was limited. The purpose of this study was to utilize high frame-rate 2D color Doppler images to investigate the importance of scanline selection in patients for the quantification of Vp. 2D color Doppler images were digitally acquired at 50 frames/s in 6 subjects from the apical 4-chamber window (System 5, GE/Vingmed, Milwaukee, WI). Vp was determined for a set of scanlines positioned through 5 locations across the mitral annulus (from the anterior to posterior mitral annulus). An analysis of variance was performed to examine the differences in Vp as a function of scanline position. Vp was not effected by scanline position in sampled locations from the center of the mitral valve towards the posterior annulus. Although not statistically significant, there was a trend to slower propagation velocities on the anterior side of the valve (60.8 +/- 16.7 vs. 54.4 +/- 13.6 cm/s). This study clinically validates our previous numerical experiment showing that Vp is insensitive to small perturbations of the scanline through the mitral valve. However, further investigation is necessary to examine the impact of ventricular geometry in pathologies including dilated cardiomyopathy.

Non-NASA Center↗

DOE OSTI2020

Computational Evaluation of Carriers in Facilitated Transport Membranes for Postcombustion Carbon Capture

Incorporating molecular amines as mobile carriers in facilitated transport membranes (FTMs) has been demonstrated to significantly enhance the CO 2 permeance and CO 2 /N 2 selectivity of the membrane for CO 2 capture from flue gas. In this study, by employing computational techniques including density functional theory calculations and molecular simulations, the role of mobile carriers has been systematically studied at a molecular level from the perspectives of the amine–CO 2 reaction chemistry, diffusivities of carriers and gases, and N 2 solubility. The latter two properties were also investigated as a function of water uptake. The water uptake values of FTMs were experimentally quantified too. The introduction of mobile carriers was shown to substantially enhance the diffusivities of CO 2 reaction products compared to FTMs without mobile carriers. The choice of mobile carriers was also demonstrated to influence the separation performance. Computationally, 2-(1-piperazinyl)ethylamine sarcosinate (PZEA-Sar) exhibited a faster reaction kinetics and slightly higher CO 2 absorption capacity as compared to piperazine glycinate (PZ-Gly). Experimentally, the FTM incorporating PZEA-Sar mobile carriers also showed a higher CO 2 permeance. The good agreement validated the computational models employed and insights generated in this study. In conclusion, the outcomes of this work shed light on the future design and selection of carrier structures, and the adopted computational approaches can be employed to discover promising mobile carrier candidates.

20 FOSSIL-FUELED POWER PLANTS↗

DOE OSTI2024

A mixed integer linear programming approach for the design of chemical process families

Tackling climate change goals requires widespread deployment of process technology variants across many decentralized sites with different geographical, environmental, and operational requirements. Conventional engineering approaches focus on unique designs for each installation (process variant), while missing opportunities for manufacturing standardization. Here, instead we seek to optimize a process platform of common unit designs while simultaneously designing an entire family of process variants that make use of that platform. This reduces engineering effort, deployment timelines, and manufacturing costs. We propose a nonlinear generalized disjunctive programming formulation and convert this to an efficient mixed-integer linear programming (MILP) formulation through discretization of the design space. We formulate our optimization in Pyomo with costing from IDAES, and we demonstrate the computational performance and solution quality on a water treatment desalination system from the PARETO framework and a carbon capture system built in Aspen Plus as part of CCSI2.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2021

Non-Proprietary Companion to the Q1 CY2021 PathForward Final Assessment (WBS 2.4.1, Milestone PM-HI-1040)

The PathForward element of the Exascale Computing Project (ECP) Hardware and Integration (HI) focus area has met its goals of preparing the US industry for exascale system procurements and generally improving US competitiveness in the worldwide computing market. This report is a companion to the deliverable for the ECP milestone PM-HI-1040, Assess PathForward Impact Against Exascale Hardware Challenges, which gives a summary of the final status of each PathForward project, describes progress achieved against PathForward contract milestones, and includes a final assessment of each vendor’s progress on key exascale challenges. This companion document details the results of the PathForward research to the extent possible without disclosing proprietary information and the impact on products and US exascale systems. In addition, it captures lessons learned in order to inform future projects in general and in high-performance computing in particular.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

PICSAR-QED: a Monte Carlo module to simulate strong-field quantum electrodynamics in particle-in-cell codes for exascale architectures

Abstract Physical scenarios where the electromagnetic fields are so strong that quantum electrodynamics (QED) plays a substantial role are one of the frontiers of contemporary plasma physics research. Investigating those scenarios requires state-of-the-art particle-in-cell (PIC) codes able to run on top high-performance computing (HPC) machines and, at the same time, able to simulate strong-field QED processes. This work presents the PICSAR-QED library, an open-source, portable implementation of a Monte Carlo module designed to provide modern PIC codes with the capability to simulate such processes, and optimized for HPC. Detailed tests and benchmarks are carried out to validate the physical models in PICSAR-QED, to study how numerical parameters affect such models, and to demonstrate its capability to run on different architectures (CPUs and GPUs). Its integration with WarpX, a state-of-the-art PIC code designed to deliver scalable performance on upcoming exascale supercomputers, is also discussed and validated against results from the existing literature.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2021

Dilute Combustion Control Using Spiking Neural Networks

Dilute combustion with exhaust gas recirculation (EGR) in spark-ignition engines presents a cost-effective method for achieving higher levels of engine efficiency. At high levels of EGR, however, cycle-to-cycle variability (CCV) of the combustion process is exacerbated by sporadic occurrences of misfires and partial burns. Previous studies have shown that temporal deterministic patterns emerge at such conditions and certain combustion cycles have a significant influence over future events. Due to the complexity of the combustion process and the nature of CCV, harnessing all the deterministic information for control purposes has remained challenging even with physics based 0-D, 1-D, and high-fidelity computational fluid dynamics (CFD) models. In this study, we present a data-driven approach to optimize the combustion process by controlling CCV adjusting the cycle-to-cycle fuel injection quantity. Readily available data from in-cylinder pressure was used to train a spiking neural network (SNN) which learns the optimal way to manage fuel injection in order to reduce CCV while maintaining acceptable levels of fuel consumption. SNNs are particularly well suited for powertrain control applications due to their ability to be deployed on FPGA-based neuromorphic hardware which are small, inexpensive, and have a low power demand. The high-performance computing (HPC) resources of Oak Ridge National Laboratory were used to run an evolutionary-based training approach for choosing the best SNN configuration that minimizes the size of the network while achieving the desired goal. The neuromorphic hardware with the optimized SNN deployed was connected to the rapid prototyping engine control system for real-time control implementation and tested on a single cylinder version of a GM LNF 4-cylinder engine. The results show a significant reduction of CCV with a small percentage of additional fuel used to stabilize the charge.

33 ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2022

Application Results on Early Exascale Hardware

This Exascale Computing Project (ECP) milestone report summarizes the status of 27 of the 31 ECP Applications Development (AD) subprojects at the end of FY21. In November and December of 2021, a comprehensive assessment of AD projects was conducted by the ECP leadership along with external subject matter experts (SMEs). (NNSA application projects are reviewed separately using the ASC milestone process.) The AD review committee—consisting of the AD lead, AD deputy, Level 3 (L3), and at least one external project SME—was tasked with evaluating each project’s progress relative to ECP project goals specified in the FY21 timeline. Key areas of focus were code maturity and performance on pre-exascale systems, an in-depth analysis of final key performance parameter (KPP) verification contracts, and future R&D priorities in the final year of ECP and beyond. As such, this report contains not only an accurate snapshot of each subproject’s current status but also represents a broad account of successes and challenges in porting large scientific applications to DOE’s next-generation high-performance computing architectures – the Frontier and Aurora systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Packaging HEP Heterogeneous Mini-apps for Portable Benchmarking and Facility Evaluation on Modern HPCs

High Energy Physics (HEP) experiments are making increasing use of GPUs and GPU dominated High Performance Computer facilities. Both the software and hardware of these systems are rapidly evolving, creating challenges for experiments to make informed decisions as to where they wish to devote resources. In its first phase, the High Energy Physics Center for Computational Excellence (HEP-CCE) produced portable versions of a number of heterogeneous HEP mini-apps, such as p2r, FastCaloSim, Patatrack and the WireCell Toolkit, that exercise a broad range of GPU characteristics, enabling cross platform and facility benchmarking and evaluation. However, these miniapps still require a significant amount of manual intervention to deploy on a new facility. We present our work in developing turn-key deployments of these mini-apps, where by means of containerization and automated configuration and build techniques such as Spack, we are able to quickly test new hardware, software, environments and entire facilities with minimal user intervention, and then track performance metrics over time.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2024

Navigating Exascale Operational Data Analytics: From Inundation to Insight

In this paper, we address the challenges in achieving sustainable data-driven efficiency by providing a detailed exploration of the end-to-end operational data analytics (ODA) framework that evolved through two generations of supercomputer systems at the Oak Ridge Leadership Computing Facility (OLCF). This framework addresses large data streams ingested from heavily instrumented HPC environment that accumulates multi-terabytes per day. We outline the multifaceted data life cycle across HPC procurement, operations, and research & development, identifying key obstacles and design decisions that shape effective strategies in building and supporting data pipelines end-to-end. By sharing key insights and lessons learned from our experience, we offer recommendations for the HPC community on enabling sustainable operational data analytics and beyond. Our contributions aim to bridge the gap between potential and real benefits of operational data, guiding future efforts towards integrated and sustainable operational intelligence in high-performance computing environments.

Shin, Woong↗