NREL HPC Eagle Jobs Data
3 months of Eagle jobs data including Slurm metadata and derived power use characterization.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
SEARCH · Search NASA
Results for “node usage”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Unveiling User Behavior on Summit Login Nodes as a User
We observe and analyze usage of the login nodes of the leadership class Summit supercomputer from the perspective of an ordinary user—not a system administrator—by periodically sampling user activities (job queues, running processes, etc.) for two full years (2020–2021). Our findings unveil key usage patterns that evidence misuse of the system, including gaming the policies, impairing I/O performance, and using login nodes as a sole computing resource. Our analysis highlights observed patterns for the execution of complex computations (workflows), which are key for processing large-scale applications.
Wilkinson, Sean↗
Optimizing Performance on Trinity Utilizing Machine Learning, Proxy Applications and Scheduling Priorities
The sheer number of nodes continues to increase in today’s supercomputers, the first half of Trinity alone contains more than 9400 compute nodes. Since the speed of today’s clusters are limited by the slowest nodes, it more important than ever to identify slow nodes, improve their performance if it can be done, and assure minimal usage of slower nodes during performance critical runs. This is an ongoing maintenance task that occurs on a regular basis and, therefore, it is important to minimize the impact upon its users by assessing and addressing slow performing nodes and mitigating their consequences while minimizing down time. These issues can be solved, in large part, through a systematic application of fast running hardware assessment tests, the application of Machine Learning, and making use of performance data to increase efficiency of large clusters. Proxy applications utilizing both MPI and OpenMP were developed to produce data as a substitute for long runtime applications to evaluate node performance. Machine learning is applied to identify underperforming nodes, and policies are being discussed to both minimize the impact of underperforming nodes and increase the efficiency of the system. In this paper, I will describe the process used to produce quickly performing proxy tests, consider various methods to isolate the outliers, and produce ordered lists for use in scheduling to accomplish this task.
97 MATHEMATICS AND COMPUTING↗
Usage Pattern Analysis for the Summit Login Nodes
High performance computing (HPC) users interact with Summit through dedicated gateways, also known as login nodes. The performance and stability of these login nodes can have a significant impact on the user experience. In this study, the performance and stability of Summit’s five login nodes are evaluated by analyzing the log data from 2020 and 2021. The analysis focuses on the computing capability (CPU average load, users and tasks) and the storage performance, along with the associated job scheduler activity. The outcome of this study can serve as the foundation of a predictive modeling framework that enables the system admin of an HPC system to preemptively deploy countermeasures before the onset of a system failure.
Eiffert, Brett↗
Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis
HPC datacenters offer a backbone to the modern digital society. Increasingly, they run Machine Learning (ML) jobs next to generic, compute-intensive workloads, supporting science, business, and other decision-making processes. However, understanding how ML jobs impact the operation of HPC datacenters, relative to generic jobs, remains desirable but understudied. In this work, we leverage long-term operational data, collected from a national-scale production HPC datacenter, and statistically compare how ML and generic jobs can impact the performance, failures, resource utilization, and energy consumption of HPC datacenters. Our study provides key insights, e.g., ML-related power usage causes GPU nodes to run into temperature limitations, median/mean runtime and failure rates are higher for ML jobs than for generic jobs, both ML and generic jobs exhibit highly variable arrival processes and resource demands, significant amounts of energy are spent on unsuccessfully terminating jobs, and concurrent jobs tend to terminate in the same state. We open-source our cleaned-up data traces on Zenodo (https://doi. org/10.5281/zenodo.13685426), and provide our analysis toolkit as software hosted on GitHub (https://github.com/atlarge-research/2024-icpads-hpc-workload-characterization). This study offers multiple benefits for data center administrators, who can improve operational efficiency, and for researchers, who can further improve system designs, scheduling techniques, etc.
crossanalysis↗
NLR HPC Eagle Node Power Data
Power time series captured from all Eagle nodes using iLO (Integrated Lights Out) The Eagle HPC operated at NLR from 2019 through 2024. Eagle was a 2,000-node, 8-petaflop system. This dataset is a comprehensive time series of instantaneous snapshots of power usage at 1 minute intervals from all nodes at the node level. Data provided in compressed Hive dataset/Parquet format. iLO Power Time Series Fields ts: Timestamp dv: Device / Node - Rack and Unit - r103u17 == r(ack)103u(nit)17 vl: Value - Value in watts (instantaneous value at sampling time) day month year
97 MATHEMATICS AND COMPUTING↗
Using heterogeneous GPU nodes with a Cabana-based implementation of MPCD
In this study, the Kokkos based library Cabana, which has been developed in the Co-design Center for Particle Applications (CoPA), is used for the implementation of Multi-Particle Collision Dynamics (MPCD), a particle-based description of hydrodynamic interactions. Cabana allows for a function portable implementation, which has been used to study the interplay between CPU and GPU usage on a multi-node system as well as analysis of said interplay with performance analysis tools. As a result, we see most advantages in a homogeneous GPU usage, but we also discuss the extent to which heterogeneous applications might be more performant, using both CPU and GPU concurrently.
97 MATHEMATICS AND COMPUTING↗
Predictor-corrector models for lightweight massive machine-type communications in Industry 4.0
Future Industry 4.0 scenarios are characterized by seamless integration between computational and physical processes. To achieve this objective, dense platforms made of small sensing nodes and other resource constraint devices are ubiquitously deployed. All these devices have a limited number of computational resources, just enough to perform the simple operation they are in charge of. The remaining operations are delegated to powerful gateways that manage sensing nodes, but resources are never unlimited, and as more and more devices are deployed on Industry 4.0 platforms, gateways present more problems to handle massive machine-type communications. Although the problems are diverse, those related to security are especially critical. To enable sensing nodes to establish secure communications, several semiconductor companies are currently promoting a new generation of devices based on Physical Unclonable Functions, whose usage grows every year in many real industrial scenarios. Those hardware devices do not consume any computational resource but force the gateway to keep large key-value catalogues for each individual node. In this context, memory usage is not scalable and processing delays increase exponentially with each new node on the platform. In this paper, we address this challenge through predictor-corrector models, representing the key-value catalogues. Models are mathematically complex, but we argue that they consume less computational resources than current approaches. The lightweight models are based on complex functions managed as Laurent series, cubic spline interpolations, and Boolean functions also developed as series. Unknown parameters in these models are predicted, and eventually corrected to calculate the output value for each given key. The initial parameters are based on the Kane Yee formula. An experimental analysis and a performance evaluation are provided in the experimental section, showing that the proposed approach causes a significant reduction in the resource consumption.
97 MATHEMATICS AND COMPUTING↗
NLR HPC Eagle GPU Node Metrics
Ganglia node metrics and iLO (Integrated Lights Out) power data captured from six representative Eagle GPU nodes The Eagle HPC operated at NLR from 2019 through 2024. Eagle was a 2,000-node, 8-petaflop system. This dataset is a representative sample of metrics for 6 of the GPU nodes. Each GPU node contained 2 CPUs and 2 GPUs. Data provided in compressed CSV format. Ganglia and iLO Power Time Series Fields ts: Timestamp dv: Device / Node - Rack and Unit - r103u17 == r(ack)103u(nit)17 mt: Metric (only present for Ganglia) vl: Value - Value in watts for iLO power (instantaneous value at sampling time) or specified Ganglia metric below Ganglia Metrics Metric name -- Metric description -- Unit cpu_aidle -- Percent of time since boot idle CPU -- Percent cpu_idle -- Percent CPU idle -- Percent cpu_nice -- Percent CPU nice -- Percent cpu_speed -- Speed in MHz of CPU -- MHz cpu_user -- Percent CPU user -- Percent cpu_wio -- The percentage of CPU Wait I/O -- Percent gpu0_bar1_memory -- Used GPU bar1 memory -- MB gpu0_decoder_util -- GPU decoder utilization -- Percent gpu0_ecc_db_error -- Total ECC error counts for the GPU -- Number gpu0_encoder_util -- GPU encoder utilization -- Percent gpu0_fan -- Fan speed -- RPM gpu0_fb_memory -- Used GPU framebuffer memory -- MB gpu0_graphics_clock_report -- Current clock speeds for the device -- MHz gpu0_mem_total -- Memory total -- MB gpu0_mem_util -- Memory utilization -- Percent gpu0_power_usage_report -- Power usage report -- Watts gpu0_temp -- GPU 1 temperature -- Celsius gpu1_bar1_memory -- Used GPU bar1 memory -- MB gpu1_decoder_util -- GPU decoder utilization -- Percent gpu1_ecc_db_error -- Total ECC error counts for the GPU -- Number gpu1_encoder_util -- GPU encoder utilization -- Percent gpu1_fan -- Fan speed -- RPM gpu1_fb_memory -- Used GPU framebuffer memory -- MB gpu1_graphics_clock_report -- Current clock speeds for the GPU -- MHz gpu1_mem_total -- Memory total -- MB gpu1_mem_util -- Memory utilization -- MB gpu1_power_usage_report -- Power usage report -- Watts gpu1_temp -- GPU 1 temperature -- Celsius ipmi_cpu1_temp -- CPU 1 temperature -- Celsius ipmi_cpu2_temp -- CPU 2 temperature -- Celsius ipmi_inlet_ambient_temp -- Temperature measured at intake -- Celsius ipmi_vr_p1_temp -- CPU 1 voltage regulator temperature -- Celsius ipmi_vr_p2_temp -- CPU 2 voltage regulator temperature -- Celsius mem_buffers -- Amount of buffered memory -- Bytes mem_cached -- Amount of cached memory -- Bytes mem_free -- Amount of available memory -- Bytes mem_shared -- Amount of shared memory -- Bytes mem_total -- Amount of available memory -- Bytes
97 MATHEMATICS AND COMPUTING↗
Low-Cost Sensor Performance Intercomparison, Correction Factor Development, and 2+ Years of Ambient PM2.5 Monitoring in Accra, Ghana
Particulate matter air pollution is a leading cause of global mortality, particularly in Asia and Africa. Addressing the high and wide-ranging air pollution levels requires ambient monitoring, but many low- and middle-income countries (LMICs) remain scarcely monitored. To address these data gaps, recent studies have utilized low-cost sensors. These sensors have varied performance, and little literature exists about sensor intercomparison in Africa. By colocating 2 QuantAQ Modulair-PM, 2 PurpleAir PA-II SD, and 16 Clarity Node-S Generation II monitors with a reference-grade Teledyne monitor in Accra, Ghana, we present the first intercomparisons of different brands of low-cost sensors in Africa, demonstrating that each type of low-cost sensor PM2.5 is strongly correlated with reference PM2.5, but biased high for ambient mixture of sources found in Accra. When compared to a reference monitor, the QuantAQ Modulair-PM has the lowest mean absolute error at 3.04 μg/m3, followed by PurpleAir PA-II (4.54 μg/m3) and Clarity Node-S (13.68 μg/m3). We also compare the usage of 4 statistical or machine learning models (Multiple Linear Regression, Random Forest, Gaussian Mixture Regression, and XGBoost) to correct low-cost sensors data, and find that XGBoost performs the best in testing (R2: 0.97, 0.94, 0.96; mean absolute error: 0.56, 0.80, and 0.68 μg/m3 for PurpleAir PA-II, Clarity Node-S, and Modulair-PM, respectively), but tree-based models do not perform well when correcting data outside the range of the colocation training. Therefore, we used Gaussian Mixture Regression to correct data from the network of 17 Clarity Node-S monitors deployed around Accra, Ghana, from 2018 to 2021. We find that the network daily average PM2.5 concentration in Accra is 23.4 μg/m3, which is 1.6 times the World Health Organization Daily PM2.5 guideline of 15 μg/m3. While this level is lower than those seen in some larger African cities (such as Kinshasa, Democratic Republic of the Congo), mitigation strategies should be developed soon to prevent further impairment to air quality as Accra, and Ghana as a whole, rapidly grow.
Humidity↗
Evaluating adaptive and predictive power management strategies for optimizing visualization performance on supercomputers
Power is becoming an increasingly scarce resource on the next generation of supercomputers, and should be used wisely to improve overall performance. One strategy for improving power usage is hardware overprovisioning, i.e., systems with more nodes than can be run at full power simultaneously without exceeding the system-wide power limit. With this study, we compare two strategies for allocating power throughout an overprovisioned system – adaptation and prediction – in the context of visualization workloads. While adaptation has been suitable for workloads with more regular execution behaviors, it may not be as suitable on visualization workloads, since they can have variable execution behaviors. This study considers a total of 104 experiments, which vary the rendering workload, power budget, allocation strategy, and node concurrency, including tests processing data sets up to 1 billion cells and using up to 18,432 cores across 512 nodes. Overall, we find that prediction is a superior strategy for this use case, improving performance up to 27% compared to an adaptive strategy.
97 MATHEMATICS AND COMPUTING↗
Singleton Sieving: Overcoming the Memory/Speed Trade-Off in Exascale k-mer Analysis
Traditional filter data structures, such as Bloom filters, do not offer necessary features that modern high-performance data analytics applications need in order to efficiently perform complex data analysis tasks. For example, MetaHipMer, a de novo metagenome assembler, can use filters to weed out singleton k-mers and reduce memory usage by 30%-70%. However, the filter needs the ability to associate values with k-mers in order to perform the analysis in a single communication pass. Bloom filters do not support value associations and cause the application to perform an extra communication pass, thereby increasing the run time. Therefore, MetaHipMer faces a trade off between memory and speed due to the limited capabilities of traditional filters. In this paper, we overcome the memory and speed trade off in MetaHipMer by integrating a GPU-based feature-rich filter, the Two-Choice filter (TCF), in the MetaHipMer pipeline. The TCF uses key-value association to approximately store k-mers with extensions. This allows MetaHipMer to perform k-mer analysis on the GPUs in a single communication pass. Our empirical analysis shows a 50% reduction in memory usage in k-mer analysis on each node in MetaHipMer without any effect on the overall run time or assembly quality. The memory reduction in turn results in a 43% reduction in the number of nodes required to assemble datasets and enables MetaHipMer to scale to much larger datasets.
McCoy, Hunter↗
Accelerating Flash-X Simulations with Asynchronous I/O
Most high-fidelity physics simulation codes, such as Flash-X, need to save intermediate results (checkpoint files) to restart or gain insights into the evolution of the simulation. These simulation codes save such intermediate files synchronously, where computation is stalled while the data is written to storage. Depending on the problem size and computational requirements, this file write time can be a substantial portion of the total simulation time. In order to hide the I/O latency of checkpointing, asynchronous I/O methods have been introduced. These methods use background threads for performing I/O while the main threads continue with the simulation. The usage of background threads can compete for resources on the node as well as with communication. In this paper, we evaluate the overheads and the overall benefit of asynchronous I/O in HDF5 to simulations. Results from real-world high-fidelity simulations on the Summit supercomputer show that I/O operation is overlapped with application communication or computation or both, effectively hiding some or all of the I/O latency. Our evaluation shows that while using asynchronous I/O adds overhead to the application, the I/O time reduction is more significant, resulting in overall up to 1.5X performance speedup.
Jain, Rajeev↗
Distributed Transforms for Efficient Data Gathering in Sensor Networks
Devices, systems, and techniques for data collecting network such as wireless sensors are disclosed. A described technique includes detecting one or more remote nodes included in the wireless sensor network using a local power level that controls a radio range of the local node. The technique includes transmitting a local outdegree. The local outdegree can be based on a quantity of the one or more remote nodes. The technique includes receiving one or more remote outdegrees from the one or more remote nodes. The technique includes determining a local node type of the local node based on detecting a node type of the one or more remote nodes, using the one or more remote outdegrees, and using the local outdegree. The technique includes adjusting characteristics, including an energy usage characteristic and a data compression characteristic, of the wireless sensor network by selectively modifying the local power level and selectively changing the local node type.
Ortega, Antonio↗
Improving scalability of electronic structure code for molecular simulations in the presence of environment
A scalable density functional electronic code with Gaussian basis set, called UTEP-NRLMOL, is developed to perform simulations of molecular systems in the presence of the environment with particular attention to the memory requirements. In the electronic structure calculations, the memory and computation time are proportional to the number of atoms. Memory requirements for density functional calculations scale as N*N, where N is the number of atoms. While the recent advances in HPC offer platforms with large numbers of cores, the limited amount of memory available on a given node and poor scalability of the electronic structure codes hinder their efficient usage of these platforms. We have introduced new scaling and parallelization paradigms using MPI-3 shared-memory functionality combined with usage of sparse algebra and storage of matrices in sparse format. This extends the range of applicability of the UTEP-NRLMOL code to large systems over 10,000 atoms, or using up to 67,000 basis functions, and making use of HPC architectures using over 6,000 processors utilizing all available cores. We have also interfaced code with effective fragment potential and polarizable continuum model libraries. The code was used in simulations of several applications which are published in reputed scientific journals.
74 ATOMIC AND MOLECULAR PHYSICS↗
A high-order computational framework for particle-resolved simulations of disperse multiphase flows
This work presents a high-order numerical approach for particle-resolved simulations of disperse multiphase flows, where the Navier-Stokes equations for fluid flow are solved using a high-order spectral element method in the Eulerian framework, and the particle phase is directly simulated with a discrete element method. The coupling between particles and fluids is explicitly handled using an adapted direct-forcing immersed boundary method. Unlike the conventional schemes, a high-order barycentric Lagrange interpolation method and a Gaussian projection kernel are used to ensure accurate momentum exchange between local boundary points and surrounding fluid nodes in the framework of high-order fluid solver. Benchmark tests of increasing complexity are conducted to demonstrate the accuracy and efficiency of our method. Here, it is found that our approach exhibits an excellent convergence performance, as the fluid element/grid is refined and the number of boundary points increases. Compared to conventional low-order methods, the proposed high-order framework enables the use of substantially larger fluid elements while maintaining high accuracy in modeling fluid-particle interactions, owing to the enhanced resolution of high-order basis functions. Moreover, since the primary unknowns are stored at element or grid nodes, the high-order approach offers improved efficiency in both CPU memory usage and total computational cost.
42 ENGINEERING↗
RAP: Resource-aware Automated GPU Sharing for Multi-GPU Recommendation Model Training and Input Preprocessing
Ensuring high-quality recommendations for newly onboarded users requires the continuous retraining of Deep Learning Recommendation Models (DLRMs) with freshly generated data. To serve the online DLRM retraining, existing solutions use hundreds of CPU computing nodes designated for input preprocessing, causing significant power consumption that surpasses even the power usage of GPU trainers. To this end, we propose RAP, an end-to-end DLRM training framework that supports Resource-aware Automated GPU sharing for DLRM input Preprocessing and Training. The core idea of RAP is to accurately capture the remaining GPU computing resources during DLRM training for input preprocessing, achieving superior training efficiency without requiring additional resources. Specifically, RAP utilizes a co-running cost model to efficiently assess the costs of various input preprocessing operations, and it implements a resource-aware horizontal fusion technique that adaptively merges smaller kernels according to GPU availability, circumventing any interference with DLRM training. In addition, RAP leverages a heuristic searching algorithm that jointly optimizes both the input preprocessing graph mapping and the co-running schedule to maximize the end-to-end DLRM training throughput. The comprehensive evaluation shows that RAP achieves 78.3× speedup on average over CPU-based DLRM input preprocessing frameworks. In addition, the end-to-end training throughput of RAP is only 2.04% lower than the ideal case, which has no input preprocessing overhead.
Wang, Zheng↗
Accurate Adaptive Level Set Method and Sharpening Technique for Three Dimensional Deforming Interfaces
In this paper, we demonstrate improved accuracy of the level set method for resolving deforming interfaces by proposing two key elements: (1) accurate level set solutions on adapted Cartesian grids by judiciously choosing interpolation polynomials in regions of different grid levels and (2) enhanced reinitialization by an interface sharpening procedure. The level set equation is solved using a fifth order WENO scheme or a second order central differencing scheme depending on availability of uniform stencils at each grid point. Grid adaptation criteria are determined so that the Hamiltonian functions at nodes adjacent to interfaces are always calculated by the fifth order WENO scheme. This selective usage between the fifth order WENO and second order central differencing schemes is confirmed to give more accurate results compared to those in literature for standard test problems. In order to further improve accuracy especially near thin filaments, we suggest an artificial sharpening method, which is in a similar form with the conventional re-initialization method but utilizes sign of curvature instead of sign of the level set function. Consequently, volume loss due to numerical dissipation on thin filaments is remarkably reduced for the test problems
Kim, Hyoungin↗