Workload-Adaptive Scheduling for Efficient Use of Parallel File Systems in High-Performance Computing Clusters
Results from work on feedback control of a queuing system for HPC jobs with shared storage resources.
Allan, Benjamin Andrew↗
SEARCH · Search NASA
Results for “parallel scheduling”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Workload-adaptive Scheduling for Efficient Use of Parallel File Systems in High-performance Computing Clusters
Explore the source record for details and available documents.
Goponenko, Alexander V.↗
Advanced Shuttle Strategies for Parallel QCCD Architectures
Trapped ions (TIs) are at the forefront of quantum computing implementation, offering unparalleled coherence, fidelity, and connectivity. However, the scalability of TI systems is hampered by the limited capacity of individual ion traps, necessitating intricate ion shuttling for advanced computational tasks. The quantum charge-coupled device (QCCD) framework has emerged as a promising solution, facilitating ion mobility for universal quantum computation. Current QCCD architectures predominantly feature a linear topology, which is increasingly recognized as inefficient for complex quantum operations. Anticipating the shift toward more efficacious designs, this article introduces an innovative quantum scheduling strategy optimized for parallel QCCD topologies. Our strategy proposes a probabilistic formula for ion movement, alongside ingenious methods for local layer generation and layer compression, yielding a significant reduction in ion shuttle times. Through simulations, we demonstrate that our strategy not only substantially outstrips the linear model but also exhibits better performance over other parallel strategies that employ greedy algorithms. This is achieved through our nuanced resolution of complexities, such as traffic blocks and trap capacity limitations. The consequent reduction in shuttle operations leads to lower energy consumption and an enhancement in the quantum computer's fidelity, ultimately accelerating program execution times.
43 PARTICLE ACCELERATORS↗
Improving I/O-aware Workflow Scheduling via Data Flow Characterization and trade-off Analysis
The scientific computing paradigm has transitioned from compute-intensive to I/O-intensive and memory-intensive in the past decade, especially when data-driven science has become common practice. Numerous empirical I/O-aware scheduling optimizations have been developed by incorporating I/O capacity and bandwidth as constraints into scheduling. Unfortunately, there is a lack of data flow (I/O) characterization tool and an understanding of trade-offs between concurrency, locality, and I/O bandwidth. To bridge the gap, this work 1) presents a set of descriptors to characterize, organize, and visualize I/O profiles, including flow size, I/O bandwidth, and operation count, which group data flows by I/O types, tasks, and files; 2) proposes an I/O Roofline model-based trade-off analysis to find the optimal trade-off between flow operational intensity, concurrency, and flow performance. The I/O descriptors generate useful insights into complicated I/O behaviors, suggesting distinct concurrency, storage, and scheduling to be used by types, tasks, and files. The proposed trade-off analysis guides scheduling decisions that generate resource assignment with the best flow parallelism. We evaluate our I/O-aware scheduling methodology on a highly I/O-intensive workflow–1000 Genomes. The experimental results demonstrate speedups of up to 2.4× compared to the state-of-the- art methods.
Guo, Luanzheng [BATTELLE (PACIFIC NW LAB)]↗
An exploration of online-simulation-driven portfolio scheduling in Workflow Management Systems
Workflow Management Systems used to automate the execution of scientific workflow applications on parallel and distributed computing platforms must make scheduling decisions at runtime. A large number of workflow scheduling algorithms have been proposed in the literature, but often these algorithms are evaluated based on simplifying assumptions that may not hold in practice. Furthermore, published algorithm evaluation and/or comparison results are necessarily only for a subset of all possible scenarios, and thus may not include scenarios relevant to particular use-cases. Consequently, it is difficult for Workflow Management Systems (WMSs) developers to decide which scheduling algorithm should be implemented. To obviate this difficulty, one possible approach is to implement a portfolio of scheduling algorithms and select the most effective algorithm at runtime. One method for performing this selection is to run an online simulation for each algorithm in the portfolio. The algorithm that leads to the best performance, in simulation, is selected for future use. The above simulation-driven portfolio scheduling (SDPS) approach has been proposed in a few parallel and distributed computing contexts. The main objective of this work is to evaluate the feasibility and potential merit of SDPS if implemented in WMSs. Here we perform this evaluation using simulated WMS executions, where the simulations are instantiated from real-world platform and workflow configurations. Our main finding is that SDPS is on par with or outperforms an approach in which a single algorithm is used, where this algorithm is the one that performs best on average across all our experimental scenarios. Furthermore, we find that SDPS remains an attractive proposition even in the presence of high levels of simulation error and for simulators with relatively low levels of sophistication. In many of our experimental scenarios we find that mitigating simulation error at runtime can further improve performance. Finally, we show that simulation overhead can be made sufficiently low for SDPS to be feasible in practice.
97 MATHEMATICS AND COMPUTING↗
Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel
GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.
Maheshwari, Ketan↗
Robustness of Deep Learning Classification to Adversarial Input on GPUs: Asynchronous Parallel Accumulation Is a Source of Vulnerability
The ability of machine learning (ML) classification models to resist small, targeted input perturbations—known as adversarial attacks—is a key measure of their safety and reliability. We show that floating-point non associativity (FPNA) coupled with asynchronous parallel programming on GPUs is sufficient to result in misclassification, without any perturbation to the input. Additionally, we show that this misclassification is particularly significant for inputs close to the decision boundary and that standard adversarial robustness results may be overestimated up to 4.6 when not considering machine-level details. We first study a linear classifier, before focusing on standard Graph Neural Network (GNN) architectures and datasets used in robustness assessments. We develop a novel black-box attack using Bayesian optimization to discover external workloads that can change the instruction scheduling which bias the output of reductions on GPUs and reliably lead to misclassification. Motivated by these results, we present a new learnable permutation (LP) gradient-based approach to learning floating-point operation orderings that lead to misclassifications. The LP approach provides a worst-case estimate in a computationally efficient manner, avoiding the need to run identical experiments tens of thousands of times over a potentially large set of possible GPU states or architectures. Finally, using instrumentation-based testing, we investigate parallel reduction ordering across different GPU architectures under external background workloads, when utilizing multi-GPU virtualization, and when applying power capping. Our results demonstrate that parallel reduction ordering varies significantly across architectures under the first two conditions, substantially increasing the search space required to fully test the effects of this parallel scheduler-based vulnerability. These results and the methods developed here can help to include machine-level considerations into adversarial robustness assessments, which can make a difference in safety and mission critical applications.
Shanmugavelu, Sanjif [Maxeler Technologies, a Groq↗
Leveraging Pre-Built Catalogs and Object-Level Scheduling to Eliminate I/O Bottlenecks in HPC Environments
Modern High-Performance Computing (HPC) environments face mounting challenges due to the shift from large to small file datasets, along with an increasing number of users and parallelized applications. As HPC systems rely on Parallel File Systems (PFS), such as Lustre for data processing, performance bottlenecks stemming from Object Storage Target (OST) contention have become a significant concern. Existing solutions, such as LADS with its object-level scheduling approach, fall short in large-scale HPC environments due to their inability to effectively address metadata I/O bottlenecks and the growing number of I/O processes. This study highlights the pressing need for a comprehensive solution that tackles both OST contention and metadata I/O challenges in diverse HPC workloads. To address these challenges, we propose SwiftLoad, an object-level I/O scheduling framework that leverages a metadata catalog to enhance the performance and efficiency of parallel HPC utilities. The adoption of the metadata catalog mitigates the metadata I/O bottlenecks that commonly occur in HPC utilities, a challenge that is particularly pronounced in object-level I/O scheduling. SwiftLoad addresses OST contention and the uneven distribution of I/O processes across different OSTs through mathematical modeling and incorporates a Loader Configuration Module to regulate the number of I/O processes. Evaluated with two representative utilities—data deduplication profiling and data augmentation—SwiftLoad achieved performance improvements of up to 5.63x and 11.0x, respectively, on a production supercomputer.
HPC↗
Priority-BF: A Task Manager for Priority-Based Scheduling
The increasing demand for computational resources, particularly in High-Performance Computing environments, necessitates to rethink how we handle job scheduling strategies. This work addresses the challenge of managing concurrent jobs with differing priorities on overloaded parallel systems, where strict QoS constraints are often difficult for users to define. Our solution relies on a qualitative description of priorities and pulls from two key approaches: the Easy-BF algorithm and the Conservative Backfilling algorithms. This solution improves the response time for high-priority jobs by 50% without affecting the overall system utilization. We show its applicability in several critical scenarios such as High-Performance Computing (HPC) resource management and in-situ computing.
Gainaru, Ana [ORNL]↗
Intelligent Partitioning based Fully Parallel AC Security-Constrained Optimal Power Flow
Today’s power grid is becoming more diverse and integrated with high-level distributed energy resources and smart control technologies that is creating a new set of grid management challenges in terms of large-scale, nonlinear, and non-convex problem modeling, complex and time-consuming computation, as well as difficult uncertainty handling. This project focused on solving a challenging multi-period security-constrained generation scheduling problem, which is of great importance for maximizing the social welfare of real-time dispatch, day-ahead market, as well as weekly planning of power systems. Our developed software explored parallel optimization algorithms for complex and realistic power system models, and develop fast, efficient, and robust grid optimization solutions on the high-performance computing platform that will enable increased grid economics, flexibility, resilience, as well as energy security in the United States.
24 POWER TRANSMISSION AND DISTRIBUTION↗
The Myth of Fungible FTE: A Quantitative Assessment of Matrixed Resource Allocation
Matrix organizations allow scientific facilities to share specialized personnel across projects, operations, maintenance, and strategic initiatives. Nominal staffing allocations, however, may not capture the schedule consequences of fragmented individual commitments, limited access to specialist groups, and intermittent availability of key decision makers. We developed a stochas- tic, daily-time-step simulation of a hypothetical medium-sized accelerator-facility project com- prising sequential phases and parallel tasks. Each task requires role-specific work measured in FTE-days. Ordinary personnel may be unavailable because they contribute concurrently to other institutional activities, while designated key roles have independently specified daily un- availability probabilities. An organization-wide priority factor scales the number of people from each functional group who can effectively contribute to the project. It is interpreted as a composite proxy for project access and workforce fragmentation across competing commit- ments. We examined project completion time as a function of this factor and Project Lead unavailability using 100 Monte Carlo runs per condition. Increasing priority factor from 0.1 to 1.0 reduced median completion time from 1708.5 days (interquartile range 1681.5–1735.25) to 390 days (interquartile range 379–399). At priority factor = 0.1, increasing Project Lead unavailability from 0.5 to 0.9 increased median completion time from 1713.5 days (interquartile range 1691–1733.25) to 4,417 days (interquartile range 4271.75–4550.5). The model quantifies the commonly expected sensitivity of project schedules to fragmented resource commitments and limited coordination availability. Within this model, the results also indicate a possible threshold regime in which small increases in workforce availability yield only modest sched- ule improvements until sufficient capacity becomes accessible, after which project performance improves sharply. With further validation and calibration, this quantitative framework could support resource-allocation decisions during initial project planning and subsequent schedule rebaselining.
Bai, Mei [SLAC National Accelerator Laboratory (SL↗
Update on Parallel Process Execution in the Next Generation System Analysis Model
As of the end of 2021, 88,880 metric tons of heavy metal (MTHM) (44,741 MTHM in dry storage; 44,139 MTHM in wet storage) of spent nuclear fuel (SNF) were stored at various reactor sites across the United States [1]. The Office of Storage and Transportation in the Department of Energy is planning for the transportation, storage, and eventual disposal of SNF and high-level radioactive waste (HLW). To aid in this effort and inform decision-makers about the backend of the spent fuel cycle, systems analysis tools capable of analyzing the various options with respect to SNF and HLW management are being used as well as continuously improved to meet the evolving needs of the program. System analysts typically use these tools to vary underlying assumptions (shipping rates, allocation priority, available facilities, start dates, etc.) and study the implications of these changes on site clearance schedules, campaign costs, transportation infrastructure acquisition, etc.
11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
Characterization and Optimization of the Fitting of Quantum Correlation Functions
This case study presents a characterization and optimization of an application code for extracting parton distribution functions from high energy electron-proton scattering data. Profiling this application code reveals that the phase-space density computation accounts for 93% of the overall execution time for a single iteration on a single core. When executing multiple iterations in parallel on a multicore system, the application spends 78% of its overall execution time idling due to load imbalance. We address these issues by first transforming the application code from Python to C++ and then tackling the application load imbalance via a hybrid scheduling strategy that combines dynamic and static scheduling. These techniques result in a 62% reduction in CPU idle time and a 2.46x speedup in overall execution time per node. In addition, the typically enabled power-management mechanisms in supercomputers (e.g., AMD Turbo Core, Intel Turbo Boost, and RAPL) can significantly impact intra-node scalability when more than 50% of the CPU cores are used. This finding underscores the importance of understanding system interactions with power management, as they can adversely impact application performance, and highlights the necessity of intra-node scaling tests to identify performance degradation that inter-node scaling tests might otherwise overlook.
Chuang, Pi-Yueh [Virginia Tech,Dept. of Computer S↗
Research Reactors Division Infrastructure Investment Plan for the High Flux Isotope Reactor
The High Flux Isotope Reactor (HFIR) is a unique national asset. Operational for nearly 60 years, continued investment into the aging infrastructure is necessary to ensure operation for another 6 decades. Additionally, growing missions require HFIR as well as important upgrades. Consequently, carefully integrated planning is required to ensure that infrastructure investments are timely executed to ensure long-term, reliable operation of HFIR. Concerns about challenges to the operational reliability of HFIR resulted in a recommendation from the 2023 Operations Review by the US Department of Energy (DOE) Office of Basic Energy Sciences that a HFIR management strategy be developed to address the infrastructure needs. This report defines the investment needs, which are evolving as new upgrade efforts are better defined. HFIR is part of the three-source strategy within the Neutron Sciences Directorate (NScD) and contributes to the five strategic science areas outlined in the NScD 10 Year Strategic Science Plan: quantum materials, soft matter, materials and engineering, chemistry, and biosciences. Fundamental to this strategy are three core values: operational excellence, responsible stewardship, and servant leadership. These values guide our mission of safe and reliable operation of the reactor and require a strong and just nuclear safety culture, a solemn respect for responsible care of the facility, good workforce development, robust procedures and processes, an effective communication strategy, world-class asset management, a determined customer focus, and a commitment to protecting the environment, the safety and health of the public and our people, and the quality of work performed within our facility. These principles are all essential to operate HFIR at a world-class level. The Research Reactors Division (RRD) will lead a new era of neutron science and isotope production at HFIR through responsible and purposeful leadership and unwavering support of the science community. The approach outlined in this plan highlights the direction leadership is taking to ensure that HFIR is ready to support the science challenges and national needs of the future and that the United States maintains world leadership in neutron sciences. The plan is in alignment with the DOE’s desire to continue operating HFIR and with the NScD strategic science goals for the future. HFIR is an aging facility with numerous infrastructure challenges and needs. It has an aging workforce in relation to the general population of Oak Ridge National Laboratory (ORNL), with many expected retirements over the next 5–10 years. With an increase in work scope caused by changing national priorities and science goals, several critical hires have been identified. To manage HFIR’s infrastructure needs, a prioritized list of equipment upgrades has been identified along with an analysis of future staffing requirements. A desire to operate HFIR at eight cycles per year will necessarily require some significant changes to procedures and processes currently in place as well as targeted staffing additions. Many of the equipment upgrades identified in this plan will significantly increase the reliability of the plant, thus contributing to the effort to reach the goal of safely operating eight cycles per year. A plan to attain eight-cycle operation is being prepared in parallel with the activities identified in this plan, although the actions identified to satisfy both plans will overlap. This plan identifies new infrastructure needs—for both plant equipment and staffing—thus necessitating formulation of future budget requests to fund the increased work scope and improvement activities. Some activities are currently being scheduled with the expectation that funding will be received. Any delays to funding or reductions of funding from the identified cost estimations will directly and negatively affect the plan’s implementation.
21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
Laser Powder Bed Fusion Additive Manufacture Nb1Zr Development
Next generation fission and fusion nuclear reactors require materials that can withstand operating temperatures greater than 500 °C, neutron irradiation doses of up to 200 displacements per atom (dpa), and potentially corrosive coolants such as the alkali liquid metals sodium, lithium, and NaK (Na33K eutectic alloy). Refractory alloys, such as Nb1Zr (Nb-1wt%Zr) and Molybdenum alloy TZM (Mo-0.5wt%Ti-0.08wt%Zr) have been traditionally considered viable candidates for advanced fission and fusion reactor concepts. However, it is relatively difficult to generate complex geometries of interest from these alloys using traditional manufacturing methods. In addition, there needs to be a concentrated effort to address refractory metal challenges at elevated temperature operation. In order to generate complex geometries of interest, modern manufacturing techniques are considered to increase the technological readiness level (TRL), cost-effectiveness, and schedule savings. This work focused on the continued development of laser powder bed fusion (L-PBF) additive manufacturing (AM) to improve both design flexibility, evaluate microstructure and properties, and ultimately accelerate the TRL and qualification of these processes and alloys for components to potentially be put into service. Niobium alloy Nb1Zr was identified through a down-selection process outlined in previous reports as a candidate to develop in L-PBF AM. Historically, Nb1Zr had been explored for high temperature fast spectrum fission reactors for both terrestrial and space applications. Molybdenum alloy TZM has also been considered for these reactor concepts due to exceptional high-temperature strength, creep resistance, and stability under irradiation. L-PBF AM of TZM has previously been investigated at LANL under the Microreactor program, NASA, ORNL, and in academia. However, due to the crack prone nature of TZM, L-PBF AM of TZM resulted in significant microcracking and additional development is required to pursue viable maturation. Other AM methods have been found to be more successful in printing TZM, and those alternatives approaches are discussed in this effort. The efforts detailed in this report focused on continued development of Nb1Zr through L-PBF and development of TZM via L-PBF and electron powder bed fusion (E-PBF). The objective of this work was to further the development of these AM techniques for the chosen refractory alloys, elucidating and addressing associated challenges through characterization of several demonstration builds. At LANL, Nb1Zr builds were completed using an EOS M290 and M400 machines, and a refractory alloy-dedicated L-PBF system, the Xact Metal XM200G, was installed. The XM200G primary purpose was to do the Nb1Zr parameter development process; however, due to difficulties associated with the machine installation and qualification process, it was decided to pivot development to the larger M400 and M290 machines. Although the supply of Nb1Zr powder was limited, it was sufficient to generate sub-scale metallographic specimens for the purpose of parameter development. This was first accomplished on the EOS M400 then the M290 due to machine schedule availability. Further development of TZM has been initiated at the University of Texas El Paso (UTEP) under contract with LANL to use both a heated build envelope L-PBF machine and E-PBF machine that have been found in the literature to mitigate microcracking. UTEP was provided with TZM powder and build plates to support parallel TZM parameter development across both machines. As part of the contract, UTEP will also be conducting microstructural characterization once optimized process parameters have been identified. The optimized process parameters for each machine will be used to generate a series of metallographic, mechanical, and surface finish specimens for subsequent characterization and testing. In the next section, we provide a detailed discussion of the methodology used for investigating the feasibility of leveraging these alloys for use in advanced reactor applications.
36 MATERIALS SCIENCE↗
Development of a Test-Bed for Testing and Refining EarthEn’s Supercritical CO 2 Based Energy Storage System
EarthEn’s energy storage concept leverages supercritical carbon dioxide (sCO 2 ) as a working fluid and relies on compact, high-performance components operating at elevated pressures and temperatures. To accelerate component development and reduce technical risk prior to larger-scale demonstrations, Oak Ridge National Laboratory (ORNL) developed a 100 kW-scale sCO 2 test-bed under a Cooperative Research and Development Agreement with EarthEn (CRADA NO. NFE-24-10050). The objective of the work was to design and construct a flexible experimental facility capable of reproducing key thermodynamic state points and heat-transfer conditions relevant to EarthEn’s thermal energy storage (TES) cycle, with particular emphasis on enabling development and evaluation of next-generation heat exchangers and TES concepts. The test-bed consists of a closed-loop sCO 2 circulation system housed within an open-topped enclosure. In its as-installed configuration, dense-phase sCO 2 is recirculated through a printed circuit recuperator, an electrically heated section, a throttling device used to simulate turbine expansion, and a water-cooled printed circuit heat exchanger that rejects heat to the building chilled-water system before returning to the pump. The pump is driven by a variable frequency drive, enabling controlled adjustment of flow and operating point. A comprehensive instrumentation suite was integrated to support both safe operation and high-quality data collection. Installed sensors include Coriolis flow meters for sCO 2 flow rate and density, resistance temperature detectors and thermocouples distributed throughout the loop (including the heated section and key heat exchanger ports), and pressure transducers for absolute and differential pressure measurements. The facility was designed to support high-pressure (19 MPa nominal) and high-temperature (575°C nominal) operation with credited overpressure protection provided by a rupture disk. Nominal operating conditions were selected to support 100 kW-class testing while maintaining flexibility for non-heated and heated shakedown, control development, and future integration of advanced TES test sections. In parallel with facility development, a system-level thermal-hydraulic model was created using Modelica-based tools to support component sizing, anticipate performance over targeted test conditions, and establish a framework for future model calibration against experimental data. At the conclusion of the project performance period, the facility was in final assembly, and the pressure boundary was nearly completed. However, several practical challenges associated with high-pressure/high-temperature systems and specialized component procurement impacted schedule and prevented initial pump-driven operation and full commissioning within the available resources. This report documents the as-built design, operating capabilities, and instrumentation, and it summarizes key lessons learned related to heater fabrication and testing, first-of-a-kind assembly factors, specialty flange supply constraints, and fill pump corrective actions. Finally, it outlines a phased plan for future commissioning and experimental campaigns, including control and instrumentation shakedown, heater characterization, model calibration, and testing at state points representative of EarthEn’s TES cycle.
25 ENERGY STORAGE↗
High Yield Xray Imager Final Design Review
The High Yield Xray Imager (HYXI) is a new NIF target diagnostic system currently under development. The goal of HYXI is to provide high-fidelity, high temporal resolution x-ray imaging capability on high yield NIF implosions at 10MJ and above. The HYXI instrument design concept is based on the combination of two technologies that have been successfully utilized at the NIF on previous instruments, electron pulse-dilation and hybrid-CMOS sensor imaging. The combination of these two techniques will give HYXI sufficient data quality to ascertain differences in hot spot formation dynamics between high and low yield implosions. This information will highlight the critical hot spot conditions needed for ignition and burn. The HYXI design leverages the successful operation of the PDIXI x-ray imager at the NIF on multi MJ yield shots. A new radiation tolerant CMOS imaging array (HYPERION) is being developed to eliminate the significant background noise which limits the data quality of PDIXI. We successfully placed the contract with Advanced hCMOS Systems (AHS) to develop the HYPERION sensor, which fulfils our criteria to place long lead time item procurements by end of FY24. The HYXI Final Design Review was completed at the end of Q4 FY24 (Sep 24 th and Sep 30 th ). The HYXI project is a multi-year effort with a phased approach to be bring up system functionality over time in parallel with the development and fabrication effort of the HYPERION CMOS imaging array. In Phase 1, time-integrated x-ray images on NIF DT experiments will be collected starting in Q3 FY25. In Phase 2 of the project, time-resolved imaging with HYXI utilizing a spare microchannel plate detector back-end will begin in Q3 FY26. Phase 3 concludes the project with the installation of the HYPERION sensor array and the final performance qualification of the HYXI instrument which is scheduled for Q3 FY27 as discussed in the PDR and MRT report on this project in FY23.
42 ENGINEERING↗
FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
Production LLM serving must simultaneously deliver high throughput, low latency, and sufficient context capacity under non-stationary traffic and mixed request requirements. Data parallelism (DP) maximizes throughput by running independent replicas, while tensor parallelism (TP) reduces per-request latency and pools memory for long-context inference. However, existing serving stacks typically commit to a static parallelism configuration at deployment; adapting to bursts, priorities, or long-context requests is often disruptive and slow. We present Flying Serving, a vLLM-based system that enables online DP-TP switching without restarting engine workers. Flying Serving makes reconfiguration practical by virtualizing the state that would otherwise force data movement: (i) a zero-copy Model Weights Manager that exposes TP shard views on demand, (ii) a KV Cache Adaptor that preserves request KV state across DP/TP layouts, (iii) an eagerly initialized Communicator Pool to amortize collective setup, and (iv) a deadlock-free scheduler that coordinates safe transitions under execution skew. Across three popular LLMs and realistic serving scenarios, Flying Serving improves performance by up to 4.79 × under high load and 3.47 × under low load while supporting latency- and memory-driven requests.
Gao, Shouwei [ORNL]↗