Workload-Adaptive Scheduling for Efficient Use of Parallel File Systems in High-Performance Computing Clusters
Results from work on feedback control of a queuing system for HPC jobs with shared storage resources.
Allan, Benjamin Andrew↗
SEARCH · Search NASA
Results for “adaptive scheduling”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Workload-adaptive Scheduling for Efficient Use of Parallel File Systems in High-performance Computing Clusters
Explore the source record for details and available documents.
Goponenko, Alexander V.↗
Multi-modal Energy-optimal Trip Scheduling in Real-time (METS-R) for Transportation Hubs (Final Report)
This report summarizes the work performed under the award number EE0008524. The project develops the Multi-modal Energy-optimal Trip Scheduling in Real-time (METS-R) platform as the next-generation transportation solution based on autonomous electric vehicles (AEV) serving passenger trips from and to urban transportation hubs, to substantially reduce transportation energy consumption. Extensive data collection and analyses were first conducted to understand the demand patterns and energy consumption of hub-based on-road trips. Then, a data-driven framework that consists of an analytical module and a simulation module was proposed. For the analytical module, five planning + operation tools were developed to support the planning and energy-efficient operations of urban AEV services: the charging station planning that robotically allocates charging supplies based on the stationary charging demand distribution; the transit planning and demand adaptive scheduling model that efficiently generates\ candidate transit routes from hubs to other places and dynamically adjusts the transit time table to fit the current demand; the online energy-efficient routing that learns the energy-optimal paths from observations of link-level energy consumption in real-time; the hub-based ridesharing that matches trip requests together with account for the uncertainty of future trip demand and vehicle supply; and finally, the integrated demand prediction and anomaly detection pipeline that leverages the flight/train time table and support other planning/operation tools. To demonstrate the performance of these tools, a scalable high-performance agent-based simulator was built. We divided the urban space into multiple service zones where each zone was considered as an agent for passenger generation and vehicle charging. Two types of AEV agents were coded to model two types of mobility services: AEV taxi and AEV transit. For the AEV taxi, the team implemented the functions of pickup/drop-off passengers, energy-efficient routing, ridesharing, fleet rebalancing, and recharging. For the AEV bus, the team implemented the functions of demand-adaptive route scheduling, passenger boarding, and recharging. A high-performance computing framework was introduced to receive various profiling information (such as link energy updates, vehicle speed) from the simulator instances and communicate the operational commands back to the instances. The numerical experiments show that each of the proposed operational algorithms can reduce energy consumption and improve system efficiency. Furthermore, there exists the need to collectively consider multiple planning + operational strategies as multiple strategies can influence each other in terms of performance impacts. Recommendations for future work related to AEV planning and simulation are discussed.
29 ENERGY PLANNING, POLICY, AND ECONOMY↗
A non-orthogonal variational quantum eigensolver
Variational algorithms for strongly correlated chemical and materials systems are one of the most promising applications of near-term quantum computers. We present an extension to the variational quantum eigensolver that approximates the ground state of a system by solving a generalized eigenvalue problem in a subspace spanned by a collection of parametrized quantum states. This allows for the systematic improvement of a logical wavefunction ansatz without a significant increase in circuit complexity. To minimize the circuit complexity of this approach, we propose a strategy for efficiently measuring the Hamiltonian and overlap matrix elements between states parametrized by circuits that commute with the total particle number operator. This strategy doubles the size of the state preparation circuits but not their depth, while adding a small number of additional two-qubit gates relative to standard variational quantum eigensolver. We also propose a classical Monte Carlo scheme to estimate the uncertainty in the ground state energy caused by a finite number of measurements of the matrix elements. We explain how this Monte Carlo procedure can be extended to adaptively schedule the required measurements, reducing the number of circuit executions necessary for a given accuracy. We apply these ideas to two model strongly correlated systems, a square configuration of H4 and the π-system of hexatriene (C6H8).
97 MATHEMATICS AND COMPUTING↗
SDN-NGenIA, a Software Defined Next Generation Integrated Architecture for HEP and Data Intensive Science
This project aims to enable the LHC and other leading programs in high energy physics and other global science domains funded by the DOE to operate with a new level of efficiency and control, through the development of a Next Generation Integrated Architecture (NGenIA) based on intelligent software defined network (SDN)-driven network systems coupled to high throughput applications. While the initial focus is on the challenging LHC use case, the systems and products being developed are general, and apply to many fields of data intensive science ranging from astrophysical sky surveys to bioinformatics and earth observation, as well as other organizations facing the challenges of extracting knowledge from distributed multi-petabyte data stores. A central concept in this development program is a new paradigm ‘consistent network operations’ among widely distributed computing and storage facilities, where stable high throughput flows, at set rates, cross load-balanced network paths, up to flexible high water marks that are adjusted in real time to accommodate other network traffic. The large smooth flows are launched and managed by SDN services that act in concert with the experiments’ site-resident data distribution and management systems, to meet the expanding needs of the science programs. The technical goals include the construction of autonomous, intelligent site-resident services that interact dynamically with network-resident services, and with the science programs’ principal data distribution and management tools, to request or command network resources in support of high throughput petascale to exascale workflow Specific work items include: 1. Developing compact Data Transfer Nodes (DTNs) with auto-tuning functions that support data transfer rates in the 100 Gbps to the 1 Tbps range when used with high throughput data transfer applications 2. Deep site orchestration among virtualized clusters, storage subsystems and subnets to successfully coschedule CPU, storage and network resources 3. Science-program designed site architectures, operational modes, and policy and resource usage priorities, adjudicated across multiple network domains and virtual organizations, using the orchestration functions and methods 4. Seamlessly extending end-to-end operation across both extra-site and intra-site boundaries through the use of Open vSwitch (OVS), FireQoS + next generation Science DMZs 5. Novel methods of system integration that enable granular control of extreme scale long distance transfers through flow matching of scattered source-destination address pairs to multi-domain dynamic circuits 6. Funneling massive sets of streams to DTNs at the site edge hosting petascale buffer pools configured for flows of 100 Gbps and up, exploiting state of the art data transfers 7. Adaptive scheduling based on pervasive end-to-end monitoring, including DTN or compute-node resident agents providing comprehensive end-system profiling 8. Developing unsupervised and supervised machine learning and modeling methods to optimize the workflow involving terabyte to multi-petabyte datasets.
47 OTHER INSTRUMENTATION↗
Quantum/AI Topology-Aware Latency-Adaptive HPC Workflow Scheduling Optimization
The growing demand for more powerful high-performance computing (HPC) systems has led to a steady rise in energy consumption by supercomputing worldwide. This study is focused on comparing our Application-Topology Mapper (ATMapper) to the popular Simple Linux Utility for Resource Management (SLURM) for the purpose of exploring methods that can further optimize job-scheduling within HPC systems. ATMapper is an Artificial-Intelligence based approach to job-scheduling that is currently being enhanced with quantum annealing (QA) to generate optimal schedules faster. We are applying QA to speedup our ATMapper process to achieve higher computing efficiency, thereby reducing HPC energy consumption. Here, we examine how four job-scheduling approaches perform in processor node assignment when using an example network architecture of 4 interconnected nodes. Using a specialized script, we are assessing the schedule of a computation flow with 11 interdependent tasks. The data movements among nodes were tracked to count for the number of interactions (network hops) between nodes needed to complete the tasks. The total number of hops and the job completion time were then used to quantify the efficiency of the different mapping approaches. In addition to SLURM, we also compare our ATMapper to the QA-enabled LBNL TIGER and the D-Wave Distributed Computing processor assignment approaches. The preliminary results showed that our topology-aware, latency-adaptive ATMapper is significantly more efficient when compared to the other scheduling approaches due to its load-imbalance network allocation. The scheduler displayed a computing efficiency of 53% by performing significantly fewer network hops than its alternatives. By reducing the number of hops, ATMapper was able to perform all 11 tasks by using only 3 nodes out of given 4. This research indicates the potential to use QA/AI for HPC job-scheduling. Later, we will test a SLURM simulator program to draw further comparisons on the effectiveness of ATMapper's scheduling approach. The results of this comparison will serve as a baseline for later improving SLURM's performance using a QA-enhanced ATMapper approach.
Caraveo, Braulio [University of Huston - Clear Lak↗
Adaptive PID Gain Scheduling Control for Hydropower Turbine Using Neural CDE and Stochastic Distribution Shaping
This paper introduces a gain-scheduling PID controller design strategy for hydroturbine frequency control mode. This scheme first uses real data to learn the nonlinear dynamics of the hydroturbine using neural controlled differential equations and then perturbs the obtained nonlinear system at different equilibrium points, based on which a static output feedback adaptive dynamic programming algorithm is then used to optimize the PID gains for each equilibrium point. Moreover, a continuous-time version of stochastic distribution control is proposed to further fine-tune the optimized PID gains. Finally, the controller is obtained by implementing linear interpolation between the optimized PID control gains. The simulation results show that the proposed gain-scheduling PID controller can control a larger range of operation points compared with the given fixed PID controller and the baseline method. Compared with the given fixed PID controller, the proposed gain-scheduling PID controller can regulate hydroturbine frequency against disturbances induced by power-load variation with over 50% less overshoot for some operation points.
13 HYDRO ENERGY↗
Online EV Scheduling Algorithms for Adaptive Charging Networks with Global Peak Constraints
Not provided.
Computer Science↗
SI–HCCI Mode Transitions Without Open-Loop Sequence Scheduling: Online Parameter Adaptation
Not provided.
Automation & Control Systems↗
Adaptive Framework for Maintenance Scheduling Based on Dynamic Preventive Intervals and Remaining Useful Life Estimation
Data-based prognostic methods exploit sensor data to forecast the remaining useful life (RUL) of industrial settings to optimize the scheduling of maintenance actions. However, implementing sensors may not be cost-effective or practical for all components. Traditional preventive approaches are not based on sensor data; however, they schedule maintenance at equally spaced intervals, which is not a cost-effective approach since the distribution of the time between failures changes with the degradation state of other parts or changes in working conditions. This study introduces a novel framework comprising two maintenance scheduling strategies. In the absence of sensor data, we propose a novel dynamic preventive policy that adjusts intervention intervals based on the most recent failure data. When sensor data are available, a method for RUL prediction, designated k-LSTM-GFT, is enhanced to dynamically account for RUL prediction uncertainty. The results demonstrate that dynamic preventive maintenance can yield cost reductions of up to 51.8% compared to conventional approaches. The predictive approach optimizes the exploitation of RUL, achieving costs that are only 3–5% higher than the minimum cost achievable while ensuring the safety of critical systems since all of the failures are avoided.
Nunes, Pedro (ORCID:0000000180012172)↗
Development of window scheduler algorithm exploiting natural ventilation and thermal mass for building energy simulation and smart home controls
Building energy simulations often rely on abstract assumptions when it comes to natural ventilation, such as ‘windows always open [or closed]’ or ‘windows open when outdoor temperature is below a certain threshold.’ However, simulations based on these assumptions fail to fully exploit the cooling potential of natural ventilation, as its effectiveness can be enhanced or diminished by various factors, including the presence of thermal mass. This issue also extends to smart home controls, where determining the window schedule becomes challenging without information about the building's response to outdoor conditions. To address these issues, this study has developed an analytical model for window operation schedules that leverages the passive cooling from natural ventilation. The analytical model was validated against a Modelica simulation. A case study utilizing the BESTEST model of ANSI/ASHRAE Standard 140 underwent validation with EnergyPlus simulations, showing strong concordance. The algorithm provides window schedule recommendations adapted to various airflow rates, thermal masses, and climate variations. Notably, the case study demonstrated that proper window scheduling could reduce indoor temperature by up to 8 °C under the given simulation settings, thereby improving resilience and indicating potential energy savings. Furthermore, the paper explores the potential opportunities and challenges this approach presents, especially for building simulation and smart home applications.
42 ENGINEERING↗
An Enhanced Method for Scheduling Observations of Large Sky Error Regions for Finding Optical Counterparts to Transients
The discovery and subsequent study of optical counterparts to transient sources is crucial for their complete astrophysical understanding. Various gamma-ray burst (GRB) detectors, and more notably the ground-based gravitational wave detectors, typically have large uncertainties in the sky positions of detected sources. Searching these large sky regions spanning hundreds of square degrees is a formidable challenge for most ground-based optical telescopes, which can usually image less than tens of square degrees of the sky in a single night. We present algorithms for better scheduling of such follow-up observations in order to maximize the probability of imaging the optical counterpart, based on the all-sky probability distribution of the source position. We incorporate realistic observing constraints such as the diurnal cycle, telescope pointing limitations, available observing time, and the rising/setting of the target at the observatory’s location. We use simulations to demonstrate that our proposed algorithms outperform the default greedy observing schedule used by many observatories. Our algorithms are applicable for follow-up of other transient sources with large positional uncertainties, such as Fermi -detected GRBs, and can easily be adapted for scheduling radio or space-based X-ray follow-up.
79 ASTRONOMY AND ASTROPHYSICS↗
Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents
Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows
The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.
Etz, Brian [ORNL] (ORCID:0000000208554863)↗
Regulatory T Cells Shape the Differential Impact of Radiation Dose-Fractionation Schedules on Host Innate and Adaptive Antitumor Immune Defenses
We examined how radiation dose per fraction (DPF) and total dose, as represented by biological effective dose (BED), can independently and differentially affect the immunomodulatory capacity of radiation therapy (RT).
62 RADIOLOGY AND NUCLEAR MEDICINE↗
Adaptive job and resource management for the growing quantum cloud
As the popularity of quantum computing continues to grow, efficient quantum machine access over the cloud is critical to both academic and industry researchers across the globe. And as cloud quantum computing demands increase exponentially, the analysis of resource consumption and execution characteristics are key to efficient management of jobs and resources at both the vendor-end as well as the client-end. While the analysis and optimization of job / resource consumption and management are popular in the classical HPC domain, it is severely lacking for more nascent technology like quantum computing.This paper proposes optimized adaptive job scheduling to the quantum cloud taking note of primary characteristics such as queuing times and fidelity trends across machines, as well as other characteristics such as quality of service guarantees and machine calibration constraints. Key components of the proposal include a) a prediction model which predicts fidelity trends across machine based on compiled circuit features such as circuit depth and different forms of errors, as well as b) queuing time prediction for each machine based on execution time estimations. Altogether, this proposal is evaluated on simulated IBM machines across a diverse set of quantum applications and system loading scenarios, and is able to reduce wait times by over 3x and improve fidelity by over 40% on specific usecases, when compared to traditional job schedulers.
97 MATHEMATICS AND COMPUTING↗
Experimental Performance of a Nonlinear Control Strategy to Regulate Temperature of a High-Temperature Solar Reactor
Abstract Despite the significant potential of solar thermochemical process technology for storing solar energy as solid-state solar fuel, several challenges have made its industrial application difficult. It is important to note that solar energy has a transient nature that causes instability and reduces process efficiency. Therefore, it is crucial to implement a robust control system to regulate the process temperature and tackle the shortage of incoming solar energy during cloudy weather. In our previous works, different model-based control strategies were developed namely a proportional integral derivative controller (PID) with gain scheduling and adaptive model predictive control (MPC). These methods were tested numerically to regulate the temperature inside a high-temperature tubular solar reactor. In this work, the proposed control strategies were experimentally tested under various operation conditions. The controllers were challenged to track different setpoints (500 °C, 1000 °C, and 1450 °C) with different amounts of gas/particle flowrates. Additionally, the flow controller was tested to regulate the reactor temperature under a cloudy weather scenario. The ultimate goal was to produce 5 kg of reduced solar fuel magnesium manganese oxide (MgMn2O4) successfully, and the controllers were able to track the required process temperature and reject disturbances despite the system's strong nonlinearity. The experimental results showed a maximum error in the temperature setpoint of less than 0.5% (6 °C), and the MPC controller demonstrated superior performance in reducing the control effort and rejecting disturbances.
Energy & Fuels↗