SEARCH · Search NASA

Results for “Pilot workload”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

31 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Flexible Pilot Jobs Framework for Distributed High Throughput Computing

Experimental particle physics has been at the forefront of analyzing the world’s largest datasets for decades. The high-energy physics (HEP) community was among the first to develop suitable software and computing tools for this purpose. GlideinWMS is a Glidein-based workload management system whose purpose is to provide experiments like CMS at CERN, DUNE at Fermilab, and others, a way to access and efficiently use vast amounts of computing resources. This system wants to provide a simple way to submit jobs to a set of computing resources, that will be provided to users behind the scenes. Glideins are the pilot jobs executed on the worker nodes at the grid sites, performing operations such as hardware detection, environment setup, and error handling. After all these operations, they will launch the actual user job. Many grid sites are supported, such as shared clusters, Google CE, and AWS. My internship aimed to design and code a flexible pilot jobs framework that will replace the one used by GlideinWMS, developing a modular and flexible skeleton of the Glidein and adding further functionalities. My project also focused on the application of machine learning techniques as support to this management system.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2020

Exploiting CRIC to streamline the configuration management of GlideinWMS factories for CMS support

GlideinWMS is a workload management and provisioning system that allows sharing computing resources distributed over independent sites. Based on the requests made by GlideinWMS frontends, a dynamically sized pool of resources is created by GlideinWMS pilot factories via pilot job submission to resource sites’ CEs. More than 400 CEs are currently serving more than ten virtual organizations through GlideinWMS, with CMS being the biggest user with 230 CEs. The complex configurations of the parameters defining resource requests, as submitted to those CEs, have been historically managed by manually editing a set of different XML files. New possibilities arise with CMS adopting the CRIC, an information system that collects, aggregates, stores, and exposes, among other things, computing resource data coming from various data providers. The paper will describe the challenges faced when CMS started to use CRIC to automatically generate the GlideinWMS factory configurations. The architecture of the prototype, and the ancillary tools developed to ease this transition, will be discussed. Finally, future plans and milestones will be outlined.

Dost, Jeffrey↗

DOE OSTI2024

Shape-shifting Elephants: Multi-modal Transport for Integrated Research Infrastructure

Data Acquisition (DAQ) workloads form an important class of scientific network traffic that by its nature (1) flows across different research infrastructure, including remote instruments and supercomputer clusters, (2) has ever-increasing throughput demands, and (3) has ever-increasing integration demands---for example, observations at one instrument could trigger a reconfiguration of another instrument. Today's DAQ transfers rely on UDP and (heavily tuned) TCP, but this is driven by convenience rather than suitability. The mismatch between Internet transport protocols and scientific workloads becomes more stark with the steady increase in link capacities, data generation, and integration across research infrastructure.This position paper argues the importance of developing specialized transport protocols for DAQ workloads. It proposes a new transport feature for this kind of elephant flow: multi-modality involves the network actively configuring the transport protocol to change how DAQ flows are processed across different underlying networks that connect scientific research infrastructure. Multi-modality is a layering violation that is proposed as a pragmatic technique for DAQ transport protocol design. It takes advantage of programmable network hardware that is increasingly being deployed in scientific research infrastructure. The paper presents an initial evaluation through a pilot study that includes a Tofino2 switch and Alveo FPGA cards, and using data from a particle detector.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Creating Unit Tests for GlideinWMS using AI tools

GlideinWMS is a workload management system that uses distributed computing to complete tasks, also known as jobs. It is particularly useful for high-throughput computing that’s used in research projects. It relies on Glideins, which are pilot jobs that pull jobs from a queue and provide resources for their completion, based on the jobs requirements. These decisions are made based on resource availability and job requirements. We used new AI tools to add unit tests to GlideinWMS.

Baburashvili, Ilya↗

DOE OSTI2022

A Methodology for Evaluating Operator Usage of Machine Learning Recommendations for Power Grid Contingency Analysis

This work presents the application of a methodology to measure domain expert trust and workload, elicit feedback, and understand the technological usability and impact when a machine learning assistant is introduced into contingency analysis for real-time power grid simulation. The goal of this framework is to rapidly collect and analyze a broad variety of human factors data in order to accelerate the development and evaluation loop for deploying machine learning applications. We describe our methodology and analysis, and we discuss insights gained from a pilot participant about the current usability state of an early technology readiness level (TRL) artificial neural network (ANN) recommender.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2018

Supercomputers, Clouds and Grids powered by BigPanDA for Brain studies

The PanDA WMS - Production and Distributed Analysis Workload Management System - has been developed and used by the ATLAS experiment at the LHC (Large Hadron Collider) for all data processing and analysis challenges. BigPanDA is an extension of the PanDA WMS to run ATLAS and non-ATLAS applications on Leadership Class Facilities and supercomputers, as well as traditional grid and cloud resources. The success of the BigPanDA project has drawn attention from other compute intensive sciences such as biology. In 2017, a pilot project was started between BigPanDA and the Blue Brain Project (BBP) of the Ecole Polytechnique Federal de Lausanne (EPFL) located in Lausanne, Switzerland. This proof of concept project is aimed at demonstrating the efficient application of the BigPanDA system to support the complex scientific workflow of the BBP, which relies on using a mix of desktop, cluster and supercomputers to reconstruct and simulate accurate models of brain tissue.

Computer Science↗

DOE OSTI2015

Evolution of CMS Workload Management Towards Multicore Job Support

The successful exploitation of multicore processor architectures is a key element of the LHC distributed computing system in the coming era of the LHC Run 2. High-pileup complex-collision events represent a challenge for the traditional sequential programming in terms of memory and processing time budget. The CMS data production and processing framework is introducing the parallel execution of the reconstruction and simulation algorithms to overcome these limitations. CMS plans to execute multicore jobs while still supporting singlecore processing for other tasks difficult to parallelize, such as user analysis. The CMS strategy for job management thus aims at integrating single and multicore job scheduling across the Grid. This is accomplished by employing multicore pilots with internal dynamic partitioning of the allocated resources, capable of running payloads of various core counts simultaneously. An extensive test programme has been conducted to enable multicore scheduling with the various local batch systems available at CMS sites, with the focus on the Tier-0 and Tier-1s, responsible during 2015 of the prompt data reconstruction. Scale tests have been run to analyse the performance of this scheduling strategy and ensure an efficient use of the distributed resources. This paper presents the evolution of the CMS job management and resource provisioning systems in order to support this hybrid scheduling model, as well as its deployment and performance tests, which will enable CMS to transition to a multicore production model for the second LHC run.

Perez-Calero Yzquierdo, A.↗

DOE OSTI2014

Light Water Reactor Sustainability Program: Digital Technology Business Case Methodology Guide

The Department of Energy’s (DOE’s) Light Water Reactor Sustainability Program aims to develop and deploy technologies that will make the existing U.S. nuclear fleet more efficient and competitive. The program has developed a standard methodology for determining the impact of new technologies in order to assist nuclear power plant (NPP) operators in building sound business cases. The Advanced Instrumentation, Information, and Control (II&C) Systems Technologies Pathway is part of the DOE’s Light Water Reactor Sustainability (LWRS) Program. It conducts targeted research and development (R&D) to address aging and reliability concerns with the legacy instrumentation and control and related information systems of the U.S. operating light water reactor (LWR) fleet. This work involves two major goals: (1) to ensure that legacy analog II&C systems are not life-limiting issues for the LWR fleet and (2) to implement digital II&C technology in a manner that enables broad innovation and business improvement in the NPP operating model. Resolving long-term operational concerns with the II&C systems contributes to the long-term sustainability of the LWR fleet, which is vital to the nation’s energy and environmental security. The II&C Pathway is conducting a series of pilot projects that enable the development and deployment of new II&C technologies in existing nuclear plants. Through the LWRS program, individual utilities and plants are able to participate in these projects or otherwise leverage the results of projects conducted at demonstration plants. Performance advantages of the new pilot project technologies are widely acknowledged, but it has proven difficult for utilities to derive business cases for justifying investment in these new capabilities. Lack of a business case is often cited by utilities as a barrier to pursuing wide-scale application of digital technologies to nuclear plant work activities. The decision to move forward with funding usually hinges on demonstrating actual cost reductions that can be credited to budgets and thereby truly reduce O&M or capital costs. Technology enhancements, while enhancing work methods and making work more efficient, often fail to eliminate workload such that it changes overall staffing and material cost requirements. It is critical to demonstrate cost reductions or impacts on non-cost performance objectives in order for the business case to justify investment by nuclear operators. The Business Case Methodology (BCM) addresses the “benefit” side of the analysis—as opposed to the cost side—and how the organization evaluates discretionary projects (net present value (NPV), accounting effects of taxes, discount rates, etc.). The cost and analysis side is not particularly difficult for the organization and can usually be determined with a fair amount of precision (not withstanding implementation project cost overruns). It is in determining the "benefits" side of the analysis that utilities have more difficulty in technology projects and that is the focus of this methodology.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2015

Pilot Project Technology Business Case: Mobile Work Packages

Performance advantages of the new pilot project technologies are widely acknowledged, but it has proven difficult for utilities to derive business cases for justifying investment in these new capabilities. Lack of a business case is often cited by utilities as a barrier to pursuing wide-scale application of digital technologies to nuclear plant work activities. The decision to move forward with funding usually hinges on demonstrating actual cost reductions that can be credited to budgets and thereby truly reduce O&M or capital costs. Technology enhancements, while enhancing work methods and making work more efficient, often fail to eliminate workload such that it changes overall staffing and material cost requirements. It is critical to demonstrate cost reductions or impacts on non-cost performance objectives in order for the business case to justify investment by nuclear operators. The Business Case Methodology (BCM) was developed in September of 2015 to frame the benefit side of II&C technologies to address the “benefit” side of the analysis—as opposed to the cost side—and how the organization evaluates discretionary projects (net present value (NPV), accounting effects of taxes, discount rates, etc.). The cost and analysis side is not particularly difficult for the organization and can usually be determined with a fair amount of precision (not withstanding implementation project cost overruns). It is in determining the “benefits” side of the analysis that utilities have more difficulty in technology projects and that is the focus of this methodology. The methodology is presented in the context of the entire process, but the tool provided is limited to determining the organizational benefits only. This report describes a the use of the BCM in building a business case for mobile work packages, which includes computer-based procedures and other automated elements of a work package. Key to those impacts will be identifying where the savings are “harvestable,” meaning they result in an actual reduction in headcount and/or cost. The report describes the specific activities conducted with a partner utility to examine the various work activities associated with mobile work packages to determine what time savings and error rate reductions are available. The report summarizes these findings in the form of a business case for the technology.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2019

Human and Robotic Testing in Gloveboxes to Reduce Worker Fatigue - 19542

A DOE-sponsored study conducted by the Los Alamos National Laboratory reported that the incidence rate of repetitive stress injuries (RSI) during manipulation tasks in glovebox and hot cell workspaces increased from 22% after 1-2 hours of work per day to 50% after 3 hours of work per day due to fatigue, repetitive motion, and hyperextension. Robotics can be a viable solution to this issue as robotic manipulators and wearable robotic devices can provide effective assistance to alleviate the workload. In order to use the robotic devices, an initial study to detect the amount of fatigue experienced by the glovebox operators is essential. Hence, in the present research two objectives have been specified and investigated. These include the use of electromyography (EMG) sensors to precisely detect the fatigue in various hands, arm and shoulder muscles and; the use of dexterous robotic arm and hand combinations to effectively perform certain repetitive and monotonous tasks conducted inside an in-house glovebox constructed for the research. A UR5 robotic arm and an i-limb artificial hand have been used. Human tests included eight subjects and two customized glovebox tasks. Pilot scale test results indicate variation in the mean frequency which is an indication of the level of fatigue that occurred. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2024

Using containers to speed up development, to run integration tests and to teach about distributed systems

GlideinWMS is a workload manager provisioning resources for many experiments including CMS and DUNE. The software is distributed both as native packages and specialized production containers. Following an approach used in other communities like web development we built our workspaces, system-like containers to ease development and testing. Developers can change the source tree or check out a different branch and quickly reconfigure the services to see the effect of their changes. In this paper, we’ll talk about what differentiates workspaces from other containers. We’ll describe our base system composed of three containers. A one-node cluster including a compute element and a batch system. A GlideinWMS Factory controlling pilot jobs. And a scheduler and Frontend, to submit jobs and provision resources. Additional containers can be used for optional components. This system can easily run on a laptop and we’ll share our evaluation of different container runtimes, with an eye for ease of use and performance. Finally, we’ll talk about our experience as developers and with students. The GlideinWMS workspaces are easily integrated with IDEs like VS Code, simplifying debugging and allowing development and testing of the system also when offline. They simplified the training and onboarding of new team members and Summer interns. And they were useful in workshops where students could have first-hand experience with the mechanisms and components that, in production, run millions of jobs.

Mambelli, Marco↗

DOE OSTI2025

Using Containers to Speed Up Development, to Run Integration Tests and to Teach About Distributed Systems

GlideinWMS is a workload manager provisioning resources for many experiments, including CMS and DUNE. The software is distributed both as native packages and specialized production containers. Following an approach used in other communities like web development, we built our workspaces, system-like containers to ease development and testing. Developers can change the source tree or check out a different branch and quickly reconfigure the services to see the effect of their changes. In this paper, we will talk about what differentiates workspaces from other containers. We will describe our base system, composed of three containers: a one-node cluster including a compute element and a batch system, a GlideinWMS Factory controlling pilot jobs, and a scheduler and Frontend to submit jobs and provision resources. Additional containers can be used for optional components. This system can easily run on a laptop, and we will share our evaluation of different container runtimes, with an eye for ease of use and performance. Finally, we will talk about our experience as developers and with students. The GlideinWMS workspaces are easily integrated with IDEs like VS Code, simplifying debugging and allowing development and testing of the system even when offline. They simplified the training and onboarding of new team members and summer interns. And they were useful in workshops where students could have first-hand experience with the mechanisms and components that, in production, run millions of jobs.

Mambelli, Marco [Fermilab] (ORCID:0000000294892681↗

DOE OSTI2024

Adoption of a token-based authentication model for the CMS Submission Infrastructure

The CMS Submission Infrastructure (SI) is the main computing resource provisioning system for CMS workloads. A number of HTCondor pools are employed to manage this infrastructure, which aggregates geographically distributed resources from the WLCG and other providers. Historically, the model of authentication among the diverse components of this infrastructure has relied on the Grid Security Infrastructure (GSI), based on identities and X509 certificates. In contrast, commonly used modern authentication standards are based on capabilities and tokens. The WLCG has identified this trend and aims at a transparent replacement of GSI for all its workload management, data transfer and storage access operations, to be completed during the current LHC Run 3. As part of this effort, and within the context of CMS computing, the Submission Infrastructure group is in the process of phasing out the GSI part of its authentication layers, in favor of IDTokens and Scitokens. The use of tokens is already well integrated into the HTCondor Software Suite, which has allowed us to fully migrate the authentication between internal components of SI. Additionally, recent versions of the HTCondor-CE support tokens as well, enabling CMS resource requests to Grid sites employing this CE technology to be granted by means of token exchange. After a rollout campaign to sites, successfully completed by the third quarter of 2022, the totality of HTCondor CEs in use by CMS are already receiving Scitoken-based pilot jobs. On the ARC CE side, a parallel campaign was launched to foster the adoption of the REST interface at CMS sites (required to enable token-based job submission via HTCondor-G), which is nearing completion as well. In this contribution, the newly adopted authentication model will be described. We will then report on the migration status and final steps towards complete GSI phase out in the CMS SI.

Pérez-Calero Yzquierdo, Antonio↗