Search NASA⌕ Search

SEARCH · Search NASA

Results for “PanDA”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory

The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment’s run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory’s needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

The.LHC, operating at CERN, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. While PanDA currently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, LHC data taking runs require more resources than grid can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, in particular with Titan supercomputer at Oak Ridge Leadership Computing Facility. Current approach utilizes modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single threaded workloads in parallel on LCFs multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms for ALICE and ATLAS experiments and it is in full pro duction for the ATLAS since September 2015. We will present our current accomplishments with running PanDA at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

INTEGRATION OF PANDA WORKLOAD MANAGEMENT SYSTEM WITH SUPERCOMPUTERS

Abstract The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the funda- mental nature of matter and the basic forces that shape our universe, and were recently credited for the dis- covery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 140 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data cen- ters are physically scattered all over the world. While PanDA currently uses more than 250000 cores with a peak performance of 0.3+ petaFLOPS, next LHC data taking runs will require more resources than Grid computing can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, Europe and Russia (in particular with Titan supercomputer at Oak Ridge Leadership Com- puting Facility (OLCF), Supercomputer at the National Research Center Kurchatov Institute , IT4 in Ostrava, and others). The current approach utilizes a modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single- threaded workloads in parallel on Titan s multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms. We will present our current accom- plishments in running PanDA WMS at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facility s infrastructure for High Energy and Nuclear Physics, as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

De, K↗
DOE OSTI2016

Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. While PanDA currently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, LHC data taking runs require more resources than Grid computing can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, Europe and Russia (in particular with Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF), MIRA supercomputer at Argonne Leadership Computing Facilities (ALCF), Supercomputer at the National Research Center Kurchatov Institute , IT4 in Ostrava and others). Current approach utilizes modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single threaded workloads in parallel on LCFs multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms for ALICE and ATLAS experiments and it is in full production for the ATLAS experiment since September 2015. We will present our current accomplishments with running PanDA WMS at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

Data-intensive science↗
DOE OSTI2024

PanDA: Production and Distributed Analysis System

The Production and Distributed Analysis (PanDA) system is a data-driven workload management system engineered to operate at the LHC data processing scale. The PanDA system provides a solution for scientific experiments to fully leverage their distributed heterogeneous resources, showcasing scalability, usability, flexibility, and robustness. The system has successfully proven itself through nearly two decades of steady operation in the ATLAS experiment, addressing the intricate requirements such as diverse resources distributed worldwide at about 200 sites, thousands of scientists analyzing the data remotely, the volume of processed data beyond the exabyte scale, dozens of scientific applications to support, and data processing over several billion hours of computing usage per year. PanDA’s flexibility and scalability make it suitable for the High Energy Physics community and wider science domains at the Exascale. Beyond High Energy Physics, PanDA’s relevance extends to other big data sciences, as evidenced by its adoption in the Vera C. Rubin Observatory and the sPHENIX experiment. As the significance of advanced workflows continues to grow, PanDA has transformed into a comprehensive ecosystem, effectively tackling challenges associated with emerging workflows and evolving computing technologies. The paper discusses PanDA’s prominent role in the scientific landscape, detailing its architecture, functionality, deployment strategies, project management approaches, results, and evolution into an ecosystem.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Integration of PanDA workload management system with Titan supercomputer at OLCF

The PanDA (Production and Distributed Analysis) workload management system (WMS) was developed to meet the scale and complexity of LHC distributed computing for the ATLAS experiment. While PanDA currently distributes jobs to more than 100,000 cores at well over 100 Grid sites, the future LHC data taking runs will require more resources than Grid computing can possibly provide. To alleviate these challenges, ATLAS is engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF). The current approach utilizes a modified PanDA pilot framework for job submission to Titan’s batch queues and local data management, with light-weight MPI wrappers to run single threaded workloads in parallel on Titan’s multi-core worker nodes. It also gives PanDA new capability to collect, in real time, information about unused worker nodes on Titan, which allows precise definition of the size and duration of jobs submitted to Titan according to available free resources. This capability significantly reduces PanDA job wait time while improving Titan’s utilization efficiency. This implementation was tested with a variety of Monte-Carlo workloads on Titan and is being tested on several other supercomputing platforms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

THE PAndAS VIEW OF THE ANDROMEDA SATELLITE SYSTEM. I. A BAYESIAN SEARCH FOR DWARF GALAXIES USING SPATIAL AND COLOR-MAGNITUDE INFORMATION

We present a generic algorithm to search for dwarf galaxies in photometric catalogs and apply it to the Pan-Andromeda Archaeological Survey (PAndAS). The algorithm is developed in a Bayesian framework and, contrary to most dwarf galaxy search codes, makes use of both the spatial and color-magnitude information of sources in a probabilistic approach. Accounting for the significant contamination from the Milky Way foreground and from the structured stellar halo of the Andromeda galaxy, we recover all known dwarf galaxies in the PAndAS footprint with high significance, even for the least luminous ones. Some Andromeda globular clusters are also recovered and, in one case, discovered. We publish a list of the 143 most significant detections yielded by the algorithm. The combined properties of the 39 most significant isolated detections show hints that at least some of these trace genuine dwarf galaxies, too faint to be individually detected. Follow-up observations by the community are mandatory to establish which are real members of the Andromeda satellite system. The search technique presented here will be used in an upcoming contribution to determine the PAndAS completeness limits for dwarf galaxies. Although here tuned to the search of dwarf galaxies in the PAndAS data, the algorithm can easily be adapted to the search for any localized overdensity whose properties can be modeled reliably in the parameter space of any catalog.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2012

Evolution of the ATLAS PanDA Production and Distributed Analysis System

Evolution of the ATLAS PanDA Production and Distributed Analysis System T Maeno1,5, K De2, T Wenaus1, P Nilsson2, R Walker3, A Stradling2, V Fine1, M Potekhin1, S Panitkin1 and G Compostella4 Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series, Volume 396, Part 3 Article PDF References Citations Metrics 101 Total downloads Cited by 8 articles Turn on MathJax Share this article Article information Abstract The PanDA (Production and Distributed Analysis) system has been developed to meet ATLAS production and analysis requirements for a data-driven workload management system capable of operating at LHC data processing scale. PanDA has performed well with high reliability and robustness during the two years of LHC data-taking, while being actively evolved to meet the rapidly changing requirements for analysis use cases. We will present an overview of system evolution including automatic rebrokerage and reattempt for analysis jobs, adaptation for the CernVM File System, support for the multi-cloud model through which Tier-2 sites act as members of multiple clouds, pledged resource management and preferential brokerage, and monitoring improvements. We will also describe results from the analysis of two years of PanDA usage statistics, current issues, and plans for the future.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Big PanDa Workflow Management on Titan for High Energy and Nuclear Physics and for Future Extreme Scale Scientific Application

Over a three year period, from 2016-2019, this project demonstrated the scientific benefits of integrating the Titan supercomputer at Oak Ridge Leadership Computing Facility into traditional high throughput grid based distributed computing systems managed by PanDA, the workflow management system used for the execution of all distributed computing applications by the ATLAS experiment at the Large Hadron Collider. PanDA manages millions of batch jobs daily at hundreds of clusters worldwide on request by thousands of physicist users, and processes more than an exabyte of data annually using grid middleware. High levels of operational use of Titan was sustained by PanDA in order to meet the physics goals of ATLAS. The success of this project led to the use of other supercomputers worldwide by ATLAS, and to the adoption of PanDA by other experiments and other scientists. Multiple innovative operational and computer science research goals were achieved supporting the use of supercomputers for scientific domains with large scale distributed data and distributed processing needs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2012

PD2P: PanDA Dynamic Data Placement for ATLAS

The PanDA (Production and Distributed Analysis) system plays a key role in the ATLAS distributed computing infrastructure. PanDA is the ATLAS workload management system for processing all Monte-Carlo (MC) simulation and data reprocessing jobs in addition to user and group analysis jobs. The PanDA Dynamic Data Placement (PD2P) system has been developed to cope with difficulties of data placement for ATLAS. We will describe the design of the new system, its performance during the past year of data taking, dramatic improvements it has brought about in the efficient use of storage and processing resources, and plans for the future.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS

Machine Learning (ML) has become one of the important tools for High Energy Physics analysis. As the size of the dataset increases at the Large Hadron Collider (LHC), and at the same time the search spaces become bigger and bigger in order to exploit the physics potentials, more and more computing resources are required for processing these ML tasks. In addition, complex advanced ML workflows are developed in which one task may depend on the results of previous tasks. How to make use of vast distributed CPUs/GPUs in WLCG for these big complex ML tasks has become a popular research area. In this paper, we present our efforts enabling the execution of distributed ML workflows on the Production and Distributed Analysis (PanDA) system and intelligent Data Delivery Service (iDDS). First, we describe how PanDA and iDDS deal with large-scale ML workflows, including the implementation to process workloads on diverse and geographically distributed computing resources. Next, we report real-world use cases, such as HyperParameter Optimization, Monte Carlo Toy confidence limits calculation, and Active Learning. Finally, we conclude with future plans.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Physics with charmonium - Highlights of BESIII and PANDA

The physics of the strong interaction is undoubtedly one of the most challenging areas of modern science. Quantum ChromoDynamics (QCD) is reproducing successfully the physics phenomena at distances much shorter than the size of the nucleon, where perturbation theory can be used yielding results of high precision and predictive power. At larger distance scales, however, perturbative methods cannot be applied anymore, although spectacular phenomena, such as the generation of hadron masses and quark confinement, occur. Studies using charmed quarks and gluon-rich matter have the potential to connect the perturbative and the non-perturbative QCD region. The annihilation of matter with antimatter in the mass regime of charmonium is an ideal environment to discover new states or transitions that could reveal the secrets of the strong interaction. Hadronic and electromagnetic transitions between charmonium states and their decays have been measured with a world-record in precision with the BESIII spectrometer at the electron-positron collider at IHEP Beijing, China. Moreover, unconventional narrow charmonium-rich states have been discovered recently in an energy regime above the open-charm threshold, thereby, possibly initiating a new era in charmonium spectroscopy. The near future experiment, PANDA, at the research facility FAIR in Germany, Darmstadt, will exploit the annihilation of cooled anti-protons with protons to perform charmonium spectroscopy with an incredible precision. I will present the most promising results that have been recently obtained with BESIII together with the future perspectives of PANDA in the field of charmonium spectroscopy.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2014

The PAndAS field of streams: Stellar structures in the milky way halo toward Andromeda and Triangulum

We reveal the highly structured nature of the Milky Way (MW) stellar halo within the footprint of the Pan-Andromeda Archaeological Survey (PAndAS) photometric survey from blue main sequence (MS) and MS turn-off stars. We map no fewer than five stellar structures within a heliocentric range of ∼5-30 kpc. Some of these are known (the Monoceros Ring, the Pisces/Triangulum globular cluster stream), but we also uncover three well-defined stellar structures that could be, at least partly, responsible for the so-called Triangulum/Andromeda and Triangulum/Andromeda 2 features. In particular, we trace a new faint stellar stream located at a heliocentric distance of ∼17 kpc. With a surface brightness of Σ {sub V} ∼ 32-32.5 mag arcsec{sup –2}, it follows an orbit that is almost parallel to the Galactic plane north of M31 and has so far eluded surveys of the MW halo as these tend to steer away from regions dominated by the Galactic disk. Investigating our follow-up spectroscopic observations of PAndAS, we serendipitously uncover a radial velocity signature from stars that have colors and magnitudes compatible with the stream. From the velocity of eight likely member stars, we show that this stellar structure is dynamically cold, with an unresolved velocity dispersion that is lower than 7.1 km s{sup –1} at the 90% confidence level. Along with the width of the stream (300-650 pc), its dynamics point to a dwarf-galaxy-accretion origin. The numerous stellar structures we can map in the MW stellar halo between 5 and 30 kpc and their varying morphology is a testament to the complex nature of the stellar halo at these intermediate distances.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

GDX-pandas [SWR 15-21]

This is a Python package to translate between gdx (GAMS data) and pandas. There are two main ways to use gdxpds. The first use case is the one that was initially supported: direct conversion between GDX files on disk and pandas DataFrames or a csv version thereof. Starting with the Version 1.0.0 rewrite, there is now a second style of use which involves interfacing with GDX files and symbols via the `gdxpds.gdx.GdxFile` and `gdxpds.gdx.GdxSymbol` classes. Please visit https://nrel.github.io/gdx-pandas for the latest documentation.

Hale, Elaine↗
DOE OSTI2020

pandas-sacct v1.0.0

pandas-sacct is a small python utility which provides a pandas dataframe based interface to Slurm's accounting database.

Cooke, Brandon↗
DOE OSTI2016

The PAndAS view of the andromeda satellite system. II. Detailed properties of 23 M31 dwarf spheroidal galaxies

We present a comprehensive analysis of the structural properties and luminosities of the 23 dwarf spheroidal galaxies that fall within the footprint of the Pan-Andromeda Archaeological Survey (PAndAS). These dwarf galaxies represent the large majority of Andromeda’s known satellite dwarf galaxies and cover a wide range in luminosity (−11.6≲M{sub V}≲−5.8 or 10{sup 4.2}≲L≲10{sup 6.5} L{sub ⊙}) and surface brightness (25.1≲μ{sub 0}≲29.3 mag arcsec{sup −2}). We confirm most previous measurements, but we find And XIX to be significantly larger than before (r{sub h}=3065{sub −935}{sup +1065} pc, M{sub V}=−10.1{sub −0.4}{sup +0.8}) and cannot derive parameters for And XXVII as it is likely not a bound stellar system. We also significantly revise downward the luminosities of And XV and And XVI, which are now M{sub V}∼−7.5 or L∼10{sup 5} L{sub ⊙}. Finally, we provide the first detailed analysis of Cas II/And XXX, a fairly faint system (M{sub V}=−8.0{sub −0.3}{sup +0.4}) of typical size (r{sub h}=270±50 pc), located in close proximity to the two bright elliptical dwarf galaxies NGC 147 and NGC 185. Combined with the set of homogeneous distances published in an earlier contribution, our analysis dutifully tracks all relevant sources of uncertainty in the determination of the properties of the dwarf galaxies from the PAndAS photometric catalog. We further publish the posterior probability distribution functions of all the parameters we fit for in the form of MCMC chains available online; these inputs should be used in any analysis that aims to remain truthful to the data and properly account for covariance between parameters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. While PanDAcurrently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, it runs around 2 million jobs per day on hundreds of Grid sites and serving thousands of ATLAS users. In 2017 about 1.5 exabytes of data were processed with PanDA.In 2012 BigPanDA project project was started with aim to introduce new types of computing resources into ATLAS computing infrastructure, but also to offering PanDA features to different data-intensive applications for projects and experiments outside of ATLAS and High-Energy and Nuclear Physics. In this article we will present accomplishments and discuss possible directions for future work.

Svirin, Pavlo↗