Search NASA⌕ Search
Home/Archive/DOE OSTI

DOE OSTI · 2283314

PanDA: Production and Distributed Analysis System

Maeno, Tadashi·Alekseev, Aleksandr·Barreiro Megino, Fernando Harald·De, Kaushik·Guan, Wen·Karavakis, Edward·Klimentov, Alexei·Korchuganova, Tatiana·Lin, FaHui·Nilsson, Paul·Wenaus, Torre·Yang, Zhaoyu·Zhao, Xin
View original source ↗Read source document ↗Export metadata ↓

Abstract

The Production and Distributed Analysis (PanDA) system is a data-driven workload management system engineered to operate at the LHC data processing scale. The PanDA system provides a solution for scientific experiments to fully leverage their distributed heterogeneous resources, showcasing scalability, usability, flexibility, and robustness. The system has successfully proven itself through nearly two decades of steady operation in the ATLAS experiment, addressing the intricate requirements such as diverse resources distributed worldwide at about 200 sites, thousands of scientists analyzing the data remotely, the volume of processed data beyond the exabyte scale, dozens of scientific applications to support, and data processing over several billion hours of computing usage per year. PanDA’s flexibility and scalability make it suitable for the High Energy Physics community and wider science domains at the Exascale. Beyond High Energy Physics, PanDA’s relevance extends to other big data sciences, as evidenced by its adoption in the Vera C. Rubin Observatory and the sPHENIX experiment. As the significance of advanced workflows continues to grow, PanDA has transformed into a comprehensive ecosystem, effectively tackling challenges associated with emerging workflows and evolving computing technologies. The paper discusses PanDA’s prominent role in the scientific landscape, detailing its architecture, functionality, deployment strategies, project management approaches, results, and evolution into an ecosystem.

Explore related subjects

97 MATHEMATICS AND COMPUTINGDistributed computingExascaleHeterogeneous computingWorkload management

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Maeno, Tadashi, Alekseev, Aleksandr, Barreiro Megino, Fernando Harald, De, Kaushik, Guan, Wen, Karavakis, Edward, Klimentov, Alexei, Korchuganova, Tatiana, Lin, FaHui, Nilsson, Paul, Wenaus, Torre, Yang, Zhaoyu, Zhao, Xin. 2024-01-23. PanDA: Production and Distributed Analysis System. https://doi.org/10.1007/s41781-024-00114-3

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

DOE OSTI2026

TANTE: Time-adaptive operator learning via neural Taylor expansion

Operator learning for time-dependent partial differential equations (PDEs) has seen rapid progress in recent years, enabling efficient approximation of complex spatiotemporal dynamics. However, most existing methods rely on fixed time step sizes during rollout, which limits their ability to adapt to varying temporal complexity and often leads to error accumulation. In this work, we propose the Time-Adaptive Transformer with Neural Taylor Expansion (TANTE), a novel operator-learning framework that produces continuous-time predictions with adaptive step sizes. TANTE predicts future states by performing a Taylor expansion at the current state, where neural networks learn both the higher-order temporal derivatives and the local radius of convergence. This allows the model to dynamically adjust its rollout based on the local behavior of the solution, thereby reducing cumulative error and improving computational efficiency. We demonstrate the effectiveness of TANTE across a wide range of PDE benchmarks, achieving superior accuracy and adaptability compared to fixed-step baselines, delivering accuracy gains of 60-80 % and speed-ups of 30-40 % at inference time.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Structured illumination for surface-resolved grazing-incidence X-ray scattering

Grazing-incidence (GI) scattering techniques are widely used to characterize thin films, offering high surface sensitivity and insight into morphology and structure. However, these approaches typically provide statistical averaged information due to elongated footprint or limited spatial resolution due to beam size. Here we introduce a method that combines structured illumination with GI X-ray scattering and leverages our computational imaging approach to resolve local structural details. We demonstrate that our method captures local features of an organic semiconductor thin film without the need for sample rotation as in tomography. The method expands GI techniques from statistical averaging to high-resolution imaging, thereby providing the capability for detailed analysis of local material properties, such as domain shape, orientation and polymorphism, which are critical for advancing material design towards more efficient and tailored materials.

97 MATHEMATICS AND COMPUTING↗