Analyzing NOvA Neutrino Data with the Perlmutter Supercomputer
Explore the source record for details and available documents.
Engineering topics
Publications and source records attributed to Doyle, Derek.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Modern experiments in high energy physics analyze millions of events recorded in particle detectors to select the events of interest and make measurements of physics parameters. These data can often be stored as tabular data in files with detector information and reconstructed quantities. Current techniques for event selection in these files lack the scalability needed for high performance computing environments. We describe our work to develop a high energy physics analysis framework suitable for high performance computing. This new framework utilizes modern tools for reading files and implicit data parallelism. Framework users analyze tabular data using standard, easy-to-use data analysis techniques in Python while the framework handles the file manipulations and parallelism without the user needing advanced experience in parallel programming. In future versions, we hope to provide a framework that can be utilized on a personal computer or a high performance computing cluster with little change to the user code.
Analysis of neutrino oscillation data involves a combination of complex fitting procedures and statistical correction techniques that are used to determine the full three-flavor PMNS parameters and constraint contours. These techniques rely on computationally intensive “multi-universe” stochastic modeling. The process of calculating these contours and corrections can dominate final stages of the data analysis and become a bottleneck for examining the ef-fect of systematic variations on the final results. As part of the DOE SciDAC-4 sponsored research program, we present a new implementation of a neutrino oscillation fitting and framework to carry out calculations of Feldman-Cousins corrections. This implementation is specifically designed and optimized to operate on modern High-Performance Computing (HPC) facilities. We present the performance of the system in calculating allowed regions for the NOvA experiment based on 8.9 x 1020 and 6.9 x 1020 protons-on-target (POT) neutrino and antineutrino datasets, and compare the performance of this new implementation run at the NERSC supercomputing facility with that from methods used previously by the NOvA collaboration running on grid computing facilities.
aThe NOvA (NuMI Off-axis $\nu_{e}$ Appearance) experiment is a two-detector, long-baseline neutrino oscillation experiment hosted by Fermilab. The primary goal is to measure neutrino oscillation parameters associated with $\nu_\mu \rightarrow \nu_{\mu}$ disappearance and $\nu_\mu \rightarrow \nu_{e} $appearance over a distance of 810 km. These measurements require correct modeling of final state particle kinematics from charged-current (CC) neutrino interactions, thus the $\nu_{e}$ CC inclusive cross section is of particular importance to the appearance measurement. The high rate of neutrino interactions in the NOvA near detector located at Fermilab presents an excellent opportunity for neutrino-nucleus cross section measurements. We present the motivations, analysis techniques and current status of $\nu_{e}$ CC inclusive double-differential cross-section measurement using neutrino interactions recorded in the NOvA near detector.
We describe our work on the use of HPC data formats and storage technologies to efficiently describe, store and access High Energy Physics (HEP) data. In the HEP community, important data sets are large, currently at the scale of tens of PiB, and are expected to grow dramatically in the next decade. These data require different data representations—ranging from complex C++ objects to tabular data—that are used at the many varied stages of data analysis. Traditionally, these data have been stored in large collections of files at each stage. This approach does not scale well on HPC resources. To obtain good scalability, without requiring that analysts have parallel programming expertise, our work stresses the use of HPCoptimized tools and implicit data-parallelism. We describe data organization and the performance and scaling of data processing using examples from three HEP experiments (LArIAT, NOvA and DUNE), and for three different stages of analysis. We compare the performance of our HPC-oriented solutions to the traditional HEP approaches.
When fitting to data with low stats and near physical boundaries, extra measures need to be taken to ensure proper statistical coverage. The method NOvA uses is called the Feldman-Cousins procedure, which entails fitting thousands of independent pseudoexperiments to generate acceptance intervals that are then used to correct our fits. The scale required by the Feldman Cousins procedure makes it extremely computationally intensive. In past analyses, it has taken up to several weeks to complete, bottlenecking our final results. Here I present recent work by members of the NOvA experiment and the SciDAC4 collaboration to enable the use of the super-computing facilities at NERSC to process our Feldman-Cousins corrections over 150x faster, allowing us to do more studies, increase the precision of our fits, and produce results quickly.
Abstract not provided.