Morozova, Viktoriya | Search NASA

DOE OSTI2023

An Extension of the Athena++ Code Framework for Radiation-magnetohydrodynamics in General Relativity Using a Finite-solid-angle Discretization

We extend the general-relativistic magnetohydrodynamics (GRMHD) capabilities of Athena++ to incorporate radiation. The intensity field in each finite-volume cell is discretized in angle, with explicit transport in both space and angle properly accounting for the effects of gravity on null geodesics, and with matter and radiation coupled in a locally implicit fashion. Here we describe the numerical procedure in detail, verifying its correctness with a suite of tests. Motivated in particular by black hole accretion in the high-accretion-rate, thin-disk regime, we demonstrate the application of the method to this problem. With excellent scaling on flagship computing clusters, the port of the algorithm to the GPU-enabled AthenaK code now allows the simulation of many previously intractable radiation-GRMHD systems.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

GR-Athena++: Puncture Evolutions on Vertex-centered Oct-tree Adaptive Mesh Refinement

Numerical relativity is central to the investigation of astrophysical sources in the dynamical and strong-field gravity regime, such as binary black hole and neutron star coalescences. Current challenges set by gravitational-wave and multimessenger astronomy call for highly performant and scalable codes on modern massively parallel architectures. We present GR-Athena++, a general-relativistic, high-order, vertex-centered solver that extends the oct-tree, adaptive mesh refinement capabilities of the astrophysical (radiation) magnetohydrodynamics code Athena++. To simulate dynamical spacetimes, GR-Athena++ uses the Z4c evolution scheme of numerical relativity coupled to the moving puncture gauge. We demonstrate stable and accurate binary black hole merger evolutions via extensive convergence testing, cross-code validation, and verification against state-of-the-art effective-one-body waveforms. GR-Athena++ leverages the task-based parallelism paradigm of Athena++ to achieve excellent scalability. We measure strong-scaling efficiencies above 95% for up to ~1.2 × 10 4 CPUs and excellent weak scaling is shown up to ~10 5 CPUs in a production binary black hole setup with adaptive mesh refinement. GR-Athena++ thus allows for the robust simulation of compact binary coalescences and offers a viable path toward numerical relativity at exascale.

79 ASTRONOMY AND ASTROPHYSICS↗