Engineering topics

Suchyta, Eric

Publications and source records attributed to Suchyta, Eric.

15 records

DOE OSTI2024

Performance Improvements of Poincaré Analysis for Exascale Fusion Simulations

Understanding the time-varying magnetic field in a fusion device is critical for the successful design and construction of clean-burning fusion power plants. Poincaré analysis provides a powerful method for the visualization of magnetic fields in fusion devices. However, Poincaré plots can be very computationally expensive making it impractical, for example, to generate these plots in situ during a simulation. In this short paper, we describe a collaboration among computer science and physics researchers to develop a new Poincaré tool that provides a significant reduction in the time to generate analysis results.

Pugmire, Dave↗

DOE OSTI2022

Hybrid Analysis of Fusion Data for Online Understanding of Complex Science on Extreme Scale Computers

The current practice for fusion scientists running first principle simulations on high performance computing plat-forms is to either run their simulations and output their data for post-hoc analysis, or to place in situ analytics into their code. In this paper we examine a complex workflow using XGC fusions simulation run on the Oak Ridge Leadership Computing Facility's supercomputer Summit, which also involve three anal-yses as part of the results necessary for scientific discovery. We discuss the challenges faced when implementing these algorithms and present an original hybrid staging technique to help enable the physicists to make discoveries during the execution of the simulation. By creating this infrastructure, we can examine complicated physics results, which may not have been possible without the infrastructure. For example, our work enables the online visualization of turbulent homoclinic tangle around the magnetic X-point, breaking the last confinement surface. This visualization could help fusion scientists to better understand and improve the turbulence spread of plasma exhaust heat, which is crucial toward realizing plasmas beyond the currently accessible physics regimes of present-day tokamak reactors. The physics of turbulent homoclinic tangle will be reported in a future physics publication, by utilizing the original online analysis/visualization framework presented in this paper.

Suchyta, Eric↗

DOE OSTI2022

Superclustering with the Atacama Cosmology Telescope and Dark Energy Survey. I. Evidence for Thermal Energy Anisotropy Using Oriented Stacking

The cosmic web contains filamentary structure on a wide range of scales. On the largest scales, superclustering aligns multiple galaxy clusters along intercluster bridges, visible through their thermal Sunyaev–Zel'dovich signal in the cosmic microwave background. We demonstrate a new, flexible method to analyze the hot gas signal from multiscale extended structures. We use a Compton y-map from the Atacama Cosmology Telescope (ACT) stacked on redMaPPer cluster positions from the optical Dark Energy Survey (DES). Cutout images from the y-map are oriented with large-scale structure information from DES galaxy data such that the superclustering signal is aligned before being overlaid. We find evidence of an extended quadrupole moment of the stacked y signal at the 3.5σ level, demonstrating that the large-scale thermal energy surrounding galaxy clusters is anisotropically distributed. We compare our ACT × DES results with the Buzzard simulations, finding broad agreement. Using simulations, we highlight the promise of this novel technique for constraining the evolution of anisotropic, non-Gaussian structure using future combinations of microwave and optical surveys.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

The Observed Evolution of the Stellar Mass–Halo Mass Relation for Brightest Central Galaxies

We quantify evolution in the cluster-scale stellar mass–halo mass (SMHM) relation's parameters using 2323 clusters and brightest central galaxies (BCGs) over the redshift range 0.03 ≤ z ≤ 0.60. The precision on the inferred SMHM parameters is improved by including the magnitude gap (m gap ) between the BCG and fourth-brightest cluster member (M14) as a third parameter in the SMHM relation. At fixed halo mass, accounting for m gap , through a stretch parameter, reduces the SMHM relation's intrinsic scatter. To explore this redshift range, we use clusters, BCGs, and cluster members identified using the Sloan Digital Sky Survey C4 and redMaPPer cluster catalogs and the Dark Energy Survey redMaPPer catalog. Through this joint analysis, we detect no systematic differences in BCG stellar mass, m gap , and cluster mass (inferred from richness) between the data sets. We utilize the Pareto function to quantify each parameter's evolution. We confirm prior findings of negative evolution in the SMHM relation's slope (3.5σ), and detect negative evolution in the stretch parameter (4.0σ) and positive evolution in the offset parameter (5.8σ). This observed evolution, combined with the absence of BCG growth, when stellar mass is measured within 50 kpc, suggests that this evolution results from changes in the cluster's m gap . For this to occur, late-term growth must be in the intracluster light surrounding the BCG. We also compare the observed results to IllustrisTNG 300-1 cosmological hydrodynamic simulations and find modest qualitative agreement. However, the simulations lack the evolutionary features detected in the real data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Dark Energy Survey Year 3 results: galaxy sample for BAO measurement

Here, we present and validate the galaxy sample used for the analysis of the baryon acoustic oscillation (BAO) signal in the Dark Energy Survey (DES) Y3 data. The definition is based on a colour and redshift-dependent magnitude cut optimized to select galaxies at redshifts higher than 0.5, while ensuring a high-quality determination. The sample covers ~ 4100 square degrees to a depth of i = 22.3 (AB) at 10σ. It contains 7,031,993 galaxies in the redshift range from z = 0.6 to 1.1, with a mean effective redshift of 0.835. Redshifts are estimated with the machine learning algorithm DNF, and are validated using the VIPERS PDR2 sample. We find a mean redshift bias of z bias ~ 0.01 and a mean uncertainty, in units of 1 + z, of σ 68 ~ 0.03$. We evaluate the galaxy population of the sample, showing it is mostly built upon Elliptical to Sbc types. Furthermore, we find a low level of stellar contamination of ≲ 4 %. We present the method used to mitigate the effect of spurious clustering coming from observing conditions and other large-scale systematics. We apply it to the BAO sample and calculate weights that are used to get a robust estimate of the galaxy clustering signal. This paper is one of a series dedicated to the analysis of the BAO signal in DES Y3. In the companion papers, we present the galaxy mock catalogues used to calibrate the analysis and the angular diameter distance constraints obtained through the fitting to the BAO scale.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

A Co-design Framework for Online Data Analysis and Reduction

Science applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation-analysis-reduction pipelines coupled in-memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application- and system-specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. Furthermore, this toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next-generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science

We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Dark energy survey year 1 results: Constraining baryonic physics in the Universe

ABSTRACT Measurements of large-scale structure are interpreted using theoretical predictions for the matter distribution, including potential impacts of baryonic physics. We constrain the feedback strength of baryons jointly with cosmology using weak lensing and galaxy clustering observables (3 × 2pt) of Dark Energy Survey (DES) Year 1 data in combination with external information from baryon acoustic oscillations (BAO) and Planck cosmic microwave background polarization. Our baryon modelling is informed by a set of hydrodynamical simulations that span a variety of baryon scenarios; we span this space via a Principal Component (PC) analysis of the summary statistics extracted from these simulations. We show that at the level of DES Y1 constraining power, one PC is sufficient to describe the variation of baryonic effects in the observables, and the first PC amplitude (Q1) generally reflects the strength of baryon feedback. With the upper limit of Q1 prior being bound by the Illustris feedback scenarios, we reach $\sim 20{{\ \rm per\ cent}}$ improvement in the constraint of $S_8=\sigma _8(\Omega _{\rm m}/0.3)^{0.5}=0.788^{+0.018}_{-0.021}$ compared to the original DES 3 × 2pt analysis. This gain is driven by the inclusion of small-scale cosmic shear information down to 2.5 arcmin, which was excluded in previous DES analyses that did not model baryonic physics. We obtain $S_8=0.781^{+0.014}_{-0.015}$ for the combined DES Y1+Planck EE+BAO analysis with a non-informative Q1 prior. In terms of the baryon constraints, we measure $Q_1=1.14^{+2.20}_{-2.80}$ for DES Y1 only and $Q_1=1.42^{+1.63}_{-1.48}$ for DESY1+Planck EE+BAO, allowing us to exclude one of the most extreme AGN feedback hydrodynamical scenario at more than 2σ.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Spatial coupling of gyrokinetic simulations, a generalized scheme based on first-principles

In this paper, we present a scheme that spatially couples two gyrokinetic codes using first-principles. Coupled equations are presented and a necessary and sufficient condition for ensuring accuracy is derived. This new scheme couples both the field and the particle distribution function. The coupling of the distribution function is only performed once every few time-steps, using a five-dimensional (5D) grid to communicate the distribution function between the two codes. This 5D grid interface enables the coupling of different types of codes and models, such as particle and continuum codes, or delta-f and total-f models. Transferring information from the 5D grid to the marker particle weights is achieved using a new resampling technique. Demonstration of the coupling scheme is shown using two XGC gyrokinetic simulations for both the core and edge. We also apply the coupling scheme to two continuum simulations for a one-dimensional advection–diffusion problem.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2020

Spatial core-edge coupling of the particle-in-cell gyrokinetic codes GEM and XGC

Two existing particle-in-cell gyrokinetic codes, GEM for the core region and XGC for the edge region, have been successfully coupled with a spatial coupling scheme at the interface in a toroidal geometry. Additionally, a mapping technique is developed for transferring data between GEM's structured and XGC's unstructured meshes. Two examples of coupled simulations are presented to demonstrate the coupling scheme. The optimization of GEM for graphics processing unit is also presented.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2020

Processing Full-Scale Square Kilometre Array Data on the Summit Supercomputer

This work presents a workflow for simulating and processing the full-scale low-frequency telescope data of the Square Kilometre Array (SKA) Phase 1. The SKA project will enter the construction phase soon, and once completed, it will be the world’s largest radio telescope and one of the world’s largest data generators. The authors used Summit to mimic an endto-end SKA workflow, simulating a dataset of a typical 6 hour observation and then processing that dataset with an imaging pipeline. This workflow was deployed and run on 4,560 compute nodes, and used 27,360 GPUs to generate 2.6 PB of data. This was the first time that radio astronomical data were processed at this scale. Results show that the workflow has the capability to process one of the key SKA science cases, an Epoch of Reionization observation. This analysis also helps reveal critical design factors for the next-generation radio telescopes and the required dedicated processing facilities.

Wang, Ruonan↗

DOE OSTI2020

Discovery of a Candidate Binary Supermassive Black Hole in a Periodic Quasar from Circumbinary Accretion Variability

Binary supermassive black holes (BSBHs) are expected to be a generic byproduct from hierarchical galaxy formation. The final coalescence of BSBHs is thought to be the loudest gravitational wave (GW) siren, yet no confirmed BSBH is known in the GW-dominated regime. While periodic quasars have been proposed as BSBH candidates, the physical origin of the periodicity has been largely uncertain. Here we report discovery of a periodicity (P=1607±7 days) at 99.95% significance (with a global p-value of ~10 –3 accounting for the look elsewhere effect) in the optical light curves of a redshift 1.53 quasar, SDSS J025214.67–002813.7. Combining archival Sloan Digital Sky Survey data with new, sensitive imaging from the Dark Energy Survey, the total ~20-yr time baseline spans ~4.6 cycles of the observed 4.4-yr (restframe 1.7-yr) periodicity. The light curves are best fit by a bursty model predicted by hydrodynamic simulations of circumbinary accretion disks. The periodicity is likely caused by accretion rate modulation by a milli-parsec BSBH emitting GWs, dynamically coupled to the circumbinary accretion disk. A bursty hydrodynamic variability model is statistically preferred over a smooth, sinusoidal model expected from relativistic Doppler boost, a kinematic effect proposed for PG1302–102. Furthermore, the frequency dependence of the variability amplitudes disfavors Doppler boost, lending independent support to the circumbinary accretion variability hypothesis. Given our detection rate of one BSBH candidate from circumbinary accretion variability out of 625 quasars, it suggests that future large, sensitive synoptic surveys such as the Vera C. Rubin Observatory Legacy Survey of Space and Time may be able to detect hundreds to thousands of candidate BSBHs from circumbinary accretion with direct implications for Laser Interferometer Space Antenna.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

Candidate Periodically Variable Quasars from the Dark Energy Survey and the Sloan Digital Sky Survey

Periodically variable quasars have been suggested as close binary supermassive black holes. We present a systematic search for periodic light curves in 625 spectroscopically confirmed quasars with a median redshift of 1.8 in a 4.6 deg 2 overlapping region of the Dark Energy Survey Supernova (DES-SN) fields and the Sloan Digital Sky Survey Stripe 82 (SDSS-S82). Our sample has a unique 20-year long multi-color ( griz ) light curve enabled by combining DES-SN Y6 observations with archival SDSS-S82 data. The deep imaging allows us to search for periodic light curves in less luminous quasars (down to r ~23.5 mag) powered by less massive black holes (with masses ≳ 10 8.5 M ⊙ ) at high redshift for the first time. We find five candidates with significant (at >99.74% single-frequency significance in at least two bands with a global p-value of ~7 × 10 -4 –3× 10 -3 accounting for the look-elsewhere effect) periodicity with observed periods of ~3–5 years (i.e., 1–2 years in rest frame) having ~4–6 cycles spanned by the observations. If all five candidates are periodically variable quasars, this translates into a detection rate of |${\sim }0.8^{+0.5}_{-0.3}$| % or |${\sim }1.1^{+0.7}_{-0.5}$| quasar per deg 2 . Our detection rate is 4–80 times larger than those found by previous searches using shallower surveys over larger areas. This discrepancy is likely caused by differences in the quasar populations probed and the survey data qualities. Furthermore, we discuss implications on the future direct detection of low-frequency gravitational waves. Additionally, continued photometric monitoring will further assess the robustness and characteristics of these candidate periodic quasars to determine their physical origins.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

Dust Reverberation Mapping in Distant Quasars from Optical and Mid-infrared Imaging Surveys

The size of the dust torus in active galactic nuclei (AGNs) and their high-luminosity counterparts, quasars, can be inferred from the time delay between UV/optical accretion disk continuum variability and the response in the mid-infrared (MIR) torus emission. This dust reverberation mapping (RM) technique has been successfully applied to ~70 z ≲ 0.3 AGNs and quasars. Here we present first results of our dust RM program for distant quasars covered in the Sloan Digital Sky Survey Stripe 82 region combining ~20 yr ground-based optical light curves with 10 yr MIR light curves from the WISE satellite. Here, we measure a high-fidelity lag between W1 band (3.4 μm) and g band for 587 quasars over 0.3 ≲ z ≲ 2 () and two orders of magnitude in quasar luminosity. They tightly follow (intrinsic scatter ~0.17 dex in lag) the IR lag–luminosity relation observed for z < 0.3 AGNs, revealing a remarkable size–luminosity relation for the dust torus over more than four decades in AGN luminosity, with little dependence on additional quasar properties such as Eddington ratio and variability amplitude. This study motivates further investigations in the utility of dust RM for cosmology and strongly endorses a compelling science case for the combined 10 yr Vera C. Rubin Observatory Legacy Survey of Space and Time (optical) and 5 yr Nancy Grace Roman Space Telescope 2 μm light curves in a deep survey for low-redshift AGN dust RM with much lower luminosities and shorter, measurable IR lags. The compiled optical and MIR light curves for 7384 quasars in our parent sample are made public with this work.

79 ASTRONOMY AND ASTROPHYSICS↗