Benchmarking Under the Hood of OpenCL FPGA Platforms
Explore the source record for details and available documents.
Engineering topics
Publications and source records attributed to Finkel, Hal.
Explore the source record for details and available documents.
Multicore scaling will end soon because of practical power limits. Dark silicon is becoming a major issue even more than the end of Moore’s law. In the post-Moore era, the energy efficiency of computing will be a major concern. FPGAs could be a key to maximizing the energy efficiency. In this paper we address severe challenges in the adoption of FPGA in HPC and describe “Re-form,” an FPGA-powered codesign flow.
The pairwise kinematic Sunyaev–Zel’dovich (kSZ) signal from galaxy clusters is a probe of their line of sight momenta, and thus a potentially valuable source of cosmological information. In addition to the momenta, the amplitude of the measured signal depends on the properties of the intracluster gas and observational limitations such as errors in determining cluster centers and redshifts. In this work, we simulate the pairwise kSZ signal of clusters at z<1, using the output from a cosmological N-body simulation and including the properties of the intracluster gas via a model that can be varied in post-processing. We find that modifications to the gas profile due to star formation and feedback reduce the pairwise kSZ amplitude of clusters by ∼50%, relative to the naive “gas traces mass” assumption. We demonstrate that miscentering can reduce the overall amplitude of the pairwise kSZ signal by up to 10%, while redshift errors can lead to an almost complete suppression of the signal at small separations. We confirm that a high-significance detection is expected from the combination of data from current generation, high-resolution cosmic microwave background experiments, such as the South Pole Telescope, and cluster samples from optical photometric surveys, such as the Dark Energy Survey. Furthermore, we forecast that future experiments such as Advanced ACTPol in conjunction with data from the Dark Energy Spectroscopic Instrument will yield detection significances of at least 20σ, and up to 57σ in an optimistic scenario. Our simulated maps are publicly available at http://www.hep.anl.gov/cosmology/ksz.html.
Large-scale simulations of cosmic structure formation play an important role in interpreting cosmological observations at high precision. The simulations must cover a parameter range beyond the standard six cosmological parameters and need to be run at high mass and force resolution. A key simulation-based task is the generation of accurate theoretical predictions for observables using a finite number of simulation runs, via the method of emulation. Using a new sampling technique, we explore an eight-dimensional parameter space including massive neutrinos and a variable equation of state of dark energy. We construct trial emulators using two surrogate models (the linear power spectrum and an approximate halo mass function). The new sampling method allows us to build precision emulators from just 26 cosmological models and to systematically increase the emulator accuracy by adding new sets of simulations in a prescribed way. Emulator fidelity can now be continuously improved as new observational data sets become available and higher accuracy is required. Finally, using one ΛCDM cosmology as an example, we study the demands imposed on a simulation campaign to achieve the required statistics and accuracy when building emulators for investigations of dark energy.
Not Available
In this work, we introduce GRChombo: a new numerical relativity code which incorporates full adaptive mesh refinement (AMR) using block structured Berger-Rigoutsos grid generation. The code supports non-trivial 'many-boxes-in-many-boxes' mesh hierarchies and massive parallelism through the message passing interface. GRChombo evolves the Einstein equation using the standard BSSN formalism, with an option to turn on CCZ4 constraint damping if required. The AMR capability permits the study of a range of new physics which has previously been computationally infeasible in a full 3 + 1 setting, while also significantly simplifying the process of setting up the mesh for these problems. We show that GRChombo can stably and accurately evolve standard spacetimes such as binary black hole mergers and scalar collapses into black holes, demonstrate the performance characteristics of our code, and discuss various physics problems which stand to benefit from the AMR technique.
Data-dependence profling is a technique that enables a com- piler to judiciously decide when the execution of a loop | which the compiler could not prove to be dependence free | should be speculated through the use of Thread Level Spec- ulation (TLS). The data collected by a data-dependence pro- fler can be used to predict if may dependencies reported by a compiler static analysis are likely to materialize at runtime. A cost analysis can then be used to decide that some loops with a lower probability of dependence should be specula- tively parallelized. This paper addresses the question as to whether a loops' dependence behaviour changes when the in- put to the program changes | a study of 57 different bench- marks indicates that it usually does not change. Then the paper describes SpecEval, an automatic speculative paral- lelization framework that uses single-input data-dependence profles to find speculation candidates in the SPEC2006 and PolyBench/C benchmarks. This paper also presents a per- formance evaluation of TLS implementation in IBM's Blue- Gene/Q supercomputer and shows that the performance of TLS is affected by several factors, including the number of speculated loops, the execution-time coverage of speculated loops, the miss-speculation overhead, the L1 cache miss rate and the effect on dynamic instruction path length.
Computing plays an essential role in all aspects of high energy physics. As computational technology evolves rapidly in new directions, and data throughput and volume continue to follow a steep trend-line, it is important for the HEP community to develop an effective response to a series of expected challenges. In order to help shape the desired response, the HEP Forum for Computational Excellence (HEP-FCE) initiated a roadmap planning activity with two key overlapping drivers -- 1) software effectiveness, and 2) infrastructure and expertise advancement. The HEP-FCE formed three working groups, 1) Applications Software, 2) Software Libraries and Tools, and 3) Systems (including systems software), to provide an overview of the current status of HEP computing and to present findings and opportunities for the desired HEP computational roadmap. The final versions of the reports are combined in this document, and are presented along with introductory material.
The halo occupation distribution (HOD) approach has proven to be an effective method for modeling galaxy clustering and bias. In this approach, galaxies of a given type are probabilistically assigned to individual halos in N-body simulations. In this paper, we present a fast emulator for predicting the fully nonlinear galaxy–galaxy auto and galaxy–dark matter cross power spectrum and correlation function over a range of freely specifiable HOD modeling parameters. The emulator is constructed using results from 100 HOD models run on a large ΛCDM N-body simulation, with Gaussian Process interpolation applied to a PCA-based representation of the galaxy power spectrum. The total error is currently ∼1% in the auto correlations and ∼2% in the cross correlations from z = 1 to z = 0, over the considered parameter range. We use the emulator to investigate the accuracy of various analytic prescriptions for the galaxy power spectrum, parametric dependencies in the HOD model, and the behavior of galaxy bias as a function of HOD parameters. Additionally, we obtain fully nonlinear predictions for tangential shear correlations induced by galaxy–galaxy lensing from our galaxy–dark matter cross power spectrum emulator. All emulation products are publicly available at http://www.hep.anl.gov/cosmology/CosmicEmu/emu.html.
Modeling large-scale sky survey observations is a key driver for the continuing development of high-resolution, large-volume, cosmological simulations. We report the first results from the “Q Continuum” cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc){sup 3} and evolves more than half a trillion particles, leading to a particle mass resolution of m{sub p}≃1.5⋅10{sup 8} M{sub ⊙}. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan’s GPU accelerators.
Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. In this paper, we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC’s design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.
Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in-situ and co-scheduling approaches for handling petabyte-scale outputs. We compare different analysis set-ups ranging from purely off-line, to purely in-situ to in-situ/co-scheduling. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.
Explore the source record for details and available documents.
Over the next decade, cosmological measurements of the large-scale structure of the Universe will be sensitive to the combined effects of dynamical dark energy and massive neutrinos. The matter power spectrum is a key repository of this information. We extend higher-order perturbative methods for computing the power spectrum to investigate these effects over quasilinear scales. Through comparison with N-body simulations, we establish the regime of validity of a time-renormalization group perturbative treatment that includes dynamical dark energy and massive neutrinos. We also quantify the accuracy of standard, renormalized and Lagrangian resummation (LPT) perturbation theories without massive neutrinos. We find that an approximation that neglects neutrino clustering as a source for nonlinear matter clustering predicts the baryon acoustic oscillation (BAO) peak position to 0.25% accuracy for redshifts 1 ≤ z ≤ 3, justifying the use of LPT for BAO reconstruction in upcoming surveys. Here, we release a modified version of the public Copter code which includes the additional physics discussed in the paper.
Supercomputing is evolving towards hybrid and accelerator-based architectures with millions of cores. The HACC (Hardware/Hybrid Accelerated Cosmology Code) framework exploits this diverse landscape at the largest scales of problem size, obtaining high scalability and sustained performance. Developed to satisfy the science requirements of cosmological surveys, HACC melds particle and grid methods using a novel algorithmic structure that flexibly maps across architectures, including CPU/GPU, multi/many-core, and Blue Gene systems. We demonstrate the success of HACC on two very different machines, the CPU/GPU system Titan and the BG/Q systems Sequoia and Mira, attaining unprecedented levels of scalable performance. We demonstrate strong and weak scaling on Titan, obtaining up to 99.2% parallel efficiency, evolving 1.1 trillion particles. On Sequoia, we reach 13.94 PFlops (69.2% of peak) and 90% parallel efficiency on 1,572,864 cores, with 3.6 trillion particles, the largest cosmological benchmark yet performed. HACC design concepts are applicable to several other supercomputer applications.
Analytical arguments suggest that a large class of scalar field potentials permit the existence of oscillons — pseudo-stable, non-topological solitons — in three spatial dimensions. In this paper we numerically explore oscillon solutions in three dimensions. We confirm the existence of these field configurations as solutions to the Klein-Gorden equation in an expanding background, and verify the predictions of Amin and Shirokoff [1] for the characteristics of individual oscillons for their model. Further, we demonstrate that significant numbers of oscillons can be generated via fragmentation of the inflaton condensate, consistent with the analysis of Amin [2]. These emergent oscillons can easily dominate the post-inflationary universe. Finally, both analytic and numerical results suggest that oscillons are stable on timescales longer than the post-inflationary Hubble time. Consequently, the post-inflationary universe can contain an effective matter-dominated phase, during which it is dominated by localized concentrations of scalar field matter.
PSpectRe is a C++ program that uses Fourier-space pseudo-spectral methods to evolve interacting scalar fields in an expanding universe. PSpectRe is optimized for the analysis of parametric resonance in the post-inflationary universe and provides an alternative to finite differencing codes, such as Defrost and LatticeEasy. PSpectRe has both second- (Velocity-Verlet) and fourth-order (Runge-Kutta) time integrators. Given the same number of spatial points and/or momentum modes, PSpectRe is not significantly slower than finite differencing codes, despite the need for multiple Fourier transforms at each timestep, and exhibits excellent energy conservation. Further, by computing the post-resonance equation of state, we show that in some circumstances PSpectRe obtains reliable results while using substantially fewer points than a finite differencing code. PSpectRe is designed to be easily extended to other problems in early-universe cosmology, including the generation of gravitational waves during phase transitions and pre-inflationary bubble collisions. Specific applications of this code will be described in future work.