Accounting for Trinity: The Computing Effort that Made Trinity Possible
Explore the source record for details and available documents.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Utilizing rarely used source materials and identifying points of bias among more commonly used sources, this critical review provides a more complete representation of wartime Los Alamos computing operations and personnel, including the Laboratory’s typically underrepresented human computers and how they contributed to the success of the Trinity test. This paper also identifies how the Laboratory’s unusual wartime computing demands served as a formative experience among many Los Alamos personnel and consultants who contributed significantly to the development and use of mechanized computing at and beyond Los Alamos after the war.
This paper describes the Trinity system, the first ASC Advanced Technology System (ATS-1). We describe the Trinity procurement timeline, the ASC computing strategy, the Trinity specific mission needs, and the Trinity system specifications.
The Trinity supercomputer broke a world record in how quickly they generated a trillion files. A Los Alamos National Laboratory scientist having trouble solving a stubborn research problem needed some help – his scientific simulations had generated a sea of data, but it took so long to search the data that he couldn’t find the information he needed. He found himself looking for the proverbial needle in a haystack. At the same time, the lab’s storage research team had been hard at work on another classic big data problem: creating massive numbers of files as quickly as possible. The day the team met with the scientist, you could say that Big Science and Big Data put their heads together – and now they’re making history.
Abstract not provided.
Researchers have created the largest simulation to date of an entire gene of DNA, a feat that required one billion atoms to model and will help researchers to better understand and develop cures for diseases like cancer. It will also give insight into autism and intellectual disabilities. Modeling genes at the atomistic level is the first step toward creating a complete explanation of how DNA expands and contracts, which controls genetic on/off switching.
When discussing research in physics and in science more generally, it is common to ascribe equal importance to the three components of the scientific trinity: theoretical, experimental, and computational studies. This review will explore the future of modern turbulence theory by tracing its history, which began in earnest with Kolmogorov’s 1941 analysis of turbulence cascade and inertial range [A.N. Kolmogorov, Dokl. Akad. Nauk SSSR, 30, 299, (1941); 32, 19, (1941)]. The 80th Anniversary of Kolmogorov’s landmark study is a welcome opportunity to survey the achievements and evaluate the future of the theoretical approach of turbulence research. Over the years, turbulence theories have been critically important in laying the foundation of our understanding of the nature of turbulent flows. In particular, the Direct Interaction Approximation (DIA) [R.H. Kraichnan, J. Fluid Mech., 5, 497 (1959)] and its subsequent development, known as the statistical closure approach, can be identified as perhaps the most profound single advancement. The remarkable success of the statistical closure has furnished a platform to study such essential concepts as the energy transfer process and interacting scales, and the roles of the straining and sweeping motions. More recently, the quasi-Lagrangian formulation of V. L’vov & I. Procaccia and Kraichnan’s solvable passive scalar model provided powerful ways to explore another fundamental aspect of turbulent flows, the phenomena of intermittency, and the associated anomalous scaling exponents. In the meantime, the theory of fluid equilibria has been developed to describe the large-scale structures that can emerge from turbulent cascades of two-dimensional and geophysical flows at a later time. And yet, despite all these successes, analytical treatments suffer from mathematical complexities. As a result, the utility of theoretical approaches has been limited to relatively idealized flows. On the other hand, in recent decades, computational abilities and experimental facilities have reached an unprecedented scale. Looking beyond the horizon, the imminent deployment of exascale supercomputers will generate complete datasets of the entire flow field of key benchmark flows, allowing researchers to extract additional measurements concerning fully developed, complex turbulent flow fields far beyond those available from the statistical closure theories. Some other developments that could potentially influence the future course of turbulence theories include the advancement of machine learning, artificial intelligence, and data science; likely disruptions arising from the advent of quantum computation; and the increasingly prominent role of turbulence research in providing more accurate climate scientific data. Finally, turbulence theorists can leverage these developments by asking the right questions and developing advanced, sophisticated frameworks that will be able to predict and correlate vast amounts of data from the other two components of the trinity.
Now one of the fastest supercomputers in the United States, the Trinity Supercomputer Phase II project was completed during the summer of 2017, and the computer became fully operational during an unclassified “open science” run. The computer has now transitioned to classified mode. The Trinity supercomputer is designed to provide increased computational capability for the NNSA Nuclear Security Enterprise in support of increasing geometric and physics fidelities in nuclear weapons simulation codes, while maintaining expectations for total time to solution. The capabilities of Trinity are required for supporting the NNSA Stockpile Stewardship program’s certification and assessments to ensure that the nation’s nuclear stockpile is safe, secure and effective. With an expected operational lifetime through 2020, Trinity is an approximately 41.5 petaflop (41.5 million billion calculations per second, peak performance) supercomputer based on the Cray XC40 architecture. It consists of 19,420 compute nodes utilizing two processor architectures, Intel Xeon Haswell and Intel Xeon Phi Knights Landing. It is a water-cooled system with a power requirement of about 10 megawatts. The Trinity project is managed and operated by Los Alamos National Laboratory and Sandia National Laboratories under the Alliance for Computing at Extreme Scale (ACES) partnership. The system is located at the Nicholas Metropolis Center for Modeling and Simulation at Los Alamos and covers approximately 5,200 square feet of floor space.
This report documents the early experiences with porting and performance analysis of the Tri-Lab Trinity benchmark applications on Intel Xeon Phi (Knights Corner) (KNC) processor. KNC, the second generation of the Intel Many Integrated Core (MIC) architectures, uses a large number of small P54C-x86 cores with wide vector units and is deployed as PCI bus attached process accelerators. Sandia has experimental test beds of small InifiniBand clusters and workstations to investigate the performance of the MIC architecture. On these experimental test beds the programming models that may be investigated are "offload", "symmetric" and "native". Among these program usage models our primary interest is in the so called "native" mode, because the planned Trinity system to be deployed in 2016 using the next generation MIC processor architecture called Knights Landing would be self-hosted. Trinity / NERSC-8 benchmark programs cover a variety of scientific disciplines and they were used to guide the procurement of these systems. Architectures such as the Intel MIC are well suited to study evolving processor architectures and a usage model commonly referred to as MPI + X that facilitates migration of our applications to use both coarse grain and fine grain parallelism. Our focus with the applications selected is on the efficacy of algorithms in these applications to take advantage of features like: large number of cores, wide vector units, higher-bandwidth and deeper memory sub-system. This is a first step towards understanding applications, algorithms and programming environments for Trinity and future exascale computing systems.
As part of its national security science mission, Los Alamos National Laboratory and HPC have a long, entwined history dating back to the earliest days of computing. From bringing the first problem to the nation’s first computer to building the first machine to break the petaflop barrier, Los Alamos holds many “firsts” in HPC breakthroughs. Today, supercomputers are integral to stockpile stewardship and the Laboratory continues to work with vendors in developing the future of HPC.
This report summarizes the work performed as part of a FY17 CSSE L2 milestone to in- vestigate the power usage behavior of ASC workloads running on the ATS-1 Trinity plat- form. Techniques were developed to instrument application code regions of interest using the Power API together with the Kokkos profiling interface and Caliper annotation library. Experiments were performed to understand the power usage behavior of mini-applications and the SNL/ATDM SPARC application running on ATS-1 Trinity Haswell and Knights Landing compute nodes. A taxonomy of power measurement approaches was identified and presented, providing a guide for application developers to follow. Controlled scaling study experiments were performed on up to 2048 nodes of Trinity along with smaller scale ex- periments on Trinity testbed systems. Additionally, power and energy system monitoring information from Trinity was collected and archived for post analysis of "in-the-wild" work- loads. Results were analyzed to assess the sensitivity of the workloads to ATS-1 compute node type (Haswell vs. Knights Landing), CPU frequency control, node-level power capping control, OpenMP configuration, Knights Landing on-package memory configuration, and algorithm/solver configuration. Overall, this milestone lays groundwork for addressing the long-term goal of determining how to best use and operate future ASC platforms to achieve the greatest benefit subject to a constrained power budget.
The Mercury Monte Carlo particle transport code is used to simulate the transport of radiation through urban environments. These challenging calculations include complicated geometries and require significant computational resources to complete. In the proposed Trinity Open Science calculations, I will investigate computer science aspects of the code which are relevant to convergence of the simulation quantities with increasing Monte Carlo particle counts.
No abstract provided.
Direct coupling between a transport solver and local, nonlinear gyrokinetic calculations using the multiscale gyrokinetic code TRINITY[M. Barnes, 'TRINITY: A unified treatment of turbulence, transport, and heating in magnetized plasmas', Ph.D. thesis, University of Maryland, 2008 (eprint arXiv:0901.2868)] is described. The coupling of the microscopic and macroscopic physics is done within the framework of multiscale gyrokinetic theory, of which we present the assumptions and key results. An assumption of scale separation in space and time allows for the simulation of turbulence in small regions of the space-time grid, which are embedded in a coarse grid on which the transport equations are implicitly evolved. This leads to a reduction in computational expense of several orders of magnitude, making first-principles simulations of the full fusion device volume over the confinement time feasible on current computing resources. Numerical results from TRINITY simulations are presented and compared with experimental data from JET [M. Keilhacker, Plasma Phys. Controlled Fusion 41, B1 (1999)] and ASDEX Upgrade [O. Gruber, Nucl. Fusion 47, S622 (2007)] plasmas.
The Trinity Test of July 16, 1945 was the first full-scale, real-world test of a nuclear weapon; with the new Trinity supercomputer Los Alamos National Laboratory's goal is to do this virtually, in 3D. Trinity was the culmination of a fantastic effort of groundbreaking science and engineering by hundreds of men and women at Los Alamos and other Manhattan Project sites. It took them less than two years to change the world. The Laboratory is marking the 70th anniversary of the Trinity Test because it not only ushered in the Nuclear Age, but with it the origin of today’s advanced supercomputing. We live in the Age of Supercomputers due in large part to nuclear weapons science here at Los Alamos. National security science, and nuclear weapons science in particular, at Los Alamos National Laboratory have provided a key motivation for the evolution of large-scale scientific computing. Beginning with the Manhattan Project there has been a constant stream of increasingly significant, complex problems in nuclear weapons science whose timely solutions demand larger and faster computers. The relationship between national security science at Los Alamos and the evolution of computing is one of interdependence.
Long computations often use checkpoint/restart to survive the inevitable failures. Optimal checkpoint/restart strategy can be derived in terms of special functions, and approximated within the regime of interest. As a further step under nominal assumptions, platforms can be optimized for maximum progress rate, computed in terms of non-dimensional parameter JMTTI/Delta, where JMTTI is the Job Mean Time To Interrupt and Delta is the time it takes to write (or read) a checkpoint. Scaling to a larger platform of the same technology requires scaling checkpoint bandwidth quadratically to maintain the same progress rate. This was observed during the design of LANL’s Roadrunner in 2007, led to burst buffers proposed in 2009 and introduced on Trinity in 2015. The need to compute reliably continues to impact the design of future exascale platforms.
Brain neuroscientists and computer scientists call this field neuromimetic computing—building computers inspired by how the cerebral cortex of the brain works. Its cortical processes rely on billions of small biological “switches,” called neurons. To learn, they store and process information. Using an approach called neural networks, researchers are developing computers that simulate neurons and their interconnections. Then computers can learn about their surroundings, interpret data, and make predictions based on it. In practice, however, researchers attempting to simulate neural processing at anything close to the scale and complexity of the brain’s cortical circuits have been stymied by limitations on computer memory and computational power. All that has changed with the new Trinity supercomputer at Los Alamos, which became fully operational in mid-2017. Trinity has unique capabilities designed for the stockpile stewardship mission, which includes highly complex nuclear simulations that have replaced the testing of nuclear weapons. All this capability means Trinity allows a fundamentally different approach to large-scale cortical simulations, enabling an unprecedented leap in the ability to model neural processing. To test that capability on a limited-scale problem, computer scientists and neuroscientists at Los Alamos created a “sparse prediction machine” that runs on a neural network on Trinity. A sparse prediction machine is designed to work like the brain: researchers expose it to data—in this case, various videos of a car driving down a road—without labeling the data in any way. Then the program sorts through that data frame by frame, focuses on the important information, and develops a prediction about the car’s motion. With Trinity’s power, the Los Alamos team simulates the way a brain handles information in its neurons but uses the fewest neurons at any given moment to explain the information at hand. That’s the “sparse” part, and it makes the brain very efficient—and, hopefully, a computer more efficient, too.
Recent high-performance computing (HPC) platforms such as the Trinity Advanced Technology System (ATS-1) feature burst buffer resources that can have a dramatic impact on an application’s I/O performance. While these non-volatile memory (NVM) resources provide a new tier in the storage hierarchy, developers must find the right way to incorporate the technology into their applications in order to reap the benefits. Similar to other laboratories, Sandia is actively investigating ways in which these resources can be incorporated into our existing libraries and workflows without burdening our application developers with excessive, platform-specific details. This FY18Q1 milestone summaries our progress in adapting the Sandia Parallel Aerodynamics and Reentry Code (SPARC) in Sandia’s ATDM program to leverage Trinity’s burst buffers for checkpoint/restart operations. We investigated four different approaches with varying tradeoffs in this work: (1) simply updating job script to use stage-in/stage out burst buffer directives, (2) modifying SPARC to use LANL’s hierarchical I/O (HIO) library to store/retrieve checkpoints, (3) updating Sandia’s IOSS library to incorporate the burst buffer in all meshing I/O operations, and (4) modifying SPARC to use our Kelpie distributed memory library to store/retrieve checkpoints. Team members were successful in generating initial implementation for all four approaches, but were unable to obtain performance numbers in time for this report (reasons: initial problem sizes were not large enough to stress I/O, and SPARC refactor will require changes to our code). When we presented our work to the SPARC team, they expressed the most interest in the second and third approaches. The HIO work was favored because it is lightweight, unobtrusive, and should be portable to ATS-2. The IOSS work is seen as a long-term solution, and is favored because all I/O work (including checkpoints) can be deferred to a single library.