Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel computing components software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

180 records · Page 10

Initial Development of Fusion Magnet Simulation Capabilities for Performance and Safety Evaluation Using the MOOSE Framework

Fusion energy holds the promise of being a transformative technology as a carbon-neutral, sustainable source of energy. Whole device modeling and the development of fusion digital twins will be increasingly important for emerging fusion device concepts at both national laboratories and within the commercial fusion industry. However, meeting the challenge of whole device modeling of fusion energy devices requires robust, multiphysics, multiscale modeling and simulation technologies capable of running on large-scale supercomputers. Detailed analysis of individual systems at-scale is also required to ensure safe and efficient operation as well as provide the safety basis for future device designs and licensing activities. In a tokamak, toroidal and poloidal magnets confine and shape the fusion plasma to promote the fusion reaction. High plasma temperatures and high magnetic field requirements in modern design concepts (leading to high amounts of energy stored within each magnet) impose electrical, thermal, and mechanical loads on the magnet components, which in turn impacts the safety considerations of the magnet and their supporting systems. Idaho National Laboratory (INL) has a history of working in this space, including development and benchmarking of the Magnetic System Circuitry Analysis Program (MSCAP) and Magnet Arcing (MAGARC) codes to study magnet quench events; notably, MAGARC was used to study quenching during the ITER Engineering Design Activity. However, these legacy codes and capabilities are not parallel and scalable, and new tools are required for future advances in this area, which leads to the INL-developed Multiphysics Object-Oriented Simulation Environment (MOOSE) framework. Developed originally for fission reactor systems under United States Department of Energy, Office of Nuclear Energy modeling and simulation programs, the MOOSE framework has been well-suited to multiscale, multiphysics modeling and simulation needs for nuclear systems. The framework is open-source, well-tested, under continuous development and deployment, and developed to a Nuclear Quality Assurance, Level 1 software quality standard. MOOSE has also been used in the fusion space previously in several projects: INL’s Tritium Migration Analysis Program, Version 8 (TMAP8) for tritium migration, UK Atomic Energy Authority’s A Unified Resource for OpenMC (fusion) Reactor Applications (AURORA) code for fusion thermo-mechanical and neutronics analysis, and Argonne National Laboratory’s Cardinal for high-fidelity computational fluid dynamics and neutronics. However, to model superconducting magnets, several MOOSE enhancements are required: additions to the current MOOSE electromagnetic capabilities, new material libraries for superconductors of interest (such as YBCO), as well as fusion-specific models for thermo-mechanics. This talk will discuss initial development activities to build these capabilities in MOOSE, focusing on initial validation and benchmarking activities. Proposed coupling workflows and future work to support the simulation of fusion magnets and magnet structural assemblies for performance and safety evaluation in MOOSE will also be discussed.

70 - PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Low-Cost Parallel Algorithms for 2:1 Octree Balance

The logical structure of a forest of octrees can be used to create scalable algorithms for parallel adaptive mesh refinement (AMR), which has recently been demonstrated for several petascale applications. Among various frequently used octree-based mesh operations, including refinement, coarsening, partitioning, and enumerating nodes, ensuring a 2:1 size balance between neighboring elements has historically been the most expensive in terms of CPU time and communication volume. The 2:1 balance operation is thus a primary target to optimize. One important component of a parallel balance algorithm is the ability to determine whether any two given octants have a consistent distance/size relation. Based on new logical concepts we propose fast algorithms for making this decision for all types of 2:1 balance conditions in 2D and 3D. Since we are able to achieve this without constructing any parent nodes in the tree that would otherwise need to be sorted and communicated, we can significantly reduce the required memory and communication volume. In addition, we propose a lightweight collective algorithm for reversing the asymmetric communication pattern induced by non-local octant interactions. We have implemented our improvements as part of the opensource “p4est” software. Benchmarking this code with both synthetic and simulation-driven adapted meshes we are able to demonstrate much reduced runtime and excellent weak and strong scalability. On our largest benchmark problem with 5.13 × 10 11 octants the new 2:1 balance algorithm executes in less than 8 seconds on 112,128 CPU cores of the Jaguar Cray XT5 supercomputer.

Computer Science↗

TU-H-BRA-02: The Physics of Magnetic Field Isolation in a Novel Compact Linear Accelerator Based MRI-Guided Radiation Therapy System

Purpose: To develop a method for isolating the MRI magnetic field from field-sensitive linear accelerator components at distances close to isocenter. Methods: A MRI-guided radiation therapy system has been designed that integrates a linear accelerator with simultaneous MR imaging. In order to accomplish this, the magnetron, port circulator, radiofrequency waveguide, gun driver, and linear accelerator needed to be placed in locations with low magnetic fields. The system was also required to be compact, so moving these components far from the main magnetic field and isocenter was not an option. The magnetic field sensitive components (exclusive of the waveguide) were placed in coaxial steel sleeves that were electrically and mechanically isolated and whose thickness and placement were optimized using E&M modeling software. Six sets of sleeves were placed 60° apart, 85 cm from isocenter. The Faraday effect occurs when the direction of propagation is parallel to the magnetic RF field component, rotating the RF polarization, subsequently diminishing RF power. The Faraday effect was avoided by orienting the waveguides such that the magnetic field RF component was parallel to the magnetic field. Results: The magnetic field within the shields was measured to be less than 40 Gauss, significantly below the amount needed for the magnetron and port circulator. Additional mu-metal was employed to reduce the magnetic field at the linear accelerator to less than 1 Gauss. The orientation of the RF waveguides allowed the RT transport with minimal loss and reflection. Conclusion: One of the major challenges in designing a compact linear accelerator based MRI-guided radiation therapy system, that of creating low magnetic field environments for the magnetic-field sensitive components, has been solved. The measured magnetic fields are sufficiently small to enable system integration. This work supported by ViewRay, Inc.

60 APPLIED LIFE SCIENCES↗

MPI + OpenACC: Accelerating radiation transport mini-application, minisweep, on heterogeneous systems

According to the article, architectures are rapidly evolving, and exascale machines are expected to offer billion-way concurrency. We need to rethink algorithms, languages and programming models among other components in order to migrate large scale applications and explore parallelism on these machines. Although directive-based programming models allow programmers to worry less about programming and more about science, expressing complex parallel patterns in these models can be a daunting task especially when the goal is to match the performance that the hardware platforms can offer. One such pattern is wavefront. This paper extensively studies a wavefront-based miniapplication for Denovo, a production code for nuclear reactor modeling. We parallelize the Koch–Baker–Alcouffe (KBA) parallel-wavefront sweep algorithm in the main kernel of Minisweep (the miniapplication) using CUDA 9.0, OpenMP 4.0 (SIMD) and OpenACC 2.6. Our OpenACC implementation running on NVIDIA’s next-generation Volta GPU boasts an 85.06x speedup over serial code, which is larger than CUDA’s 83.72x speedup over the same serial implementation. We also explore the scalability of our solution using MPI to decompose our simulation domain, allowing us to run on many nodes and accelerators present in state-of-the-art HPC systems. Our parallelization effort across platforms also motivated us to define an abstract parallelism model that is architecture independent, with a goal of creating software abstractions that can be used by applications employing the wavefront sweep motif.

Computer Science↗

Cybersecurity for Grid Connected eXtreme Fast Charging (XFC) Station (CyberX) (Final Scientific/Technical Report)

This report summarizes the activities conducted under the DOE VTO funded project DE- EE0008451, where ABB Inc. (ABB), in collaboration with Idaho National Laboratory (INL), APS Global (APS), and XOS Trucks (XOS) pursued the development of a cyber-resilient extreme fast charging (XFC) management system. This project entitled Cybersecurity for Grid Connected eXtreme Fast Charging (XFC) Station (CyberX) focuses on a resilient architecture for smart charging EV Supply Equipment (EVSE) device control and Coordinated Anomaly Detection System (CADS) features that can be added at the charging site depot level to increase cybersecurity. The project was split into two budget periods focused first on developing the threat model and resilient control concepts and second on testing, improving, and validating those developed resilient control algorithms and features with a focus on key vulnerabilities identified during the threat assessment portion of the project. During the first budget period of the CyberX project, the ABB led team focused on activities to identify, model, and quantitatively prioritize high-impact attack scenarios with potential cyber-physical effects while also modeling and developing concepts for a resilient control system that could securely address integration of DERs and other resources with EV charging. Development of the security focused XFC management system (XMS) was accomplished first by offline simulation using a developed XFC station or depot with 480V input level and simulating measurement inputs to monitoring and control systems in concept development. A representative distribution grid model was developed supporting an EV charging site model with BESS and 6 general EV charging models. These EV charging models allowed multiple configurations of charging level, multiple connected protection and measurement devices, and simulation function to show general compromise of EV, BESS, and protection features based on parallel threat analysis. During the second budget period, the EV site and supporting systems model was developed in more detail and converted from offline model to real-time to real-time with EV charging hardware in the loop (HIL). The resilient control architecture developed as concept in the first part of the project was further tested and validated for integration of local energy resources and XFC charging station site equipment while maintaining cybersecure operating principles. The proposed resilient architecture for smart charging and cybersecurity features consists of two main concepts developed and tested within the project. The first concept is an XFC management system (XMS) consisting of a hardware gateway, software platform, and Supervisory Control and Data Acquisition (SCADA) or Distribution Management System integration components. The second concept is a Coordinated Anomaly Detection System (CADS) which forms a primarily software-related subsystem of the total CyberX solution focused on monitoring system measurements, estimation of measurement states, and predicting current at the utility point of interaction based on machine learning for anomaly detection.

33 ADVANCED PROPULSION SYSTEMS↗

A Public-Private-Acadmic Partnership to Advance Solar Power Forecasting

The National Center for Atmospheric Research (NCAR) is pleased to have led a partnership to advance the state-of-the-science of solar power forecasting by designing, developing, building, deploying, testing, and assessing the SunCast™ Solar Power Forecasting System. The project has included cutting edge research, testing in several geographically- and climatologically-diverse high penetration solar utilities and Independent System Operators (ISOs), and wide dissemination of the research results to raise the bar on solar power forecasting technology. The partners include three other national laboratories, six universities, and industry partners. This public-private-academic team has worked in concert to perform use-inspired research to advance solar power forecasting through cutting-edge research to advance both the necessary forecasting technologies and the metrics for evaluating them. The project has culminated in a year-long, full-scale demonstration of provide irradiance and power forecasts to utilities and ISOs to use in their operations. The project focused on providing elements of a value chain, beginning with the weather that causes a deviation from clear sky irradiance and progresses through monitoring and observations, modeling, forecasting, dissemination and communication of the forecasts, interpretation of the forecast, and through decision-making, which produces outcomes that have an economic value. The system has been evaluated using metrics developed specifically for this project, which has provided rich information on model and system performance. Research was accomplished on the very short range (0-6 hours) Nowcasting system as well as on the longer term (6-72 hour) forecasting system. The shortest range forecasts are based on observations in the field. The shortest range system, built by Brookhaven National Laboratory (BNL) and based on Total Sky Imagers (TSIs) is TSICast, which operates on the shortest time scale with a latency of only a few minutes and forecasts that currently go out to about 15 min. This project has facilitated research in improving the hardware and software so that the new high definition cameras deployed at multiple nearby locations allow discernment of the clouds at varying levels and advection according to the winds observed at those levels. Improvements over “smart persistence” are about 29% for even these very short forecasts. StatCast is based on pyranometer data measured at the site as well as concurrent meteorological observations and forecasts. StatCast is based on regime-dependent artificial intelligence forecasting techniques and has been shown to improve on “smart persistence” forecasts by 15-50%. A second category of short-range forecasting systems employ satellite imagery and use that information to discern clouds and their motion, allowing them to project the clouds, and the resulting blockage of irradiance, in time. CIRACast (the system produced by the Cooperative Institute for Atmospheric Research [CIRA] at Colorado State University) was already one of the more advanced cloud motion systems, which is the reason that team was brought to this project. During the project timeframe, the CIRA team was able to advance cloud shadowing, parallax removal, and implementation of better advecting winds at different altitudes. CIRACast shows generally a 25-40% improvement over Smart Persistence between sunrise and approximately 1600 UTC (Coordinated Universal Time) . A second satellite-based system, MADCast (Multi-sensor Advective Diffusive foreCast system), assimilates data from multiple satellite imagers and profilers to assimilate a fully three-dimensional picture of the cloud into the dynamic core of WRF. During 2015, MADCast (provided at least 70% improvement over Smart Persistence, with most of that skill being derived during partly cloudy conditions. That allows advection of the clouds via the Weather Research and Forecasting (WRF) model dynamics directly. After WRF-Solar™ showed initial success, it was also deployed in nowcasting mode with coarser runs out to 6 hours made hourly. It provided improvements on the order of 50-60% over Smart Persistence for forecasts up to 1600 UTC. The advantages of WRF-Solar-Nowcasting and MADCast were then blended to develop the new MAD-WRF model that incorporates the most important features of each of those models, both assimilating satellite cloud fields and using WRF-So far physics to develop and dissipate clouds. MAE improvements for MAD-WRF for forecasts from 3-6 hours are improved over WRF-Solar-Now by 20%. While all the Nowcasting system components by themselves provide improvement over Smart Persistence, the largest benefit is derived when they are smartly blended together by the Nowcasting Integrator to produce an integrated forecast. The development of WRF-Solar™ under this project has provided the first numerical weather prediction (NWP) model specifically designed to meet the needs of irradiance forecasting. The first augmentation improved the solar tracking algorithm to account for deviations associated with the eccentricity of the Earth’s orbit and the obliquity of the Earth. Second, WRF-Solar™ added the direct normal irradiance (DNI) and diffuse (DIF) components from the radiation parameterization to the model output. Third, efficient parameterizations were implemented to either interpolate the irradiance in between calls to the expensive radiative transfer parameterization, or to use a fast radiative transfer code that avoids computing three-dimensional heating rates but provides the surface irradiance. Fourth, a new parameterization was developed to improve the representation of absorption and scattering of radiation by aerosols (aerosol direct effect). A fifth advance is that the aerosols now interact with the cloud microphysics, altering the cloud evolution and radiative properties, an effect that has been traditionally only implemented in atmospheric computationally costly chemistry models. A sixth development accounts for the feedbacks that sub-grid scale clouds produce in shortwave irradiance as implemented in a shallow cumulus parameterization Finally, WRF-Solar™ also allows assimilation of infrared irradiances from satellites to determine the three dimensional cloud field, allowing for an improved initialization of the cloud field that increases the performance of short-range forecasts. We find that WRF-Solar™ can improve clear sky irradiance prediction by 15-80% over a standard version of WRF, depending on location and cloud conditions. In a formal comparison to the NAM baseline, WRF-Solar™ showed improvements in the Day-Ahead forecast of 22-42%. The SunCast™ system requires substantial software engineering to blend all of the new model components as well as existing publically available NWP model runs. To do this we use an expert system for the Nowcasting blender and the Dynamic Integrated foreCast (DICast®) system for the NWP models. These two systems are then blended, we use an empirical power conversion method to convert the irradiance predictions to power, then apply an analog ensemble (AnEn) approach to further tune the forecast as well as to estimate its uncertainty. The AnEn module decreased RMSE (root mean squared error) by 17% over the blended SunCast™ power forecasts and provided skill in the probabilistic forecast with a Brier Skill Score of 0.55. In addition, we have also developed a Gridded Atmospheric Forecast System (GRAFS) in parallel, leveraging cost share funds. An economic evaluation based on Production Cost Modeling in the Public Service Company of Colorado showed that the observed 50% improvement in forecast accuracy will save their customers $819,200 with the projected MW deployment for 2024. Using econometrics, NCAR has scaled this savings to a national level and shown that an annual expected savings for this 50% forecast error reduction ranges from $11M in 2015 to $43M expected in 2040 with increased solar deployment. This amounts to a $455M discounted savings over the 26 year period of analysis.

14 SOLAR ENERGY↗

Performance Assessment of a Generic Nuclear Waste Repository in Shale - 18392

A performance assessment (PA) is an important component of a comprehensive safety analysis for a nuclear waste repository. In a PA, probabilistic simulations of the total repository system are performed, and results are evaluated against performance metrics. Uncertainty and sensitivity analyses help prioritize additional research and model development. The United States Department of Energy (DOE) has been developing a state of the art PA simulation software tool-kit, Geologic Disposal Safety Assessment (GDSA) Framework, that couples increasingly higher fidelity models of subsystem processes into total system PA simulations. Over the past several years, PAs of generic repositories in three geologic media (salt, shale, and crystalline rock) have demonstrated ongoing developments in capability. The current PA of a nuclear waste repository in a generic shale formation showcases GDSA Framework, including capabilities in domain discretization (Cubit), multi-physics simulations (PFLOTRAN), uncertainty and sensitivity analysis (Dakota), and visualization (Paraview). The generic shale reference case considers the disposal of 22,000 metric tons heavy metal of commercial spent nuclear fuel (SNF) in a generic shale formation. PA simulations account for the thermal load and radionuclide inventory of the waste form, components of the engineered barrier system including waste package and buffer, and components of the natural barrier system including the host rock shale and underlying and overlying stratigraphic units. Two repository layouts are considered, one for emplacement of waste packages containing 12 pressurized water reactor (PWR) assemblies, and one for 4-PWR waste package emplacement. Model domains are half-symmetry and contain between 7 and 22 million grid cells. Grid refinement captures the detail of individual waste packages, emplacement drifts, access drifts, and shafts. Simulations are run in a high-performance computing (HPC) environment on 512 or 2048 processes. The governing equations describing coupled heat and fluid flow and reactive transport are solved with PFLOTRAN, an open-source, massively parallel multiphase flow and reactive transport code. Additional simulated processes include waste package degradation; waste form dissolution; radioactive decay and ingrowth in aqueous, solid, and adsorbed phases; and a simple biosphere based on a pumping well. Simulations are run to 106 y; dose and radionuclide concentrations are observed within aquifers at a point approximately 5 km downgradient of the repository. Dakota is used to sample likely ranges of input parameters including waste form and waste package degradation rates and properties of engineered and natural materials to quantify uncertainty in predicted concentrations and sensitivity to input parameters. Given the assumptions of the reference case, results of 12-PWR and 4-PWR simulations are very similar. Dose at the pumping well 5-km downgradient of the repository does not exceed 10-9 Sv/y. I-129 is the largest contributor to dose. The dose due to Cl-36, the next largest contributor, is orders of magnitude smaller. At the pumping well, the concentration of I-129 5-km downgradient does not exceed 10-14 mol/L. Rank correlation coefficients indicate that the maximum concentration of I-129 within the aquifer is sensitive to shale (repository host rock) porosity and aquifer permeability. These results affirm that HPC-capable codes can be used to simulate important multi-physics couplings directly in a total system performance assessment of a geologic repository, and can be used in prioritization of future research and model development. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

SciDAC's Earth System Grid Center for Enabling Technologies Semi-Annual Progress Report for the Period April 1, 2009 through September 30, 2009

This report summarizes work carried out by the ESG-CET during the period April 1, 2009 through September 30, 2009. It includes discussion of highlights, overall progress, period goals, collaborations, papers, and presentations. To learn more about our project, and to find previous reports, please visit the Earth System Grid Center for Enabling Technologies (ESG-CET) website. This report will be forwarded to the DOE SciDAC program management, the Office of Biological and Environmental Research (OBER) program management, national and international collaborators and stakeholders (e.g., the Community Climate System Model (CCSM), the Intergovernmental Panel on Climate Change (IPCC) 5th Assessment Report (AR5), the Climate Science Computational End Station (CCES), the SciDAC II: A Scalable and Extensible Earth System Model for Climate Change Science, the North American Regional Climate Change Assessment Program (NARCCAP), and other wide-ranging climate model evaluation activities). During this semi-annual reporting period, the ESG-CET team continued its efforts to complete software components needed for the ESG Gateway and Data Node. These components include: Data Versioning, Data Replication, DataMover-Lite (DML) and Bulk Data Mover (BDM), Metrics, Product Services, and Security, all joining together to form ESG-CET's first beta release. The launch of the beta release is scheduled for late October with the installation of ESG Gateways at NCAR and LLNL/PCMDI. Using the developed ESG Data Publisher, the ESG II CMIP3 (IPCC AR4) data holdings - approximately 35 TB - will be among the first datasets to be published into the new ESG enterprise system. In addition, the NCAR's ESG II data holdings will also be published into the new system - approximately 200 TB. This period also saw the testing of the ESG Data Node at various collaboration sites, including: the British Atmospheric Data Center (BADC), the Max-Planck-Institute for Meteorology, the University of Tokyo Center for Climate System Research, and the Australian National University. This period, a total of 14 national and international sites installed an ESG Data Node for testing. During this period, we also continued to provide production-level services to the community, providing researchers worldwide with access to CMIP3 (IPCC AR4), CCES, and CCSM, Parallel Climate Model (PCM), Parallel Ocean Program (POP), and Cloud Feedback Model Intercomparison Project (CFMIP), and NARCCAP data.

54 ENVIRONMENTAL SCIENCES↗

The Data Transfer Kit: A geometric rendezvous-based tool for multiphysics data transfer

The Data Transfer Kit (DTK) is a software library designed to provide parallel data transfer services for arbitrary physics components based on the concept of geometric rendezvous. The rendezvous algorithm provides a means to geometrically correlate two geometric domains that may be arbitrarily decomposed in a parallel simulation. By repartitioning both domains such that they have the same geometric domain on each parallel process, efficient and load balanced search operations and data transfer can be performed at a desirable algorithmic time complexity with low communication overhead relative to other types of mapping algorithms. With the increased development efforts in multiphysics simulation and other multiple mesh and geometry problems, generating parallel topology maps for transferring fields and other data between geometric domains is a common operation. The algorithms used to generate parallel topology maps based on the concept of geometric rendezvous as implemented in DTK are described with an example using a conjugate heat transfer calculation and thermal coupling with a neutronics code. In addition, we provide the results of initial scaling studies performed on the Jaguar Cray XK6 system at Oak Ridge National Laboratory for a worse-case-scenario problem in terms of algorithmic complexity that shows good scaling on 0(1 x 104) cores for topology map generation and excellent scaling on 0(1 x 105) cores for the data transfer operation with meshes of O(1 x 109) elements. (authors)

97 MATHEMATICS AND COMPUTING↗

Measurement of Flow Phenomena in a Lower Plenum Model of a Prismatic Gas-Cooled Reactor

Mean-velocity-field and turbulence data are presented that measure turbulent flow phenomena in an approximately 1:7 scale model of a region of the lower plenum of a typical prismatic gas-cooled reactor (GCR) similar to a General Atomics Gas-Turbine-Modular Helium Reactor (GTMHR) design. The data were obtained in the Matched-Index-of-Refraction (MIR) facility at Idaho National Laboratory (INL) and are offered for assessing computational fluid dynamics (CFD) software. This experiment has been selected as the first Standard Problem endorsed by the Generation IV International Forum. Results concentrate on the region of the lower plenum near its far reflector wall (away from the outlet duct). The flow in the lower plenum consists of multiple jets injected into a confined cross flow - with obstructions. The model consists of a row of full circular posts along its centerline with half-posts on the two parallel walls to approximate geometry scaled to that expected from the staggered parallel rows of posts in the reactor design. The model is fabricated from clear, fused quartz to match the refractive-index of the working fluid so that optical techniques may be employed for the measurements. The benefit of the MIR technique is that it permits optical measurements to determine flow characteristics in complex passages in and around objects to be obtained without locating intrusive transducers that will disturb the flow field and without distortion of the optical paths. An advantage of the INL system is its large size, leading to improved spatial and temporal resolution compared to similar facilities at smaller scales. A three-dimensional (3-D) Particle Image Velocimetry (PIV) system was used to collect the data. Inlet jet Reynolds numbers (based on the jet diameter and the time-mean bulk velocity) are approximately 4,300 and 12,400. Uncertainty analyses and a discussion of the standard problem are included. The measurements reveal developing, non-uniform, turbulent flow in the inlet jets and complicated flow patterns in the model lower plenum. Data include three-dimensional vector plots, data displays along the coordinate planes (slices) and presentations that describe the component flows at specific regions in the model. Information on inlet conditions is also presented.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Initial CEED API (CEED-MS10)

Key components of CEED software development involve fast finite element operator storage and evaluation, architecture optimizations, performant algorithms for all orders, global kernels for finite element operators, efficient use of the memory sub-system, optimal data locality and motion, enhanced scalability and parallelism, and fast tensor contractions. As ultimate goals of the project, CEED is exploring and identifying the best algorithms for the full range of discretizations and applying algorithmic and software development to support ECP applications’ needs. In this milestone, we initiated the development and implementation of a common CEED API consisting of two main components: low-level and high-level APIs. We accomplished the goal of having the low-level component ready for the release of CEED v1.0 in the next quarter, see CEED-MS13. In this report we also provide a highlight from CEED’s integration with the MARBL app and the corresponding Laghos miniapp. A main theme in this work has been the systematic transition of advanced algorithms, developed and tested in Laghos, to the large-scale multi-physics settings of the MARBL code. The low-level CEED API provides a set of FE kernels and components for writing new low-level kernels. Examples include: vector and sparse linear algebra, element matrix assembly over a batch of elements, partial assembly and action for efficient high-order operators like mass, diffusion, advection, etc. The main goal of the low-level API is to establish the basis for the high-level API. Also, identifying such low-level kernels and providing a reference implementation for them serves as the basis for collaboration with vendors and STs. Another goal for the low-level API is to make it easy for applications with their own discretization infrastructure, who do not want to switch directly to using the high-level CEED API, to still evaluate and use the core operations provided by the low-level API. One of the challenges with high-order methods is that a global sparse matrix is no longer a good representation of a high-order linear operator, both with respect to the FLOPs needed for its evaluation, as well as the memory transfer needed for a matvec. Thus, another important question we tried to address while developing the low-level API is: is it possible to abstract out in a natural and efficient way the different implementations and data structures that arise when dealing with various computational device types: CPUs, GPUs, etc. Fully answering this question is beyond the scope of this particular milestone, since that requires longer time as we develop and expand the devices we support. The future high-level API will be the frontend provided by CEED consisting of high-level abstractions such as meshes, finite element spaces and solutions, bilinear forms, solvers, etc. At this high-level, an important aspect of the API will be to provide a uniform abstraction layer for writing code in a portable manner. To achieve that, we plan to utilize the MPI programming model combined with a device abstraction layer (DAL) for finite element computations that we will develop. The DAL will serve as the bridge between the low- and the high-level APIs, allowing a variety of programming models to be used for the low-level kernel implementations. The software artifacts delivered as part of this milestone include an initial CEED API software implementation (libCEED) provided through the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q1 of FY18, including: the Nek5000 v17.0 MFEM v3.3.2 and PETSc v3.8 releases, collaborations with ECP/ST and SciDAC projects, the inaugural Nek5000 hackathon and various outreach activities.

97 MATHEMATICS AND COMPUTING↗

CyDER: A Cyber Physical Co-simulation Platform for Distributed Energy Resources in Smartgrids

The CyDER project aimed at developing an open-source, modular and scalable co-simulation platform for power grids with large shares of Distributed Energy Resources (DERs). The project partners are the Lawrence Berkeley National Lab (LBNL), Lawrence Livermore National Lab (LLNL), PG&E, SolarCity, and ChargePoint. The prime recipient is LBNL; SolarCity and ChargePoint were partners for the project’s first two years. Increased DER integration introduces a number of challenges in power grid operation including a more dynamic interaction between the transmission grid and distribution grids, and increased modeling complexity. Although specialized software exists to precisely model different components of the power system, it is far from trivial to integrate all various models and perform a holistic simulation. Instead of replicating all models in a common simulation program, a commonly accepted approach to tackle this model diversity is to couple third-party simulators and models through a co-simulation platform that coordinates information exchange among the various components. Following this line of research, this project’s objective was to develop a co-simulation platform based on a widely accepted industrial standard called Functional Mockup Interface (FMI). Within this process, the project developed models compliant with the FMI standard, called Functional Mockup Units (FMUs), and used them to perform various operational and planning power system analyses. Relying and building upon an industrial standard is the main differentiation of this project compared with previous or parallel efforts in the co-simulation area. Particular emphasis was put on delivering software utilities to facilitate setting up and running co-simulations by end-users. Furthermore, a strong aspect of this project is demonstrating that co-simulation techniques can be used to perform Hardware-in-the-Loop (HIL) simulations that couple software components (e.g., simulated models) with hardware components (e.g., real devices such PV systems and batteries). The long-term goal of CyDER project is to help establish FMI as a powerful standard for co-simulation and promote adoption by electric utilities and other interested stakeholders. The main accomplishments of the project include the development of several FMUs including distribution and transmission grid models, PV inverters with Volt/Var/Watt controllers, batteries, and predictive optimal controllers. Additionally, a unique software package was developed, called SimulatorToFMU, which is capable of exporting any Python-driven simulator or Python script as an FMU. This is an important contribution towards establishing FMI as one of the main co-simulation standards, because more and more third-party programs for sub-system modeling and simulation are delivered with Python APIs. The CyDER platform was used to perform PV hosting capacity analyses in real utility feeders with and without smart inverter controls, battery storage, and EV charging. Smart inverter controls include conventional Volt/Var/Watt controls for reactive power support and active power curtailment, but also predictive controls that optimize the charging and discharging profile of the battery connected on the DC side in order to minimize the customer’s economic benefit. Finally, an important result of this project is delivering an experimental setup that consists of residential-scale PV inverters with battery storage, a real-time grid simulator with an ideal voltage source as grid emulator, and micro Phasor Measurement Units (PMUs). All these components and additional software modules are coupled to one another using the FMI standard and can be co-simulated with the CyDER platform.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Reassessing the MCNP Random Number Generator

Random number generators are integral components to Monte Carlo codes. They provide the pseudorandom number sequence used to actually sample the distributions of interest. As a result, they are one of the most important components to the software. The current recommended MCNP random number generator is a 63-bit linear congruential generator (LCG). This generator is quite fast, but it has some drawbacks. First, it only has a period of 2 63 . Due to the necessarily non-optimal usage of random numbers to ensure parallel reproducibility, this amount is too few to guarantee random number sequences are not reused in all configurations the code runs under. As simulation size increases, users will need to be aware of the limitations of the generator and tune configuration variables to best suit their simulations, or they will need to assume that reuse is not negatively affecting their answers. Neither of these are optimal. Second, small LCGs are fairly weak in bit generation quality, and this can have an unknown impact on the quality of the simulation. This paper is an investigation into whether or not more modern random number generators can supersede the current ones. The goal is to find a generator that is similar or superior in speed to the LCGs, has a state space large enough to make strong guarantees about random number reuse, and passes all modern random number test suites. If such a generator is found, it would eliminate the need for the user to even be aware of the limitations of the random number generator and would simplify the use of the code. This paper will be broken into several parts. Sec. 2 will discuss the evolution of the random number generator within the MCNP code. Sec. 3 will go over what a Monte Carlo code needs from a generator to be reproducible and portable and how the current generator behaves in that light. Sec. 4 goes through how each generator was tested. Finally, Sec. 5 will discuss improvements that could be made to the code.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Summary of Documentation for DYNA3D-ParaDyn's Software Quality Assurance Regression Test Problems

The Software Quality Assurance (SQA) regression test suite for DYNA3D (Zywicz and Lin, 2015) and ParaDyn (DeGroot, et al., 2015) currently contains approximately 600 problems divided into 21 suites, and is a required component of ParaDyn’s SQA plan (Ferencz and Oliver, 2013). The regression suite allows developers to ensure that software modifications do not unintentionally alter the code response. The entire regression suite is run prior to permanently incorporating any software modification or addition. When code modifications alter test problem results, the specific cause must be determined and fully understood before the software changes and revised test answers can be incorporated. The regression suite is executed on LLNL platforms using a Python script and an associated data file. The user specifies the DYNA3D or ParaDyn executable, number of processors to use, test problems to run, and other options to the script. The data file details how each problem and its answer extraction scripts are executed. For each problem in the regression suite there exists an input deck, an eight-processor partition file, an answer file, and various extraction scripts. These scripts assemble a temporary answer file in a specific format from the simulation results. The temporary and stored answer files are compared to a specific level of numerical precision, and when differences are detected the test problem is flagged as failed. Presently, numerical results are stored and compared to 16 digits. At this accuracy level different processor types, compilers, number of partitions, etc. impact the results to various degrees. Thus, for consistency purposes the regression suite is run with ParaDyn using 8 processors on machines with a specific processor type (currently the Intel Xeon E5530 processor). For non-parallel regression problems, i.e., the two XFEM problems, DYNA3D is used instead. When environments or platforms change, executables using the current source code and the new resource are created and the regression suite is run. If differences in answers arise, the new answers are retained provided that the differences are inconsequential. This bootstrap approach allows the test suite answers to evolve in a controlled manner with a high level of confidence. Developers also run the entire regression suite with (serial) DYNA3D. While these results normally differ from the stored (parallel) answers, abnormal termination or wildly different values are strong indicators of potential issues.

42 ENGINEERING↗

Node failure resiliency for Uintah without checkpointing

The frequency of failures in upcoming exascale supercomputers may well be greater than at present due to many-core architectures if component failure rates remain unchanged. This potential increase in failure frequency coupled with I/O challenges at exascale may prove problematic for current resiliency approaches such as checkpoint restarting, although the use of fast intermediate memory may help. Algorithm-Based Fault Tolerance (ABFT) using Adaptive Mesh Refinement (AMR) is one resiliency approach used to address these challenges. For adaptive mesh codes, a coarse mesh version of the solu- tion may be used to restore the fine mesh solution. This paper addresses the implementation of the ABFT approach within the Uintah software framework: both at a software level within Uintah and in the data reconstruction method used for the recovery of lost data. This method has two problems: inaccuracies introduced during the reconstruction propagate forward in time, and the physical consistency of variables such as positivity or boundedness may be violated during interpolation. These challenges can be addressed by the combination of two techniques: 1. a fault-tolerant MPI implementation to recover from runtime node failures, and 2. high-order interpolation schemes to preserve the physical solution and reconstruct lost data. Here, the approach considered here uses a "Limited Essentially Non-Oscillatory" (LENO) scheme along with AMR to rebuild the lost data without checkpointing using Uintah. Experiments were carried out using a fault-tolerant MPI - ULFM to recover from runtime failure, and LENO to recover data on patches belonging to failed ranks, while the simulation was continued to the end. Results show that this ABFT approach is up to 10x faster than the traditional checkpointing method. The new interpolation approach is more accurate than linear interpolation and not subject to the overshoots found in other interpolation methods.

97 MATHEMATICS AND COMPUTING↗

Nuclear Energy Advanced Modeling and Simulation (NEAMS) waste Integrated Performance and Safety Codes (IPSC) : gap analysis for high fidelity and performance assessment code development.

This report describes a gap analysis performed in the process of developing the Waste Integrated Performance and Safety Codes (IPSC) in support of the U.S. Department of Energy (DOE) Office of Nuclear Energy Advanced Modeling and Simulation (NEAMS) Campaign. The goal of the Waste IPSC is to develop an integrated suite of computational modeling and simulation capabilities to quantitatively assess the long-term performance of waste forms in the engineered and geologic environments of a radioactive waste storage or disposal system. The Waste IPSC will provide this simulation capability (1) for a range of disposal concepts, waste form types, engineered repository designs, and geologic settings, (2) for a range of time scales and distances, (3) with appropriate consideration of the inherent uncertainties, and (4) in accordance with rigorous verification, validation, and software quality requirements. The gap analyses documented in this report were are performed during an initial gap analysis to identify candidate codes and tools to support the development and integration of the Waste IPSC, and during follow-on activities that delved into more detailed assessments of the various codes that were acquired, studied, and tested. The current Waste IPSC strategy is to acquire and integrate the necessary Waste IPSC capabilities wherever feasible, and develop only those capabilities that cannot be acquired or suitably integrated, verified, or validated. The gap analysis indicates that significant capabilities may already exist in the existing THC codes although there is no single code able to fully account for all physical and chemical processes involved in a waste disposal system. Large gaps exist in modeling chemical processes and their couplings with other processes. The coupling of chemical processes with flow transport and mechanical deformation remains challenging. The data for extreme environments (e.g., for elevated temperature and high ionic strength media) that are needed for repository modeling are severely lacking. In addition, most of existing reactive transport codes were developed for non-radioactive contaminants, and they need to be adapted to account for radionuclide decay and in-growth. The accessibility to the source codes is generally limited. Because the problems of interest for the Waste IPSC are likely to result in relatively large computational models, a compact memory-usage footprint and a fast/robust solution procedure will be needed. A robust massively parallel processing (MPP) capability will also be required to provide reasonable turnaround times on the analyses that will be performed with the code. A performance assessment (PA) calculation for a waste disposal system generally requires a large number (hundreds to thousands) of model simulations to quantify the effect of model parameter uncertainties on the predicted repository performance. A set of codes for a PA calculation must be sufficiently robust and fast in terms of code execution. A PA system as a whole must be able to provide multiple alternative models for a specific set of physical/chemical processes, so that the users can choose various levels of modeling complexity based on their modeling needs. This requires PA codes, preferably, to be highly modularized. Most of the existing codes have difficulties meeting these requirements. Based on the gap analysis results, we have made the following recommendations for the code selection and code development for the NEAMS waste IPSC: (1) build fully coupled high-fidelity THCMBR codes using the existing SIERRA codes (e.g., ARIA and ADAGIO) and platform, (2) use DAKOTA to build an enhanced performance assessment system (EPAS), and build a modular code architecture and key code modules for performance assessments. The key chemical calculation modules will be built by expanding the existing CANTERA capabilities as well as by extracting useful components from other existing codes.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

TH-E-9A-01: Medical Physics 1.0 to 2.0, Session 4: Computed Tomography, Ultrasound and Nuclear Medicine

Medical Physics 2.0 is a bold vision for an existential transition of clinical imaging physics in face of the new realities of value-based and evidencebased medicine, comparative effectiveness, and meaningful use. It speaks to how clinical imaging physics can expand beyond traditional insular models of inspection and acceptance testing, oriented toward compliance, towards team-based models of operational engagement, prospective definition and assurance of effective use, and retrospective evaluation of clinical performance. Organized into four sessions of the AAPM, this particular session focuses on three specific modalities as outlined below. CT 2.0: CT has been undergoing a dramatic transition in the last few decades. While the changes in the technology merits discussions of their own, an important question is how clinical medical physicists are expected to effectively engage with the new realities of CT technology and practice. Consistent with the upcoming paradigm of Medical Physics 2.0, this CT presentation aims to provide definitions and demonstration of the components of the new clinical medical physics practice pertaining CT. The topics covered include physics metrics and analytics that aim to provide higher order clinicallyrelevant quantification of system performance as pertains to new (and not so new) technologies. That will include the new radiation and dose metrics (SSDE, organ dose, risk indices), image quality metrology (MTF/NPS/d’), task-based phantoms, and the effect of patient size. That will follow with a discussion of the testing implication of new CT hardware (detectors, tubes), acquisition methods (innovative helical geometries, AEC, wide beam CT, dual energy, inverse geometry, application specialties), and image processing and analysis (iterative reconstructions, quantitative CT, advanced renditions). The presentation will conclude with a discussion of clinical and operational aspects of Medical Physics 2.0 including training and communication, use optimization (dose and technique factors), automated analysis and data management (automated QC methods, protocol tracking, dose monitoring, issue tracking), and meaningful QC considerations. US 2.0: Ultrasound imaging is evolving at a rapid pace, adding new imaging functions and modes that continue to enhance its clinical utility and benefits to patients. The ultrasound talk will look ahead 10–15 years and consider how medical physicists can bring maximal value to the clinical ultrasound practices of the future. The roles of physics in accreditation and regulatory compliance, image quality and exam optimization, clinical innovation, and education of staff and trainees will all be considered. A detailed examination of expected technology evolution and impact on image quality metrics will be presented. Clinical implementation of comprehensive physics services will also be discussed. Nuclear Medicine 2.0: Although the basic science of nuclear imaging has remained relatively unchanged since its inception, advances in instrumentation continue to advance the field into new territories. With a great number of these advances occurring over the past decade, the role and testing strategies of clinical nuclear medicine physicists must evolve in parallel. The Nuclear Medicine 2.0 presentation is designed to highlight some of the recent advances from a clinical medical physicist perspective and provide ideas and motivation for designing better evaluation strategies. Topics include improvement of traditional physics metrics and analytics, testing implications of hybrid imaging and advanced detector technologies, and strategies for effective implementation into the clinic. Learning Objectives: Become familiar with new physics metrics and analytics in nuclear medicine, CT, and ultrasound. To become familiar with the major new developments of clinical physics support. To understand the physics testing implications of new technologies, hardware, software, and applications. Identify approaches for implementing comprehensive medical physics services in future imaging practices.

60 APPLIED LIFE SCIENCES↗

Coastal Ohio Wind Project

The Coastal Ohio Wind Project intends to address problems that impede deployment of wind turbines in the coastal and offshore regions of Northern Ohio. The project evaluates different wind turbine designs and the potential impact of offshore turbines on migratory and resident birds by developing multidisciplinary research, which involves wildlife biology, electrical and mechanical engineering, and geospatial science. Firstly, the project conducts cost and performance studies of two- and three-blade wind turbines using a turbine design suited for the Great Lakes. The numerical studies comprised an analysis and evaluation of the annual energy production of two- and three-blade wind turbines to determine the levelized cost of energy. This task also involved wind tunnel studies of model wind turbines to quantify the wake flow field of upwind and downwind wind turbine-tower arrangements. The experimental work included a study of a scaled model of an offshore wind turbine platform in a water tunnel. The levelized cost of energy work consisted of the development and application of a cost model to predict the cost of energy produced by a wind turbine system placed offshore. The analysis found that a floating two-blade wind turbine presents the most cost effective alternative for the Great Lakes. The load effects studies showed that the two-blade wind turbine model experiences less torque under all IEC Standard design load cases considered. Other load effects did not show this trend and depending on the design load cases, the two-bladed wind turbine showed higher or lower load effects. The experimental studies of the wake were conducted using smoke flow visualization and hot wire anemometry. Flow visualization studies showed that in the downwind turbine configuration the wake flow was insensitive to the presence of the blade and was very similar to that of the tower alone. On the other hand, in the upwind turbine configuration, increasing the rotor blade angle of attack reduced the wake size and enhanced the vortices in the flow downstream of the turbine-tower compared with the tower alone case. Mean and rms velocity distributions from hot wire anemometer data confirmed that in a downwind configuration, the wake of the tower dominates the flow, thus the flow fields of a tower alone and tower-turbine combinations are nearly the same. For the upwind configuration, the mean velocity shows a narrowing of the wake compared with the tower alone case. The downwind configuration wake persisted longer than that of an upwind configuration; however, it was not possible to quantify this difference because of the size limitation of the wind tunnel downstream of the test section. The water tunnel studies demonstrated that the scale model studies could be used to adequately produce accurate motions to model the motions of a wind turbine platform subject to large waves. It was found that the important factors that affect the platform is whether the platform is submerged or surface piercing. In the former, the loads on the platform will be relatively reduced whereas in the latter case, the structure pierces the wave free surface and gains stiffness and stability. The other important element that affects the movement of the platform is depth of the sea in which the wind turbine will be installed. Furthermore, the wildlife biology component evaluated migratory patterns by different monitoring systems consisting of marine radar, thermal IR camera and acoustic recorders. The types of radar used in the project are weather surveillance radar and marine radar. The weather surveillance radar (1988 Doppler), also known as Next Generation Radar (NEXRAD), provides a network of weather stations in the US. Data generated from this network were used to understand general migratory patterns, migratory stopover habitats, and other patterns caused by the effects of weather conditions. At a local scale our marine radar was used to complement the datasets from NEXRAD and to collect additional monitoring parameters such as passage rates, flight paths, flight directions, and flight altitudes of nocturnal migrating species. Our work focused on the design and development of custom built marine radar that used t-bar and parabolic dish antennas. The marine radar used in the project was Furuno (XANK250) which was coupled with a XIR3000B digitizing card from Russell Technologies for collection of the radar data. The radar data was processed by open source radR processing software using different computational techniques and methods. Additional data from thermal IR imaging cameras were collected to detect heat emitted from objects and provide information on movements of birds and bats, data which we used for different animal flight behavior analysis. Lastly, the data from the acoustic recorders were used to provide the number of bird calls for assessing patterns and peak passage rates during migration. The development of the geospatial database included collection of different data sources that are used to support offshore wind turbine development. Many different data sets were collected and organized using initial version of web-based repository software tools that can accommodate distribution of rectified pertinent data sets such as the lake depth, lake bottom engineering parameters, extent of ice, navigation pathways, wind speed, important bird habitats, fish efforts and other layers that are relevant for supporting robust offshore wind turbine developments. Additional geospatial products developed during the project included few different prototypes for offshore wind farm suitability which can involve different stakeholders and participants for solving complex planning problems and building consensus. Some of the prototypes include spatial decision support system (SDSS) for collaborative decision making, a web-based Participatory Geographic Information System (PGIS) framework for evaluating importance of different decision alternatives using different evaluation criteria, and an Android application for collection of field data using mobile and tablet devices . In summary, the simulations of two- and three-blade wind turbines suggested that two-bladed machines could produce comparable annual energy as the three-blade wind turbines but have a lighter tower top weight, which leads to lower cost of energy. In addition, the two-blade rotor configuration potentially costs 20% less than a three blade configuration that produces the same power at the same site. The cost model analysis predicted a potential cost savings of approximately 15% for offshore two-blade wind turbines. The foundation design for a wind turbine in Lake Erie is likely to be driven by ice loads based on the currently available ice data and ice mechanics models. Hence, for Lake Eire, the cost savings will be somewhat smaller than the other lakes in the Great Lakes. Considering the size of cranes and vessels currently available in the Great Lakes, the cost optimal wind turbine size should be 3 MW, not larger. The surveillance data from different monitoring systems suggested that bird and bat passage rates per hour were comparable during heavy migrations in both spring and fall seasons while passage rates were significantly correlated to wind directions and wind speeds. The altitude of migration was higher during heavy migrations and higher over water relative to over land. Notable portions of migration on some spring nights occurred parallel the shoreline, often moving perpendicular to southern winds. The birds approaching the Western basin have a higher propensity to cross than birds approaching the Central basin of Lake Erie and as such offshore turbine development might be a better option further east towards Cleveland than in the Western basin. The high stopover density was more strongly associated with migration volume the following night rather than the preceding night. The processed mean scalar wind speeds with temporal resolutions as fine as 10-minute intervals near turbine height showed that August is the month with the weakest winds while December is the month, which typically has the strongest winds. The ice data suggests that shallow western basin of Lake Erie has higher ice cover duration many times exceeding 90 days during some winters.

17 WIND ENERGY↗