Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel codes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

GeantV: Results from the Prototype of Concurrent Vector Particle Transport Simulation in HEP

Full detector simulation was among the largest CPU consumers in all CERN experiment software stacks for the first two runs of the Large Hadron Collider. In the early 2010s, it was projected that simulation demands would scale linearly with increasing luminosity, with only partial compensation from increasing computing resources. The extension of fast simulation approaches to cover more use cases that represent a larger fraction of the simulation budget is only part of the solution, because of intrinsic precision limitations. The remainder corresponds to speeding up the simulation software by several factors, which is not achievable by just applying simple optimizations to the current code base. In this context, the GeantV R&D project was launched, aiming to redesign the legacy particle transport code in order to benefit from features of fine-grained parallelism, including vectorization and increased locality of both instruction and data. This paper provides an extensive presentation of the results and achievements of this R&D project, as well as the conclusions and lessons learned from the beta version prototype.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

FLARE: field line analysis and reconstruction for 3D boundary plasma modeling

The FLARE code is a magnetic mesh generator that is integrated within a suite of tools for the analysis of the magnetic geometry in toroidal fusion devices. A magnetic mesh is constructed from field line segments and permits fast reconstruction of field lines in 3D boundary plasma codes such as EMC3-EIRENE. Both intrinsically non-axisymmetric configurations (stellarators) and those with symmetry breaking perturbations of an axisymmetric equilibrium (tokamaks) are supported. The code itself is written in Modern Fortran with MPI support for parallel computing, and it incorporates object-oriented programming for the definition of the magnetic field and the material surface geometry. Extended derived types for a number of different magnetohydrodynamic equilibrium and plasma response models are implemented. The core element of FLARE is a field line tracer with adaptive step-size control, and this is integrated into tools for the construction of Poincaré maps and invariant manifolds of X-points. A collection of high-level procedures that generate output files for visualization is build on top of that. The analysis modules are build with Python frontends that facilitate customization of tasks and/or scripting of parameter scans.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

Kimberlina 1.2 CCUS Geophysical Models and Synthetic Data Sets

This synthetic multi-scale and multi-physics data set was produced in collaboration with teams at the Lawrence Berkeley National Laboratory, National Energy Technology Laboratory, Los Alamos National Laboratory, and Colorado School of Mines through the Science-informed Machine Learning for Accelerating Real-Time Decisions in Subsurface Applications (SMART) Initiative. Data are associated with the following publication: Alumbaugh, D., Gasperikova, E., Crandall, D., Commer, M., Feng, S., Harbert, W., Li, Y., Lin, Y., and Samarasinghe, S., “The Kimberlina Synthetic Geophysical Model and Data Set for CO2 Monitoring Investigations”, The Geoscience Data Journal, 2023, DOI: 10.1002/gdj3.191. The dataset uses the Kimberlina 1.2 CO2 reservoir flow model simulations based on a hypothetical CO2 storage site in California (Birkholzer et al., 2011; Wainwright et al., 2013). Geophysical properties models (P- and S-wave seismic velocities, saturated density, and electrical resistivity) were produced with an approach similar to that of Yang et al. (2019) and Gasperikova et al. (2022) for 100 Kimberlina 1.2 reservoir models. Links to individual resources are provided below: [CO2 Saturation Models](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-co2-saturation-models); Resistivity Models – [part 1](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-resistivity-models-part-1), [part 2](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-resistivity-models-part-2), and [part 3](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-resistivity-models-part-3); [Vp Velocity Models](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-vp-velocity-models); [Vs Velocity Models](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-vs-velocity-models); [Density Models](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-density-models). The 3D distributions of geophysical properties for the 33 time stamps of the SIM001 model were used to generate synthetic seismic, gravity, and electromagnetic (EM) responses for 33 times between zero and 200 years. Synthetic surface seismic data were generated using 2D and 3D finite-difference codes that simulate the acoustic wave equation (Moczo et al., 2007). 2D data were simulated for six point-pressure sources along a 2D line with 10 m receiver spacing and a time spacing of 0.0005 s. 3D simulations were completed for 25 surface pressure sources using a source separation of 1 km in both the x and y directions and a time spacing of 0.001 s. Links to individual resources are provided below: [2D velocity models](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-2d-velocity-models) and [2D surface seismic data](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-2d-surface-seismic-data). [3D velocity models](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-velocity-models), and 3D seismic data [year0](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year0), [year1](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year1), [year2](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year2), [year5](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year5), [year10](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year10), [year15](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year15), [year20](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year20), [year25](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year25), [year30](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year30), [year35](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year35), [year40](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year40), [year45](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year45), [year49](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year49), [year50](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year50), [year51](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year51), [year52](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year52), [year55](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year55), [year60](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year60), [year65](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year65), [year70](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year70), [year75](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year75), [year80](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year80), [year85](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year85), [year90](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year90), [year95](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year95), [year100](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year100), [year110](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year110), [year120](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year120), [year130](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year130), [year140](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year140), [year150](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year150), [year175](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year175), [year200](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-seismic-data-year200). The Python scripts to read these models and data are provided [here](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-python-scripts). EM simulations used a borehole-to-surface survey configuration, with the source located near the reservoir level and receivers on the surface using the code developed by Commer and Newman (2008). Pseudo-2D data for the source at [2500 m](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-pseudo-2d-csem-data-tz2500m) and [3025 m](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-pseudo-2d-csem-data-tz3025m), used a 2D inline receiver configuration to simulate a response over 3D resistivity models. The [3D data](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-3d-csem-data) contain electric fields generated by borehole sources at monitoring well locations and measured over a surface receiver grid. Vector gravity data, both on the surface and in boreholes, were simulated using a modeling code developed by Rim and Li (2015). The simulation scenarios were parallel to those used for the EM: [pseudo-2D data](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-gravity-data) were calculated along the same lines and within the same boreholes, and [3D data](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-gravity-data) were simulated over 3D models on the surface and in three monitoring wells. A series of [synthetic well logs](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-well-logs) of CO2 saturation, acoustic velocity, density, and induction resistivity in the injection well and three monitoring wells are also provided at 0, 1, 2, 5, 10, 15, and 20 years after the initiation of injection. These were constructed by combining the low-frequency trend of the geophysical models with the high-frequency variations of actual well logs collected in the Kimberlina 1 well that was drilled at the proposed site. Measurements of permeability and pore connectivity were made on cores of Vedder Sandstone, which forms the primary reservoir unit: [CT micro scans](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-ct-micro-scans-of-vedder-formation) and [Industrial CT Images](https://edx.netl.doe.gov/dataset/kimberlina-1-2-ccus-geophysical-models-and-synthetic-data-sets-industrial-ct-images-vedder-formation). These measurements provide the range of scales in the otherwise synthetic data set to be as close to a real-world situation as possible. References: Birkholzer, J.T., Zhou, Q., Cortis, A. and Finsterle, S., 2011. A sensitivity study on regional pressure buildup from large-scale CO2 storage projects. Energy Procedia, 4, 4371-4378. Commer, M., and Newman, G.A., 2008. New advances in three-dimensional controlled-source electromagnetic inversion, Geophysical Journal International, 172, 513-535. Gasperikova, E., Appriou, D., Bonneville, A., Feng, Z., Huang, L., Gao, K., Yang, X., Daley, T., 2022, Sensitivity of geophysical techniques for monitoring secondary CO2 storage plumes, Int. J. Greenh. Gas Control, Volume 114, 103585, ISSN 1750-5836, https://doi.org/10.1016/j.ijggc.2022.103585. Moczo, P., J.O. Robertsson and L. Eisner, 2007, The finite-difference time-domain method for modeling of seismic wave propagation: Advances in geophysics, 48, 421-516. Rim, H., and Y. Li, 2015, Advantages of borehole vector gravity in density imaging, Geophysics, 80, G1-G13. Wainwright, H. M.; Finsterle, S.; Zhou, Q.; Birkholzer, J. T., 2013. Modeling the Performance of Large-Scale CO2 Storage Systems: A Comparison of Different Sensitivity Analysis Methods. International Journal of Greenhouse Gas Control, 17, 189205. https://doi.org/10.1016/j.ijggc.2013.05.007, DOI: 10.18141/1603331. Yang, X., Buscheck, T.A., Mansoor, K., Wang, Z., Gao, K., Huang, L., Appriou, D., and Carroll, S.A., 2019. Assessment of geophysical monitoring methods for detection of brine and CO2 leakage in drinking water aquifers, International Journal of Greenhouse Gas Control, 90, 102803, https://doi.org/10.1016/j.ijggc.2019.102803.

CCUS↗
DOE OSTI2020

Distributed fiber sensing systems for 3D combustion temperature field monitoring in coal-fired boilers using optically generated acoustic waves (Final Report)

In this project, we have developed and tested three kinds of fiber optic sensing systems for real time monitoring of temperature variations within an industrial scale boiler furnace. The fiber optic sensing systems target spatial and temporal distributions of high temperature profiles in a boiler furnace in fossil power plants. The reconstructed temperature profile will provide critical input for the control mechanisms to optimize the combustion process. This temperature profile will address the essential problem for fossil power plants in achieving higher efficiency and fewer pollutant emissions. Acoustic pyrometer systems have been used to reconstruct temperature field of power plant boilers based on measuring TOF (times-of-flight) of sound waves along some straight paths in a 2D cross-section of the boiler. In this project, optically generated acoustic signals from a fiber optic sensing system have replaced the acoustic signals generated from an electrical transducer. A 3D reconstruction algorithm replaced the previous 2D model. In this project, three kinds of fiber optic sensing systems have been developed and tested. They are fiber optic sensing system I, fiber optic sensing system II (Distributed Sensing System I) and fiber optic sensing system III (Distributed Sensing System II). For fiber optic sensing system I, the fiber optic ultrasound generator acts as a signal generator. A microphone, hydrophone or other electronic devices serve as a signal receiver. In this system, there are one generator and one receiver. Distance test, water temperature test, air temperature test, air temperature reconstruction, and GE ISBF pilot test were performed by Fiber optic sensing system I. The fiber optic sensing system I successfully detected temperature in all these tests. We got 2D temperature reconstruction results by using the fiber optic sensing system I and it matched the reference data. The fiber optic sensing system I successfully survived in GE ISBF boiler environment (480 °F). For fiber optic sensing system II (Distributed Sensing System I), it is an all optical ultrasound system. The fiber optic ultrasound generator acts as a signal generator. Fiber Bragg Grating (FBG) and Fabry-Perot (FP) sensor act as a signal receiver. In this system, there is one generator and one receiver. Aluminum plate temperature test, furnace high temperature test, and GE ISBF pilot test were performed by the fiber optic sensing system II. Fiber optic sensing system II successfully detected the temperature in all these tests. The fiber optic sensing system II successfully survived at up to 700 °C furnace environment and 320 °C GE ISBF boiler environment. For fiber optic sensing system III (Distributed Sensing System II), it is also an all optical ultrasound system. The fiber optic ultrasound generator acts as a signal generator. Multiple FP fiber sensors act as signal receivers. In this system, there is one generator and three receivers. Three GE ISBF pilot tests were performed by fiber optic sensing system III. The fiber optic sensing system III survived in the cold flow tests in GE’s ISBF pilot test facility. However, we didn’t get high temperature data by using this system since the nanosecond laser issues. During the period of the project, test trials, data simulation and algorithm optimization was performed successfully. For real time temperature field construction, the sampling rate must be fast enough to capture the field variations. The technology of Code-division multiple access (CDMA) is well studied which could allow parallel multiplexing, even if signals overlap in time or frequencies. Moreover, it has been known that extending the length of signal significantly improves SNR. For acoustic signals, these multiplexing techniques have also been widely used, mainly for sonar and acoustic communications. The CDMA modulation technique has been proposed and studied to guarantee high network throughput, low channel access delay and low energy consumption. We have studied the temperature field reconstruction using Gaussian Radial Basis Functions (GRBF)-based approximation approach. Reconstruction of 3D temperature field using Neural Networks with measured TOF and known propagation paths is feasible. 2D and 3D temperature field reconstruction simulation results are achieved. The milestone status is shown in Table 1. We finished milestone 1-8 and milestone 10. For milestone 9, we did three pilot tests by using the fiber optic sensing system III (Distributed Sensing System II) at GE Power. However, due to the failure of the ns laser, we did not get the temperature results. We conducted some additional tasks that were not originally proposed: 1) We fabricated a fiber optic sensing system I and did a pilot test based on this system. 2) In the proposal, we proposed two pilot tests at GE Power. In reality, we finished at least seven pilot tests at GE Power. GE Power has made a lot of efforts for supporting the pilot tests. 3) We got a simulation results based on CDMA. In summary, most of the tasks have been accomplished. The outcome of this project removed a few barriers that hinder the achievement of the final product of the distributed sensing systems. With the successful accomplishment of this project, a prototype of the fiber optic sensing system can be fabricated to attract more interests from companies and other funding agencies.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

Common Information Model for Electromagnetic Transients (CIM for EMT): CRADA 533 [Abstract only}

The control and protection functions of inverter-based resources (IBR) have raised concerns with bulk system reliability. Most of the current interest lies with solar photovoltaic generation but increasing amounts of storage would pose the same risks. Newer North American Electric Reliability Corporation (NERC) guidelines call for electromagnetic transient (EMT) studies of IBR and recommend that transmission operators collect distributed energy resource (DER) data to support such modeling. IEEE Standard P2800.1 is defining tests for model parameterization, so good model data should become available from inverter vendors. (EMT studies also apply to large power transformer reliability, and transformer vendors can provide EMT models.) Utilities don’t currently have the rest of the bulk system represented for EMT studies at large scale. An International Electrotechnical Commission (IEC) standard Common Information Model (CIM) provides a way of supporting these detailed models from physical asset data, e.g., conductors, towers, transformer data sheets, control block diagrams, while avoiding software vendor lock-in. CIM-for-EMT, with proposed schema extensions and open-source converters, provides a way to exchange EMT data between organizations and tools. This project leverages Office of Electricity (OE) funding of CIM-for-EMT code base through the GridAPPS-DTM project, and of GridPACKTM (parallelized transmission solver), for interoperability testing in CIM-for-EMT. The project also leverages partner PGSTech investments in EMTP® interoperability with CIM.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

ARPA-E Grid Optimization (GO) Competition Challenge 1

The ARPA-E Grid Optimization (GO) Competition Challenge 1, from 2018 to 2019, focused on the basic Security Constrained AC Optimal Power Flow problem (SCOPF) for a single time period. The Challenge utilized sets of unique datasets generated by the ARPA-E GRID DATA program. Each dataset consisted of a collection of power system network models of different sizes with associated operating scenarios (snapshots in time defining instantaneous power demand, renewable generation, generator and line availability, etc.). The datasets were of two types: Real-Time, which included starting-point information, and Online, which did not. Week-Ahead data is also provided for some cases but was not used in the Competition. Although most datasets were synthetic and generated by GRIDDATA, a few came from industry and were only used in the Final Event. All synthetic Input Data and Team Results for the GO Competition Challenge 1 for the Sandbox, Trial Events 1 to 3, and the Final Event along with problem, format, scoring and rules descriptions are available here. Data for industry scenarios will not be made public. Challenge 1, a minimization problem, required two computational steps. Solver 1 or Code 1 solved the base SCOPF problem under a strict wall clock time limit, as would be the case in industry, and reported the base case operating point as output, which was used to compute the Objective Function value that was used as the scenario score. The feasibility of the solution was provided by the Solver 2 or Code 2, which solves the power flow problem for all contingencies based on the results from Solver 1. This is not normally done in industry, so the time limits were relaxed. In fact, there were no time limits for Trial Event 1. This proved to be a mistake, with some codes running for more than 90 hours, and a time limit of 2 seconds per contingency was imposed for all other events. Entrants were free to use their own Solver 2 or use an open-source version provided by the Competition. Containers, such as Docker, were considered to improve the portability of codes, but none that could reliably support a multi-node parallel computing environment, e.g., MPI, could be found. For more information on the competition and challenge see the "GO Competition Challenge 1 Information" and "GO Competition Challenge 1 Additional Information" resources below.

ACOPF↗
DOE OSTI2019

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

In this paper, we develop software for decomposing sparse tensors that is portable to and performant on a variety of multicore, manycore, and GPU computing architectures. The result is a single code whose performance matches optimized architecture-specific implementations. The key to a portable approach is to determine multiple levels of parallelism that can be mapped in different ways to different architectures, and we explain how to do this for the matricized tensor times Khatri--Rao product (MTTKRP), which is the key kernel in canonical polyadic tensor decomposition. Our implementation leverages the Kokkos framework, which enables a single code to achieve high performance across multiple architectures that differ in how they approach fine-grained parallelism. We also introduce a new construct for portable thread-local arrays, which we call compile-time polymorphic arrays. Not only are the specifics of our approaches and implementation interesting for tuning tensor computations, but they also provide a roadmap for developing other portable high-performance codes. As a last step in optimizing performance, we modify the MTTKRP algorithm itself to do a permuted traversal of tensor nonzeros to reduce atomic-write contention. Lastly, we test the performance of our implementation on 16- and 68-core Intel CPUs and the K80 and P100 NVIDIA GPUs, showing that we are competitive with state-of-the-art architecture-specific codes while having the advantage of being able to run on a variety of architectures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

MFIX DEM Enhancement for Industry-Relevant Flows (Final Report)

The overall goal of this two-phase project is to implement performance improvements of the Multiphase Flow with Interphase Exchanges (MFIX) Discrete Element Model (DEM) code that enable a transformative shift for industrial use. Prior to this effort, the largest simulations performed using MFIX are O(10 7 ) particles. This falls short of the O(10 9 ) particle simulations that must be completed on a timescale of days or weeks (vs. months or years) to enable simulations with physically-relevant domain sizes to be incorporated into industrial design cycles within five years. This was accomplished by tailoring best-in-class practices to bear on the unique challenges posed by the MFIX-DEM algorithm and code base. Scientific simulations (e.g., in cosmology, turbulent combustion) routinely use massively parallel computing to update far more particles in short wall clock times. Results from Phase 1 (1.5 years in duration) indicated significant gains in speed were possible for a wide range of benchmark cases. Moreover, a survey sent to >35 companies indicates that the timing is ideal for such an enhanced tool, with >80% of the respondents indicating that DEM is already value-added or will be within the next 5 years, and >70% of the respondents indicating that improved speed is the top computational priority. In Phase 2 (3.5 years in duration), the two major barriers that hinder industry from effectively using multiphase Computational Fluid Dynamics (CFD) to cut costs and improve performance, namely computational overhead and confidence in predictions, continued to be addressed. Regarding the former, the results from Phase 1 to guide the effort, with enhancements focused on an improved time-stepping algorithm and particle sorting. Four target problems of 1 billion particles each and increasing complexity were identified: homogeneous cooling, tumbler with continuous particle size distribution, discharge from a rectangular hopper and a cylindrical riser. Each of these were successfully simulated for relevant time scales (on order of seconds) using less than 24 hours of wall clock time. These represent the first 1-billion particle DEM simulations performed with MFIX, namely using the MFIX-Exa code. This code is currently under development at NETL in collaboration with Lawrence Berkeley National Laboratory. Regarding the second barrier on predictive uncertainty, experiments from Phase 1 (interacting nozzles - hydrodynamics only) and Phase 2 (very small-scale segregation experiments) were used to demonstrate the ability of two simplified approaches to uncertainty quantification (UQ). By limiting the number of particles, UQ based on the simplified treatment was compared to standard UQ, which was shown to have much higher computational demands. Experiments were also performed on a pilot-scale stripper unit to provide validation data for future CFD-DEM simulations and UQ.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2020

Program Verification for Extreme-Scale Applications (Final Scientific/Technical Report)

The project seeks to develop tools and techniques to help software developers verify the correctness and accuracy of their code. The target domain is scientific software of the kind widely used and developed in the Department of Energy research community, with a particular focus on "extreme scale" programs - those that are expected to involve possibly millions of parallel threads of execution. The report covers the University of Delaware contribution to the collaborative project. The project had a number of successful outcomes, especially regarding the development and extension of the CIVL software verification framework. CIVL is a verification tool for C or Fortran programs that use MPI, OpenMP, CUDA, and/or Pthreads for parallelization. CIVL went through vast improvements and extensions, and was successfully applied to a number of challenging codes. It found a subtle bug in the Devito PDE framework. It was able to verify the functional correctness of a conjugate gradient solver using a novel probabilistic technique with vanishingly small chance of error. CIVL was used very successfully in verification competitions, and the CIVL solutions to the competition challenges were presented and published. CIVL was also used successfully by other researchers on a computational chemistry kernel.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Analysis of the impact of parallel magnetic fluctuations on linear gyrokinetic stability in NSTX-U and verification of gyro-fluid models

In this work, we use the CGYRO gyrokinetic code to analyze two L- and one H-mode discharges from the National Spherical Torus Experiment (NSTX) and NSTX-Upgrade (NSTX-U) selected due to their different mix of ion-scale driftwaves, ion temperature gradient (ITG) mode and trapped electron mode (TEM), and electromagnetic instabilities, kinetic ballooning mode (KBM), and micro-tearing mode (MTM) in the plasma core. It is found that the effect of parallel magnetic fluctuations is strongly destabilizing to the unstable KBMs compared to calculations with only perpendicular magnetic fluctuations. Two discharges have a mix of ITG/TEM and MTMs that are predicted to be dominant instability across the plasma radius. The parallel magnetic fluctuations are found to have little effect on the MTM stability but are destabilizing to ITG/TEM modes. To test the validity of the gyro-fluid linear stability codes TGLF and GFS at low aspect ratio, a database of linear growth rates has been created using the CGYRO gyrokinetic code. The database is comprised of various parameter scans around a standardized set of NSTX-U core parameters. It contains a group of electrostatic cases and an electromagnetic group that includes the effects of perpendicular and parallel magnetic fluctuations. Comparing the results from the GFS and TGLF models, we find that GFS exhibits the best agreement with the database of CGYRO linear growth rates. Comparing the model results for the electromagnetic scans shows that GFS captures the effects of parallel magnetic fluctuations accurately, while the TGLF model does not, as it lacks sufficient perpendicular energy resolution.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

Automatic Differentiation of C++ Codes on Emerging Manycore Architectures with Sacado

Automatic differentiation (AD) is a well-known technique for evaluating analytic derivatives of calculations implemented on a computer, with numerous software tools available for incorporating AD technology into complex applications. However, a growing challenge for AD is the efficient differentiation of parallel computations implemented on emerging manycore computing architectures such as multicore CPUs, GPUs, and accelerators as these devices become more pervasive. In this work, we explore forward mode, operator overloading-based differentiation of C++ codes on these architectures using the widely available Sacado AD software package. In particular, we leverage Kokkos, a C++ tool providing APIs for implementing parallel computations that is portable to a wide variety of emerging architectures. Here we describe the challenges that arise when differentiating code for these architectures using Kokkos, and two approaches for overcoming them that ensure optimal memory access patterns as well as expose additional dimensions of fine-grained parallelism in the derivative calculation. We describe the results of several computational experiments that demonstrate the performance of the approach on a few contemporary CPU and GPU architectures. We then conclude with applications of these techniques to the simulation of discretized systems of partial differential equations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

XScan: An Integrated Tool for Understanding Open Source Community-Based Scientific Code

Many scientific communities have adopted community-based models that integrate multiple components to simulate whole system dynamics. The community software projects’ complexity, stems from the integration of multiple individual software components that were developed under different application requirements and various machine architectures, has become a challenge for effective software system understanding and continuous software development. The paper presents an integrated software toolkit called X-ray Software Scanner (in abbreviation, XScan) for a better understanding of large-scale community-based scientific codes. Our software tool provides support to quickly summarize the overall information of scientific codes, including the number of lines of code, programming languages, external library dependencies, as well as architecture-dependent parallel software features. The XScan toolkit also realizes a static software analysis component to collect detailed structural information and provides an interactive visualization and analysis of the functions. We use a large-scale community-based Earth System Model to demonstrate the workflow, functions and visualization of the toolkit. We also discuss the application of advanced graph analytics techniques to assist software modular design and component refactoring.

Zheng, Weijian↗
DOE OSTI2019

Scalable Reactive Molecular Dynamics Simulations for Computational Synthesis

Reactive molecular dynamics (MD) simulation is a powerful research tool for describing chemical reactions. We eliminate the speed-limiting charge iteration in MD with a novel extended-Lagrangian scheme. The extended-Lagrangian reactive MD (XRMD) code drastically improves energy conservation while substantially reducing time-to-solution. Furthermore, we introduce a new polarizable charge equilibration (PQEq) model to accurately predict atomic charges and polarization. The XRMD code based on hybrid message passing+multithreading achieves a weak-scaling parallel efficiency of 0.977 on 786 432 IBM Blue Gene/Q cores for a 67.6 billion-atom system. The performance is portable to the second-generation Intel Xeon Phi, Knights Landing. Blue Gene/Q simulations for the computational synthesis of materials via novel exfoliation mechanisms for synthesizing atomically thin transition metal dichalcogenide layers will dominate nanomaterials science in this century.

Simulation, Modeling, Performance, and Visualizati↗
DOE OSTI2022

CEAZ: Accelerating Parallel I/O Via Hardware-Algorithm Co-Designed Adaptive Lossy Compression

As supercomputers continue to grow to exa-scale, the amount of data that needs to be saved or transmitted is exploding. To this end, many previous works have studied using error-bounded lossy compressors to reduce the data size and improve the I/O performance. However, little work has been done for effectively offloading lossy compression onto FPGA-based SmartNICs to reduce the compression overhead. In this paper, we propose a hardware-algorithm co-design of efficient and adaptive lossy compressor for scientific data on FPGAs (called CEAZ) to accelerate parallel I/O. Our contribution is fourfold: (1) We propose an efficient Huffman coding approach that can adaptively update Huffman codewords online based on codewords generated offline (from a variety of representative scientific datasets). (2) We derive a theoretical analysis to support a precise control of compression ratio under an error-bounded compression mode, enabling accurate offline Huffman codewords generation. This also help us create a fixed-ratio compression mode for consistent throughput. (3) We develop an efficient compression pipeline by adopting cuSZ’s dual-quantization algorithm to our hardware use case. (4) We evaluate CEAC on five real-world datasets with both a single FPGA board and 256 nodes from Bridges2 supercomputer. Experiments show that CEAZ outperforms the second-best FPGA-based lossy compressor by 2× of throughput and 9.6× of compression ratio. It also improves MPI_File_write and MPI_Gather throughputs by up to 32.7× and 31.4×, respectively.

Zhang, Chengming↗
DOE OSTI2025

LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the LLaMA 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation, and unit tests. Here, our results indicate that while LLaMA 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

ECP Milestone Memo WBS 2.3.4.13 ECP/VTK-m FY18Q4 [MS-18/09-10] Dynamic Types / Rendering Topologies STDA05-16

The STDA05-16 milestone comprises the following 3 distinct deliverables. OpenMP VTK-m currently supports three types of devices: serial CPU, TBB, and CUDA. To run algorithms on multicore CPU-type devices (such as Xeon and Xeon Phi), TBB is required. However, there are known issues with integrating a software product using TBB with another one using OpenMP. Therefore, we will add an OpenMP device to the VTK-m software. When engaged, this device will run parallel algorithms using OpenMP directives. This will mesh more nicely with other code also using OpenMP. Rendering Topological Entities VTK-m currently supports surface rendering by tessellation of data structures,and rendering the resulting triangles. We will extend current functionality to include face, edge, and point rendering. Better Dynamic Types Impl For the best efficiency across all platforms, VTK-m algorithms use static typing with C++ templates. However, many libraries like VTK, ParaView, and Visit use dynamic types with virtual functions because data types often cannot be determined at compile time. We have an interface in VTK-m to merge these two typing mechanisms by generating all possible combinations of static types when faced with a dynamic type. Although this mechanism works, it generates very large executables and takes a very long time to compile. As we move forward, it is clear that these problems will get worse and become infeasible at exascale. We will rectify the problem by introducing some level of virtual methods, which require only a single code path, within VTK-m algorithms. This first milestone produces a design document to propose an approach to the new system.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

YAHFC: A Code Framework to Model Nuclear Reactions and Estimate Correlated Uncertainties

Reaction modeling is a key ingredient in designing experiments and interpreting their results, and is an essential component in the process of evaluating nuclear data and assembling nuclear data libraries used in nuclear technology applications. Typically, experimental data are available only for a handful of reaction channels and theory models are used to fill in the gaps. In addition, theory is often called upon as the arbitrator between discrepant data. Most importantly, theory and modeling are required for an accurate determination of uncertainties in the evaluated data and the correlations between the multiple channels. A fast, accurate, and flexible modeling capability has been developed at LLNL with the code system YAHFC (Yet Another Hauser-Feshbach Code). YAHFC is a Monte Carlo, Hauser-Feshbach code framework, making full use of dynamic memory allocation, derived types, and parallel computing. YAHFC can generate events to simulate experiments and is guiding experiments designed to measure inelastic neutron scattering from actinide targets. YAHFC is also being used to analyze decays from surrogate experiments, thereby enabling the inference of reaction cross sections inaccessible by direct measurement. Finally, by modeling nuclear reactions with constraints from experimental data, YAHFC can deliver complete nuclear data libraries, with evaluated uncertainties, using the modernized Generalized Nuclear Data Structure (GNDS).

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Human Host Cellular Response to HCoV-229E Infection Proteomics (ACS-JM-DP2)

The purpose of this experiment was to evaluate the human host cellular response to wild-type Human coronavirus strain 229E (HCoV-229E) infection. Sample data was obtained for mock and infected immortalized human lung epithelial cells (A549) (MOI 5) nuclear extracts, immortalized human lung fibroblasts cells (MRC5) (MOI5) nuclear extracts, and primary human airway epithelial (HAE) (MOI 3) cells from lung tissue and processed for proteome analysis. Processed datasets are openly accessible from the download button and contain secondary processed proteomic results files and supporting metadata materials. Experimental proteomics samples were prepared using Limited Proteolysis (LiP) methods for Label-free quantification (LFQ) and global proteomic evaluation. Sample data was acquired using a Q-Exactive HF-X mass spectrometer and was processed and compiled using MaxQuant software (v.1.6.17.0). Processed proteomic data downloads include a sample naming key, processed MaxQuant results/parameters, and protein annotated relative abundance files. See corresponding primary data accessions below and Viral Experiment LiP Analysis source code supporting data transparency and reuse. Experimental transcriptomics samples were collected in parallel and processed for RNA sequencing (RNA-Seq) as summarized under ACS-DP1 (https://data.pnnl.gov/group/nodes/dataset/34069).

59 BASIC BIOLOGICAL SCIENCES↗