Search NASA⌕ Search

Engineering topics

Petersen, Mark Roger

Publications and source records attributed to Petersen, Mark Roger.

At least 19 records

Performance Results on CPU/GPU Exascale Architectures for OMEGA: The Ocean Model for E3SM Global Applications

The US Department of Energy (DOE) conducts climate simulations on some of the world’s largest supercomputers. These exascale machines use heterogeneous architectures with both CPUs and GPUs, and scientific codes must adapt to make full use of this computing power. Los Alamos National Lab is developing Omega: The Ocean Model for E3SM Global Applications, which is specifically designed for modern exascale computers. It uses external libraries that have been optimized for a variety of architectures to run on different supercomputers. Omega is an unstructured-mesh ocean model based on TRiSK numerical methods. It will be the new ocean component of the DOE’s Energy Exascale Earth System Model (E3SM). The algorithms in Omega follow those of the current ocean component, MPAS-Ocean, but it will be written in C++ rather than Fortran to take advantage of the Kokkos performance portability library. Omega spatial operators are written as Kokkos kernels to run efficiently on both CPUs and GPUs. Work on Omega began in 2023 with a new C++ framework for unstructured mesh partitioning, halo exchanges, parallel IO, and Kokkos interfaces. The current version, Omega-0, is being developed to solve the shallow water equations and at present includes all of the tendency terms but not time stepping. Here we share the results of Omega-0 verification and performance testing. Verification includes unit tests implemented with CTest as well as convergence tests in Polaris, an in-house python package with a large suite of test problems. Performance tests compare simulations conducted on CPUs versus GPUs and across different architectures: tests are run on Frontier, which has AMD “Optimized 3rd Gen EPYC” CPUs and AMD MI250X GPUs, as well as Perlmutter, which is composed of AMD EPYC 7763 CPUs and NVIDIA A100 GPUs.

58 GEOSCIENCES↗

LANL Institutional Computing Close-out Report for Project t22_ocean_time_step

This year, my team made a very productive use of LANL IC time. Four papers were published that used IC resources, and two more are under review. These publications fall into three categories: 1.) Improving tide modeling in the global ocean by adding self attraction and loading (Barton et. al. 2022), ice shelf cavities (Pal et al. 2023), and local time stepping (Lilly et al. 2023). 2.) A new sea ice numerical formulation (Capodaglio 2023). 3.) Performance comparisons, methods, and test cases for ocean model development, This includes a verification suite for ocean models (Bishnu, submitted) and a comparison between Julia and Fortran (Strauss, submitted). A new time-stepping method was introduced in Calandrini et al. (2022). These publications, and the use of LANL IC resources, go hand-in-hand with our mentoring efforts to train young scientists. Two of the lead authors are graduate students who are conducting their PhD research: Kristin Barton at the University of Michigan and Jeremy Lilly at Oregon State. These are both their first publications, and they both have DOE funding and DOE mentors. In addition, Bishnu is a post-doctoral researcher at LANL; Strauss conducted his research as a senior in high school; and Capodaglio, Calandrini and Pal are all early-career scientists who were converted to staff in 2021 or 2022. Here we highlight two publications on improvements in tidal modeling: Barton et al. 2022 and Lilly et al. 2023.

58 GEOSCIENCES↗

An improved B-grid formulation for the dynamics of MPAS-Seaice [Slides]

Sea ice plays an important role in the global climate system. The internal ice stress regulates the ice deformation and acts as a resistance to the motion. An accurate approximation of the internal ice stress is therefore crucial to determine the sea ice evolution.

54 ENVIRONMENTAL SCIENCES↗

An evaluation of the E3SMv1 Arctic ocean and sea-ice regionally refined model

The Energy Exascale Earth System Model (E3SM) is a state-of-the-science Earth system model (ESM) with the ability to focus horizontal resolution of its multiple components in specific areas. Regionally refined global ESMs are motivated by the need to explicitly resolve, rather than parameterize, relevant physics within the regions of refined resolution, while offering significant computational cost savings relative to the respective cost of configurations with high-resolution (HR) everywhere on the globe. In this paper, we document results from the first Arctic regionally refined E3SM configuration for the ocean and sea-ice components (E3SM-Arctic-OSI), while employing data-based atmosphere, land, and hydrology components. Our aim is an improved representation of the Arctic coupled ocean and sea-ice state, its variability and trends, and the exchanges of mass and property fluxes between the Arctic and the sub-Arctic. We find that E3SM-Arctic-OSI increases the realism of simulated Arctic ocean and sea-ice conditions compared to a similar low-resolution E3SM simulation without the Arctic regional refinement in ocean and sea-ice components (E3SM-LR-OSI). In particular, exchanges through the main Arctic gateways are greatly improved with respect to E3SM-LR-OSI. Other aspects, such as the Arctic freshwater content variability and sea-ice trends, are also satisfactorily simulated. Yet, other features, such as the upper-ocean stratification and the sea-ice thickness distribution, need further improvements, involving either more advanced parameterizations, model tuning, or additional grid refinements. Overall, E3SM-Arctic-OSI offers an improved representation of the Arctic system relative to E3SM-LR-OSI, at a fraction (15 %) of the computational cost of comparable global high-resolution configurations, while permitting exchanges with the lower-latitude oceans that cannot be directly accounted for in Arctic regional models.

54 ENVIRONMENTAL SCIENCES↗

Improved Time-Stepping Methods in Global to Regional Ocean Modeling (Annual Status Report)

Time stepping algorithms are an important part of ocean models, and strongly influence both the accuracy of solution and performance. There have been a number of projects investigating various improvements for ocean time-stepping schemes in the Model for Prediction Across Scales-Ocean (MPAS-Ocean), a component of the DOE Energy Exascale Earth System Model. Ocean dynamics include fast surface gravity waves, which are two-dimensional, and slower internal waves, which are three-dimensional, so ocean models use a split time-stepping scheme that separates these barotropic and baroclinic modes for efficiency. MPAS-Ocean runs on variable-resolution horizontal meshes, and must scale to tens of thousands of cores and millions of horizontal gridcells. Ocean models require time stepping algorithms that are customized to these needs, and which are tuned for performance on various resolutions and architectures.

58 GEOSCIENCES↗

A Comparison of Machine Learning Methods to Forecast Tropospheric Ozone Levels in Delhi

Ground-level ozone is a pollutant that is harmful to urban populations, particularly in developing countries where it is present in significant quantities. It greatly increases the risk of heart and lung diseases and harms agricultural crops. This study hypothesized that, as a secondary pollutant, ground-level ozone is amenable to 24 h forecasting based on measurements of weather conditions and primary pollutants such as nitrogen oxides and volatile organic compounds. We developed software to analyze hourly records of 12 air pollutants and 5 weather variables over the course of one year in Delhi, India. To determine the best predictive model, eight machine learning algorithms were tuned, trained, tested, and compared using cross-validation with hourly data for a full year. The algorithms, ranked by R2 values, were XGBoost (0.61), Random Forest (0.61), K-Nearest Neighbor Regression (0.55), Support Vector Regression (0.48), Decision Trees (0.43), AdaBoost (0.39), and linear regression (0.39). When trained by separate seasons across five years, the predictive capabilities of all models increased, with a maximum R 2 of 0.75 during winter. Bidirectional Long Short-Term Memory was the least accurate model for annual training, but had some of the best predictions for seasonal training. Out of five air quality index categories, the XGBoost model was able to predict the correct category 24 h in advance 90% of the time when trained with full-year data. Separated by season, winter is considerably more predictable (97.3%), followed by post-monsoon (92.8%), monsoon (90.3%), and summer (88.9%). These results show the importance of training machine learning methods with season-specific data sets and comparing a large number of methods for specific applications.

54 ENVIRONMENTAL SCIENCES↗

An unstructured C-grid type variational formulation for the sea ice dynamics

Historically, B-grid formulations of sea ice dynamics have been dominant because they have matched the grid type used by ocean models. The reason for the grid match is simple – it facilitates penetration of the curl of ice-ocean stress into the deep ocean with minimal numerical diffusivity because sea ice and ocean velocity are co-located. In recent years, as ocean models have increasingly progressed to C-grids, sea ice models have followed suit on quadrilateral meshes, but the implementation of an unstructured C-grid sea ice models is new. We present an unstructured C-grid discretization of the Elastic Viscous Plastic (EVP) rheology, where the velocity unknowns are discretized at the edges of the mesh cells with n-sides, where typically n is greater than or equal to four, rather than at the vertices, as in the B-grid. Our framework of choice is the Model for Prediction Across Scales (MPAS) within E3SM, the climate model of the U.S. Department of Energy, although our approach is general and could be applied to other models as well. While MPAS-Seaice is currently defined on a B-grid, MPAS-Ocean runs on a C-grid, hence interpolation operators are heavily used when coupled simulations are performed. In this work, we describe a mathematical formulation to transition the dynamics of MPAS-Seaice to a C-grid, in order to ultimately facilitate the coupling with MPAS-Ocean and reduce numerical errors associated with this communication. Numerical results are reported to highlight the features of the method.

58 GEOSCIENCES↗

Detecting Arsenic Contamination Using Satellite Imagery and Machine Learning

Arsenic, a potent carcinogen and neurotoxin, affects over 200 million people globally. Current detection methods are laborious, expensive, and unscalable, being difficult to implement in developing regions and during crises such as COVID-19. This study attempts to determine if a relationship exists between soil’s hyperspectral data and arsenic concentration using NASA’s Hyperion satellite. It is the first arsenic study to use satellite-based hyperspectral data and apply a classification approach. Four regression machine learning models are tested to determine this correlation in soil with bare land cover. Raw data are converted to reflectance, problematic atmospheric influences are removed, characteristic wavelengths are selected, and four noise reduction algorithms are tested. The combination of data augmentation, Genetic Algorithm, Second Derivative Transformation, and Random Forest regression (R 2 =0.840 and normalized root mean squared error (re-scaled to [0,1]) = 0.122) shows strong correlation, performing better than past models despite using noisier satellite data (versus lab-processed samples). Three binary classification machine learning models are then applied to identify high-risk shrub-covered regions in ten U.S. states, achieving strong accuracy (=0.693) and F1-score (=0.728). Overall, these results suggest that such a methodology is practical and can provide a sustainable alternative to arsenic contamination detection.

63 RADIATION, THERMAL, AND OTHER ENVIRON. POLLUTAN↗

Creating an MPAS Ocean Shallow Water Core in Julia [Slides]

An MPAS model was created in Julia to test its potential for scientific HPC. Tradeoff between execution speed and development speed: Development languages (e.g. Python) are easy, but slow and production languages (e.g. C) are hard, but fast. Julia aims to be the best of both.

58 GEOSCIENCES↗

Comparing numerical accuracy and stability for different horizontal discretizations in MPAS-Ocean

This manuscript investigates the effectiveness of two possible horizontal discretizations for the global ocean model MPAS-Ocean, both applied to Spherical Centroidal Voronoi Tessellations (SCVTs). The first discretization is TRiSK, a C-grid, finite-volume method, that possesses many desirable mimetic properties, but has a low order accuracy. The second discretization was introduced for the first time by Peixoto (2016), and consists of modifications to the TRiSK scheme designed to achieve at least first-order accuracy in the L ∞ norm, with the loss of some of the mimetic properties. Tests on shallow-water and primitive-equation models show that the scheme due to Peixoto is indeed more accurate, but presents stability issues with respect to TRiSK. Here, TRiSK is indeed found to be often more stable in time and more robust with respect to errors in the geometric properties of the grid.

97 MATHEMATICS AND COMPUTING↗

LANL Institutional Computing Highlight: Local Time-Stepping in Global to Regional Ocean Modeling (Annual Report) [Slides]

Multi-resolution global ocean simulations are possible with the Model for Prediction Across Scales-Ocean (MPAS-Ocean), although the model is currently constrained to a uniform time step. The time step is determined by the size of the smallest cell in the grid, according to the CFL condition. To overcome this issue, a local time stepping (LTS) scheme has been designed for MPAS-Ocean. We have made progress on implementing the LTS scheme, as well as quantitative results and performance tests.

58 GEOSCIENCES↗