ALPINE: In Situ Visualization, Analysis, and Infrastructure for ECP Science.
Abstract not provided.
Bennet, Janine↗
SEARCH · Search NASA
Results for “in situ analysis and visualization”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Catalyst Revised: Rethinking the ParaView in Situ Analysis and Visualization API
As in situ analysis goes mainstream, ease of development, deployment, and maintenance becomes essential, perhaps more so than raw capabilities. In this paper, we present the design and implementation of Catalyst, an API for in situ analysis using ParaView, which we refactored with these objectives in mind. Furthermore, our implementation combines design ideas from in situ frameworks and HPC tools like Ascent and MPICH.
97 MATHEMATICS AND COMPUTING↗
DevOps for Catalyst In Situ Visualization and Analysis at Sandia National Laboratories
Explore the source record for details and available documents.
Mauldin, Jeffrey A.↗
Towards elastic in situ analysis for high-performance computing simulations
In situ analysis and visualization have grown increasingly popular for enabling direct access to data from high-performance computing (HPC) simulations. As a simulation progresses and interesting physical phenomena emerge, however, the data produced may become increasingly complex, and users may need to dynamically change the type and scale of in situ analysis tasks being carried out and consequently adapt the amount of resources allocated to such tasks. To date, none of the production in situ analysis frameworks offer such an elasticity feature, and for good reason: the assumption that the number of processes could vary during run time would force developers to rethink software and algorithms at every level of the in situ analysis stack. In this paper we present Colza, a data staging service with elastic in situ visualization capabilities. We demonstrate the use of Colza with the Deep Water Impact and the AMR-Wind simulations, coupling them with the ParaView Catalyst and Ascent in situ libraries, and show that Colza enables dynamic rescaling of these widely-used frameworks with no interruption to the simulation or staging service. Here, we highlight the challenges of enabling such elasticity, which requires overcoming these frameworks' reliance on MPI, using distinct engineering approaches, namely dependency injection and dependency overload. To the best of our knowledge, this work is the first to enable elastic in situ visualization capabilities for HPC applications on top of existing production analysis tools.
97 MATHEMATICS AND COMPUTING↗
The ECP ALPINE project: In situ and post hoc visualization infrastructure and analysis capabilities for exascale
A significant challenge on an exascale computer is the speed at which we compute results exceeds by many orders of magnitude the speed at which we save these results. Therefore the Exascale Computing Project (ECP) ALPINE project focuses on providing exascale-ready visualization solutions including in situ processing. In situ visualization and analysis runs as the simulation is run, on simulations results are they are generated avoiding the need to save entire simulations to storage for later analysis. The ALPINE project made post hoc visualization tools, ParaView and VisIt, exascale ready and developed in situ algorithms and infrastructures. The suite of ALPINE algorithms developed under ECP includes novel approaches to enable automated data analysis and visualization to focus on the most important aspects of the simulation. Many of the algorithms also provide data reduction benefits to meet the I/O challenges at exascale. ALPINE developed a new lightweight in situ infrastructure, Ascent.
97 MATHEMATICS AND COMPUTING↗
Collaborative: in situ visual analytics technologies for extreme scale combustion simulations
This project aims to drastically enhance the usability of in situ analysis and visualization for extreme-scale scientific simulations. Current exascale computing capabilities promise to offer greater predictive ability of simulations and to further push the frontiers of science and technology. However, to validate the simulation output at extreme scale, examine the modeled phenomena, and discover previously unknowns from the output data, the output must be reduced or transformed in situ as it is being generated during the simulation such that the amount of data to examine and store is kept to a minimum. Such in situ approaches allow us to process and analyze the data and any embedded geometry to an extent that would be prohibitively expensive, if not impossible, to perform as a post hoc task. While in situ processing has been demonstrated to be a feasible and promising approach, its full potential has not yet been leveraged. In this project, we have developed comprehensive enhancements to in situ technology based on probability distributions in data. Our research focuses on jointly developing new ways of interacting with massive statistical samples while creatively utilizing new state-of-the-art computational resources to push the boundaries of in situ exploration. Moreover, we have developed new time-dependent techniques to enable previously unattainable capabilities in areas such as intelligent simulation steering and precise feature identification. We have experimentally studied our design and implementation at NERSC and OLCF, and are able to leverage existing in situ infrastructures whenever possible. While the exemplar in this project is combustion, many other fields for which turbulent transport is important, e.g., fusion, climate, astrophysics among others, encounter similar issues as simulations scale up to the exascale. This project shows its potential to generate high impact on DOE missions since the resulting technology promises to improve scientists’ ability to rapidly and correctly interpret and tune extreme-scale simulations, leading to new scientific understanding and advancements.
97 MATHEMATICS AND COMPUTING↗
Streaming Data in HPC Workflows Using ADIOS
The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.
Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗
Demonstrating the viability of Lagrangian in situ reduction on supercomputers
Performing exploratory analysis and visualization of large-scale time-varying computational science applications is challenging due to inaccuracies that arise from under-resolved data. In recent years, Lagrangian representations of the vector field computed using in situ processing are being increasingly researched and have emerged as a potential solution to enable exploration. However, prior works have offered limited estimates of the encumbrance on the simulation code as they consider “theoretical” in situ environments. Further, the effectiveness of this approach varies based on the nature of the vector field, benefitting from an in-depth investigation for each application area. With this study, an extended version of Sane et al. (2021), we contribute an evaluation of Lagrangian analysis viability and efficacy for simulation codes executing at scale on a supercomputer. We investigated previously unexplored cosmology and seismology applications as well as conducted a performance benchmarking study by using a hydrodynamics mini-application targeting exascale computing. Here, to inform encumbrance, we integrated in situ infrastructure with simulation codes, and evaluated Lagrangian in situ reduction in representative homogeneous and heterogeneous HPC environments. To inform post hoc accuracy, we conducted a statistical analysis across a range of spatiotemporal configurations as well as a qualitative evaluation. Additionally, our study contributes cost estimates for distributed-memory post hoc reconstruction. In all, we demonstrate viability for each application — data reduction to less than 1% of the total data via Lagrangian representations, while maintaining accurate reconstruction and requiring under 10% of total execution time in over 90% of our experiments.
97 MATHEMATICS AND COMPUTING↗
Capturing Carbonation: Understanding Kinetic Complexities through a New Era of Electron Microscopy
Cryogenic plasma focused ion beam (PFIB) electron microscopy analysis is applied to visualizing ex situ (surface industrial) and in situ (subsurface geologic) carbonation products, to advance understanding of carbonation kinetics. Ex situ carbonation is investigated using NIST fly ash standard #2689 exposed to aqueous sodium bicarbonate solutions for brief periods of time. In situ carbonation pathways are investigated using volcanic flood basalt samples from Schaef et al. (2010) exposed to aqueous CO 2 solutions by them. The fly ash reaction products at room temperature show small amounts of incipient carbonation, with calcite apparently forming via surface nucleation. Reaction products at 75° C show beginning stages of an iron carbonate phase, e.g., siderite or ankerite, common phases in subsurface carbon sequestration environments. This may suggest an alternative to calcite in carbonation low calcium-bearing fly ashes. Flood basalt carbonation reactions show distinct zonation with high calcium and calcium-magnesium bearing zones alternating with high iron-bearing zones. The calcium-magnesium zones are notable with occurrence of localized pore space. Oscillatory zoning in carbonate minerals is distinctly associated with far-from-equilibrium conditions where local chemical environments fluctuate via a coupling of reaction with transport. The high porosity zones may reflect a precursor phase (e.g., aragonite) with higher molar volume that then “ripens” to the high-Mg calcite phase-plus-porosity. These observations reveal that carbonation can proceed with evolving local chemical environments, formation and disappearance of metastable phases, and evolving reactive surface areas. Together this work shows that future application of cryo-PFIB in carbonation studies would provide advanced understanding of kinetic mechanisms for optimizing industrial-scale and commercial-scale applications.
36 MATERIALS SCIENCE↗
Crystallographic Insight of Reduced Lattice Volume Expansion in Mesoporous Cu 2+ -Doped TiNb 2 O 7 Microspheres during Li + Insertion
TiNb 2 O 7 represents a promising anode material for lithium-ion batteries (LIBs), but its practical applications are currently hampered by the non-negligible volumetric expansion and contraction during the charge/discharge process and the sluggish ion/electron kinetics. Here a combination technique is reported by systematically optimizing the porous and spherical morphology, crystal structure, and surface decoration of mesoporous Cu 2+ -doped TiNb 2 O 7 microspheres to enhance the electrochemical Li + storage performance and stability simultaneously. The Cu 2+ dopants preferentially replace Ti 4+ in crystal lattices, which decreases the Li + diffusion barrier and increases the electronic conductivity, as confirmed by density functional theory (DFT) calculation and demonstrated by diverse electrochemical characterizations. The successful Cu 2+ doping significantly reduces the lattice expansion coefficient from 7.26% to 4.61% after Li + insertion along the b-axis of TiNb 2 O 7 , as visualized from in situ and ex situ XRD analysis. The optimal 5% Cu 2+ -doped TiNb 2 O 7 with surface coating of N-doped carbon exhibits significantly enhanced specific capacity and rate and cyclic performances in both half- and full-cell configurations, demonstrating an excellent electrochemical behavior for fast-charging LIB applications.
36 MATERIALS SCIENCE↗
AEflow (Autoencoder fluid flow compression network) [SWR-22-29]
As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.
King, Ryan↗
CALORIE: A Constraint Language and Optimizing Runtime for Exascale Power Management (Final Report)
This final technical report summarizes the key accomplishments on the CALORIE project, a DOE Early Career award received by PI Henry Hoffmann at the University of Chicago. CALORIE’s main goal was to create principled methodologies, tools, and practices to help scientists and high-performance computing (HPC) operators maximize the performance and insights obtained from scientific computing applications in the face of exascale power constraints. The project accomplished these goals by completing the following objectives: • Designing a language for describing application goals (including constraints and objectives) and system capabilities. The application goals will include things like which simulations or simulations plus in situ analysis will be run together, what the power constraints are, and what requirements there are for in situ analysis (for example, a desired frame rate for visualization). The system capabilities include all components that can be adjusted to tradeoff power and performance. • Designing a runtime system that takes specified goals and capabilities and dynamically determines what capabilities to use to meet the goals. This runtime adapts to changes in goals, application behavior, or available capabilities to automatically maintain the goals despite unexpected disturbances. • Developing a foundational understanding of how power constraints affect the problem of scheduling applications in large-scale systems. This report provides an overview of the accomplishments related to each of these key objectives.
97 MATHEMATICS AND COMPUTING↗
Enabling Command-and-Control in Advanced In Situ Workflows
Scientific discovery is progressing towards autonomous science with the combination of scientific instruments, high-performance computing, and artificial intelligence in complex workflows. This evolution introduces new requirements for managing scientific workflows, including feedback loops, near real-time constraints, and the ability to dynamically control workflow execution. In situ workflows that analyze and visualize data as it is generated are well-suited to satisfy stringent time constraints and their iterative nature offers greater opportunities for command-and-control. However, only a few of the many workflow management systems available have been specifically designed to manage in situ workflows and often lack support for automated feedback loops that allow analysis and visualization components to interact with the main scientific data producer. To address this need, we present in this paper how to add command-and-control capabilities to a workflow management system. We identify the functional design requirements of such a command-and-control system, detail its architecture, interface, and core mechanisms, and illustrate how advanced in situ workflows can leverage command-and-control in three use cases: graceful termination with checkpoint, dynamic and adaptive data reduction, and event-triggered analysis.
Mehta, Kshitij [ORNL] (ORCID:0000000297149981)↗
ALPINE Overview
Data-driven sampling enables probabilistic identification of interesting regions in the data automatically, prioritizing important regions. Applied in situ to Nyx, important halo regions are preserved.
97 MATHEMATICS AND COMPUTING↗
Sim-Situ: A Framework for the Faithful Simulation of in situ Processing
The amount of data generated by numerical simulations in various scientific domains led to a fundamental redesign of how the analysis and visualization of simulation outputs are performed. The throughput and capacity of storage subsystems have not evolved as fast as the computing power in extreme-scale supercomputers, making the classical post-hoc approach highly inefficient. In situ processing has then emerged as a solution in which simulation and data analysis/visualization are intertwined for better performance and greater interactivity.Determining the best allocation, i.e., how many resources to allocate to simulation and analysis respectively, mapping, i.e., where and at which frequency to run the analysis/visualization, and data transfer mode is a complex task whose performance assessment is crucial to the efficient execution of in situ processing. However, such a performance evaluation of different strategies usually relies either on directly running them on the targeted execution environments, which can rapidly become extremely time- and resource-consuming, or on resorting to simplified models of the components of an in situ application, which can lack of realism. In both cases, the validity of the performance evaluation is limited.In this paper, we present Sim-Situ, a simulation-based framework for the faithful performance evaluation of in situ processing strategies. We designed Sim-Situ to reflect the typical features of in situ processing systems. Thanks to its modular design, Sim-situ has the necessary flexibility to easily and faithfully evaluate the behavior and performance of various allocation, mapping, and data transfer strategies. We illustrate the simulation capabilities of Sim-Situ on a Molecular Dynamics use case. We study the impact of different strategies on performance and show how users can leverage Sim-Situ to determine interesting tradeoffs when adding analysis/visualization components to their application.
Honoré, Valentin↗
Scoreboard
Emerging HPC machines have given rise to enhanced compute power that far outstrips the machine's ability to save large scale results for post-processing. To combat this, in situ data analysis techniques are slowly being adopted. With in situ data management favoring workflows composed of multiple simulations and analyses connected in transit on heterogeneous machines, scientists and engineers need a tool that enables them to create data extracts, visualizations, and interactively monitor and steer their simulations. Scoreboard Phase II is a next generation analysis software that supports composite in transit workflows on heterogeneous architectures and restores interactivity to in situ data analysis through simulation monitoring and computational steering. Scoreboard provides a simulation dashboard with graphs of metrics over time, controls for setting custom simulation steering parameters, controls for managing the set of data extracts being produced in the simulation, as well as the ability to explore data extracts, all from a web browser. Realizing the vision outlined in this project required research into making a system that integrates end to end from simulations all the way to the user. In situ tools generally suffer from complexity and excessive software dependencies. Scoreboard, by contrast, is easy to build and integrate into simulation codes and it provides first class FORTRAN support. The Scoreboard library is capable of in situ and in transit data analysis that can produce data extracts commonly needed for Computational Fluid Dynamics (CFD) analysis. Simulations can transparently stage data in transit to a Scoreboard Endpoint program, which can accept their data and produce the requested data extracts. This lets simulations return to their work while the Endpoint works on the analysis. Efficiently staging the data at scale was a topic of this research. Scoreboard provides the means to let the user manage data extracts and monitor/steer many simulations from a web browser. This area of the research focused on discovery of in transit network components to expose and control their steering parameters within an interactive browser-based user interface that includes: system topology, gathered metrics, notifications, dynamically-generated steering controls, and exploration of visualization data products.
Whitlock, BradJoseph [Intelligent Light] (00000001↗
Performance analysis and data reduction for exascale scientific workflows
Chimbuko is the first in situ, scalable, workflow-level performance analysis tool for trace-level analysis and visualization of application performance. This tool was developed by the Co-design Center for Online Data Analysis and Reduction and funded by the U.S. Department of Energy’s Exascale Computing Project. We provide a detailed description of Chimbuko’s architecture and illustrate our online and offline visualization with multiple use cases. We also present results for the deployment and scalability of the tool as applied to a high-energy physics workflow running at large scale on the Frontier supercomputer.
97 MATHEMATICS AND COMPUTING↗
Operando video microscopy of Li plating and re-intercalation on graphite anodes during fast charging
Despite the demand for fast-charging lithium (Li)-ion batteries, high-energy-density batteries with thick graphite anodes are limited by Li plating when charging at >4C rates. In this work, plan-view operando video microscopy is applied on >3 mA h cm –2 calendared graphite electrodes to study the dynamic evolution of local state-of-charge (SoC) and Li plating during fast charging. This technique allows for visualization of the spatial heterogeneity in SoC across the electrode, nucleation and growth of Li filaments, Li re-intercalation into graphite, “dead Li” formation, and SoC equilibration. The operando microscopy analysis is complemented by ex situ imaging of through-plane gradients in SoC to gain a three-dimensional visualization of spatial heterogeneity. We demonstrate that (1) Li plating preferentially nucleates on the graphite particles that lithiate fastest during fast charging; (2) the onset of Li plating correlates with the local minimum of the graphite electrode potential; (3) galvanic corrosion currents are responsible for Li re-intercalation, dead Li formation, and SoC re-equilibration after fast charging; and (4) electrochemical signatures during OCV rest or discharge are associated with Li re-intercalation into graphite. Furthermore, this work provides insight into the Li–graphite interactions at the composite electrode level and can be used to inform strategies to diagnose and mitigate Li plating during fast charging.
25 ENERGY STORAGE↗