ALPINE: In Situ Visualization, Analysis, and Infrastructure for ECP Science.
Abstract not provided.
Bennet, Janine↗
SEARCH · Search NASA
Results for “in situ analysis and visualization”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Catalyst Revised: Rethinking the ParaView in Situ Analysis and Visualization API
As in situ analysis goes mainstream, ease of development, deployment, and maintenance becomes essential, perhaps more so than raw capabilities. In this paper, we present the design and implementation of Catalyst, an API for in situ analysis using ParaView, which we refactored with these objectives in mind. Furthermore, our implementation combines design ideas from in situ frameworks and HPC tools like Ascent and MPICH.
97 MATHEMATICS AND COMPUTING↗
DevOps for Catalyst In Situ Visualization and Analysis at Sandia National Laboratories
Explore the source record for details and available documents.
Mauldin, Jeffrey A.↗
Towards elastic in situ analysis for high-performance computing simulations
In situ analysis and visualization have grown increasingly popular for enabling direct access to data from high-performance computing (HPC) simulations. As a simulation progresses and interesting physical phenomena emerge, however, the data produced may become increasingly complex, and users may need to dynamically change the type and scale of in situ analysis tasks being carried out and consequently adapt the amount of resources allocated to such tasks. To date, none of the production in situ analysis frameworks offer such an elasticity feature, and for good reason: the assumption that the number of processes could vary during run time would force developers to rethink software and algorithms at every level of the in situ analysis stack. In this paper we present Colza, a data staging service with elastic in situ visualization capabilities. We demonstrate the use of Colza with the Deep Water Impact and the AMR-Wind simulations, coupling them with the ParaView Catalyst and Ascent in situ libraries, and show that Colza enables dynamic rescaling of these widely-used frameworks with no interruption to the simulation or staging service. Here, we highlight the challenges of enabling such elasticity, which requires overcoming these frameworks' reliance on MPI, using distinct engineering approaches, namely dependency injection and dependency overload. To the best of our knowledge, this work is the first to enable elastic in situ visualization capabilities for HPC applications on top of existing production analysis tools.
97 MATHEMATICS AND COMPUTING↗
The ECP ALPINE project: In situ and post hoc visualization infrastructure and analysis capabilities for exascale
A significant challenge on an exascale computer is the speed at which we compute results exceeds by many orders of magnitude the speed at which we save these results. Therefore the Exascale Computing Project (ECP) ALPINE project focuses on providing exascale-ready visualization solutions including in situ processing. In situ visualization and analysis runs as the simulation is run, on simulations results are they are generated avoiding the need to save entire simulations to storage for later analysis. The ALPINE project made post hoc visualization tools, ParaView and VisIt, exascale ready and developed in situ algorithms and infrastructures. The suite of ALPINE algorithms developed under ECP includes novel approaches to enable automated data analysis and visualization to focus on the most important aspects of the simulation. Many of the algorithms also provide data reduction benefits to meet the I/O challenges at exascale. ALPINE developed a new lightweight in situ infrastructure, Ascent.
97 MATHEMATICS AND COMPUTING↗
Collaborative: in situ visual analytics technologies for extreme scale combustion simulations
This project aims to drastically enhance the usability of in situ analysis and visualization for extreme-scale scientific simulations. Current exascale computing capabilities promise to offer greater predictive ability of simulations and to further push the frontiers of science and technology. However, to validate the simulation output at extreme scale, examine the modeled phenomena, and discover previously unknowns from the output data, the output must be reduced or transformed in situ as it is being generated during the simulation such that the amount of data to examine and store is kept to a minimum. Such in situ approaches allow us to process and analyze the data and any embedded geometry to an extent that would be prohibitively expensive, if not impossible, to perform as a post hoc task. While in situ processing has been demonstrated to be a feasible and promising approach, its full potential has not yet been leveraged. In this project, we have developed comprehensive enhancements to in situ technology based on probability distributions in data. Our research focuses on jointly developing new ways of interacting with massive statistical samples while creatively utilizing new state-of-the-art computational resources to push the boundaries of in situ exploration. Moreover, we have developed new time-dependent techniques to enable previously unattainable capabilities in areas such as intelligent simulation steering and precise feature identification. We have experimentally studied our design and implementation at NERSC and OLCF, and are able to leverage existing in situ infrastructures whenever possible. While the exemplar in this project is combustion, many other fields for which turbulent transport is important, e.g., fusion, climate, astrophysics among others, encounter similar issues as simulations scale up to the exascale. This project shows its potential to generate high impact on DOE missions since the resulting technology promises to improve scientists’ ability to rapidly and correctly interpret and tune extreme-scale simulations, leading to new scientific understanding and advancements.
97 MATHEMATICS AND COMPUTING↗
Streaming Data in HPC Workflows Using ADIOS
The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.
Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗
Celestial Mapping System and Digital Lunar Library Initiative
We are preparing to create an interactive, global 3D lunar environment with integrated dataset and AI/ML tools to provide unique value to mission planners, scientists and the entire lunar community. This lunar environment will be based on NASA Ames Celestial Mapping System (CMS) [1] and Digital Lunar Library (DLL) Initiative. CMS provides a 3D virtual Lunar Globe with extensive user friendly tool sets, that include high resolution terrain visualization, elevation profiles, measurement kits, slope analysis, path optimization, line of sight analysis, equipment planning and placement tools and many other functionalities [1]. It has a thick client with less overhead to access hardware resources. This allows features such as terrain profiling and distance calculations to be performed on the client and on the fly. The application is developed to provide situational and domain awareness on the Lunar surface, planning capabilities for equipment placements and traverse path optimization. As data becomes available, CMS has the capabilities to integrate data sets that change dynamically in real-time, which will be useful for monitoring satellites and remotely-sensed data on the Lunar surface. CMS supports importing synthetic features in a variety of 3D, 2D, vector and raster formats. In the future, these capabilities will be enhanced by incorporating AI/ML tools and a plug-in architecture to enable customization by the user groups. With the help of DLL we will be able to : 1) Amplify the value of lunar information with AI-powered data enhancements 2) Acquire and integrate lunar data with AI-assisted georectification and homogenization 3) Analyze lunar data with advanced 3D visualization, intelligent search-by-example 4) Apply lunar data insights to specific use cases with an open plug-in architecture. The CMS-DLL initiative will have several potential use cases for NASA and the lunar community in general, including subsurface lava tube visualization and analysis, soil analysis, in-situ lunar resource visualization and representation on 3D globe, and data analytics for utilization. REFERENCES: [1] https://celestial.arc.nasa.gov/
3D Globe↗
Demonstrating the viability of Lagrangian in situ reduction on supercomputers
Performing exploratory analysis and visualization of large-scale time-varying computational science applications is challenging due to inaccuracies that arise from under-resolved data. In recent years, Lagrangian representations of the vector field computed using in situ processing are being increasingly researched and have emerged as a potential solution to enable exploration. However, prior works have offered limited estimates of the encumbrance on the simulation code as they consider “theoretical” in situ environments. Further, the effectiveness of this approach varies based on the nature of the vector field, benefitting from an in-depth investigation for each application area. With this study, an extended version of Sane et al. (2021), we contribute an evaluation of Lagrangian analysis viability and efficacy for simulation codes executing at scale on a supercomputer. We investigated previously unexplored cosmology and seismology applications as well as conducted a performance benchmarking study by using a hydrodynamics mini-application targeting exascale computing. Here, to inform encumbrance, we integrated in situ infrastructure with simulation codes, and evaluated Lagrangian in situ reduction in representative homogeneous and heterogeneous HPC environments. To inform post hoc accuracy, we conducted a statistical analysis across a range of spatiotemporal configurations as well as a qualitative evaluation. Additionally, our study contributes cost estimates for distributed-memory post hoc reconstruction. In all, we demonstrate viability for each application — data reduction to less than 1% of the total data via Lagrangian representations, while maintaining accurate reconstruction and requiring under 10% of total execution time in over 90% of our experiments.
97 MATHEMATICS AND COMPUTING↗
Capturing Carbonation: Understanding Kinetic Complexities through a New Era of Electron Microscopy
Cryogenic plasma focused ion beam (PFIB) electron microscopy analysis is applied to visualizing ex situ (surface industrial) and in situ (subsurface geologic) carbonation products, to advance understanding of carbonation kinetics. Ex situ carbonation is investigated using NIST fly ash standard #2689 exposed to aqueous sodium bicarbonate solutions for brief periods of time. In situ carbonation pathways are investigated using volcanic flood basalt samples from Schaef et al. (2010) exposed to aqueous CO 2 solutions by them. The fly ash reaction products at room temperature show small amounts of incipient carbonation, with calcite apparently forming via surface nucleation. Reaction products at 75° C show beginning stages of an iron carbonate phase, e.g., siderite or ankerite, common phases in subsurface carbon sequestration environments. This may suggest an alternative to calcite in carbonation low calcium-bearing fly ashes. Flood basalt carbonation reactions show distinct zonation with high calcium and calcium-magnesium bearing zones alternating with high iron-bearing zones. The calcium-magnesium zones are notable with occurrence of localized pore space. Oscillatory zoning in carbonate minerals is distinctly associated with far-from-equilibrium conditions where local chemical environments fluctuate via a coupling of reaction with transport. The high porosity zones may reflect a precursor phase (e.g., aragonite) with higher molar volume that then “ripens” to the high-Mg calcite phase-plus-porosity. These observations reveal that carbonation can proceed with evolving local chemical environments, formation and disappearance of metastable phases, and evolving reactive surface areas. Together this work shows that future application of cryo-PFIB in carbonation studies would provide advanced understanding of kinetic mechanisms for optimizing industrial-scale and commercial-scale applications.
36 MATERIALS SCIENCE↗
Crystallographic Insight of Reduced Lattice Volume Expansion in Mesoporous Cu 2+ -Doped TiNb 2 O 7 Microspheres during Li + Insertion
TiNb 2 O 7 represents a promising anode material for lithium-ion batteries (LIBs), but its practical applications are currently hampered by the non-negligible volumetric expansion and contraction during the charge/discharge process and the sluggish ion/electron kinetics. Here a combination technique is reported by systematically optimizing the porous and spherical morphology, crystal structure, and surface decoration of mesoporous Cu 2+ -doped TiNb 2 O 7 microspheres to enhance the electrochemical Li + storage performance and stability simultaneously. The Cu 2+ dopants preferentially replace Ti 4+ in crystal lattices, which decreases the Li + diffusion barrier and increases the electronic conductivity, as confirmed by density functional theory (DFT) calculation and demonstrated by diverse electrochemical characterizations. The successful Cu 2+ doping significantly reduces the lattice expansion coefficient from 7.26% to 4.61% after Li + insertion along the b-axis of TiNb 2 O 7 , as visualized from in situ and ex situ XRD analysis. The optimal 5% Cu 2+ -doped TiNb 2 O 7 with surface coating of N-doped carbon exhibits significantly enhanced specific capacity and rate and cyclic performances in both half- and full-cell configurations, demonstrating an excellent electrochemical behavior for fast-charging LIB applications.
36 MATERIALS SCIENCE↗
AEflow (Autoencoder fluid flow compression network) [SWR-22-29]
As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.
King, Ryan↗
CALORIE: A Constraint Language and Optimizing Runtime for Exascale Power Management (Final Report)
This final technical report summarizes the key accomplishments on the CALORIE project, a DOE Early Career award received by PI Henry Hoffmann at the University of Chicago. CALORIE’s main goal was to create principled methodologies, tools, and practices to help scientists and high-performance computing (HPC) operators maximize the performance and insights obtained from scientific computing applications in the face of exascale power constraints. The project accomplished these goals by completing the following objectives: • Designing a language for describing application goals (including constraints and objectives) and system capabilities. The application goals will include things like which simulations or simulations plus in situ analysis will be run together, what the power constraints are, and what requirements there are for in situ analysis (for example, a desired frame rate for visualization). The system capabilities include all components that can be adjusted to tradeoff power and performance. • Designing a runtime system that takes specified goals and capabilities and dynamically determines what capabilities to use to meet the goals. This runtime adapts to changes in goals, application behavior, or available capabilities to automatically maintain the goals despite unexpected disturbances. • Developing a foundational understanding of how power constraints affect the problem of scheduling applications in large-scale systems. This report provides an overview of the accomplishments related to each of these key objectives.
97 MATHEMATICS AND COMPUTING↗
Visual Data Analysis for Satellites
The Visual Data Analysis Package is a collection of programs and scripts that facilitate visual analysis of data available from NASA and NOAA satellites, as well as dropsonde, buoy, and conventional in-situ observations. The package features utilities for data extraction, data quality control, statistical analysis, and data visualization. The Hierarchical Data Format (HDF) satellite data extraction routines from NASA's Jet Propulsion Laboratory were customized for specific spatial coverage and file input/output. Statistical analysis includes the calculation of the relative error, the absolute error, and the root mean square error. Other capabilities include curve fitting through the data points to fill in missing data points between satellite passes or where clouds obscure satellite data. For data visualization, the software provides customizable Generic Mapping Tool (GMT) scripts to generate difference maps, scatter plots, line plots, vector plots, histograms, timeseries, and color fill images.
Lau, Yee↗
Enabling Command-and-Control in Advanced In Situ Workflows
Scientific discovery is progressing towards autonomous science with the combination of scientific instruments, high-performance computing, and artificial intelligence in complex workflows. This evolution introduces new requirements for managing scientific workflows, including feedback loops, near real-time constraints, and the ability to dynamically control workflow execution. In situ workflows that analyze and visualize data as it is generated are well-suited to satisfy stringent time constraints and their iterative nature offers greater opportunities for command-and-control. However, only a few of the many workflow management systems available have been specifically designed to manage in situ workflows and often lack support for automated feedback loops that allow analysis and visualization components to interact with the main scientific data producer. To address this need, we present in this paper how to add command-and-control capabilities to a workflow management system. We identify the functional design requirements of such a command-and-control system, detail its architecture, interface, and core mechanisms, and illustrate how advanced in situ workflows can leverage command-and-control in three use cases: graceful termination with checkpoint, dynamic and adaptive data reduction, and event-triggered analysis.
Mehta, Kshitij [ORNL] (ORCID:0000000297149981)↗
ALPINE Overview
Data-driven sampling enables probabilistic identification of interesting regions in the data automatically, prioritizing important regions. Applied in situ to Nyx, important halo regions are preserved.
97 MATHEMATICS AND COMPUTING↗
Developing in situ flood estimators using multi-date Landsat imagery
Landsat satellite imagery is being used as the primary source of information on flooding in the Black River Basin of northern New York State. Landsat images (Band 7) depicting flood conditions during several flood seasons since 1973 were obtained for analysis. Visual interpretation of these images is providing the basis for quantitatively relating in situ measurements of river discharge with the total area and geographic locations of inundation. This, in turn, will provide real-time estimation of flood losses over the entire river basin. This practical and inexpensive approach can provide sufficiently reliable information, and is applicable in other similar river basins.
Mcleester, J. N.↗
Determining the Importance of Energy Transfer between Magnetospheric Regions via MHD Waves using Constellations of Spacecraft
This grant was focused on research in two specific areas: (1) development of new techniques and software for assimilation, analysis and visualization of data from multiple satellites making in-situ measurements; and (2) determination of the role of MHD waves in energy transport during storms and substorms. Results were obtained in both areas and presented at national meetings and in publications. The talks and papers that were supported in part or fully by this grant are listed in this paper.
Cattell, Cynthia A.↗