Final Report: NEAMS Data Analysis
Explore the source record for details and available documents.
Engineering topics
Publications and source records attributed to O'Leary, Patrick.
Explore the source record for details and available documents.
Physical aquifer models are a highly effective teaching tool for hydrology education, however they come with inherent limitations that include the high cost to purchase, the static configuration of the model materials, the time required to visualize hydrogeological phenomena, and the effort to reset and clean them over time. To address these and other limitations, we have developed an interactive computer simulation of a physical aquifer model called the ParFlow Sandtank. In this gamified interface, users run the simulation using a familiar web-app like interface with sliders and buttons while learning real hydrologic concepts. Our user interface allows participants to dive into the world of hydrology, understanding assumptions about model parameters such as hydraulic conductivity, making decisions about inputs to groundwater aquifer systems such as pumping rates, visualizing outputs such as stream flow, transport, and saturation, and exploring various factors that impact real environmental systems such as climate change. The ParFlow Sandtank has already been used in a variety of educational settings with more than 9,000 users per year, and we feel this emerging educational tool can be used broadly in educational environments and can be scaled-up to provide greater accessibility for students and educators. Here we present the capabilities and workflow of the ParFlow Sandtank, two use cases, and additional tools and custom templates that have been developed to support and enhance the reach of the ParFlow Sandtank.
As in situ analysis goes mainstream, ease of development, deployment, and maintenance becomes essential, perhaps more so than raw capabilities. In this paper, we present the design and implementation of Catalyst, an API for in situ analysis using ParaView, which we refactored with these objectives in mind. Furthermore, our implementation combines design ideas from in situ frameworks and HPC tools like Ascent and MPICH.
Ubiquitous web-based visualization is essential to delivering large-scale data visualization to various stakeholders, from the scientist to the board member. These stakeholders will not tolerate a stalled application or a pop-up window asking them to wait for the processing to complete. They require a responsive and interactive visualization environment with high-quality imagery suitable for detailed analysis and boardroom presentations. At Kitware, Inc., we have accomplished web visualization to this point, leveraging state-of-the-art tools like HTML5, CSS3, SVG, Canvas, and WebGL. Solutions that leverage a combination of these technologies are necessary to handle workloads that vary significantly in data size efficiently. However, it is not always practical to move large data to the web client for visualization. Kitware's ParaView as a Service combines client-side visualization using both distributed processing and remote rendering on big data impractical to move. Existing distributed processing and remote rendering solution's interactivity is below the expectations of web-based applications. Our project examined proposed solutions to the areas outlined above in ParaView as a Service. We have investigated concurrent pipelines, streaming images, progressive rendering, and optimization of algorithms and data movement to address these concerns. For the Phase I project, we completed the proposed work plan. As a result, the project produced three prototypes of essential importance for web visualization and the ParaView as a Service community. We created a simple desktop application for an interactive streamline placement prototype, a web-based interactive streamline placement prototype, and a web-based progressive rendering utilizing raytracing prototype. These prototypes relied on the hardening of emerging software toolkits funded by the Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) program (such as ParaView, VTK-m, and Mochi). We blended these components into web-based visualization prototypes that meet the industry's expectations for interactivity and responsiveness. The Phase I project had four essential focus areas: 1. Develop prototype ParaView as a Service backend server using asynchronous, non-blocking design principles. 2. Develop a prototype web application that uses the ParaView as a Service backend server for remote data visualization. 3. Implement image streaming with encoding/compression and progressive rendering capabilities in the proposed platform. 4. Evaluate the prototype developed and summarize observations, including the challenges and pitfalls of our approach. After our successful completion of Phase I, we are strongly positioned to propose a successful Phase II project.
Advanced manufacturing and engineering are fundamental to the economic strength and national security of the United States. With the introduction of advanced modeling and simulation practices in manufacturing and engineering over the last several decades, we are producing better products, more quickly, and at decreased cost. Through the adoption of high-performance computing (HPC) enabled workflows, in small- to medium-sized manufacturers and engineering firms, which employ fewer than 500 people, is renewing United States manufacturing and engineering. Despite successes, there are significant barriers to widespread adoption of these tools. Our Phase I SBIR project, Next-Generation Analysis and Visualization Platform for Manufacturing and Engineering Workflows, aims to address several of these barriers. Our project helped us evaluate the feasibility and benefits and mitigate risks of asynchronous processing for analysis and visualization workflows. We evaluated the impact of asynchronous, non-blocking communication can have on the user experience in a complex visualization application. Besides the impact on users, we wanted to understand the impacts on developers (development application programming interfaces (APIs))and generated applications (robustness and stability). Any asynchronous platform implementation is bound to rely on multiple threads which can make the programming model extraordinarily complex and prone to bugs. If not managed carefully, the complexity and resulting instability would be an unacceptable evolution for widely used production tools like ParaView and putting the success of the Phase II project at risk.
A key trend facing extreme-scale computational science is the widening gap between computational and I/O rates, and the challenge that follows is how to best gain insight from simulation data when it is increasingly impractical to save it to persistent storage for subsequent visual exploration and analysis. One approach to this challenge is centered around the idea of in situ processing, where visualization and analysis processing is performed while data is still resident in memory. This paper examines several key design and performance issues related to the idea of in situ processing at extreme scale on modern platforms: Scalability, overhead, performance measurement and analysis, comparison and contrast with a traditional post hoc approach, and interfacing with simulation codes. We illustrate these principles in practice with studies, conducted on large-scale HPC platforms, that include a miniapplication and multiple science application codes, one of which demonstrates in situ methods in use at greater than 1M-way concurrency.
The introduction of advanced modeling and simulation practices over the last several decades has proven to produce better products, more quickly, and at the reduced cost. This computational experimentation has enabled the design and development of a variety of innovative products by reducing the need to build expensive prototypes, and by allowing rapid, comprehensive design studies across a wide range of products. Despite these successes, there are significant barriers to widespread adoption of these tools. This project aims to address several of these barriers.
Complex, coupled physical systems that occur in various domain sciences present several serious challenges in implementing an efficient and seamless simulation workflow. These challenges impact the productivity of scientific research and the overall time it takes to verify simulation results. To overcome these computational modeling barriers, specifically in nuclear reactor analysis, one must rely on flexible and scalable tools that simplify modeling complexity.
Data-intensive and compute-intensive applications are the two essential methods supporting the scientific discovery process. Until recently, the development of technologies supporting the corresponding schemes evolved along two independent paths. Born and matured before the Big Data era, high-performance computing (HPC) scientific-oriented applications were not designed for Big Data streaming frameworks. But an efficient processing framework offered by the Spark computational platform has resolved many of the barriers to enable compute-intensive applications from within a data-intensive environment. Our Phase I project extended Spark with pipelines that utilize HPC applications. As a result, the project immediately addressed emerging high-throughput experiments driven by a new generation of X-ray instruments. After successful completion of Phase I, we are in a strong position for proposing a successful Phase II project.
Summarization and compression at current and future scales requires a framework for developing and benchmarking algorithms. We present a framework created by integrating existing, production-ready projects and provide timings of two particular algorithms that serve as exemplars for summarization: a wavelet-based data reduction filter and a generator for creating image-like databases of extracted features (isocontours in this case). Both support browser-based, post-hoc, interactive visualization of the summary for decision-making. A study of their weak-scaling on a distributed multi-GPU system is included in this article.
The primary challenge motivating this project is the widening gap between the ability to compute information and to store it for subsequent analysis. This gap adversely impacts science code teams, who can perform analysis only on a small fraction of the data they calculate, resulting in the substantial likelihood of lost or missed science, when results are computed but not analyzed. Our approach is to perform as much analysis or visualization processing on data while it is still resident in memory, which is known as in situ processing. The idea in situ processing was not new at the time of the start of this effort in 2014, but efforts in that space were largely ad hoc, and there was no concerted effort within the research community that aimed to foster production-quality software tools suitable for use by Department of Energy (DOE) science projects. Our objective was to produce and enable the use of production-quality in situ methods and infrastructure, at scale, on DOE high-performance computing (HPC) facilities, though we expected to have an impact beyond DOE due to the widespread nature of the challenges, which affect virtually all large-scale computational science efforts. To achieve this objective, we engaged in software technology research and development (R&D), in close partnerships with DOE science code teams, to produce software technologies that were shown to run efficiently at scale on DOE HPC platforms.
According to the World Nuclear Association, the U.S. is the largest producer of nuclear power worldwide. In fact, the U.S. provides over 30 percent of the world's nuclear generation of electricity. Although few new reactors have been built in the past 30 years, the association anticipates that four to six reactors may be built by 2020. The first of the reactors will be built in response to 16 license applications, which have been completed since 2007, to build 24 additional nuclear reactors. Currently, both government and industry are working to accelerate approval for the construction and design of new nuclear energy plants. The U.S. nuclear industry has already achieved significant developments in the implementation of nuclear power plants due to advancements in refueling, maintenance, and safety systems at current power plants. Meanwhile, changes in government policy, which have occurred since the late 1990s, have provided for noteworthy expansion in nuclear capacity. For example, the Energy Policy Act of 2005 evoked investment in electricity infrastructure such as nuclear power. To design future systems and continue operational improvements at existing U.S. plants, advanced modeling and simulation of nuclear reactors are crucial. We developed a Cloud/Web-based tool, HPCCloud, that enables nuclear energy firms, or other manufacturing and engineering companies, to innovate using advanced modeling and simulation. Through the Web-Based Computational Model Builder for Nuclear Energy Advanced Modeling and Simulation grant, we develop HPCCloud to simplify and democratize advanced modeling and simulation in the nuclear energy industry on a range of nuclear engineering applications. It leverages millions of investment dollars from the Department of Energy's Consortium for Advanced Simulation of Light Water Reactors (CASL) and the Office of Nuclear Energy's research and development in Nuclear Energy Advanced Modeling and Simulation (NEAMS). Our approach simplifies the workflow, alleviates the need for in-house computational science and engineering experts, and lowers the capital investments required for advanced modeling and simulation. The Cloud/Web-based simulation environment guides end-users and developers through the advanced modeling and simulation lifecycle. In addition to providing significantly improved, intuitive software, the environment offers reproducible workflows where one can save, share, and even publish the full path of data from input to final analyzed results. HPCCloud addresses identified challenges to the nuclear energy industry's use of advanced modeling and simulation. With this unique Cloud/Web-based environment, we enable an ecosystem of synergistic activities, fueling new research and development for nuclear energy. HPCCloud lays the groundwork for commercialization, through customization, technology integration, and advanced research and development services.
Modern scientific, engineering and medical computational sim- ulations, as well as experimental and observational data sens- ing/measuring devices, produce enormous amounts of data. While statistical analysis provides insight into this data, scientific vi- sualization is tactically important for scientific discovery, prod- uct design and data analysis. These benefits are impeded, how- ever, when scientific visualization algorithms are implemented from scratch—a time-consuming and redundant process in im- mersive application development. This process can greatly ben- efit from leveraging the state-of-the-art open-source Visualization Toolkit (VTK) and its community. Over the past two (almost three) decades, integrating VTK with a virtual reality (VR) environment has only been attempted to varying degrees of success. In this pa- per, we demonstrate two new approaches to simplify this amalga- mation of an immersive interface with visualization rendering from VTK. In addition, we cover several enhancements to VTK that pro- vide near real-time updates and efficient interaction. Finally, we demonstrate the combination of VTK with both Vrui and OpenVR immersive environments in example applications.
The framework created through the Open-Source Integrated Design-Analysis Environment (IDAE) for Nuclear Energy Advanced Modeling & Simulation grant has simplify and democratize advanced modeling and simulation in the nuclear energy industry that works on a range of nuclear engineering applications. It leverages millions of investment dollars from the Department of Energy's Office of Nuclear Energy for modeling and simulation of light water reactors and the Office of Nuclear Energy's research and development. The IDEA framework enhanced Kitware’s Computational Model Builder (CMB) while leveraging existing open-source toolkits and creating a graphical end-to-end umbrella guiding end-users and developers through the nuclear energy advanced modeling and simulation lifecycle. In addition, the work deliver strategic advancements in meshing and visualization for ensembles.
High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In this paper, we describe the development of the Python-based cumulus, which addresses many of these barriers. cumulus is a platform for the dynamic provisioning and execution of HPC workflows. cumulus provides the infrastructure needed to build applications that leverage traditional or Cloud-based HPC resources in their workflows. Lastly, we demonstrate the use of cumulus in both web and desktop simulation applications, as well as in an Apache Spark-based analysis application.
Abstract not provided.
Computer simulations are growing in sophistication and producing results of ever greater fidelity. This trend has been enabled by advances in numerical methods and increasing computing power. However these advances come with several costs including massive increases in data size, difficulties examining output data, challenges in configuring simulation runs, and difficulty debugging running codes. Interactive visualization tools, like ParaView, have been used for post-processing of simulation results. However, the increasing data sizes, and limited storage and bandwidth make high fidelity post-processing impractical. In situ analysis is recognized as one of the ways to address these challenges. In situ analysis moves some of the post-processing tasks in line with the simulation code thus short circuiting the need to communicate the data between the simulation and analysis via storage. ParaView Catalyst is a data processing and visualization library that enables in situ analysis and visualization. Built on and designed to interoperate with the standard visualization toolkit VTK and the ParaView application, Catalyst enables simulations to intelligently perform analysis, generate relevant output data, and visualize results concurrent with a running simulation. Here, we provide an overview of the Catalyst framework and some of the success stories.