Provenance-Enabling DOE Visualization Applications
VisTrails, Inc.’s goal main charter has been to develop provenance management solutions for software tools. As the use of computation increases, keeping track of computational processes and the data they consume and derive is essential for many applications. Provenance (also referred to as audit trail, lineage, and pedigree) captures information about the steps used to generate a given data product. Such information provides important documentation that is key to preserving the data, to determining the data’s quality and authorship, and to reproducing as well as validating the results. Given the growing complexity of the computational processes and the volume of data they manipulate, manual approaches to maintaining provenance are both tedious and error-prone. At VisTrails, Inc., we have implemented infrastructure that can be used to provenance-enable existing tools. We have developed a series of plugins for specific applications, including ParaView and VisIt, which enable these applications to capture provenance. While developing these plugins, we identified general patterns across different applications, and these were used to inform the design of a general framework: the VisTrails Provenance Software Development Kit (SDK). The SDK encapsulates a set of core functionality including methods for capture, navigation, analysis and re-use of provenance. It allows application or plugin developers to more effectively provide provenance support: they can use the core functionality and write code only to handle application specific components. The SDK includes features that can be used in any application, for example, a refined interface for browsing the intermediate and final results captured. While the application-specific plugins were written in Python, the SDK was written from scratch in C++. This has led not only to substantial gains in performance but also the ability to integrate the SDK with applications written in other languages, including Java. To ensure that the SDK supported all of the functionality in the application-specific plugins, we have rewritten some of the existing plugins using the SDK. The SDK can be downloaded from www.vistrals.com/sdk.html.