Processing Spacecraft Data Without Confusion

Abstract

Producing multiple versions of the same data product for the same time frame with the same remotely sensed inputs can be a recipe for disaster. Yet, amidst the commotion of satellite launch and early operations (LEO), such data processing is needed. After LEO, the situation gets worse. Processing newly arriving data ("forward processing") is augmented with reprocessing and algorithm development, comparison, evaluation, and testing -- often happening all at the same time. The problem can be analyzed in three main parts -- maintaining multiple versions of algorithms and data so that end-product users are not overwhelmed,allocating computer resources efficiently, and simplifying production operations so that va st amounts of data can be processed with minimal staff and fewer errors. OMIDAPS provides a framework for execution of algorithms that transform lower level data acquired by OMI on NASA's Aura satellite into higher level science data products. In contrast to traditional science data processing systems, we address all parts of the problem with an innovative approach allowing multiple data processing to run within a single physical system. The data products, imports, exports, and execution planning are all segregated into distinct "ArchiveSets." This paper describes reasons for multiple concurrent productions on a typical satellite data processing project using OMI as an example. It describes the virtual data processing system concept and its advantages over separate physical processing strings. It explores the specific implementation of the virtual systems within OMIDAPS and discusses some of the implications of our approach and describes how virtual processing is used to accomplish the overall mission of OMI data processing.