Streaming Analytics and Data Engineering

Abstract

SADE is a software package for rapidly assembling analytic pipelines to manipulate data. The packages consists of the engine that manages the data and coordinates the movement of data between the tasks performing a function, a set of core libraries consisting of plugins that perform common tasks, and a framework to extend the system supporting the development of new plugins. Currently through configuration files, a pipeline can be defined that maps the routing of data through a series of plugins. Pipelines can be run in a batch mode or can process streaming data; they can be executed from the command line or run through a Windows background service. There currently exists over a hundred plugins, over fifty pipeline configurations, and the software is now being used by about a half-dozen projects.