Evaluation of Alignment: Precision, Recall, Weighting and Limitations

Abstract

In the real world, data does not come neatly packaged. Instead, it typically comes as small updates from many sources with different conventions. Building a single, cohesive knowledge-base to work from requires merging small updates from many different sources. This paper outlines methods we have investigated for scoring merging routines. Given a challenge problem consisting of a large knowledge-base and a set of smaller documents, algorithms are asked to identify alignment points between the smaller document and the knowledge base. This paper surveys options for evaluating such algorithms, providing notes on strengths, weaknesses and considerations for interpretation.