A Large Scale Dataset for the Evaluation of Matching Systems

Yatskevich, Mikalai and Giunchiglia, Fausto and Avesani, Paolo (2006) A Large Scale Dataset for the Evaluation of Matching Systems. Technical Report DIT-06-035, Ingegneria e Scienza dell'Informazione, University of Trento.

Abstract

Ontology matching is one of the biggest challenges of Semantic Web research. In the last years the number of matching techniques and systems has significantly increased, and this, in turn, has raised the issue of their evaluation and comparison. In this paper we present a mapping dataset extracted from the Google, Yahoo and Looksmart web directories. This dataset allows for the evaluation of both Precision and Recall, and it is an order of magnitude larger than the state of the art datasets with the same capabilities. We have evaluated this dataset on nine state of the art matching solutions. The evaluation results highlight the fact that the dataset has three key properties, namely it is error-free, it is hard to solve, and it can discriminate among systems.