Implications of inter-rater agreement on a student information retrieval evaluation
In: LWA 2010 : Lernen, Wissen & Adaptivität ; workshop proceedings, p. 1-7
This paper is about an information retrieval evaluation on three different retrieval-supporting
services. All three services were designed to
compensate typical problems that arise in metadata-
driven Digital Libraries, which are not adequately
handled by a simple tf-idf based retrieval.
The services are: (1) a co-word analysis based
query expansion mechanism and re-ranking via
(2) Bradfordizing and (3) author centrality. The
services are evaluated with relevance assessments
conducted by 73 information science students.
Since the students are neither information
professionals nor domain experts the question of
inter-rater agreement is taken into consideration.
Two important implications emerge: (1) the inter-
rater agreement rates were mainly fair to
moderate and (2) after a data-cleaning step which
erased the assessments with poor agreement rates
the evaluation data shows that the three retrieval
services returned disjoint but still relevant result sets. (author's abstract)