Next: Evaluation Measures Up: Experimental Environment Previous: Queries and Judgments
4. To be fair and to make sure that our implementation of the TREC ad-hoc algorithm is not broken, we test our implementation on several TREC ad-hoc tasks and verify that it is indeed state-of-the-art (see Section 4). Our evaluation procedure needs to accommodate the fact that the TREC ad-hoc algorithms retrieve pages, not sites, whereas most of the queries used in this study seek particular sites. Our implementation, like any other keyword-based system, has a tendency to retrieve multiple pages from a site. To have a site-oriented retrieval, we group the pages by the host they reside on and select the top twenty sites for evaluation. A site/host is given the same rank as the rank of the best page residing on that site. This is in-line with what many commercial search engines do.
Amit Singhal 2001-02-18