Results and Discussion next up previous
Next: Use of RealNames Up: A Case Study in Previous: TREC Ad-Hoc Algorithm

Results and Discussion

As described in Section 3, we use 100 queries in this study that seek a certain web page/site. These queries vary from finding company web sites, e.g. ``jordanian airlines'', ``Volkswagon'', to finding college home pages as in ``Walla Walla College'', ``Brigham Young University'', to finding individual pages, e.g. ``mari ostendorf'', ``Vangelis Natsios'', and so on. The results from both the first pass (no query expansion) and the second pass (with query expansion) of our TREC algorithm are compared to four commercial search engines: Excite, Google, Lycos, and AltaVista Raging.

Figure 1: Performance of TREC algorithm compared to four commercial search engines.
\begin{figure*}\centerline{\epsfxsize=4.5in \epsfbox{plot.eps}}\end{figure*}

Figure 1 shows the results of our experiments. The x-axis of Figure 1 shows the rank at which the desired site was retrieved by a system. The y-axis shows the cumulative number of queries for which the desired site was retrieved at or before the corresponding rank on the x-axis. For example, a point $<$6,82$>$ on the plot indicates that the corresponding search engine retrieved the desired site at rank 6 or better for 82 out of the 100 queries. The higher the plot, the better the engine is. For example, the best engine (Engine 4) retrieves the relevant page at rank 1 for 81 out of the 100 queries. Whereas our two-pass TREC algorithm retrieves the relevant page at rank 1 for only 22 out of the 100 queries.

We would like to emphasize that for the TREC algorithms, any page that resides on the same host as the relevant page is counted as relevant. We assume that it should be fairly simple to do the host based grouping and present the root page for the host. This assumption might not always be true and the results presented in Figure 1 for the TREC algorithms are, in some sense, best-case. Despite the best-case scenario for the TREC algorithms, Figure 1 shows that the TREC algorithms are far behind these four commercial web search engines for the kind of queries used in this study. The best commercial engine finds the relevant page in top 10 for 94/100 queries whereas the better performing (one-pass) TREC algorithm finds the relevant page for only 60 of the 100 queries. In short, the performance analysis presented in Figure 1 shows that the results from the TREC algorithms are consistently and notably below even the poorest of the commercial web search engines used in our study. This indicates that best TREC ad-hoc algorithms are by no means state-of-the-art for web search if our objective is to find a specific web site. These results contradict the results presented in [5,6,7]. However, we should say that these previous studies do not use the type of queries used in our study.

An even more surprising result is that adding a more complex query-expansion second pass does not improve the results, instead it makes the results somewhat worse. Using query expansion and doing two-pass retrieval we only find the relevant page for 50/100 queries in top 10 results as opposed to 60 pages for the one-pass algorithm. This result is in direct contradiction to the results obtained by TREC participants for the TREC ad-hoc benchmark tasks. In those results, it has been widely shown that in terms of average precision, which is how results are measured at TREC, a two-pass algorithm is almost always notably better than using just the first pass.

next up previous
Next: Use of RealNames Up: A Case Study in Previous: TREC Ad-Hoc Algorithm
Amit Singhal 2001-02-18