Zhiping Zheng: AnswerBus News Engine

AnswerBus News Engine

Zhiping Zheng
Saarland University
Department of Computational Linguistics
Postfach 15 11 50
D-66041 Saarbrücken, Germany
+49 681 302 3829
zheng@coli.uni-sb.de

ABSTRACT

AnswerBus News Engine uses news stories published on CNN Web sites as its knowledge base and intends to answer questions on just-happened facts.

Keywords

Question answering, QA specific indexing

1. INTRODUCTION

AnswerBus Question Answering System is a Web-based open-domain system. It successfully uses NLP/IR techniques and reaches high correct answer rate. Although it is not designed for TREC, it still correctly answers over 70% of TREC-8 questions with Web resources ([2,3]). The question remains if a special indexing system will work better for the QA tasks.

In the experiment, we locally indexed over 700,000 news stories published on CNN Web sites since 1996. We developed a search engine for the new QA system. The goal of this experiment is to use most techniques used in AnswerBus QA system together with some new techniques, such as QA specific indexing, described in [2,3] but not fully implemented in original AnswerBus system, and build a QA system to answer time sensitive questions in the real world.

2. NEW FEATURES

Comparing to other QA systems, AnswerBus News Engine has some new features not seen in other QA systems including its previous versions.

2.1 Scalable to a large Size

The current size of indexed data has been over 700K Web pages from CNN Web site and some of its sub sites. We believe that it has been the largest size of knowledge base for QA tasks at current time. And the designed size can be much bigger than the size we have already reached. It is possible for the future system to index the whole Web and answer questions.

Partially because of the local indexing, AnswerBus News Engine is now able to extract the possible answers for a user question from CNN news stories in 2-4 seconds. This makes the system fast enough to process more documents to mine the answers.

System load has been largely decreased than its previous systems and the system can answer more questions at the same time than its previous versions with same resource.

2.2 Embedded Search Engine

A QA systems usually uses some search tools to retrieve documents. Many systems use commercial search engines while others use local search engines for local data, for example, local Web contents or TREC QA corpus. For this experiment, we partially deployed the techniques used in Seven Tones Search Engine ([7,6]) for the search task, since it has a high indexing speed and it is possible to update the indexed database part by part. Some new functions including sentence level indexing, temporal indexing have been also implemented in the system.

As the results of the new techniques, AnswerBus News Engine is now able to answer some time sensitive questions about the some factual issues just happened half an hour ago.

3. WEB INTERFACE

The system has a similar Web interface as its original version. As in Figure 1, the system lists up to ten possible answers to a specific user question. Each of these answers has a dynamic link back to a specific CNN Web page containing the answer sentence. The navigation bar at the end provides an easy way to try user question with other online systems.


Figure 1. Screen Shot of AnswerBus News Engine

Some times a QA system cannot find any answer from the working knowledge base for a question. This doesn't mean there is no answer for the question. In this case, AnswerBus News Engine redirects the question to the embedded search engine so users will get a bunch of documents instead of answers. Very likely, if there is an answer to the question, the user can dig it out from the documents given by the search engine.

4. EVALUATION

It gets more difficult to evaluate the system because we don't have any baseline or comparable systems. And also because of the dynamic content, it is difficult to design a question set to do the evaluation.

However, the techniques used in this system and in its previous local archive version ([8]) are almost same. The evaluation data of the local archive version should be able to level the performance of the system.

We refer to the milestones described in [1] and provided questions, which covered all 16 Arthur Graesser's questions categories and 3 other question categories that ranged from easy to very difficult. Table 1 shows the encouraging test result. The accuracy of is 72% in top 1 and 80% in top 5 (Table 1).

Table 1. Evaluation on AnswerBus Local Archive
Question TypeNumberTop1Top5Wrong
1. Verification3111
2. Comparison2011
3. Disjunctive2200
4. Concept Completion6501
5. Definition6501
6. Example3300
7. Interpretation3210
8. Feature Specification5500
9. Quantification6402
10. Causal antecedent3201
11. Cause Consequence0000
12. Goal orientation1100
13. Enablement0000
14. Instrumental/Procedural1100
15. Expectational1100
16. Judgemental3100
17. Assertion3111
18. Request/Directive0000
19. Nils question2002

We also compare our search engine results with the search result from the LookSmart Search Engine used by CNN Web site, and the result from the Google site search. We conclude that our system outperforms these systems in terms of recall and precision.

Question-sentence matching formula used in original AnswerBus system was proved effective in Web-based QA system. However, in the new QA system, it is not working as good as in original AnswerBus QA system. Probably it is because 1) The text in CNN Web site is very formal and the style is almost unique. 2) Few redundant information can be found in CNN Web site.

5. CONCLUSIONS AND FUTURE WORK

Based on our experiment of our new QA system, we found that QA specific indexing and searching are quite feasible. Most techniques used in original AnswerBus System are scalable to large size knowledge base. A question answering system uses these techniques can reach a high speed.

Some more new tasks have been already in our future plan. One of them is to add more Web news resource to the system and make the system itself into a news portal. It could be something like a combination of a question answering system and Google News or other online news systems.

6. REFERENCES

  1. John Burger et al. Issues, Tasks and Program Structures to Roadmap Research in Question & Answering (Q&A). NIST. 2001.
  2. Zhiping Zheng. AnswerBus Question Answering System. Human Language Technology Conference (HLT 2002). San Diego, CA. March 24-27, 2002.
  3. Zhiping Zheng. Developing a Web-based Question Answering System. The Eleventh World Wide Conference (WWW 2002). Honolulu, HI. May 7-11, 2002.
  4. Zhiping Zheng. Rule-based Sentence Segmentation for HTML/TEXT Documents. The Thirteenth meeting of Computational Linguistics in the Netherlands (CLIN 2002). Groningen, Netherlands. November 29 2002.
  5. Zhiping Zheng. International News Connection: A real-time online news filtering and classification system. The 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM/SIGIR 2001). New Orleans, Louisiana. September 9-14, 2001.
  6. Zhiping Zheng. Seven Tones: Search for Linguistics and Languages. The 2nd Meeting of the North American Chapter of Association for Computational Linguistics (NAACL 2001). Pittsburgh, PA. June 2-7, 2001.
  7. Zhiping Zheng and Gregor Erbach. Specialized search in linguistics and languages. XI International Conference on Computing (CIC 2002). Mexico City, Mexico. November 25-29, 2002.
  8. Zhiping Zheng, Huiyan Huang and Sven Schmeier. Deploying Web-based Question Answering System to Local Archive. Fifth International Conference on TEXT, SPEECH and DIALOGUE (TSD 2002). Brno, Czech Republic. September 9-12, 2002.