Web Search Behavior of Internet Experts and Newbies
Christoph Hölscher & Gerhard Strube
Center for Cognitive Science, Institute for Computer Science & Social Research, University of Freiburg, Germany
Email: firstname.lastname@example.org, email@example.com
Searching for relevant information on the World Wide Web is often a laborious and frustrating task for casual and experienced users. To help improve searching on the Web based on a better understanding of user characteristics, we investigate what types of knowledge are relevant for Web-based information seeking, and which knowledge structures and strategies are involved. Two experimental studies are presented, which address these questions from different angles and with different methodologies. In the first experiment 12 established Internet experts are first interviewed about search strategies and then perform a series of realistic search tasks on the WWW. From this study a model of information seeking on the WWW is derived and then tested in a second study. In the second experiment two types of potentially relevant types of knowledge are compared directly. Effects of Web experience and domain-specific background knowledge are investigated with a series of search tasks in an economics-related domain (introduction of the EURO currency). We find differential and combined effects of both Web experience and domain knowledge: While successful search performance requires the combination of the two types of expertise, specific strategies directly related to Web experience or domain knowledge can be identified.
Keywords: Expertise, Information Retrieval, Internet Search Engines, Logfile Analysis
The accelerated growth of the World Wide Web has turned the Internet into an immense information space with diverse and often poorly organized content. Online users are confronted with rapidly increasing amounts of information as epitomized by the buzzword "information overload." While skills necessary for browsing individual websites seem to be available to users after only minimal training (Hurtienne and Wandtke, 1997), considerably more experience is required for query-based searching (Pollock and Hockley, 1997) and intersite navigation.
The underlying question of the research presented in this paper is, what types of knowledge are relevant for Web-based information seeking, and which knowledge structures and strategies are involved. Two experimental studies are presented, which address these questions from different angles and with different methodologies.
Search engines such as Altavista or Excite are a central part of information seeking on the Internet. Their efficient use requires sophisticated knowledge. Since experienced users make use of search engines regularly for diverse information needs, i.e., using them quite often, it is reasonable to assume that they will develope particular expert knowledge in mastering these more complex services. Thus the research presented here is focused on interactions with search engines and related services. In addition, query-based searching allows for comparisons with research on search behavior of end-users in traditional IR systems.
Investigations on the search behavior of both expert and novice Web users have several practical applications. First and foremost, a model of search behavior can serve as the basis for improving interfaces and functionality of existing search systems. The varied needs of experts and novices can be identified and considered by more sophisticated future systems. Also, help-systems and Internet education (e.g., courses and tutorials) can also benefit from a better understanding of users' difficulties with the search process.
1.1 Related Research on Web Search:
The first influential studies on Web user behavior mainly investigated aspects of Browsing when navigating the WWW (Cockburn & Jones, 1996; Catledge & Pitkow, 1995; Tauscher & Greenberg, 1997). Byrne, John and Crow (1999) have recently proposed a "taxonomy of WWW user tasks" that span a user's complete range of behaviors while surfing the Web, but does not have a focus on information seeking or Web search.
Choo, Detlor and Turnbull (1999) have investigated the information seeking behavior of knowledge workers over a period of two weeks. Combining surveys, interviews and client-side logging they were able to characterize a number of information seeking behaviors of Web users that are summarized in a model of behavioral modes and moves.
Navarro-Prieto, Scaife and Rogers (1999) identified cognitive strategies related to Web searching. They compared Web searchers with high and low experience and concluded that expert searchers plan ahead in their searching behavior based on their knowledge about the Web, while novice searchers hardly plan at all and are rather driven by external representations (what they see on the screen).
Several researchers (e.g., Jansen et al., 1998; Silverstein et al., 1998) have collected impressively large datasets derived from the logs of Internet search engines like Excite or AltaVista. Their studies give a detailed picture of how the average Web user approaches a search service, but they also have drawbacks: Since the data is anonymous, we do not know anything about the context of the individual user, that is, we do not know what information problem he or she was trying to find or how experienced a user is with respect to the Internet in general or searching in particular. In the present study we use aggregated data from a large German search service to complement data collected from individual users.
In the User Modeling community the behavior of Web users has also attracted some attention. Lau and Horvitz (1999), for example, constructed Bayesian networks to model the successive search queries issued by users of a search engine. Augmenting the search engine logfile with manually assigned categories of presumed information goals they are able to predict query modifications. Similarly, Zukerman, Albrecht and Nicholson (1999) propose the use of Markov models to predict a Web user's next request based on the timing and location of past requests. Again, these studies do not address personal characteristics of the user and his level of expertise.
2. Experiment 1: Exploratory Investigation of Expert Knowledge and Search Behavior
The behavior of experienced Internet users and their specific knowledge has not been systematically investigated. Thus the first study is exploratory, aiming at a detailed description of Web expertise, describing typical search behavior of Web experts and constructing a descriptive model of information seeking with search engines. Comparable models for searching in electronic information systems were proposed by Marchionini, Dwiggins, Katz and Lin (1993) and Shneiderman, Byrd and Croft (1997), but did not consider, for example, the specific differences between the World Wide Web and bibliographic database systems.
We define Web expertise as a type of media competence, i.e., the knowledge and skills necessary to utilize the WWW and other Internet resources successfully to solve information problems. This has to be clearly distinguished from background-knowledge related to the topic area of a specific Web search (see Experiment 2).
Well established Internet professionals were recruited for this study. All had at least 3 years of intensive experience with this medium and a daily use of the Internet as a source of information at their workplace. Among the 12 participants each of them participated in both parts of the expert study were information brokers, Web masters, Internet consultants, Web content designers, librarians and authors of books about online searching. It is noteworthy that most participants had not received formal training in Internet use, they are cleary to be characterized as self-trained experts.
2.1 Phase I: Interviews.
First the participants were asked to describe their experience with the available search services, their search behavior and their intentions and rationales for using certain sources and strategies. With the help of mental walk-throughs the process of searching for online information was then discussed step by step. To reveal those experts' conceptual structures, the interview was augmented with a specialised card-sorting task (Janetzko, 1998; Strube, Janetzko & Knauff, 1996): During the interview, relevant terminology and actions were made explicit by recording them on colour-coded cards. Afterwards, the experts were asked to build a graphic structure with these cards. This structure is supposed to represent an expert's personal conceptualisation of the search process. To support the participants in this task, some appropriate concept categories and relations were predefined and presented to the experts.
2.2 Phase II: Experiment using web-based information tasks
Web-based information-seeking tasks. In the second phase of this expert study, a number of real-life information-seeking tasks were employed that had to be performed by the experts on the Internet. Examples: 'Which finger is unaffected in RSI?' or 'Find a sound archive for the VIRUS music synthesiser'. The experts were not limited in their choices for searching the Web and could freely choose which search engines - if any - they wanted to use.
All inputs to the computer were mediated through an assistant of the experimenter who had to be orally instructed by the expert for each action. This procedure forced the expert to make every step of the interaction process verbally explicit, including those that might otherwise be missed because of rapid interaction sequences. Additionally, the experts were asked to think aloud about their search activities. The method can be categorized as being between a classic thinking-aloud and a teaching-aloud scenario (Ericsson and Simon, 1993). All utterances were audio-taped and later transcribed for the analysis. Web-page requests and search queries were also included in the protocol.
2.3 Results of the expert study
The experts reported a wealth of Internet-related knowledge, most of it highly idiosyncratic. Therefore, their statements relating to the search process were collected from the transcripts and entered into a matrix to determine which concepts, heuristics and strategies were common to the majority of the experts. Likewise, the concept-card models were inspected for interindividually common knowledge structures. The statement matrix and the card models were aggregated into an initial process model of information seeking with search engines. This model describes the search process from the experts' shared perspective.
Web-based information-seeking tasks
We distinguish two levels of data analysis, the level of information seeking steps, and the level of individual search queries. For the analysis of information seeking steps, a set of rules was derived from the experts' process model for segmentation and categorization of the protocol into action units. A total of 56 information problems was tackled by the subjects, two thirds of these successfully. A total of 1956 action units were identified, each corresponding to a step in the process model. The matrix of transition probabilities between all steps of the model was computed, allowing for an analysis of interaction sequences. The main results are summarized below.
Figure 1: Global level of the process model of information seeking in Experiment 1: Browsing vs. Searching. (values represent transition probabilities to the next unit. The transition probabilities going out from a given step of the model add up to 100%, but transition probabilities of .03 and below are omitted here to reduce visual clutter)
Figure 1 shows the experts information seeking behavior on a global level of browsing and searching. In two thirds of the search tasks, the experts initially choose to use a search engine. Only in one third of the cases did they opt for browsing as the initial strategy. Finding potentially relevant documents with a search engine led to browsing episodes of varying length in about 47 percent of the cases. Once the searchers were in "browsing mode" they continued browsing for several clicks, hence the .73 probability of one browsing move leading to the next. Such browsing episodes could lead directly to a solution, but often enough, a return to the search engine for further queries was observed. This indicates that the experts in our study quite frequently switched back and forth between browsing and querying if necessary.
Figure 2: Close-up of direct interaction with a search engine. (values represent transition probabilities to the next unit. Transition probabilities of .03 and below are omitted to reduce visual clutter)
Figure 2 shows a close-up of actions directly involved in search-engine interaction. The straight downward arrows represent the default handling of the search engine with correspondingly high transitions probabilities. Additionally, the experts showed more complex behavior if no relevant documents were found, including reformulations or reformatting of existing queries, changing search engines, requesting additional result pages as well as backtracking to earlier result pages or queries. Again we observe opportunistic behaviour making use of all the options a search engine provides. Flexible use of availlable search behaviors is a characteristic feature of expert searchers.
Individual search queries were analysed as well, and compared to available data on user behavior. Jansen et al. (1998) report a quantitative analysis of a large sample of search requests from the EXCITE search engine, representing the search queries of the average Internet user. Similar data has been reported by Silverstein et al (1998) for the AltaVista search service.
A corresponding sample from German search-engine users was made available to us by the managers of the FIREBALL search engine, representing some 16 million queries and 27 million non-unique terms. A comparison of data sets from average users with the experts' queries in our study revealed several differences: First of all, the average length of a query in FIREBALL is only 1.66 words, while the experts used an average of 3.64 words, twice as many.
Table 1:Usage of query formating in experiment 1 (the Expert study) and aggregated statistics from the Fireball search engine.
We also found that web experts make use of advanced search options like Boolean operators, modifiers, phrase search etc., much more frequently than the average user (see Table 1). A noteworthy exception is the "+" operator. It is equally popular among the general public, making it the most important query formatting tool for non-expert users.
This first expert study confirmes the significance of media-specific skills of Internet users, and gives a detailed picture of Internet expertise. While IR skills were the focus of this study we found numerous hints of the importance of content specific knowledge. Experts frequently complained about lacking relevant domain knowledge regarding individual search questions and were highly aware of this obstacle while being confident of their technical competence.
3. Experiment 2: The EURO study
Several authors, for example Hsieh-Yee (1993), were able to show that technical competencies in using bibliographic database systems are necessary for successful information retrieval, but that such knowledge has to be combined with background knowledge about the topic area to be searched. This finding is in line with observations from the verbal protocols obtained in our expert study, where Web experts complained that they lacked domain-specific background knowledge for particular search tasks. The following experiment addressed these two types of knowledge that contribute to the success of searching on the Web, and how the two interact.
Experiment 2 is designed to compare directly the contributions that technical Internet skills and content-area specific domain knowledge make to the search process. A current topic from the domain of economics - the European Monetary Union - was chosen for this laboratory experiment. The subjects were given a set of information-search problems from this domain. A 2 x 2 design of the independent factors Web expertise and domain knowledge results in four experimental groups. Participants with domain knowledge were recruited from students of economics. Web expertise was assessed by interview and pre-test, allowing us to clearly identify novices and advanced web users, thereby excluding intermediate level web users from data analysis.
In the experiment, two kinds of tasks were used, simulated search tasks and tasks that had to be performed live on the Web.
3.1 Simulated Search tasks
Based on the process model developed in the expert study above, complex search tasks were broken down into sub-tasks corresponding to individual steps of the process, such as search term selection or query revision. The resulting sub-tasks allowed for a focussed investigation of the direct effects that different types of expertise have on individual steps of the model.These simulated tasks were collated in a questionnaire. The approach made sure that each participant worked on the same stimuli (words, queries, result pages), allowing for comparisons that are not readily available from observing unrestricted task performance on the Web. In "real" searches on the Web participants follow different paths trying to solve given tasks and hardly ever face exactly the same pages of results or have to reformulate the exact same search queries as another participant.
3.2 Web-based Search tasks
In the second part of the experiment, the actual Web searches, we tried to impose as few restrictions as possible, and did not employ thinking aloud techniques. Participants were asked to solve five information problems directly via the WWW. The only restriction imposed on the participants was a time limit of 10 minutes per task. All interactions were recorded by a proxy server (Siemens WebWasher) and a traditional observer protocol to complement the proxy log. Again, subjects could freely choose how to tackle the search tasks and which search engines to consult.
While in experiment 1 interaction sequences and search statements were reconstructed from the audio protocol of the thinking aloud tasks, in experiment 2 the same measures are recorded directly with the proxy-server installed on the client computer. The proxy logfile contains most of the necessary information like the date and time of each access, the Uniform Resource Locator (URL) of each file viewed and its length. Additionally we have the HTTP result code (indicating, e.g., if the file to be accessed was physically unavailable) and for most cases also the Referrer URL that indicates from which URL a users requests another page and is an important tool for reconstructing the behavioral trace. The logfile data can be processed to reveal the majority of the users interaction. Nonetheless a traditional observer protocol was written during the experiment to complement the logfile, because certain interactions are not adequately recorded in the proxy logfile, mainly concerning navigation in FRAME-Sets, use of the Back-Button in the Browser and queries submitted via the POST method. For the analysis, the proxy log and the observer protocol were combined for categorizing the user actions in terms of the process model developed above.
Browsing and searching behaviors which manifest in the interaction sequences during the search are compared to identify differences in information seeking strategies and tactics.
3.3 Results of the double comparison of advanced and novice searchers
The data presented below is based on a sample of 24 participants, 6 from each cell of the 2 x 2 design of Web expertise (high/low) and domain knowledge (high/low). We analyzed four types of data: rate of success, action sequences (expressed as transition probabilities), time data and formal properties of search queries.
Rate of success
The web-based search tasks proved to be rather difficult for all participants, resulting in low overall success rates. In three of the experimental groups, participants solved no more than 2 of 5 tasks on average. Only those users who could rely both on high Web expertise and high domain knowledge ("double experts") were able to solve an average of 3.2 out of the 5 tasks.
Across all experimental groups the pattern of action sequences is comparable to the data from the expert study.
Figure 3: Global level of the process model of information seeking: all four groups of the EURO study combined. (transition probabilities)
Figure 4: Close-up of direct interaction with a search engine: all four groups (novices & experts) of the EURO study combined. (transition probabilities)
One important difference is the fact that participants now obviously found less useful pages and had to reiterate their searches more frequently to find relevant information (see Fig. 3 and 4 for details). This increased difficulty most likely reflects both differences in the tasks (harder) and the participants (overall lower levels of expertise, since 50% were novices) of the two studies. Please note that the coding scheme was slightly revised from the expert study to the EURO study. This accounts for differences between the studies at the process stages "Access web site directly" (Fig. 3) and "Select + Launch Search Engine" (Fig. 4).
Differences between experts and novices
Figure 5: Initial behavior - the first action performed after receiving a task. (Web +/- refers to Web expertise, Econo +/- refers to domain knowledge)
Looking at the actions subjects choose as their initial behavior (Fig.5) we find several important differences between groups. Only "double experts" initially tried to access directly web-sites related to economics, while all others immediately accessed a search engine in one way or the other. Web experts would type in the URL of their favorite search engine, while the "double novices" were highly inclined to simply click on the Netscape Search button (these effects - and all others discussed below - prove to be significant in HILOGLINEAR analysis, unless stated otherwise).
Figure 6:Actions selected on a search engine result page. (Web +/- refers to Web expertise, Econo +/- refers to domain knowledge)
Once a Web search has led to a page of results (Fig. 6), Web experts were significantly more likely to choose a target document for closer inspection than Web novices (35% vs. 25%), while Web novices more often reiterate their search queries. We also found significant interactions of domain knowledge and Web expertise: When Web experts had little domain knowledge, they were most likely to pick a target document (possibly for lack of clear selection criteria). Double novices showed the highest proportion of query re-formulations while choosing the smallest number of target documents for closer examination and of these documents the highest proportion turned out to be irrelevant. A qualitative inspection of the query re-formulations that were issued by the double novices indicated that they often make only small and ineffective changes to their queries, forcing them to reiterate repeatedly.
Figure 7:Transitions while Browsing. (Web +/- refers to Web expertise, Econo +/- refers to domain knowledge)
Looking at browsing behavior, we can once again identify some clear patterns. Figure 7 shows what the participants choose to do next once they have accessed a document in a browsing episode. The behavior of the double experts with technical and domain-specific knowledge can be characterised like this: They are most likely to continue browsing (follow another link) to explore more content from a Web site or to change their strategy and use a different search engine. They are the group least likely to engage in backward-oriented behavior like clicking the backbutton to browse back or return to previous search engine result. Such backward-oriented behavior is very common for the less experienced users, with double novices showing it most often. It is not fully clear, if novices browse less usefull material than the experts, but once they face a dead end their only way out is to go backwards, while experts have more flexible ways fo reacting.
From the proxy logfile one can reconstruct how much time has elapsed between page transmissions. These intervals cannot be translated into steps of our search model equally well for all steps. For example, when a user refines an initial query, the corresponding interval between page loadings contains three steps: reviewing the initial result set, generating terms, and formatting the revised query, and submission of that query. This makes an analysis of time spent in direct search engine interaction difficult and statistical results for these measures are not so clear.
This problem is far less pronounced for the timing of content pages. Consequently, the stronger differences could be established for the time users spent with content pages (Figure 8).
Figure 8:Time spent with Web pages. (Web +/- refers to Web expertise, Econo +/- refers to domain knowledge)
For the time spent with a document that was directly selected from a page of search engine results, we find a clear independent main effect of domain knowledge (MANOVA: F=11.44, p<.003). People with considerable background knowledge about the domain spent significantly less time with a document from that domain. It takes them less time to read it and make a decision about the next move. Descriptively, Web expertise also reduces the time spent in content documents, but this effect was not found to be significant.
No significant differences between the groups were found for pages accessed during browsing episodes. Quite likely this can be attributed to the nature of pages included in this category. The category not only includes content pages relevant to the domain of economics, but also a number of function pages like navigation pages (which lead to content pages, but do not contain longer paragraphs of task-related information) and even search engine help pages. This may have diminished the influence of domain knowledge on reading times for these kinds of documents. Descriptively we again find shorter reading times for Web experts, but no significant effect. Thus the influence of Web expertise on the time measure is rather weak, but less dependent on whether or not a topic-related page is accessed.
We found the same general pattern of query formulations for both the web-based search and the simulated search tasks, with the data from the search simulations being somewhat more clear-cut (Table 2).
Table 2: Query formatting in Simulated Search tasks (percent of all queries).
Web experts relied significantly more on query formatting tools than Web novices (87 % vs. 47 %), while higher domain knowledge corresponded to a lower number of Boolean operators and modifiers being used. A very clear effect of Web expertise was found for the number of queries with formatting errors (19.6% vs. 1.9%).
Effects of the experimental conditions could also be established for the number of search terms per query, and the sources of search terms, but only in the searches actually performed on the Web, not in the simulated search.
From the expert study one would have expected Web experts to use longer queries. This hypothesis was not confirmed: the queries issued by Web experts were only marginally longer than those of Web novices (2.61 vs. 2.32 words/query). Instead we found a significant effect of domain knowledge: Participants with little domain knowledge made significantly longer queries (average query length: 2.96 vs 1.97 words). Maybe domain experts know more appropriate terms and hence need fewer of them.
The analysis of query formatting (Table 2) revealed that participants who know a lot about the subject domain, but lack Web expertise are quite reluctant to use query formatting (see above). But they seem to compensate for this by showing more verbal creativity and flexibility than the other groups: They most likely used their own terminology instead of relying on the words that were already in the original task statement. Also they more often than others used completely different terminology from one query to the next.
In the Expert study we investigated how Internet professional conceptualize the search process and derived a process model of search engine interaction. This model was first applied to the search behavior of the same Internet professionals and we believe that it has shown its value as a tool for capturing expert searching behavior.
In the second study, the EURO study, we focus on a direct comparison of expert and novice web searchers. It turns out that the process model can be applied to the behavior of both expert and novice searchers and that it also captures differences between these groups.
Expertise was further differentiated into technical Web expertise and domain-specific background knowledge, in this case the field of economics. The two types of expertise have shown independent and combined effects. Participants which could rely on both types of expertise were overall most successful in their search behavior. Deficits in one or the other type of expertise led to compensatory behavior, for example, domain-expert/web-novices relying heavily on terminology and avoiding query formatting. Participants with lower levels of knowledge are less flexible in their strategies and return to previous stages of their search more often rather than trying new approaches (like changing the search engine).
The severe troubles that the "double novices" faced when dealing with the tasks in the EURO study again point at the joint contribution that both domain-knowledge and Internet expertise make to the search process.
Overall Web-based information seeking turned out to be rather difficult for the participants of both Experiment 1 and Experiment 2. This indicates that there still is much room for improvement in Web-based searching. The behavioral differences we found between the experimental groups clearly show that search engine users are a heterogenous crowd and may need to be catered to differently. Novice users had severe problems with formulating a reasonable query and tools that support the query formulation process would seem desirable.
Because successful search on the Web turns out to be so difficult for novice users, learning how to use search engines efficiently should be a central part of any Internet skills training. Novices in our study were ignorant about a number of core problems of Web searching, e.g. the limited scope of individual search engines or the necessity to state a search query at an adequate level of specificity. The differences found between the Web novices and the Web experts point at specific deficiencies in the novices' knowledge and could be directly addressed in Internet skills training.
The first author is a Ph.D. student in the Virtual PhD Program (VGK: Knowledge Acquisition and Knowledge Exchange with New Media - see http://www.vgk.de for details) and the work was funded by the German Research Council (DFG). We would like to thank Gruner+Jahr EMS, Hamburg for supporting the expert study and giving us access to the Fireball statistics reported in this paper.
 Byrne, M. D., John, B. E., Wehrle, N. S., & Crow, D. C. (1999). The tangled web we wove: A taskonomy of WWW use. In Human Factors in Computing Systems: Proceedings of CHI 99 (pp. 544-551). Reading, MA: Addison Wesley.
 Catledge, L. D., & Pitkow, J. E. (1995). Characterizing Browsing Strategies in the World-Wide Web. Computer Networks and ISDN Systems, 27, 1065-1073.
 Choo, C. W., Detlor, B., & Turnbull, D. (1999). Information Seeking on the Web - An integrated model of browsing and searching. Proceedings of the Annual Meeting of the American Society for Information Science (ASIS). [WWW document]. Available: URL http://choo.fis.utoronto.ca/fis/respub/aisis99/
 Cockburn, A., & Jones, S. (1996). Which way now? Analysing and easing inadequacies in WWW navigation. International Journal of Human-Computer Studies, 45, 105-129.
 Ericsson, K. A. & Simon, H. A. (1993). Protocol analysis: verbal reports as data. Cambridge, MA: MIT Press.
 Hsieh-Yee, I. (1993). Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. Journal of the American Society for Information Science, 45(3), 161-174.
 Hurtienne, J., & Wandke, H. (1997). Wie effektiv und effizient navigieren Benutzer im World Wide Web? Eine empirische Studie [How effectively and efficiently do users navigate in the WWW? An empirical study]. In D. Janetzko, B. Batanic, D. Schoder, M. Mattingley-Scott & G. Strube (Eds.), CAW-97: Workshop Cognition & Web. (pp. 93-104). Freiburg. IIG Berichte 1/97
 Jansen, B. J., Spink, A., Bateman, J., & Saracevic, T. (1998). Real life information retrieval: A study of user queries on the Web. SIGIR Forum, 32(1), 5-17.
 Janetzko, Dietmar (1998). Card-sorting and knowledge elicitation. unpublished manuscript.
 Lau, T., & Horvitz, E. (1999). Patterns of Search: Analyzing and Modeling Web Query Dynamics. Proceedings of the Seventh International Conference User Modelling - UM99, 119-128.
 Marchionini, G., Dwiggins, S., Katz, A., & Lin, X. (1993). Information Seeking in Full-Text End-User-Oriented Search Systems: The Roles of Domain and Search Expertise. LISR, 15, 35-69.
 Navarro-Prieto, R., Scaife, M., & Rogers, Y. (1999). Cognitive Strategies in Web Searching. Proceedings of the 5th Conference on Human Factors & the Web, June 1999. [WWW document] Available: URL http://zing.ncsl.nist.gov/hfweb/proceedings/navarro-prieto/index.html
 Pollock, A., & Hockley, A. (1997). What's wrong with Internet Searching. D-Lib Magazine.[WWW document]. Available: URL http://www.dlib.org/dlib/march97/bt/03pollock.html
 Shneidermann, B., Byrd, D., & Croft, W.B., (1997). Clarifying Search: A User-Interface Framework for Text Searches. D-Lib Magazine. [WWW document] Available: URL http://www.dlib.org/dlib/january97/retrieval/01shneiderman.html
 Silverstein,C., Henzinger, M., Marais, H., & Moricz, M. (1998). Analysis of a very large Alta Vista Query Log (SRC Technical Note 1998-014). Palo Alto, CA: digital Systems Research Center.
 Strube, G., Janetzko, D., & Knauff, M. (1996). Cooperative construction of expert knowledge: the case of knowledge engineering. In P. B. Baltes & U. M. Staudinger (Eds.), Interactive minds (pp. 366-393). Cambridge: Cambridge University Press.
 Tauscher, L., & Greenberg, S. (1997). How people revisit web pages: Empirical findings and implications for the design of history systems. International Journal of Human-Computer Studies, 47, 97-137.
 Zukerman, I., Albrecht, D. W., & Nicholson, A. E. (1999). Predicting Users' Requests on the WWW. User Modeling: Proceedings of the Seventh International Conference, UM99, 275-284.