Overview

The goal of web search clustering, a type of web content mining system, is to organize the results of a search into groups of topics. As mentioned in Chapter 1, this is done in order to allow the user to more easily find the desired web pages from among the results by displaying them by topic area rather than as a ranked list. In this chapter we give the details of such a system which we have created. Our system initially used a binary vector representation for web pages and cluster labels, but we upgraded it in a straightforward manner to work with graphs instead, as we will describe below.

Web page clustering as performed by humans was examined by Macskassy et al. [MBDH98], Ten subjects were involved in the experiments, and each was asked to manually cluster the results of five different queries submitted to a web search engine at Rutgers University. The queries were selected from the most popular submitted to this particular web search engine: accounting, career services, employment, library, and off campus housing. All subjects received the pages URLs and titles, however four of the ten subjects were also given the full text of each page for each query. The subjects then clustered the group of documents associated with each query. The investigators examined the size of clusters created, the number of clusters created, the similarity of created clusters, the amount of cluster overlap, and documents not clustered. The results indicated that the size...