View in EDS

Crawling the Hidden Web

Saved in:
Bibliographic Details
Title: Crawling the Hidden Web
Authors: Sriram Raghavan, Hector Garcia-molina
Contributors: The Pennsylvania State University CiteSeerX Archives
Source: http://www.cise.ufl.edu/~cgrant/projects/public/morpheus/files/(32)Crawling the HiddenWeb(2001).pdf.
Publication Year: 2000
Collection: CiteSeerX
Subject Terms: Crawling, Hidden Web, Content extraction, HTML Forms
Description: Current-day crawlers retrieve content only from the publicly indexable Web, i.e., the set of web pages reachable purely by following hypertext links, ignoring search forms and pages that require authorization or prior regis-tration. In particular, they ignore the tremendous amount of high quality content “hidden ” behind search forms, in large searchable electronic databases. In this paper, we provide a framework for addressing the problem of extracting content from this hidden Web. At Stanford, we have built a task-specific hidden Web crawler called the Hidden Web Exposer (HiWE). We describe the architecture of HiWE and present a number of novel tech-niques that went into its design and implementation. We also present results from experiments we conducted to test and validate our techniques.
Document Type: text
File Description: application/pdf
Language: English
Relation: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.517.8573; http://www.cise.ufl.edu/~cgrant/projects/public/morpheus/files/(32)Crawling the HiddenWeb(2001).pdf
Availability: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.517.8573
http://www.cise.ufl.edu/~cgrant/projects/public/morpheus/files/(32)Crawling the HiddenWeb(2001).pdf
Rights: Metadata may be used without restrictions as long as the oai identifier remains attached to it.
Accession Number: edsbas.49E1F0F7
Database: BASE
Description
Abstract:Current-day crawlers retrieve content only from the publicly indexable Web, i.e., the set of web pages reachable purely by following hypertext links, ignoring search forms and pages that require authorization or prior regis-tration. In particular, they ignore the tremendous amount of high quality content “hidden ” behind search forms, in large searchable electronic databases. In this paper, we provide a framework for addressing the problem of extracting content from this hidden Web. At Stanford, we have built a task-specific hidden Web crawler called the Hidden Web Exposer (HiWE). We describe the architecture of HiWE and present a number of novel tech-niques that went into its design and implementation. We also present results from experiments we conducted to test and validate our techniques.