TEXT: Automatic Template Extraction from Heterogeneous Web Pages

World Wide Web is the most useful source of information. In order to achieve high productivity of publishing, the webpages in many websites are automatically populated by using the common templates with contents. The templates provide readers easy access to the contents guided by consistent structur...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on knowledge and data engineering Vol. 23; no. 4; pp. 612 - 626
Main Authors: Kim, Chulyun, Shim, Kyuseok
Format: Journal Article
Language:English
Published: New York, NY IEEE 01.04.2011
IEEE Computer Society
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Applied sciences
Artificial intelligence
Clustering
Clustering algorithms
Clusters
Computer science; control theory; systems
Computer systems and distributed systems. User interface
Data mining
Data models
Data processing. List processing. Character string processing
Exact sciences and technology
HTML
Information systems. Data bases
Memory organisation. Data processing
Merging
MinHash
minimum description length principle
Readers
Search engines
Similarity
Software
Speech and sound recognition and synthesis. Linguistics
Studies
Template extraction
Web pages
Websites
World Wide Web
XML
Electronic document
Information extraction
Information source
Classification
World wide web
Robustness
Document analysis
Pattern extraction
Productivity
Content access
Template extraction
Useful information
Search engine
Cluster
Information retrieval
Text
Case based reasoning
minimum description length principle
Document structure
Minimum principle
clustering
Internet
Web site
MinHash
Algorithm analysis
ISSN:1041-4347, 1558-2191
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:World Wide Web is the most useful source of information. In order to achieve high productivity of publishing, the webpages in many websites are automatically populated by using the common templates with contents. The templates provide readers easy access to the contents guided by consistent structures. However, for machines, the templates are considered harmful since they degrade the accuracy and performance of web applications due to irrelevant terms in templates. Thus, template detection techniques have received a lot of attention recently to improve the performance of search engines, clustering, and classification of web documents. In this paper, we present novel algorithms for extracting templates from a large number of web documents which are generated from heterogeneous templates. We cluster the web documents based on the similarity of underlying template structures in the documents so that the template for each cluster is extracted simultaneously. We develop a novel goodness measure with its fast approximation for clustering and provide comprehensive analysis of our algorithm. Our experimental results with real-life data sets confirm the effectiveness and robustness of our algorithm compared to the state of the art for template detection algorithms.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ObjectType-Article-2
ObjectType-Feature-1
content type line 23
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2010.140