Document indexing: a concept-based approach to term weight estimation

Traditional index weighting approaches for information retrieval from texts depend on the term frequency based analysis of the text contents. A shortcoming of these indexing schemes, which consider only the occurrences of the terms in a document, is that they have some limitations in extracting sema...

Full description

Saved in:
Bibliographic Details
Published in:Information processing & management Vol. 41; no. 5; pp. 1065 - 1080
Main Authors: Kang, Bo-Yeong, Lee, Sang-Jo
Format: Journal Article
Language:English
Published: Oxford Elsevier Ltd 01.09.2005
Elsevier Science
Elsevier Science Ltd
Subjects:
Automatic indexing
Cluster analysis
Concepts
Data mining
Exact sciences and technology
Formalism
Index weight estimation
Indexes
Indexing
Information and communication sciences
Information processing and retrieval
Information retrieval
Information retrieval. Man machine relationship
Information science. Documentation
Lexical semantics
Lexicon
Online information retrieval
Query formulation
Ranking
Research process. Evaluation
Retrieval
Sciences and techniques of general use
Search output
Semantics
Studies
Terms
Wall Street
Weighting
Weighting function
United States > US
Information retrieval
Index weight estimation
Weighting function
Automatic indexing
Automatic classification
Weighting
Estimation
Cluster
Experimentation
Algorithm
Index (documentation)
ISSN:0306-4573, 1873-5371
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Traditional index weighting approaches for information retrieval from texts depend on the term frequency based analysis of the text contents. A shortcoming of these indexing schemes, which consider only the occurrences of the terms in a document, is that they have some limitations in extracting semantically exact indexes that represent the semantic content of a document. To address this issue, we developed a new indexing formalism that considers not only the terms in a document, but also the concepts. In this approach, concept clusters are defined and a concept vector space model is proposed to represent the semantic importance degrees of lexical items and concepts within a document. Through an experiment on the TREC collection of Wall Street Journal documents, we show that the proposed method outperforms an indexing method based on term frequency (TF), especially in regard to the few highest-ranked documents. Moreover, the index term dimension was 80% lower for the proposed method than for the TF-based method, which is expected to significantly reduce the document search time in a real environment.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
content type line 23
ISSN:0306-4573
1873-5371
DOI:10.1016/j.ipm.2004.08.005