An encoder-decoder based framework for hindi image caption generation

In recent times, research activity on image caption generation has attracted several researchers. The present work attempt to address the problem of Hindi image caption generation using Hindi Visual genome dataset. Hindi is the official and most spoken language in India. In a linguistically diverse...

Full description

Saved in:
Bibliographic Details
Published in:Multimedia tools and applications Vol. 80; no. 28-29; pp. 35721 - 35740
Main Authors: Singh, Alok, Singh, Thoudam Doren, Bandyopadhyay, Sivaji
Format: Journal Article
Language:English
Published: New York Springer US 01.11.2021
Springer Nature B.V
Subjects:
1166: Advances of machine learning in data analytics and visual information processing
Artificial neural networks
Coders
Computer Communication Networks
Computer Science
Data Structures and Information Theory
Datasets
Encoders-Decoders
Genomes
Multimedia Information Systems
Special Purpose and Application-Based Systems
India
CNN
VGG19
Stacked LSTM
VGG16
Hindi image caption generation
ISSN:1380-7501, 1573-7721
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In recent times, research activity on image caption generation has attracted several researchers. The present work attempt to address the problem of Hindi image caption generation using Hindi Visual genome dataset. Hindi is the official and most spoken language in India. In a linguistically diverse country like India, it is essential to provide a means that can help the people to understand the visual entities in their native languages. In this paper, an encoder-decoder based architecture is proposed where Convolutional Neural Network (CNN) is employed for encoding visual features of an image and stacked Long Short-Term Memory (sLSTM) in combination with both uni-directional LSTM and bi-directional LSTM for generating the captions in Hindi. For encoding the visual feature representation of an image, V G G 19 based pre-trained model is used and sLSTM architecture is employed for caption generation at the decoder side. The model is tested over Hindi visual genome dataset to validate the proposed approach’s performance and cross-verification is carried out for English captions with Flickr dataset. The experimental results of the proposed approach manifest that the model is qualitatively and quantitatively better than state-of-the-art approaches for Hindi caption generation.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1380-7501
1573-7721
DOI:10.1007/s11042-021-11106-5