BabyTalk: Understanding and Generating Simple Image Descriptions

We present a system to automatically generate natural language descriptions from images. This system consists of two parts. The first part, content planning, smooths the output of computer vision-based detection and recognition algorithms with statistics mined from large pools of visually descriptiv...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on pattern analysis and machine intelligence Ročník 35; číslo 12; s. 2891 - 2903
Hlavní autori: Kulkarni, Girish, Premraj, Visruth, Ordonez, Vicente, Dhar, Sagnik, Siming Li, Yejin Choi, Berg, Alexander C., Berg, Tamara L.
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Los Alamitos, CA IEEE 01.12.2013
IEEE Computer Society
Predmet:
Algorithms
Applied sciences
Artificial intelligence
Computer science; control theory; systems
Computer vision
Context awareness
Exact sciences and technology
Humans
image description generation
Image segmentation
Information analysis
Natural language processing
Pattern recognition. Digital image processing. Computational geometry
Speech and sound recognition and synthesis. Linguistics
Computer vision
Description language
Computer output
Statistical analysis
Similarity
Image processing
Image content
Visual servoing
Text
image description generation
Natural scenes
Natural language
Metric
Planning
Sentence
ISSN:0162-8828, 1939-3539, 2160-9292, 1939-3539
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:We present a system to automatically generate natural language descriptions from images. This system consists of two parts. The first part, content planning, smooths the output of computer vision-based detection and recognition algorithms with statistics mined from large pools of visually descriptive text to determine the best content words to use to describe an image. The second step, surface realization, chooses words to construct natural language sentences based on the predicted content and general statistics from natural language. We present multiple approaches for the surface realization step and evaluate each using automatic measures of similarity to human generated reference descriptions. We also collect forced choice human evaluations between descriptions from the proposed generation system and descriptions from competing approaches. The proposed system is very effective at producing relevant sentences for images. It also generates descriptions that are notably more true to the specific image content than previous work.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0162-8828
1939-3539
2160-9292
1939-3539
DOI:10.1109/TPAMI.2012.162