A Neuro-Symbolic ASP Pipeline for Visual Question Answering

We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) training neural networks for object classification and bounding-box prediction of...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Theory and practice of logic programming Ročník 22; číslo 5; s. 739 - 754
Hlavní autori: EITER, THOMAS, HIGUERA, NELSON, OETSCH, JOHANNES, PRITZ, MICHAEL
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Cambridge, UK Cambridge University Press 01.09.2022
Predmet:
Answer set programming
Computation theory
Datasets
Determinism
Deterministics
Encoding (symbols)
Encodings
Forecasting
Logic programming
Natural language processing
Network prediction
Neural networks
neuro-symbolic computation
Object classification
Original Article
Pipelines
Program translators
Question Answering
Questions
Statistical analysis
Symbolic computation
visual question answering
neuro-symbolic computation
answer-set programming
visual question answering
ISSN:1471-0684, 1475-3081, 1475-3081
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) training neural networks for object classification and bounding-box prediction of the CLEVR scenes, (ii) statistical analysis on the distribution of prediction values of the neural networks to determine a threshold for high-confidence predictions, and (iii) a translation of CLEVR questions and network predictions that pass confidence thresholds into logic programmes so that we can compute the answers using an answer-set programming solver. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect. Furthermore, we show that restricting non-determinism to reasonable choices allows for more efficient implementations in comparison with related neuro-symbolic approaches without losing much accuracy.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1471-0684
1475-3081
1475-3081
DOI:10.1017/S1471068422000229