Associating Facial Expressions and Upper-Body Gestures with Learning Tasks for Enhancing Intelligent Tutoring Systems

Learning involves a substantial amount of cognitive, social and emotional states. Therefore, recognizing and understanding these states in the context of learning is key in designing informed interventions and addressing the needs of the individual student to provide personalized education. In this...

Full description

Saved in:
Bibliographic Details
Published in:International journal of artificial intelligence in education Vol. 30; no. 2; pp. 236 - 270
Main Authors: Behera, Ardhendu, Matthew, Peter, Keidel, Alexander, Vangorp, Peter, Fang, Hui, Canning, Susan
Format: Journal Article
Language:English
Published: New York Springer New York 01.06.2020
Springer
Springer Nature B.V
Subjects:
Algorithms
Artificial Intelligence
Behavior Patterns
Cognitive tasks
Computer Science
Computer Science Education
Computer vision
Computers and Education
Context
Deep learning
Difficulty Level
Distance learning
Education
Educational Environment
Educational Researchers
Educational Strategies
Educational Technology
Emotional factors
Emotions
Eye Movements
Head
Image classification
Intelligent Tutoring Systems
Learner Engagement
Learning Processes
Nonverbal Communication
Online instruction
Problem Solving
Students
Teaching
Teaching Methods
Tutoring
Undergraduate Students
User Interfaces and Human Computer Interaction
Video Technology
Neural models applied to AIED systems
Adaptive and intelligent multimedia and hypermedia systems
Intelligent Tutoring Systems (ITSs)
Computer-supported collaborative learning
ISSN:1560-4292, 1560-4306
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Learning involves a substantial amount of cognitive, social and emotional states. Therefore, recognizing and understanding these states in the context of learning is key in designing informed interventions and addressing the needs of the individual student to provide personalized education. In this paper, we explore the automatic detection of learner’s nonverbal behaviors involving hand-over-face gestures, head and eye movements and emotions via facial expressions during learning. The proposed computer vision-based behavior monitoring method uses a low-cost webcam and can easily be integrated with modern tutoring technologies. We investigate these behaviors in-depth over time in a classroom session of 40 minutes involving reading and problem-solving exercises. The exercises in the sessions are divided into three categories: an easy, medium and difficult topic within the context of undergraduate computer science. We found that there is a significant increase in head and eye movements as time progresses, as well as with the increase of difficulty level. We demonstrated that there is a considerable occurrence of hand-over-face gestures (on average 21.35%) during the 40 minutes session and is unexplored in the education domain. We propose a novel deep learning approach for automatic detection of hand-over-face gestures in images with a classification accuracy of 86.87%. There is a prominent increase in hand-over-face gestures when the difficulty level of the given exercise increases. The hand-over-face gestures occur more frequently during problem-solving (easy 23.79%, medium 19.84% and difficult 30.46%) exercises in comparison to reading (easy 16.20%, medium 20.06% and difficult 20.18%).
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1560-4292
1560-4306
DOI:10.1007/s40593-020-00195-2