Multimodality in online education: a comparative study - Immadisetty, Praneeta; Rajesh, Pooja; Gupta, Akshita; M. R., Dr. Anala; A., Dr. Soumya; Subramanya, Dr. K. N. (Multimedia tools and applications, 2025)
The commencement of the decade brought along with it a grave pandemic and in response the movement of education forums predominantly into the online world. With a surge in the usage of online video conferencing platforms and tools to better gauge student understanding, there needs to be a mechanism to assess whether instructors can grasp the extent to which students understand the subject and their response to the educational stimuli. The current systems consider only a single cue with a lack of focus in the educational domain. Thus, there is a necessity for the measurement of an all-encompassing holistic overview of the students’ reaction to the subject matter. This paper highlights the need for a multimodal approach to affect recognition and its deployment in the online classroom while considering four cues, posture and gesture, facial, eye tracking and verbal recognition. In this paper, various machine learning models suitable for each cue are analyzed based on key metrics such as accuracy, ease of data collection, sensitivity, and limitations. The comparison of models revealed the most effective techniques for each cue. A multimodal approach derived from weighted majority voting is proposed by combining the top-performing models for each aspect of student engagement in online learning environments. This paper proposes a novel approach that integrates multiple modalities for affect recognition, aiming to improve the accuracy of student response assessments.