Assessing Teamwork Skills: Can a Computer Algorithm Match Human Experts?

Teamwork skills are commonly evaluated by human assessors, which can be logistically challenging and resource intensive. Technological advancements provide an opportunity for a new assessment method – virtual behavioural simulations with self-scoring algorithms. This study explores whether a rule-ba...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International journal of artificial intelligence in education Jg. 33; H. 4; S. 955 - 991
Hauptverfasser: Kotlyar, Igor, Sharifi, Tina, Fiksenbaum, Lisa
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York Springer New York 01.12.2023
Springer Nature B.V
Schlagworte:
Algorithms
Artificial Intelligence
Behavioral Objectives
Chatbots
Class Size
College Students
Communication (Thought Transfer)
Computer Science
Computer simulation
Computers and Education
Educational Technology
Mathematics
Natural language
Natural language (computers)
Observation
Secondary Education
Self Evaluation (Individuals)
Semantics
Simulated Environment
Simulation
Skills
Stimuli
Student Evaluation
Students
Teams
Teamwork
Undergraduate study
User Interfaces and Human Computer Interaction
Verbal communication
Simulations
Computer algorithm
Teamwork skills
Assessment
ISSN:1560-4292, 1560-4306
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Teamwork skills are commonly evaluated by human assessors, which can be logistically challenging and resource intensive. Technological advancements provide an opportunity for a new assessment method – virtual behavioural simulations with self-scoring algorithms. This study explores whether a rule-based algorithm can match human assessors at evaluating teamwork skills. 206 undergraduate students completed a virtual simulation assessment, where they interacted with “teammates” (represented by chatbots) using natural language. In this study, students’ teamwork skills were assessed independently by a computer algorithm and two human experts based on the transcripts of their conversations with “teammates” (chatbots). The relative accuracy of these assessments was evaluated against peer- and self-evaluations of teamwork. The assessment scores generated by the algorithm and human experts were highly correlated with each other and were comparable in their ability to predict teamwork. The scores generated by the algorithm were slightly more correlated with peer-evaluations than those generated by human experts ( r  = .25 and r  = .17, respectively; p  = .21). The results indicate that AI-based techniques offer a promising method of skill assessment to support learning and acquisition teamwork skills.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1560-4292
1560-4306
DOI:10.1007/s40593-022-00318-x