An AI-based Approach for Grading Students' Collaboration

Soft skills (such as communication and collaboration) are rarely addressed in programming courses, mostly because they are difficult to teach, assess, and grade. A quantitative, modular, AI-based approach for assessing and grading students' collaboration has been examined in this paper. The ped...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	IEEE transactions on learning technologies Ročník 16; číslo 3; s. 1 - 15
Hlavní autoři:	Tomic, Bojan B., Kijevcanin, Anisja D., Sevarac, Zoran V., Jovanovic, Jelena M.
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	Piscataway IEEE 01.06.2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:	Automatic assessment tools Codes Collaboration Computer Assisted Instruction computer science education Cooperation Electronic mail Fuzzy sets fuzzy systems Grading Java Machine learning Measurement Pedagogy Programming Scoring Rubrics Soft Skills Software engineering Students Teachers Teaching Methods Teamwork Training
ISSN:	1939-1382, 2372-0050
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Abstract	Soft skills (such as communication and collaboration) are rarely addressed in programming courses, mostly because they are difficult to teach, assess, and grade. A quantitative, modular, AI-based approach for assessing and grading students' collaboration has been examined in this paper. The pedagogical underpinning of the approach includes a pedagogical framework and a quantitative soft skill assessment rubric, which have been adapted and used in an extracurricular Java programming course. The objective was to identify pros and cons of using different AI methods within this approach when it comes to assessing and grading collaboration in group programming projects. More specifically, fuzzy rules and several machine learning methods (ML onward) have been examined to see which one would yield the best results regarding performance, interpretability/explainability of recommendations and feasibility/practicality. The data used for training and testing span four academic years, and the results suggest that almost all of the examined AI methods, when used within the proposed AI-based approach, can provide adequate grading recommendations as long as teachers cover other aspects of the assessment not covered by the rubrics: code quality, plagiarism, and project completion. The fuzzy-rule-based method requires time and effort to be spent on (manual) creation and tuning of fuzzy rules and sets, whereas the examined ML methods require lesser initial investments but do need historical data for training. On the other hand, the fuzzy-rule-based method can provide the best explanations on how the assessment/grading was made - something that proved to be very important to teachers.
AbstractList	Soft skills (such as communication and collaboration) are rarely addressed in programming courses, mostly because they are difficult to teach, assess, and grade. A quantitative, modular, AI-based approach for assessing and grading students' collaboration has been examined in this article. The pedagogical underpinning of the approach includes a pedagogical framework and a quantitative soft skill assessment rubric, which have been adapted and used in an extracurricular Java programming course. The objective was to identify pros and cons of using different AI methods within this approach when it comes to assessing and grading collaboration in group programming projects. More specifically, fuzzy rules and several machine learning methods (ML onward) have been examined to see which one would yield the best results regarding performance, interpretability/explainability of recommendations, and feasibility/practicality. The data used for training and testing span four academic years, and the results suggest that almost all of the examined AI methods, when used within the proposed AI-based approach, can provide adequate grading recommendations as long as teachers cover other aspects of the assessment not covered by the rubrics: code quality, plagiarism, and project completion. The fuzzy-rule-based method requires time and effort to be spent on (manual) creation and tuning of fuzzy rules and sets, whereas the examined ML methods require lesser initial investments but do need historical data for training. On the other hand, the fuzzy-rule-based method can provide the best explanations on how the assessment/grading was made—something that proved to be very important to teachers. Soft skills (such as communication and collaboration) are rarely addressed in programming courses, mostly because they are difficult to teach, assess, and grade. A quantitative, modular, AI-based approach for assessing and grading students' collaboration has been examined in this paper. The pedagogical underpinning of the approach includes a pedagogical framework and a quantitative soft skill assessment rubric, which have been adapted and used in an extracurricular Java programming course. The objective was to identify pros and cons of using different AI methods within this approach when it comes to assessing and grading collaboration in group programming projects. More specifically, fuzzy rules and several machine learning methods (ML onward) have been examined to see which one would yield the best results regarding performance, interpretability/explainability of recommendations and feasibility/practicality. The data used for training and testing span four academic years, and the results suggest that almost all of the examined AI methods, when used within the proposed AI-based approach, can provide adequate grading recommendations as long as teachers cover other aspects of the assessment not covered by the rubrics: code quality, plagiarism, and project completion. The fuzzy-rule-based method requires time and effort to be spent on (manual) creation and tuning of fuzzy rules and sets, whereas the examined ML methods require lesser initial investments but do need historical data for training. On the other hand, the fuzzy-rule-based method can provide the best explanations on how the assessment/grading was made - something that proved to be very important to teachers.
Author	Tomic, Bojan B. Sevarac, Zoran V. Jovanovic, Jelena M. Kijevcanin, Anisja D.
Author_xml	– sequence: 1 givenname: Bojan B. surname: Tomic fullname: Tomic, Bojan B. organization: Department of Software Engineering, Faculty of Organizational Sciences, University of Belgrade, Serbia – sequence: 2 givenname: Anisja D. surname: Kijevcanin fullname: Kijevcanin, Anisja D. organization: Department of Software Engineering, Faculty of Organizational Sciences, University of Belgrade, Serbia – sequence: 3 givenname: Zoran V. surname: Sevarac fullname: Sevarac, Zoran V. organization: Department of Software Engineering, Faculty of Organizational Sciences, University of Belgrade, Serbia – sequence: 4 givenname: Jelena M. orcidid: 0000-0002-1904-0446 surname: Jovanovic fullname: Jovanovic, Jelena M. organization: Department of Software Engineering, Faculty of Organizational Sciences, University of Belgrade, Serbia
BookMark	eNp9kDFPwzAQRi1UJNrCjsQSiYEpxT7Hjj1GFZRKlRgos-U4NqQKcbGdgX9PSisGBqZbvnff3ZuhSe97i9A1wQtCsLzfbrYLwAALCsAKCmdoCrSEHGOGJ2hKJJU5oQIu0CzGHcYcSglTJKo-q9Z5raNtsmq_D16b98z5kK2Cbtr-LXtJQ2P7FO-ype86XfugU-v7S3TudBft1WnO0evjw3b5lG-eV-tltckNJTTl4LTkjltuJK0FI1oWRhorhGYCE6iFrAUpdIE14YZokMJRV4pGGFZTbgmdo9vj3vG0z8HGpHZ-CP1YqUAAL8pClmJM4WPKBB9jsE7tQ_uhw5ciWB38qNGPOvhRJz8jwv8gpk0_r6Wg2-4_8OYIttba3x4pOWOS028SdXIw
CODEN	ITLTAT
CitedBy_id	crossref_primary_10_1109_TLT_2024_3456447 crossref_primary_10_1177_10464964241279164 crossref_primary_10_1016_j_caeai_2025_100451 crossref_primary_10_1016_j_caeai_2024_100355 crossref_primary_10_1177_00472395241231815 crossref_primary_10_30831_akukeg_1644354 crossref_primary_10_1109_TLT_2023_3324714 crossref_primary_10_1080_10447318_2024_2338330 crossref_primary_10_1109_ACCESS_2025_3608158 crossref_primary_10_3390_math13172828 crossref_primary_10_1177_01626434231165977 crossref_primary_10_1080_10999922_2024_2414957 crossref_primary_10_1088_1742_6596_3105_1_012017 crossref_primary_10_1080_03634523_2025_2477479 crossref_primary_10_1016_j_caeo_2025_100278 crossref_primary_10_1038_s41598_025_93052_y
Cites_doi	10.1109/FIE.2015.7344177 10.1016/j.jss.2020.110613 10.9790/0837-19560103 10.1007/978-3-319-91602-6_4 10.1109/FIE.2016.7757406 10.1145/2445196.2445351 10.1145/3134744 10.1016/j.jss.2013.12.037 10.1109/FIE.2006.322591 10.3991/ijep.v5i1.4047 10.18543/tjhe-3(2)-2016pp389-427 10.2307/249686 10.1080/14703297.2015.1105754 10.1007/s11412-015-9226-y 10.1109/ICSE.2007.40 10.18608/jla.2021.7227 10.1007/BF01257890 10.1080/18756891.2013.818190 10.1613/jair.953 10.1080/08957347.2018.1495212 10.17140/PCSOJ-4-e011 10.1109/CHASE.2013.6614749 10.3389/fcomp.2021.728801 10.1109/MITP.2012.7 10.1016/j.ijer.2014.07.001 10.1007/s10618-012-0295-5 10.1109/APSECW.2017.19 10.1109/FIE.2018.8658983
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID	97E RIA RIE AAYXX CITATION
DOI	10.1109/TLT.2022.3225432
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef
DatabaseTitle	CrossRef
DatabaseTitleList
Database_xml	– sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Education
EISSN	2372-0050
EndPage	15
ExternalDocumentID	10_1109_TLT_2022_3225432 9965596
Genre	orig-research
GroupedDBID	0R~ 29I 5GY 5VS 6IK 97E AAJGR AAKDD AAKPC AASAJ AAWTH ABJNI ABOPQ ABQJQ ABVLG ACGFO ACHQT ACIWK ADDVE AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD HZ~ IFIPE IPLJI JAVBF M43 O9- OCL P2P PQQKQ RIA RIE RNS RZB AAYXX CITATION AARMG ABAZT
ID	FETCH-LOGICAL-c313t-2fa96f6e6c93b851a94c9ce88a58012b89b814a40a16c1a298f3f78d8c5b36e13
IEDL.DBID	RIE
ISICitedReferencesCount	14
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001012684000001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1939-1382
IngestDate	Mon Jun 30 06:34:45 EDT 2025 Tue Nov 18 22:11:36 EST 2025 Sat Nov 29 06:08:00 EST 2025 Tue Nov 25 14:44:28 EST 2025
IsPeerReviewed	true
IsScholarly	true
Issue	3
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c313t-2fa96f6e6c93b851a94c9ce88a58012b89b814a40a16c1a298f3f78d8c5b36e13
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-1904-0446 0000-0001-5292-5804 0000-0001-9418-6915 0000-0001-6853-4322
PQID	2826474978
PQPubID	85505
PageCount	15
ParticipantIDs	crossref_primary_10_1109_TLT_2022_3225432 crossref_citationtrail_10_1109_TLT_2022_3225432 proquest_journals_2826474978 ieee_primary_9965596
PublicationCentury	2000
PublicationDate	2023-06-01
PublicationDateYYYYMMDD	2023-06-01
PublicationDate_xml	– month: 06 year: 2023 text: 2023-06-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE transactions on learning technologies
PublicationTitleAbbrev	TLT
PublicationYear	2023
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref15 ref37 ref31 ref30 ref33 ref2 binkley (ref12) 2011 ref1 (ref10) 0 ref17 ref39 ref16 ref38 (ref9) 0 (ref35) 0 ref19 ref18 molnar (ref27) 0 ref24 ref23 lee (ref14) 2012; 15 ref26 ref20 ref42 ref41 ref22 (ref36) 0 beckett (ref40) 2018 (ref11) 0 (ref34) 0 ref28 tomi? (ref25) 2017; 50 ref29 ref8 ref4 ref3 gitinabard (ref21) 2020 ref6 matturro (ref7) 2019; 25 ref5 schneider (ref32) 2016
References_xml	– year: 0 ident: ref35 article-title: GRASS D2.2 supporting document GRASS pedagogical rubric – volume: 50 start-page: 518 year: 2017 ident: ref25 article-title: Grading students' programming and soft skills with open badges: A case study publication-title: Brit J Educ Technol – ident: ref18 doi: 10.1109/FIE.2015.7344177 – ident: ref22 doi: 10.1016/j.jss.2020.110613 – ident: ref1 doi: 10.9790/0837-19560103 – ident: ref6 doi: 10.1007/978-3-319-91602-6_4 – ident: ref20 doi: 10.1109/FIE.2016.7757406 – ident: ref4 doi: 10.1145/2445196.2445351 – ident: ref16 doi: 10.1145/3134744 – ident: ref28 doi: 10.1016/j.jss.2013.12.037 – ident: ref17 doi: 10.1109/FIE.2006.322591 – volume: 15 start-page: 214 year: 2012 ident: ref14 article-title: Peer evaluation in blended team project-Based learning: What do students find important? publication-title: Educ Technol Soc – ident: ref13 doi: 10.3991/ijep.v5i1.4047 – start-page: 409 year: 2020 ident: ref21 article-title: Student teamwork on programming projects: What can GitHub logs show us? publication-title: Proc 13th Int Conf Educ Data Mining – ident: ref8 doi: 10.18543/tjhe-3(2)-2016pp389-427 – ident: ref41 doi: 10.2307/249686 – volume: 25 start-page: 16 year: 2019 ident: ref7 article-title: A systematic mapping study on soft skills in software engineering publication-title: JUCS - Journal of Universal Computer Science – ident: ref15 doi: 10.1080/14703297.2015.1105754 – ident: ref30 doi: 10.1007/s11412-015-9226-y – year: 0 ident: ref11 article-title: 21CLD student work rubrics – year: 0 ident: ref10 article-title: 21CLD learning activity rubrics – year: 0 ident: ref27 article-title: Interpretable machine learning: A guide for making black box models explainable – ident: ref29 doi: 10.1109/ICSE.2007.40 – start-page: 17 year: 2011 ident: ref12 article-title: Defining twenty-first century skills publication-title: Assessment and Teaching of 21st Century Skills – year: 0 ident: ref36 article-title: GRASS D3.2 supporting document didactic materials for GRASS application cases – ident: ref33 doi: 10.18608/jla.2021.7227 – year: 0 ident: ref9 article-title: PISA 2015 draft collaborative problem solving framework – ident: ref42 doi: 10.1007/BF01257890 – ident: ref37 doi: 10.1080/18756891.2013.818190 – year: 0 ident: ref34 article-title: GRASS D2.2 pedagogical approaches – ident: ref38 doi: 10.1613/jair.953 – ident: ref26 doi: 10.1080/08957347.2018.1495212 – ident: ref5 doi: 10.17140/PCSOJ-4-e011 – year: 2018 ident: ref40 article-title: Rfviz: An interactive visualization package for random forests in R – ident: ref3 doi: 10.1109/CHASE.2013.6614749 – ident: ref31 doi: 10.3389/fcomp.2021.728801 – ident: ref2 doi: 10.1109/MITP.2012.7 – start-page: 99 year: 2016 ident: ref32 article-title: Real-time mutual gaze perception enhances collaborative learning and collaboration quality publication-title: Educational Media and Technology Yearbook – ident: ref19 doi: 10.1016/j.ijer.2014.07.001 – ident: ref39 doi: 10.1007/s10618-012-0295-5 – ident: ref24 doi: 10.1109/APSECW.2017.19 – ident: ref23 doi: 10.1109/FIE.2018.8658983
SSID	ssj0062792
Score	2.3879402
Snippet	Soft skills (such as communication and collaboration) are rarely addressed in programming courses, mostly because they are difficult to teach, assess, and...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	1
SubjectTerms	Automatic assessment tools Codes Collaboration Computer Assisted Instruction computer science education Cooperation Electronic mail Fuzzy sets fuzzy systems Grading Java Machine learning Measurement Pedagogy Programming Scoring Rubrics Soft Skills Software engineering Students Teachers Teaching Methods Teamwork Training
Title	An AI-based Approach for Grading Students' Collaboration
URI	https://ieeexplore.ieee.org/document/9965596 https://www.proquest.com/docview/2826474978
Volume	16
WOSCitedRecordID	wos001012684000001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE/IET Electronic Library (IEL) (UW System Shared) customDbUrl: eissn: 2372-0050 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0062792 issn: 1939-1382 databaseCode: RIE dateStart: 20080101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwGP3Yhg--eJvidEofBBHM1jZtmjyW4VQYQ3DC3kpzA0E62cXfb5KmQ1EE3_qQ0HJyOd_XfDkH4CpSOjU0HyIdaoGsgBeiSSSQkqEw7GeSIu50ZifZdErnc_bUgtvtXRillCs-UwP76M7y5UJs7K-yoYnNTQBM2tDOMlLf1Wp2XWKF8JpjyJANZ5OZSf7ieGAnbILjb7TjfFR-bL6OUcb7__uWA9jzkWOQ10N9CC1VHVnTZV-g0QWaV0H-iCwvySD3WuGBCUqD-6WrlA-eayHL1XUw-jr8x_AyvpuNHpA3RkACR3iNYl0yookigmFuQqaSJYIJRWmZWsLhlHEaJWUSlhERURkzqrHOqKQi5ZioCJ9Ap1pU6hQCbjIEQQgTqbRqYJpyFUosWcyleYEUPRg2uBXCq4Zb84q3wmUPISsM0oVFuvBI9-Bm2-O9Vsz4o23XIrtt50HtQb8ZmsIvqVVhckOSZNYQ7-z3Xuewa73g6zquPnTWy426gB3xsX5dLS_dbPkE13O8yA
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwGP2YU9AXb1OcTs2DIILZes2SxzKcG9YhWGFvpU1SEKSTXfz9Jmk6FEXwrQ8JLSeX833Nl3MArlxZhIrmHVw4BcdawAvTwOVYCocr9lNJUW50ZuP-ZEKnU_bUgNv1XRgppSk-k139aM7yxYyv9K-ynorNVQBMNmBTO2fZ21r1vku0FF59EOmwXhInKv3zvK6esoHvfSMe46TyY_s1nDLc-9_X7MOujR1RVA32ATRkeahtl22JRgtoVKJojDUzCRRZtXCkwlJ0Pze18ui5krJcXKPB1wlwBC_Du2QwwtYaAXPf9ZfYKzJGCiIJZ36ugqaMBZxxSWkWasrJKcupG2SBk7mEu5nHaOEXfSooD3OfSNc_hmY5K-UJoFzlCJwQxkOh9cAKmktH-IJ5uVAvELwNvRq3lFvdcG1f8Zaa_MFhqUI61UinFuk23Kx7vFeaGX-0bWlk1-0sqG3o1EOT2kW1SFV2SIK-tsQ7_b3XJWyPksc4jceThzPY0c7wVVVXB5rL-Uqewxb_WL4u5hdm5nwC2j7AEQ
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+AI-based+Approach+for+Grading+Students%E2%80%99+Collaboration&rft.jtitle=IEEE+transactions+on+learning+technologies&rft.au=Tomi%C4%87%2C+Bojan+B.&rft.au=Kijev%C4%8Danin%2C+Anisja+D.&rft.au=%C5%A0evarac%2C+Zoran+V.&rft.au=Jovanovi%C4%87%2C+Jelena+M.&rft.date=2023-06-01&rft.issn=1939-1382&rft.eissn=2372-0050&rft.volume=16&rft.issue=3&rft.spage=292&rft.epage=305&rft_id=info:doi/10.1109%2FTLT.2022.3225432&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TLT_2022_3225432
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1939-1382&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1939-1382&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1939-1382&client=summon