Machine Learning-Based Classification of Programming Logic Understanding Levels by Mouse-Tracking Heatmaps

In the past decade, there has been a significant increase in the growth of the global technology employment market, primarily owing to the digital age, artificial intelligence expansion, cloud computing, and cybersecurity requirements. Consequently, the demand for IT professionals has increased. Int...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE access Vol. 13; pp. 89905 - 89914
Main Authors:	Khaesawad, Attaporn, Fukuchi, Yosuke, Yem, Vibol, Nishiuchi, Nobuyuki
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 2025 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Algorithms Artificial intelligence Block codes Block coding Classification Cloud computing Codes Colleges & universities Computer languages Cybersecurity Decision trees Education Encoding Heating systems Logic Machine learning mouse-tracking heatmap pattern classification Problem-solving Programming Programming profession Students teacher support Teachers Tracking
ISSN:	2169-3536, 2169-3536
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	In the past decade, there has been a significant increase in the growth of the global technology employment market, primarily owing to the digital age, artificial intelligence expansion, cloud computing, and cybersecurity requirements. Consequently, the demand for IT professionals has increased. Introductory programming is a crucial early course in IT higher education as it aids in developing problem-solving skills. However, such courses tend to result in substantially high failure and dropout rates. Some studies have proposed solutions such as peer learning and personalized feedback, but challenges persist. One key assumption is that teacher support is likely to help mitigate these rates. However, with limited class time, teachers find it challenging to support all the students. This study aims to automatically assess the programming logic understanding level (PLUL) during introductory programming courses. We proposed a method for classifying PLUL through a block coding learning platform based on mouse-tracking heatmaps using machine learning techniques. We collected actual learning-task data from first-year university students and compared the performance of five machine learning models. The decision tree algorithm achieved an accuracy score of 76.00%, F1 score of 77.36%, sensitivity score of 75.93%, and specificity score of 76.09%. These findings demonstrate that our method can effectively classify students based on their PLUL of tasks on a block coding learning platform.
AbstractList	In the past decade, there has been a significant increase in the growth of the global technology employment market, primarily owing to the digital age, artificial intelligence expansion, cloud computing, and cybersecurity requirements. Consequently, the demand for IT professionals has increased. Introductory programming is a crucial early course in IT higher education as it aids in developing problem-solving skills. However, such courses tend to result in substantially high failure and dropout rates. Some studies have proposed solutions such as peer learning and personalized feedback, but challenges persist. One key assumption is that teacher support is likely to help mitigate these rates. However, with limited class time, teachers find it challenging to support all the students. This study aims to automatically assess the programming logic understanding level (PLUL) during introductory programming courses. We proposed a method for classifying PLUL through a block coding learning platform based on mouse-tracking heatmaps using machine learning techniques. We collected actual learning-task data from first-year university students and compared the performance of five machine learning models. The decision tree algorithm achieved an accuracy score of 76.00%, F1 score of 77.36%, sensitivity score of 75.93%, and specificity score of 76.09%. These findings demonstrate that our method can effectively classify students based on their PLUL of tasks on a block coding learning platform.
Author	Khaesawad, Attaporn Nishiuchi, Nobuyuki Fukuchi, Yosuke Yem, Vibol
Author_xml	– sequence: 1 givenname: Attaporn orcidid: 0009-0001-6775-4689 surname: Khaesawad fullname: Khaesawad, Attaporn organization: Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan – sequence: 2 givenname: Yosuke orcidid: 0000-0002-7514-9040 surname: Fukuchi fullname: Fukuchi, Yosuke organization: Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan – sequence: 3 givenname: Vibol orcidid: 0000-0002-7254-6614 surname: Yem fullname: Yem, Vibol organization: Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Ibaraki, Japan – sequence: 4 givenname: Nobuyuki orcidid: 0000-0002-6527-0138 surname: Nishiuchi fullname: Nishiuchi, Nobuyuki email: nnishiuc@tmu.ac.jp organization: Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan
BookMark	eNpNUV1PGzEQtCqQSoFfUB5O6vOl_ojt3CM90YIUBBLwbK1969QhsVP7Uin_vg6H2u6LV-OZ2bXnEzmJKSIhnxmdMUa7r9d9f_P0NOOUy5mQmlFJP5AzzlTXCinUyX_9R3JZyprWWlRI6jOyvgf3M0Rslgg5hrhqv0HBoek3UErwwcEYUmySbx5zWmXYbiunWaZVcM1LHDCXEeLwhuFv3JTGHpr7tC_YPmdwr8eLW4RxC7tyQU49bApevp_n5OX7zXN_2y4fftz118vWCcXG1tLBdpqxhVsMVkkQKCgyqbnUXs_9nFqugDPvGOWWS9ROsAHQKSs9x06Lc3I3-Q4J1maXwxbywSQI5g1IeWUgj8Ft0CgpFdXcegV07pB3tLNzxSUXEiloUb2-TF67nH7tsYxmnfY51vWN4IwvZP11XlliYrmcSsno_05l1BwzMlNG5piRec-oqq4mVUDEfwpGqaqPFX8A_KGOjQ
CODEN	IAECCG
Cites_doi	10.4307/jsee.70.1_41 10.1186/s41235-022-00414-5 10.1108/itp-08-2021-0590 10.24059/olj.v24i1.1956 10.1109/iv56949.2022.00053 10.1109/icpr48806.2021.9412492 10.1109/icsgrc.2011.5991826 10.1186/s40594-020-00222-7 10.3390/app10134566 10.1109/CVPR46437.2021.00290 10.16910/jemr.11.6.4 10.1109/access.2018.2854966 10.3102/0034654317710096 10.1177/07356331221102312 10.1109/blocks.2015.7369000 10.1177/07356331211027387 10.1145/3159450.3159476 10.1007/978-3-030-71846-6_3 10.1109/telfor.2017.8249483 10.1016/0098-3004(93)90090-r 10.1108/aaouj-07-2019-0027 10.3758/s13428-021-01703-5 10.1109/ijcnn.2003.1223867 10.17275/per.23.24.10.1 10.1016/j.chb.2015.05.047 10.1109/iccv.2007.4409066 10.1007/978-3-031-16865-9_30 10.12731/2070-7568-2024-13-1-237 10.3390/electronics10101180 10.1109/5254.708428 10.1016/j.cola.2021.101075 10.1109/incit50588.2020.9310787 10.3390/educsci9030181 10.1145/3323503.3360623 10.1109/fie44824.2020.9273982 10.1145/3459665 10.15388/ioi.2022.02 10.1016/j.tics.2018.03.012 10.1002/imt2.43 10.3758/s13414-019-01889-z 10.1109/siie53363.2021.9583623 10.1145/3328778.3372699 10.1177/0735633116685852 10.1145/2960310.2960338 10.1145/3397271.3401031 10.3390/digital3020009
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
DBID	97E ESBDL RIA RIE AAYXX CITATION 7SC 7SP 7SR 8BQ 8FD JG9 JQ2 L7M L~C L~D DOA
DOI	10.1109/ACCESS.2025.3571050
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE Xplore Open Access Journals IEEE All-Society Periodicals Package (ASPP) Online IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Engineered Materials Abstracts METADEX Technology Research Database Materials Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional DOAJ Directory of Open Access Journals
DatabaseTitle	CrossRef Materials Research Database Engineered Materials Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace METADEX Computer and Information Systems Abstracts Professional
DatabaseTitleList	Materials Research Database
Database_xml	– sequence: 1 dbid: DOA name: DOAJ Directory of Open Access Journals url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Education
EISSN	2169-3536
EndPage	89914
ExternalDocumentID	oai_doaj_org_article_6556072bf6a04ce2909b4625235e0a73 10_1109_ACCESS_2025_3571050 11006725
Genre	orig-research
GrantInformation_xml	– fundername: Japan Society for the Promotion of Science (JSPS) Grants-in-Aid for Scientific Research (KAKENHI) grantid: JP23K11744 funderid: 10.13039/501100000646 – fundername: Graduate School of Systems Design, Tokyo Metropolitan University (TMU)
GroupedDBID	0R~ 4.4 5VS 6IK 97E AAJGR ABAZT ABVLG ACGFS ADBBV AGSQL ALMA_UNASSIGNED_HOLDINGS BCNDV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD ESBDL GROUPED_DOAJ IPLJI JAVBF KQ8 M43 M~E O9- OCL OK1 RIA RIE RNS AAYXX CITATION 7SC 7SP 7SR 8BQ 8FD JG9 JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c361t-b0db97118c8db65a3e30e157257f74f40b26a21fc102b25e7c31daec6b5f2e973
IEDL.DBID	DOA
ISICitedReferencesCount	0
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001498285200039&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2169-3536
IngestDate	Fri Oct 03 12:47:14 EDT 2025 Sat Nov 01 15:14:07 EDT 2025 Sat Nov 29 07:52:08 EST 2025 Wed Aug 27 01:51:17 EDT 2025
IsDoiOpenAccess	true
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Language	English
License	https://creativecommons.org/licenses/by-nc-nd/4.0
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c361t-b0db97118c8db65a3e30e157257f74f40b26a21fc102b25e7c31daec6b5f2e973
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0009-0001-6775-4689 0000-0002-7514-9040 0000-0002-7254-6614 0000-0002-6527-0138
OpenAccessLink	https://doaj.org/article/6556072bf6a04ce2909b4625235e0a73
PQID	3212852022
PQPubID	4845423
PageCount	10
ParticipantIDs	ieee_primary_11006725 proquest_journals_3212852022 doaj_primary_oai_doaj_org_article_6556072bf6a04ce2909b4625235e0a73 crossref_primary_10_1109_ACCESS_2025_3571050
PublicationCentury	2000
PublicationDate	20250000 2025-00-00 20250101 2025-01-01
PublicationDateYYYYMMDD	2025-01-01
PublicationDate_xml	– year: 2025 text: 20250000
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE access
PublicationTitleAbbrev	Access
PublicationYear	2025
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref35 ref12 ref34 ref15 ref37 ref14 ref36 ref31 ref30 ref11 ref33 ref10 ref32 ref1 ref17 ref39 ref16 ref38 (ref2) 2024 ref19 ref18 Tafvizi (ref44) 2022 ref24 ref46 ref23 ref45 ref26 ref48 ref25 ref47 ref20 ref42 ref41 ref22 ref21 ref43 ref28 ref27 ref29 ref8 ref7 ref9 ref4 ref3 ref6 ref5 ref40
References_xml	– ident: ref21 doi: 10.4307/jsee.70.1_41 – ident: ref32 doi: 10.1186/s41235-022-00414-5 – ident: ref3 doi: 10.1108/itp-08-2021-0590 – ident: ref12 doi: 10.24059/olj.v24i1.1956 – ident: ref34 doi: 10.1109/iv56949.2022.00053 – ident: ref45 doi: 10.1109/icpr48806.2021.9412492 – ident: ref41 doi: 10.1109/icsgrc.2011.5991826 – ident: ref5 doi: 10.1186/s40594-020-00222-7 – ident: ref6 doi: 10.3390/app10134566 – ident: ref27 doi: 10.1109/CVPR46437.2021.00290 – ident: ref29 doi: 10.16910/jemr.11.6.4 – ident: ref25 doi: 10.1109/access.2018.2854966 – ident: ref10 doi: 10.3102/0034654317710096 – ident: ref47 doi: 10.1177/07356331221102312 – ident: ref20 doi: 10.1109/blocks.2015.7369000 – ident: ref48 doi: 10.1177/07356331211027387 – ident: ref15 doi: 10.1145/3159450.3159476 – ident: ref13 doi: 10.1007/978-3-030-71846-6_3 – ident: ref22 doi: 10.1109/telfor.2017.8249483 – ident: ref36 doi: 10.1016/0098-3004(93)90090-r – ident: ref14 doi: 10.1108/aaouj-07-2019-0027 – ident: ref30 doi: 10.3758/s13428-021-01703-5 – ident: ref43 doi: 10.1109/ijcnn.2003.1223867 – ident: ref24 doi: 10.17275/per.23.24.10.1 – ident: ref18 doi: 10.1016/j.chb.2015.05.047 – ident: ref42 doi: 10.1109/iccv.2007.4409066 – ident: ref23 doi: 10.1007/978-3-031-16865-9_30 – ident: ref1 doi: 10.12731/2070-7568-2024-13-1-237 – ident: ref17 doi: 10.3390/electronics10101180 – ident: ref39 doi: 10.1109/5254.708428 – ident: ref16 doi: 10.1016/j.cola.2021.101075 – ident: ref46 doi: 10.1109/incit50588.2020.9310787 – ident: ref8 doi: 10.3390/educsci9030181 – ident: ref31 doi: 10.1145/3323503.3360623 – volume-title: Future of Jobs Report 2023 year: 2024 ident: ref2 – ident: ref7 doi: 10.1109/fie44824.2020.9273982 – ident: ref40 doi: 10.1145/3459665 – ident: ref19 doi: 10.15388/ioi.2022.02 – ident: ref26 doi: 10.1016/j.tics.2018.03.012 – ident: ref35 doi: 10.1002/imt2.43 – ident: ref37 doi: 10.3758/s13414-019-01889-z – ident: ref9 doi: 10.1109/siie53363.2021.9583623 – ident: ref28 doi: 10.1145/3328778.3372699 – ident: ref4 doi: 10.1177/0735633116685852 – ident: ref11 doi: 10.1145/2960310.2960338 – year: 2022 ident: ref44 article-title: Attributing AUC-ROC to analyze binary classifier performance publication-title: arXiv:2205. 11781 – ident: ref33 doi: 10.1145/3397271.3401031 – ident: ref38 doi: 10.3390/digital3020009
SSID	ssj0000816957
Score	2.3340676
Snippet	In the past decade, there has been a significant increase in the growth of the global technology employment market, primarily owing to the digital age,...
SourceID	doaj proquest crossref ieee
SourceType	Open Website Aggregation Database Index Database Publisher
StartPage	89905
SubjectTerms	Algorithms Artificial intelligence Block codes Block coding Classification Cloud computing Codes Colleges & universities Computer languages Cybersecurity Decision trees Education Encoding Heating systems Logic Machine learning mouse-tracking heatmap pattern classification Problem-solving Programming Programming profession Students teacher support Teachers Tracking
SummonAdditionalLinks	– databaseName: IEEE Electronic Library (IEL) dbid: RIE link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NTxQxFG-EcNCDAmJYRdKDRwudfm6PsJFwgXCQhFvTT6OJu4TdJfG_971OwU2MB2-TSTvT9jed33uv74OQTyji1ymXLGsTmRrUwIJ2gQG3VxORIFxuxSbs9fX07s7d9GD1FgtTSmnOZ-UEL9tZfl6kNZrKTjG9mbFCb5Eta80YrPVsUMEKEk7bnllo4O70bDaDSYAOKPSJ1EClGFy_wT4tSX-vqvLXr7jxy8Wb_xzZLnndBUl6NiK_R16U-T7WYO7-Gvvk1Uaqwbfkx1Xzmiy0J1T9xs6BvzJtRTHRXaj1ootKb0aPrZ_QhmIl5kRvNwNg4AGPwKc0_qJXi_WyMGC7hPZ2eoku6-F-eUBuL758nV2yXmeBJWmGFYs8R2dB00jTHI0OskheBg3TsdWqqngUJoihJhBGotDFJjnkUJKJuorirHxHtueLeTkkNMuqdI6gBFWtAq9OZZtqyHqooUaZJ-Tz0_r7-zGdhm9qCHd-hMsjXL7DNSHniNFzU8yF3W7A4vu-tbzRILVZEasJXKUiHHdRgVonpC48WDkhBwjYn_d1rCbk6Aly3zfu0kug8qmGMYj3_-j2gbzEIY5mmCOyvXpYl49kJz2uvi8fjts3-RvLPt9- priority: 102 providerName: IEEE
Title	Machine Learning-Based Classification of Programming Logic Understanding Levels by Mouse-Tracking Heatmaps
URI	https://ieeexplore.ieee.org/document/11006725 https://www.proquest.com/docview/3212852022 https://doaj.org/article/6556072bf6a04ce2909b4625235e0a73
Volume	13
WOSCitedRecordID	wos001498285200039&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVAON databaseName: DOAJ Directory of Open Access Journals customDbUrl: eissn: 2169-3536 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0000816957 issn: 2169-3536 databaseCode: DOA dateStart: 20130101 isFulltext: true titleUrlDefault: https://www.doaj.org/ providerName: Directory of Open Access Journals – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2169-3536 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0000816957 issn: 2169-3536 databaseCode: M~E dateStart: 20130101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV09TxwxELUQSgEFSgKIIwS5oMTgb69LOIFoDlGARGf5ExEpd4g7kGjy2zP2LslKFGnSbLFare03a78Z7_gNQkfVxS8dFSQpHYhkkhGvrCfA7UWHShA2tWIT5vq6u7-3N6NSXzUnrJcH7oE71Qo42fBQtKcyZm6pDRKcdi5Upt40nU9q7CiYamtwx7RVZpAZYtSenk2nMCIICLk6EQp4tZ60H1FRU-wfSqx8WJcb2Vx-RluDl4jP-t59QWt5_hVtjrQDt9GPWUuDzHhQSH0g50BICbcqlzX_p0GOFwXf9ClYP-EZXEsrR3w3PtECL3gFgsThDc8WL8tMgL5i3UDHVzUH3T8td9Dd5cXt9IoMhRNIFJqtSKApWAOhQ-xS0MqLLGhmysD0LEYWSQPXnrMSwbsIXGUTBUs-Rx1U4dkasYvW54t53kM4iSJVChDVFCU9LVYmE4tPihVfgkgTdPyOoXvq9TFciyuodT3krkLuBsgn6Lzi_OfRKm7dboDJ3WBy9y-TT9BOtdLf9lj7oawm6ODdbG6YiUsngJs7BX3g-_-j7W9oo46n34Q5QOur55f8HX2Kr6vH5fNh-wjhOvt1cdiOEv4GupHe-Q
linkProvider	Directory of Open Access Journals
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LbxMxEB5BQQIOPEoRgQI-cGRbr5_xsY2ogmiiHlqpN8tPBBJJ1SSV-PeMvW6JhDhwW63sXdvfer-Z8TwAPhYRP48p76JUvhO96DsnjeuQ27PyhSBMrMUm9Hw-vrw0Zy1YvcbCpJSq81k6KJf1LD8uw6aYyg5LejOlmbwPD6QQjA7hWncmlVJDwkjdcgv11BweTSY4DdQCmTzgEsm0hNdv8U9N09_qqvz1M64Mc_LsP8f2HJ42UZIcDdi_gHtpsVuqMDePjV14spVs8CX8mFW_yURaStVv3TEyWCS1LGZxGKq9yDKTs8Fn6ye2IaUWcyAX2yEw-IAbZFTif5HZcrNKHfJdKBZ3Mi1O6-5qtQcXJ5_PJ9OuVVroAlf9uvM0eqNR1wjj6JV0PHGaeonT0VmLLKhnyrE-BxRHPJNJB95Hl4LyMrNkNH8FO4vlIr0GEnkWMnpUg7IUjmYjog7ZRdlnlz2PI_h0u_72akioYasiQo0d4LIFLtvgGsFxweiuacmGXW_g4tu2uaySKLdp5rNyVITEDDVeoGLHuEzUaT6CvQLYn_c1rEawfwu5bVt3ZTmS-VjiGNibf3T7AI-m57NTe_pl_vUtPC7DHYwy-7Czvt6kd_Aw3Ky_r67f1-_zN46q4sU
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Machine+Learning-Based+Classification+of+Programming+Logic+Understanding+Levels+by+Mouse-Tracking+Heatmaps&rft.jtitle=IEEE+access&rft.au=Khaesawad%2C+Attaporn&rft.au=Fukuchi%2C+Yosuke&rft.au=Yem%2C+Vibol&rft.au=Nishiuchi%2C+Nobuyuki&rft.date=2025&rft.issn=2169-3536&rft.eissn=2169-3536&rft.volume=13&rft.spage=89905&rft.epage=89914&rft_id=info:doi/10.1109%2FACCESS.2025.3571050&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_ACCESS_2025_3571050
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2169-3536&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2169-3536&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2169-3536&client=summon