Improving Assessment of Programming Pattern Knowledge through Code Editing and Revision
How well do code-writing tasks measure students' knowledge of programming patterns and anti-patterns? How can we assess this knowledge more accurately? To explore these questions, we surveyed 328 intermediate CS students and measured their performance on different types of tasks, including writ...
Saved in:
| Published in: | IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (Online) pp. 58 - 69 |
|---|---|
| Main Authors: | , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
01.05.2023
|
| Subjects: | |
| ISSN: | 2832-7578 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | How well do code-writing tasks measure students' knowledge of programming patterns and anti-patterns? How can we assess this knowledge more accurately? To explore these questions, we surveyed 328 intermediate CS students and measured their performance on different types of tasks, including writing code, editing someone else's code, and, if applicable, revising their own alternatively-structured code. Our tasks targeted returning a Boolean expression and using unique code within an if and else.We found that code writing sometimes under-estimated student knowledge. For tasks targeting returning a Boolean expression, over 55% of students who initially wrote with non-expert structure successfully revised to expert structure when prompted - even though the prompt did not include guidance on how to improve their code. Further, over 25% of students who initially wrote non-expert code could properly edit someone else's non-expert code to expert structure. These results show that non-expert code is not a reliable indicator of deep misconceptions about the structure of expert code. Finally, although code writing is correlated with code editing, the relationship is weak: a model with code writing as the sole predictor of code editing explains less than 15% of the variance. Model accuracy improves when we include additional predictors that reflect other facets of knowledge, namely the identification of expert code and selection of expert code as more readable than non-expert code. Together, these results indicate that a combination of code writing, revising, editing, and identification tasks can provide a more accurate assessment of student knowledge of programming patterns than code writing alone. |
|---|---|
| AbstractList | How well do code-writing tasks measure students' knowledge of programming patterns and anti-patterns? How can we assess this knowledge more accurately? To explore these questions, we surveyed 328 intermediate CS students and measured their performance on different types of tasks, including writing code, editing someone else's code, and, if applicable, revising their own alternatively-structured code. Our tasks targeted returning a Boolean expression and using unique code within an if and else.We found that code writing sometimes under-estimated student knowledge. For tasks targeting returning a Boolean expression, over 55% of students who initially wrote with non-expert structure successfully revised to expert structure when prompted - even though the prompt did not include guidance on how to improve their code. Further, over 25% of students who initially wrote non-expert code could properly edit someone else's non-expert code to expert structure. These results show that non-expert code is not a reliable indicator of deep misconceptions about the structure of expert code. Finally, although code writing is correlated with code editing, the relationship is weak: a model with code writing as the sole predictor of code editing explains less than 15% of the variance. Model accuracy improves when we include additional predictors that reflect other facets of knowledge, namely the identification of expert code and selection of expert code as more readable than non-expert code. Together, these results indicate that a combination of code writing, revising, editing, and identification tasks can provide a more accurate assessment of student knowledge of programming patterns than code writing alone. |
| Author | Rafferty, Anna N. Nurollahian, Sara Wiese, Eliane |
| Author_xml | – sequence: 1 givenname: Sara surname: Nurollahian fullname: Nurollahian, Sara email: sara.nurollahian@utah.edu organization: University of Utah,School of Computing,Salt Lake City,Utah – sequence: 2 givenname: Anna N. surname: Rafferty fullname: Rafferty, Anna N. email: arafferty@carleton.edu organization: Computer Science Carleton College,Northfield,MN – sequence: 3 givenname: Eliane surname: Wiese fullname: Wiese, Eliane email: eliane.wiese@utah.edu organization: University of Utah,School of Computing,Salt Lake City,Utah |
| BookMark | eNotjNFKwzAUQKMoOOf-wIf4Aa03SdMkj6NUHQ4cbuLjSJu7rrAmI6kT_16GPp2Hczi35MoHj4Q8MMgZA_O4qNZ1tq7rjdSlljkHLnIAYPyCzIwyWkgQnEsDl2TCteCZkkrfkFlKfQOSS60UZxPyuRiOMZx639F5SpjSgH6kYUdXMXTRDsPZrOw4YvT01YfvA7oO6biP4avb0yo4pLXrx3NmvaPveOpTH_wdud7ZQ8LZP6fk46neVC_Z8u15Uc2XmeWmHDMGbVGgA9W2u0ZoqRAMmtK0wKSWzMrGYIlaiabAtrBWF1xI6ZwFo6VptZiS-79vj4jbY-wHG3-2DJjiSinxC-eDVuQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/ICSE-SEET58685.2023.00012 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE Xplore IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9798350322590 |
| EISSN | 2832-7578 |
| EndPage | 69 |
| ExternalDocumentID | 10172777 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Science Foundation funderid: 10.13039/100000001 |
| GroupedDBID | 6IE 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK OCL RIE RIL |
| ID | FETCH-LOGICAL-a296t-10c44ed07ccfb3857e09e969c015851a5b9e6e873b4ec4aa842355dda09859c83 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 2 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001032813100007&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:22:08 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a296t-10c44ed07ccfb3857e09e969c015851a5b9e6e873b4ec4aa842355dda09859c83 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_10172777 |
| PublicationCentury | 2000 |
| PublicationDate | 2023-May |
| PublicationDateYYYYMMDD | 2023-05-01 |
| PublicationDate_xml | – month: 05 year: 2023 text: 2023-May |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (Online) |
| PublicationTitleAbbrev | ICSE-SEET |
| PublicationYear | 2023 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssib052587721 ssj0003211719 |
| Score | 2.233764 |
| Snippet | How well do code-writing tasks measure students' knowledge of programming patterns and anti-patterns? How can we assess this knowledge more accurately? To... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 58 |
| SubjectTerms | Code Editing Code quality code readability code refactoring Code Revising code structure Code writing Codes Knowledge engineering Predictive models programming patterns and anti-patterns Reliability Task analysis Training Writing |
| Title | Improving Assessment of Programming Pattern Knowledge through Code Editing and Revision |
| URI | https://ieeexplore.ieee.org/document/10172777 |
| WOSCitedRecordID | wos001032813100007&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB1sEfGkYsVvVvCaNtlkv45SUhShFK3YW9nsTqAHE-mHv9_dbVLx4MFbSPYQJsu82cm89wDuHcrYDNM4KqwX1U4oj2RR8shBoStGFGp3mgtmE2I8lrOZmjRk9cCFQcQwfIZ9fxn-5dvabHyrbOC3DxVCdKAjBN-StdrNwyiTotX68mk4dUcbkagDuGt0NQdPw9fcZal8yiSXrO-Nw_uhE_PLWSUAy-jon690DL0fih6Z7MDnBPawOoX3XYuAPOwUN0ld-pV-CuvDP5kERc2KPLfdNNJ49ZBhbZHkduEnoYmuLHlpqOc9eBvl0-Fj1DgnRJoqvna51WQZ2lgYUxapZAJjhYor48DflViaFQo5SpEWGZpMa-mKKsas1bGSTBmZnkG3qis8B2K1xpLTMrG2yGiitYhpobRwmUKUUukL6PmozD-34hjzNiCXf9y_gkMf-O3M4DV018sN3sC--VovVsvb8Em_AfZFoRs |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwED1BQcAEiCK-MRJr2nw5tkdUpWrVUlVQRLfKsS9ShyaotPx-bDcpYmBgi5wMkW3dO5_vvQfwaFBGxxj5XqatqHYQJh7P8sQzUGiSEYHSnOac2QQbjfh0KsYVWd1xYRDRNZ9hyz66u3xdqrUtlbXt9gkZY7uwZ62zKrpWvX1oSDmr1b5sII7M4YYF4gAeKmXNdr_zmpo4lU4oTzhtWevwlqvF_PJWcdDSPf7nT51A84ekR8Zb-DmFHSzO4H1bJCBPW81NUub2S9uHtbBvxk5TsyCDup5GKrce0ik1klTPbS80kYUmLxX5vAlv3XTS6XmVd4InQ5GsTHRVcYzaZ0rlWcQpQ1-gSIQy8G-SLEkzgQlyFmUxqlhKbtIqSrWWvuBUKB6dQ6MoC7wAoqXEPAnzQOssDgMpmR9mQjITK1jOhbyEpp2V2cdGHmNWT8jVH-P3cNibPA9nw_5ocA1HdhE2HYQ30Fgt13gL--prNf9c3rnl_QaLeqRk |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Software+Engineering%3A+Software+Engineering+Education+and+Training+%28Online%29&rft.atitle=Improving+Assessment+of+Programming+Pattern+Knowledge+through+Code+Editing+and+Revision&rft.au=Nurollahian%2C+Sara&rft.au=Rafferty%2C+Anna+N.&rft.au=Wiese%2C+Eliane&rft.date=2023-05-01&rft.pub=IEEE&rft.eissn=2832-7578&rft.spage=58&rft.epage=69&rft_id=info:doi/10.1109%2FICSE-SEET58685.2023.00012&rft.externalDocID=10172777 |