Does GenAI Make Usability Testing Obsolete?
Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality. This paper presents UX-LLM, a novel tool powered by a Large Vision-Language Model that predicts usability issues in iOS apps. To evaluate the...
Saved in:
| Published in: | Proceedings / International Conference on Software Engineering pp. 437 - 449 |
|---|---|
| Main Authors: | , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
26.04.2025
|
| Subjects: | |
| ISSN: | 1558-1225 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality. This paper presents UX-LLM, a novel tool powered by a Large Vision-Language Model that predicts usability issues in iOS apps. To evaluate the performance of UX-LLM, we predicted usability issues in two open-source apps of a medium complexity and asked two usability experts to assess the predictions. We also performed traditional usability testing and expert review for both apps and compared the results to those of UX-LLM. UX-LLM demonstrated precision ranging from 0.61 and 0.66 and recall between 0.35 and 0.38, indicating its ability to identify valid usability issues, yet failing to capture the majority of issues. Finally, we conducted a focus group with an app development team of a capstone project developing a transit app for visually impaired persons. The focus group expressed positive perceptions of UX-LLM as it identified unknown usability issues in their app. However, they also raised concerns about its integration into the development workflow, suggesting potential improvements. Our results show that UX-LLM cannot fully replace traditional usability evaluation methods but serves as a valuable supplement particularly for small teams with limited resources, to identify issues in less common user paths, due to its ability to inspect the source code. |
|---|---|
| AbstractList | Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality. This paper presents UX-LLM, a novel tool powered by a Large Vision-Language Model that predicts usability issues in iOS apps. To evaluate the performance of UX-LLM, we predicted usability issues in two open-source apps of a medium complexity and asked two usability experts to assess the predictions. We also performed traditional usability testing and expert review for both apps and compared the results to those of UX-LLM. UX-LLM demonstrated precision ranging from 0.61 and 0.66 and recall between 0.35 and 0.38, indicating its ability to identify valid usability issues, yet failing to capture the majority of issues. Finally, we conducted a focus group with an app development team of a capstone project developing a transit app for visually impaired persons. The focus group expressed positive perceptions of UX-LLM as it identified unknown usability issues in their app. However, they also raised concerns about its integration into the development workflow, suggesting potential improvements. Our results show that UX-LLM cannot fully replace traditional usability evaluation methods but serves as a valuable supplement particularly for small teams with limited resources, to identify issues in less common user paths, due to its ability to inspect the source code. |
| Author | Maalej, Walid Pourasad, Ali Ebrahimi |
| Author_xml | – sequence: 1 givenname: Ali Ebrahimi surname: Pourasad fullname: Pourasad, Ali Ebrahimi email: ali.ebrahimi.pourasad@uni-hamburg.de organization: Universität Hamburg,Department of Informatics,Hamburg,Germany – sequence: 2 givenname: Walid surname: Maalej fullname: Maalej, Walid email: walid.maalej@uni-hamburg.de organization: Universität Hamburg,Department of Informatics,Hamburg,Germany |
| BookMark | eNotz8FOwkAQgOHVaCIgb8Chd9M6s9Pp7pwMQcQmGA7CmWxxaqq1NWwvvL0kePpuf_KPzU3Xd2rMDCFDBHksF-9LZspdZsFyBoDkr8xUnHgiZOBC8NqMkNmnaC3fmXGMXwBQ5CIj8_Dca0xW2s3L5C18a7KLoWraZjglW41D030mmyr2rQ76dG9u69BGnf47MbuX5Xbxmq43q3IxX6fBFjCk4m2OgYTQkoTgD6RcuKrOqRYKUnD4cCg1KsMZR2rlQOiQAtYIlmhiZpduo6r732PzE46n_fnWiqCnP7pbQhk |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICSE55347.2025.00138 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798331505691 |
| EISSN | 1558-1225 |
| EndPage | 449 |
| ExternalDocumentID | 11029918 |
| Genre | orig-research |
| GroupedDBID | -~X .4S .DC 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-a260t-98241a3931239aa8c3e567bf43f93a965ad719f1e5019f73e29c31713a1f10233 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001538318100034&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 01:40:27 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a260t-98241a3931239aa8c3e567bf43f93a965ad719f1e5019f73e29c31713a1f10233 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_11029918 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-April-26 |
| PublicationDateYYYYMMDD | 2025-04-26 |
| PublicationDate_xml | – month: 04 year: 2025 text: 2025-April-26 day: 26 |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Conference on Software Engineering |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0006499 |
| Score | 2.3034673 |
| Snippet | Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality.... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 437 |
| SubjectTerms | AI-Inspired Design AI4SE App Development Foundation Models Large Language Model Large language models Mobile applications Predictive models Quality Requirements Recommender systems Reviews Software engineering Source coding Testing Usability Usability Engineering User experience |
| Title | Does GenAI Make Usability Testing Obsolete? |
| URI | https://ieeexplore.ieee.org/document/11029918 |
| WOSCitedRecordID | wos001538318100034&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEG6EePCED4zv9OC1QpntdnsyhoByEEmEhBtpd6fGmCwEFhP_vdNlQS8evDVNk2b6-r5p-80wdptBlLlEg6DF4kQUgxY2aXuhAvf32hIiuzLZhB4Ok-nUjCqxeqmFQcTy8xnehWL5lp_N03W4KmsRVNHpKZMaq2kdb8Rau2M3Ju5eaeNk27QG3deeUhBp8gE74d5EBgnKrwwqJYD0G__s-pA1f6R4fLQDmSO2h_kxa2xzMfBqa54wYsK44o-YPwz4s_1APtnEzi2--DgE0sjf-IujdUYc-b7JJv3euPskqkQIwpK7UQiTEM5aMEAwY6xNUkAVa-cj8AasiZXNtDReIo2w8RqwY1LiBRKs9CE0A5yyej7P8YxxlynjEJy1GEdtAEutnANLzMFJlaXnrBmMny02sS5mW7sv_qi_ZAdhfMP7Sie-YvViucZrtp9-Fu-r5U05Q99IKI7_ |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEJ4omugJHxjf9uB1Zctst9uTMQSECEgiJNxIu9s1xmQxsJj4750uC3rx4K1pmjTT1_dN228G4DbBIDGRRI8Wi_GCEKWnIz_1hOP-qdSEyKZINiEHg2gyUcNSrF5oYay1xecze-eKxVt-MouX7qqsTlBFpyePtmFHBEHDX8m1NgdvSOy9VMdxX9W7zZeWEBhI8gIb7uaEOxHKrxwqBYS0q__s_ABqP2I8NtzAzCFs2ewIqutsDKzcnMdAXNgu2KPNHrqsr98tG6-i5-ZfbORCaWSv7NnQSiOWfF-Dcbs1ana8MhWCp8nhyD0VEdJqVEhAo7SOYrQilCYNMFWoVSh0IrlKuaUxVqlE21AxMQOOmqcuOAOeQCWbZfYUmEmEMhaN1jYMfERNrYxBTdzBcJHEZ1Bzxk8_VtEupmu7z_-ov4G9zqjfm_a6g6cL2Hdj7V5bGuElVPL50l7BbvyZvy3m18VsfQPtd5JG |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=Does+GenAI+Make+Usability+Testing+Obsolete%3F&rft.au=Pourasad%2C+Ali+Ebrahimi&rft.au=Maalej%2C+Walid&rft.date=2025-04-26&rft.pub=IEEE&rft.eissn=1558-1225&rft.spage=437&rft.epage=449&rft_id=info:doi/10.1109%2FICSE55347.2025.00138&rft.externalDocID=11029918 |