Discovering UI Display Issues with Visual Understanding

GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on difference devices due to the software or hardware compatibility. They n...

Full description

Saved in:
Bibliographic Details
Published in:2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) pp. 1373 - 1375
Main Author: Liu, Zhe
Format: Conference Proceeding
Language:English
Published: ACM 01.09.2020
Subjects:
Computer bugs
Deep Learning
Graphical user interfaces
Mobile App
Software
Software engineering
UI display
UI testing
Usability
User experience
Visualization
ISSN:2643-1572
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on difference devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot.Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues. We develop a heuristics-based data augmentation method and a GAN-based data augmentation method for boosting the performance of our OwlEye. At present, the evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues.
ISSN:2643-1572
DOI:10.1145/3324884.3418917