Comparison of Crowdsourced and Remote Subjective User Studies: A Case Study of Investigative Child Interviews

Crowdsourced and remote user studies have recently gained popularity as alternatives to traditional laboratory studies. However, they are subject to unreliability, and it is challenging to ensure that valid results are collected, especially when conducting user studies with experts. Experts are a sp...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International Workshop on Quality of Multimedia Experience s. 1 - 6
Hlavní autoři: Sabet, Saeed Shafiee, Midoglu, Cise, Hassan, Syed Zohaib, Salehi, Pegah, Baugerud, Gunn Astrid, Griwodz, Carsten, Johnson, Miriam, Riegler, Michael Alexander, Halvorsen, Pal
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 05.09.2022
Témata:
Avatars
Correlation
CPS
Crowdsourcing
game engine
Games
Interviews
Reliability
Schedules
talking avatar
Training
ISSN:2472-7814
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Crowdsourced and remote user studies have recently gained popularity as alternatives to traditional laboratory studies. However, they are subject to unreliability, and it is challenging to ensure that valid results are collected, especially when conducting user studies with experts. Experts are a sparse resource, usually having busy schedules and heavy workloads, and are not necessarily geographically close. They are therefore often unwilling to participate in studies which require physical attendance. In this paper, we compare three alternative methods: crowd sourced user study with non-experts, remote user study with non-experts, and remote user study with domain experts, for a use case involving investigative child interview training. We present the results from three subjective studies about the perception of AI-generated child avatars, which is developed using various technologies such as dialogue models, game engine, text-to-speech and speech-to-text components. The study was conducted with three different user groups, and our results indicate the importance of using best practice measures for ensuring the collection of reliable results in crowdsourced settings as compared to remote studies, and highlight the difference between the perspectives of domain experts and non-experts.
ISSN:2472-7814
DOI:10.1109/QoMEX55416.2022.9900900