Leveraging Large Language Models to Annotate Activities of Daily Living Captured with Egocentric Vision

Developing a system that automatically and passively recognizes activities of daily living (ADLs) would be transformative for numerous health applications. However, engineering approaches for building such a classifier today requires the availability of large and rich annotated datasets representing...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (Online) s. 182 - 186
Hlavní autoři: Shrestha, Sloke, Thomaz, Edison
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 19.06.2024
Témata:
Activities of Daily Living
Annotations
Buildings
Data Annotation
Detectors
Human Activity Recognition
Large language models
Manuals
NASA
Pipelines
Vison Language Models
ISSN:2832-2975
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Developing a system that automatically and passively recognizes activities of daily living (ADLs) would be transformative for numerous health applications. However, engineering approaches for building such a classifier today requires the availability of large and rich annotated datasets representing ADLs in a generalizable way. In this work, we evaluated state of the art large language models (LLMs) to perform fully-automated and assisted manual annotations of first-person images with ADLs. We performed automatic evaluations on four different vision language pipelines (VLPs): concept detector, concept detector + GPT-3.5, BLIP2, and GPT-4. Three of them were tested on 31,849 first person images and one of them, GPT-4, was tested on 3,446 images first person images. Among the four VLPs, BLIP2 scored the highest cosine similarity of 0.86. Furthermore, we evaluated assisted manual annotation with 20 participants who annotated 100 ADL images with recommended labels from three different VLPs. We show that annotation with BLIP2 assistance has highest pick rate of 0.698 and a subjective workload (NASA Task Load Index) score of 39.41 in a scale of 100. Despite limitations, our work demonstrates how large language model can be leveraged to optimize the difficult task of data annotation for building ADL classifiers.
ISSN:2832-2975
DOI:10.1109/CHASE60773.2024.00031