Evaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study.
Uloženo v:
| Název: | Evaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study. |
|---|---|
| Autoři: | Herschbach L; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany., Festl-Wietek T; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany., Stegemann-Philipps C; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany., Sonanini A; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany., Herrmann B; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany., Erschens R; Department of Psychosomatic Medicine and Psychotherapy, University Hospital Tuebingen, Tuebingen, Germany., Herrmann-Werner A; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany., Holderried F; TIME - Tuebingen Institute for Medical Education, University of Tuebingen, Tuebingen, Germany. |
| Zdroj: | Journal of medical Internet research [J Med Internet Res] 2025 Nov 28; Vol. 27, pp. e82818. Date of Electronic Publication: 2025 Nov 28. |
| Způsob vydávání: | Journal Article; Observational Study |
| Jazyk: | English |
| Informace o časopise: | Publisher: JMIR Publications Country of Publication: Canada NLM ID: 100959882 Publication Model: Electronic Cited Medium: Internet ISSN: 1438-8871 (Electronic) Linking ISSN: 14388871 NLM ISO Abbreviation: J Med Internet Res Subsets: MEDLINE |
| Imprint Name(s): | Publication: <2011- > : Toronto : JMIR Publications Original Publication: [Pittsburgh, PA? : s.n., 1999- |
| Výrazy ze slovníku MeSH: | Artificial Intelligence* , Communication* , Health Personnel*/education, Humans ; Feedback ; Male ; Proof of Concept Study ; Female ; Adult ; Physician-Patient Relations ; Generative Artificial Intelligence |
| Abstrakt: | Background: Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)-based tools offer promising support for practicing and improving communication techniques among physicians. Objective: This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice? Methods: The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice. Results: The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice. Conclusions: The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures. (©Lea Herschbach, Teresa Festl-Wietek, Christian Stegemann-Philipps, Alessandra Sonanini, Bernd Herrmann, Rebecca Erschens, Anne Herrmann-Werner, Friederike Holderried. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 28.11.2025.) |
| Contributed Indexing: | Keywords: artificial intelligence; clinical competence; communication; feedback; medical education; physician-patient relations; simulated patients |
| Entry Date(s): | Date Created: 20251128 Date Completed: 20251128 Latest Revision: 20251128 |
| Update Code: | 20251129 |
| DOI: | 10.2196/82818 |
| PMID: | 41314643 |
| Databáze: | MEDLINE |
Full Text 
Full Text Finder 
Nájsť tento článok vo Web of Science | Abstrakt: | Background: Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)-based tools offer promising support for practicing and improving communication techniques among physicians.<br />Objective: This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice?<br />Methods: The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice.<br />Results: The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice.<br />Conclusions: The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures.<br /> (©Lea Herschbach, Teresa Festl-Wietek, Christian Stegemann-Philipps, Alessandra Sonanini, Bernd Herrmann, Rebecca Erschens, Anne Herrmann-Werner, Friederike Holderried. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 28.11.2025.) |
|---|---|
| ISSN: | 1438-8871 |
| DOI: | 10.2196/82818 |