In EDS ansehen

RiskRAG: A Data-Driven Solution for Improved AI Model Risk Reporting

Gespeichert in:
Bibliographische Detailangaben
Titel: RiskRAG: A Data-Driven Solution for Improved AI Model Risk Reporting
Autoren: Pooja S. B. Rao, Sanja Šćepanović, Ke Zhou, Edyta Paulina Bogucka, Daniele Quercia
Quelle: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. :1-26
Publication Status: Preprint
Verlagsinformationen: ACM, 2025.
Publikationsjahr: 2025
Schlagwörter: Software Engineering (cs.SE), FOS: Computer and information sciences, Computer Science - Software Engineering, Computer Sciences, Graphics and Human Computer Interfaces, Computer Science - Human-Computer Interaction, Physical Sciences and Mathematics, Human-Computer Interaction (cs.HC)
Beschreibung: Risk reporting is essential for documenting AI models, yet only 14% of model cards mention risks, out of which 96% copying content from a small set of cards, leading to a lack of actionable insights. Existing proposals for improving model cards do not resolve these issues. To address this, we introduce RiskRAG, a Retrieval Augmented Generation based risk reporting solution guided by five design requirements we identified from literature, and co-design with 16 developers: identifying diverse model-specific risks, clearly presenting and prioritizing them, contextualizing for real-world uses, and offering actionable mitigation strategies. Drawing from 450K model cards and 600 real-world incidents, RiskRAG pre-populates contextualized risk reports. A preliminary study with 50 developers showed that they preferred RiskRAG over standard model cards, as it better met all the design requirements. A final study with 38 developers, 40 designers, and 37 media professionals showed that RiskRAG improved their way of selecting the AI model for a specific application, encouraging a more careful and deliberative decision-making. The RiskRAG project page is accessible at: https://social-dynamics.net/ai-risks/card.
Publikationsart: Article
Other literature type
DOI: 10.1145/3706598.3713979
DOI: 10.48550/arxiv.2504.08952
DOI: 10.17605/osf.io/chjgp
Zugangs-URL: http://arxiv.org/abs/2504.08952
Rights: CC BY
Dokumentencode: edsair.doi.dedup.....e59c46adb75894500a52112a94da3e6c
Datenbank: OpenAIRE
Beschreibung
Abstract:Risk reporting is essential for documenting AI models, yet only 14% of model cards mention risks, out of which 96% copying content from a small set of cards, leading to a lack of actionable insights. Existing proposals for improving model cards do not resolve these issues. To address this, we introduce RiskRAG, a Retrieval Augmented Generation based risk reporting solution guided by five design requirements we identified from literature, and co-design with 16 developers: identifying diverse model-specific risks, clearly presenting and prioritizing them, contextualizing for real-world uses, and offering actionable mitigation strategies. Drawing from 450K model cards and 600 real-world incidents, RiskRAG pre-populates contextualized risk reports. A preliminary study with 50 developers showed that they preferred RiskRAG over standard model cards, as it better met all the design requirements. A final study with 38 developers, 40 designers, and 37 media professionals showed that RiskRAG improved their way of selecting the AI model for a specific application, encouraging a more careful and deliberative decision-making. The RiskRAG project page is accessible at: https://social-dynamics.net/ai-risks/card.
DOI:10.1145/3706598.3713979