Social Bias in Vision Transformers: A Comparative Study Across Architectures and Learning Paradigms: A Comparative Study Across Architectures and Learning Paradigms
Saved in:
| Title: | Social Bias in Vision Transformers: A Comparative Study Across Architectures and Learning Paradigms: A Comparative Study Across Architectures and Learning Paradigms |
|---|---|
| Authors: | Takehiro Tsurumi, Elena Beretta |
| Source: | Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. :2934-2973 |
| Publisher Information: | ACM, 2025. |
| Publication Year: | 2025 |
| Subject Terms: | vision transformers, social bias, unsupervised learning, computer vision |
| Description: | Vision Transformers (ViTs) have revolutionized computer vision but their susceptibility to encoding social biases remains a pressing ethical concern. These biases, often inherited from pre-Training datasets or embedded within learning architectures, can amplify societal stereotypes when deployed in sensitive contexts. This paper explores how biases manifest across 10 state-of-The-Art ViTs, focusing on the intersection between different learning objectives, the architectures and their impact on bias encoding. By applying the Image Embedding Association Test (iEAT), Gradient-weighted Class Activation Mapping (Grad-CAM) for interpretability and t-SNE visualizations, we reveal persistent biases such as gender-career associations, gender-race intersectionality and valence disparities. Our findings reveal discriminative models amplify biases in their representation and decision layers, while contrastive models exacerbate stereotypical alignments in embedding spaces. Furthermore, we identify actionable pathways for bias mitigation by enhancing representational fairness within latent spaces. |
| Document Type: | Article Conference object |
| DOI: | 10.1145/3715275.3732189 |
| Access URL: | https://research.vu.nl/en/publications/241b370a-b545-4ed5-95c1-724cdad5f4f8 https://doi.org/10.1145/3715275.3732189 https://hdl.handle.net/1871.1/241b370a-b545-4ed5-95c1-724cdad5f4f8 |
| Rights: | CC BY |
| Accession Number: | edsair.doi.dedup.....dc620c7d63a5278cc7cac3b465b98934 |
| Database: | OpenAIRE |
Nájsť tento článok vo Web of Science | Abstract: | Vision Transformers (ViTs) have revolutionized computer vision but their susceptibility to encoding social biases remains a pressing ethical concern. These biases, often inherited from pre-Training datasets or embedded within learning architectures, can amplify societal stereotypes when deployed in sensitive contexts. This paper explores how biases manifest across 10 state-of-The-Art ViTs, focusing on the intersection between different learning objectives, the architectures and their impact on bias encoding. By applying the Image Embedding Association Test (iEAT), Gradient-weighted Class Activation Mapping (Grad-CAM) for interpretability and t-SNE visualizations, we reveal persistent biases such as gender-career associations, gender-race intersectionality and valence disparities. Our findings reveal discriminative models amplify biases in their representation and decision layers, while contrastive models exacerbate stereotypical alignments in embedding spaces. Furthermore, we identify actionable pathways for bias mitigation by enhancing representational fairness within latent spaces. |
|---|---|
| DOI: | 10.1145/3715275.3732189 |