Neural collapse under cross-entropy loss

We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minim...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Applied and computational harmonic analysis Jg. 59; S. 224 - 241
Hauptverfasser: Lu, Jianfeng, Steinerberger, Stefan
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier Inc 01.07.2022
Schlagworte:
Cross-entropy loss
Frame potential
Neural collapse
Neural networks
Neural collapse
Frame potential
Cross-entropy loss
Neural networks
ISSN:1063-5203
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.
ISSN:1063-5203
DOI:10.1016/j.acha.2021.12.011