Neural collapse under cross-entropy loss

We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minim...

Celý popis

Uložené v:

Podrobná bibliografia
Vydané v:	Applied and computational harmonic analysis Ročník 59; s. 224 - 241
Hlavní autori:	Lu, Jianfeng, Steinerberger, Stefan
Médium:	Journal Article
Jazyk:	English
Vydavateľské údaje:	Elsevier Inc 01.07.2022
Predmet:	Cross-entropy loss Frame potential Neural collapse Neural networks Neural collapse Frame potential Cross-entropy loss Neural networks
ISSN:	1063-5203
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!

Popis
Shrnutí:	We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.
ISSN:	1063-5203
DOI:	10.1016/j.acha.2021.12.011