Neural collapse under cross-entropy loss

We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minim...

Full description

Saved in:
Bibliographic Details
Published in:Applied and computational harmonic analysis Vol. 59; pp. 224 - 241
Main Authors: Lu, Jianfeng, Steinerberger, Stefan
Format: Journal Article
Language:English
Published: Elsevier Inc 01.07.2022
Subjects:
Cross-entropy loss
Frame potential
Neural collapse
Neural networks
Neural collapse
Frame potential
Cross-entropy loss
Neural networks
ISSN:1063-5203
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.
ISSN:1063-5203
DOI:10.1016/j.acha.2021.12.011