Learning From Mistakes: A Multilevel Optimization Framework

Bi-level optimization methods in machine learning are popularly effective in subdomains of neural architecture search, data reweighting, etc. However, most of these methods do not factor in variations in learning difficulty, which limits their performance in real-world applications. To address the a...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on artificial intelligence Jg. 6; H. 6; S. 1651 - 1663
Hauptverfasser: Zhang, Li, Garg, Bhanu, Sridhara, Pradyumna, Hosseini, Ramtin, Xie, Pengtao
Format: Journal Article
Sprache:Englisch
Veröffentlicht: IEEE 01.06.2025
Schlagworte:
Artificial intelligence
Computer architecture
Data models
Data reweighting (DR)
learning from mistakes (LFM)
Machine learning
multilevel optimization
Neural architecture search
neural architecture search (NAS)
Noise
Noise measurement
Optimization
Training
Training data
ISSN:2691-4581, 2691-4581
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Bi-level optimization methods in machine learning are popularly effective in subdomains of neural architecture search, data reweighting, etc. However, most of these methods do not factor in variations in learning difficulty, which limits their performance in real-world applications. To address the above problems, we propose a framework that imitates the learning process of humans. In human learning, learners usually focus more on the topics where mistakes have been made in the past to deepen their understanding and master the knowledge. Inspired by this effective human learning technique, we propose a multilevel optimization framework, learning from mistakes (LFM), for machine learning. We formulate LFM as a three-stage optimization problem: 1) the learner learns, 2) the learner relearns based on the mistakes made before, and 3) the learner validates his learning. We develop an efficient algorithm to solve the optimization problem. We further apply our method to differentiable neural architecture search and data reweighting. Extensive experiments on CIFAR-10, CIFAR-100, ImageNet, and other related datasets powerfully demonstrate the effectiveness of our approach. The code of LFM is available at: https://github.com/importZL/LFM .
ISSN:2691-4581
2691-4581
DOI:10.1109/TAI.2025.3534151