Learning From Mistakes: A Multilevel Optimization Framework

Bi-level optimization methods in machine learning are popularly effective in subdomains of neural architecture search, data reweighting, etc. However, most of these methods do not factor in variations in learning difficulty, which limits their performance in real-world applications. To address the a...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on artificial intelligence Vol. 6; no. 6; pp. 1651 - 1663
Main Authors: Zhang, Li, Garg, Bhanu, Sridhara, Pradyumna, Hosseini, Ramtin, Xie, Pengtao
Format: Journal Article
Language:English
Published: IEEE 01.06.2025
Subjects:
Artificial intelligence
Computer architecture
Data models
Data reweighting (DR)
learning from mistakes (LFM)
Machine learning
multilevel optimization
Neural architecture search
neural architecture search (NAS)
Noise
Noise measurement
Optimization
Training
Training data
ISSN:2691-4581, 2691-4581
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Bi-level optimization methods in machine learning are popularly effective in subdomains of neural architecture search, data reweighting, etc. However, most of these methods do not factor in variations in learning difficulty, which limits their performance in real-world applications. To address the above problems, we propose a framework that imitates the learning process of humans. In human learning, learners usually focus more on the topics where mistakes have been made in the past to deepen their understanding and master the knowledge. Inspired by this effective human learning technique, we propose a multilevel optimization framework, learning from mistakes (LFM), for machine learning. We formulate LFM as a three-stage optimization problem: 1) the learner learns, 2) the learner relearns based on the mistakes made before, and 3) the learner validates his learning. We develop an efficient algorithm to solve the optimization problem. We further apply our method to differentiable neural architecture search and data reweighting. Extensive experiments on CIFAR-10, CIFAR-100, ImageNet, and other related datasets powerfully demonstrate the effectiveness of our approach. The code of LFM is available at: https://github.com/importZL/LFM .
ISSN:2691-4581
2691-4581
DOI:10.1109/TAI.2025.3534151