An Efficient and Effective Second-Order Training Algorithm for LSTM-Based Adaptive Learning

We study adaptive (or online) nonlinear regression with Long-Short-Term-Memory (LSTM) based networks, i.e., LSTM-based adaptive learning. In this context, we introduce an efficient Extended Kalman filter (EKF) based second-order training algorithm. Our algorithm is truly online, i.e., it does not as...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on signal processing Ročník 69; s. 2541 - 2554
Hlavní autoři: Vural, N. Mert, Ergut, Salih, Kozat, Suleyman S.
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 2021
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Adaptation models
Adaptive algorithms
Adaptive learning
Adaptive systems
Algorithms
Approximation algorithms
Extended Kalman filter
Kalman filtering
long short term memory (LSTM)
Machine learning
MLR-DEEP
online learning
Prediction algorithms
regression
Signal processing algorithms
stochastic gradient descent (SGD) EDICS Category: MLR-SLER
Training
truly online
ISSN:1053-587X, 1941-0476
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:We study adaptive (or online) nonlinear regression with Long-Short-Term-Memory (LSTM) based networks, i.e., LSTM-based adaptive learning. In this context, we introduce an efficient Extended Kalman filter (EKF) based second-order training algorithm. Our algorithm is truly online, i.e., it does not assume any underlying data generating process and future information, except that the target sequence is bounded. Through an extensive set of experiments, we demonstrate significant performance gains achieved by our algorithm with respect to the state-of-the-art methods. Here, we mainly show that our algorithm consistently provides 10 to 45% improvement in the accuracy compared to the widely-used adaptive methods Adam, RMSprop, and DEKF, and comparable performance to EKF with a 10 to 15 times reduction in the run-time.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1053-587X
1941-0476
DOI:10.1109/TSP.2021.3071566