Generating feature spaces for linear algorithms with regularized sparse kernel slow feature analysis

Without non-linear basis functions many problems can not be solved by linear algorithms. This article proposes a method to automatically construct such basis functions with slow feature analysis (SFA). Non-linear optimization of this unsupervised learning method generates an orthogonal basis on the...

Full description

Saved in:
Bibliographic Details
Published in:Machine learning Vol. 89; no. 1-2; pp. 67 - 86
Main Authors: Böhmer, Wendelin, Grünewälder, Steffen, Nickisch, Hannes, Obermayer, Klaus
Format: Journal Article
Language:English
Published: Boston Springer US 01.10.2012
Springer Nature B.V
Subjects:
Algorithms
Artificial Intelligence
Basis functions
Computer Science
Control
Kernels
Linear programming
Mathematical analysis
Mathematical models
Mechatronics
Natural Language Processing (NLP)
Nonlinearity
Robotics
Simulation and Modeling
Time series
Latent variables
Slow feature analysis
Sparse kernel methods
Linear classification
Unsupervised learning
Time series
ISSN:0885-6125, 1573-0565
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Without non-linear basis functions many problems can not be solved by linear algorithms. This article proposes a method to automatically construct such basis functions with slow feature analysis (SFA). Non-linear optimization of this unsupervised learning method generates an orthogonal basis on the unknown latent space for a given time series. In contrast to methods like PCA, SFA is thus well suited for techniques that make direct use of the latent space. Real-world time series can be complex, and current SFA algorithms are either not powerful enough or tend to over-fit. We make use of the kernel trick in combination with sparsification to develop a kernelized SFA algorithm which provides a powerful function class for large data sets. Sparsity is achieved by a novel matching pursuit approach that can be applied to other tasks as well. For small data sets, however, the kernel SFA approach leads to over-fitting and numerical instabilities. To enforce a stable solution, we introduce regularization to the SFA objective. We hypothesize that our algorithm generates a feature space that resembles a Fourier basis in the unknown space of latent variables underlying a given real-world time series . We evaluate this hypothesis at the example of a vowel classification task in comparison to sparse kernel PCA . Our results show excellent classification accuracy and demonstrate the superiority of kernel SFA over kernel PCA in encoding latent variables.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-2
content type line 23
ObjectType-Article-1
ObjectType-Feature-2
ISSN:0885-6125
1573-0565
DOI:10.1007/s10994-012-5300-0