Deep Contrastive Representation Learning With Self-Distillation

Recently, contrastive learning (CL) is a promising way of learning discriminative representations from time series data. In the representation hierarchy, semantic information extracted from lower levels is the basis of that captured from higher levels. Low-level semantic information is essential and...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on emerging topics in computational intelligence Ročník 8; číslo 1; s. 3 - 15
Hlavní autori: Xiao, Zhiwen, Xing, Huanlai, Zhao, Bowen, Qu, Rong, Luo, Shouxi, Dai, Penglin, Li, Ke, Zhu, Zonghai
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Piscataway IEEE 01.02.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Algorithms
Clustering
Clustering algorithms
Contrastive learning
Data augmentation
Deep learning
Distillation
Feature extraction
Knowledge discovery
knowledge distillation
Knowledge management
Machine learning
Performance enhancement
Representation learning
Representations
Semantics
Similarity
Time series
Time series analysis
time series classification
time series clustering
ISSN:2471-285X, 2471-285X
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Recently, contrastive learning (CL) is a promising way of learning discriminative representations from time series data. In the representation hierarchy, semantic information extracted from lower levels is the basis of that captured from higher levels. Low-level semantic information is essential and should be considered in the CL process. However, the existing CL algorithms mainly focus on the similarity of high-level semantic information. Considering the similarity of low-level semantic information may improve the performance of CL. To this end, we present a deep contrastive representation learning with self-distillation (DCRLS) for the time series domain. DCRLS gracefully combine data augmentation, deep contrastive learning, and self distillation. Our data augmentation provides different views from the same sample as the input of DCRLS. Unlike most CL algorithms that concentrate on high-level semantic information only, our deep contrastive learning also considers the contrast similarity of low-level semantic information between peer residual blocks. Our self distillation promotes knowledge flow from high-level to low-level blocks to help regularize DCRLS in the knowledge transfer process. The experimental results demonstrate that the DCRLS-based structures achieve excellent performance on classification and clustering on 36 UCR2018 datasets.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2471-285X
2471-285X
DOI:10.1109/TETCI.2023.3304948