WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal repre...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of selected topics in signal processing Vol. 16; no. 6; pp. 1505 - 1518
Main Authors: Chen, Sanyuan, Wang, Chengyi, Chen, Zhengyang, Wu, Yu, Liu, Shujie, Chen, Zhuo, Li, Jinyu, Kanda, Naoyuki, Yoshioka, Takuya, Xiao, Xiong, Wu, Jian, Zhou, Long, Ren, Shuo, Qian, Yanmin, Qian, Yao, Zeng, Michael, Yu, Xiangzhan, Wei, Furu
Format: Journal Article
Language:English
Published: New York IEEE 01.10.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Benchmark testing
Benchmarks
Convolution
Noise reduction
Predictive models
Self-supervised learning
Signal processing
Speech
speech pre-training
Speech processing
Speech recognition
Supervised learning
Training
ISSN:1932-4553, 1941-0484
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60 k hours to 94 k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1932-4553
1941-0484
DOI:10.1109/JSTSP.2022.3188113