ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs

Transformers have become keystone models in natural language processing over the past decade. They have achieved great popularity in deep learning applications, but the increasing sizes of the parameter spaces required by transformer models generate a commensurate need to accelerate performance. Nat...

Full description

Bibliographic Details

Published in:	Proceedings - IEEE International Parallel and Distributed Processing Symposium pp. 344 - 355
Main Authors:	Zhai, Yujia, Jiang, Chengquan, Wang, Leyuan, Jia, Xiaoying, Zhang, Shang, Chen, Zizhong, Liu, Xin, Zhu, Yibo
Format:	Conference Proceeding
Language:	English
Published:	IEEE 01.05.2023
Subjects:	BERT Bit error rate CUTLASS Deep learning Distributed processing Graphics processing units Large Language Models Multi-head Attention Natural Language Processing NVIDIA GPU Optimization methods Technological innovation Training Transformer
ISSN:	1530-2075
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!