Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

Transformers have transformed the field of natural language processing. Their superior performance is largely attributed to the use of stacked "self-attention" layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the...

Full description

Bibliographic Details

Published in:	2021 58th ACM/IEEE Design Automation Conference (DAC) pp. 469 - 474
Main Authors:	Stevens, Jacob R., Venkatesan, Rangharajan, Dai, Steve, Khailany, Brucek, Raghunathan, Anand
Format:	Conference Proceeding
Language:	English
Published:	IEEE 05.12.2021
Subjects:	Deep learning Design automation Hardware hardware/software codesign Natural language processing neural network accelerators Neural networks Software Transformers
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!