Extensions of recurrent neural network language model

We present several modifications of the original recurrent neural net work language model (RNN LM). While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we...

Full description

Saved in:
Bibliographic Details
Published in:2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 5528 - 5531
Main Authors: Mikolov, Tomas, Kombrink, Stefan, Burget, Lukas, Cernocky, Jan Honza, Khudanpur, Sanjeev
Format: Conference Proceeding
Language:English
Published: IEEE 01.05.2011
Subjects:
Artificial neural networks
Backpropagation
Computational modeling
language modeling
Probability distribution
Recurrent neural networks
speech recognition
Training
Vocabulary
ISBN:9781457705380, 1457705389
ISSN:1520-6149
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract We present several modifications of the original recurrent neural net work language model (RNN LM). While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one.
AbstractList We present several modifications of the original recurrent neural net work language model (RNN LM). While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one.
Author Cernocky, Jan Honza
Mikolov, Tomas
Burget, Lukas
Kombrink, Stefan
Khudanpur, Sanjeev
Author_xml – sequence: 1
  givenname: Tomas
  surname: Mikolov
  fullname: Mikolov, Tomas
  email: imikolov@fit.vutbr.cz
  organization: Brno Univ. of Technol., Brno, Czech Republic
– sequence: 2
  givenname: Stefan
  surname: Kombrink
  fullname: Kombrink, Stefan
  email: kombrink@fit.vutbr.cz
  organization: Brno Univ. of Technol., Brno, Czech Republic
– sequence: 3
  givenname: Lukas
  surname: Burget
  fullname: Burget, Lukas
  email: burget@fit.vutbr.cz
  organization: Brno Univ. of Technol., Brno, Czech Republic
– sequence: 4
  givenname: Jan Honza
  surname: Cernocky
  fullname: Cernocky, Jan Honza
  email: cernocky@fit.vutbr.cz
  organization: Brno Univ. of Technol., Brno, Czech Republic
– sequence: 5
  givenname: Sanjeev
  surname: Khudanpur
  fullname: Khudanpur, Sanjeev
  email: khudanpur@jhu.edu
  organization: Dept. of Electr. & Comput. Eng., Johns Hopkins Univ., Baltimore, MD, USA
BookMark eNo1j81OwzAQhI0oEm3JE_SSF0jw2l47PqKq_EiVQCqcKztZV4HUQU4q4O2JRJnLN3MZzSzYLPaRGFsBLwG4vX1a3-12L6XgACVaZTTABVuAQmM4SmsuWWZN9Z8rPmNzQMELDcpes2wY3vkkLYxBO2e4-R4pDm0fh7wPeaL6lBLFMY90Sq6bMH716SPvXDyc3IHyY99Qd8OugusGys5csrf7zev6sdg-P0z7tkULBsfCg2zEtNGTd8GSblSlMeDkbRASmrpCHiDUyrtaYq3JyeBReK-8CEaSXLLVX29LRPvP1B5d-tmfX8tfSY5MVg
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/ICASSP.2011.5947611
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISBN 1457705397
9781457705373
9781457705397
1457705370
EndPage 5531
ExternalDocumentID 5947611
Genre orig-research
GroupedDBID 23M
29P
6IE
6IF
6IH
6IK
6IL
6IM
6IN
AAJGR
AAWTH
ABLEC
ACGFS
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
IPLJI
M43
OCL
RIE
RIL
RIO
RNS
ID FETCH-LOGICAL-i175t-b13d2594bebaf9e6d4865f5af99f231dc850f1fc4bac35c6ea3fb52bb4b2f73e3
IEDL.DBID RIE
ISBN 9781457705380
1457705389
ISICitedReferencesCount 784
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000296062406059&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 1520-6149
IngestDate Wed Aug 27 02:54:03 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i175t-b13d2594bebaf9e6d4865f5af99f231dc850f1fc4bac35c6ea3fb52bb4b2f73e3
PageCount 4
ParticipantIDs ieee_primary_5947611
PublicationCentury 2000
PublicationDate 2011-May
PublicationDateYYYYMMDD 2011-05-01
PublicationDate_xml – month: 05
  year: 2011
  text: 2011-May
PublicationDecade 2010
PublicationTitle 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
PublicationTitleAbbrev ICASSP
PublicationYear 2011
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0000627759
ssj0008748
Score 2.4372094
Snippet We present several modifications of the original recurrent neural net work language model (RNN LM). While this model has been shown to significantly outperform...
SourceID ieee
SourceType Publisher
StartPage 5528
SubjectTerms Artificial neural networks
Backpropagation
Computational modeling
language modeling
Probability distribution
Recurrent neural networks
speech recognition
Training
Vocabulary
Title Extensions of recurrent neural network language model
URI https://ieeexplore.ieee.org/document/5947611
WOSCitedRecordID wos000296062406059&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED6VigEWHi3iLQ-MhCbxKx5R1QqWqlJB6lbZsS1VQi0qKerP5-ykBSQWlsTJkId9yt19ufs-gDuMYJXVLE8w9MeN8zYxuckSjfYjGC-EsSyKTcjRqJhO1bgF97teGOdcLD5zD2EY_-XbZbkOUFmPK4ZZN-Y6e1KKuldrh6cEut1IXdd8hQsZlbPQPYX0iKnY1MWlRKMr1JbrqTlOGzqiLFW95_7jZDKuuT2b-_0SXol-Z3j0vyc-hu53Ax8Z71zTCbTc4hQOf3APdoAPNrF4Ha2OLD1ZBdw9MDWRwHCp33AX68PJFtAkUTOnC6_DwUv_KWk0FJI5BgZVYjJqMcNhxhntlROWFYJ7jmPlMbSzZcFTn_mSGV1SXgqnqTc8N4aZ3Evq6Bm0F8uFOweC82V87hReMGVaSGOtpBTXWVHGNLcX0AkTMHuvaTJmzbtf_n36Cg5qeDbUDl5Du1qt3Q3sl5_V_GN1G9f2CxUtnd8
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB5KFdSLj1Z8m4NHY5PsbjZ7lNLSYi2FVuit7GZ3oSCN1Fb8-c5u0qrgxUuyySGP3SEz82Xm-wDuMIIVWtIkxNAfN8bqUCUqDiXaT0pZlipNvdgEHw6z6VSManC_7YUxxvjiM_Pghv5fvi7ytYPKWkxQzLox19lhlCZR2a21RVQc4a4nr6u-wxn32lnooFyCRIVv62Kco9llYsP2VB1HFSFRHIlWv_04Ho9Kds_qjr-kV7zn6R7-75mPoPndwheMts7pGGpmcQIHP9gHG8A6n758He0uKGywdMi742oKHMelfMWdrxAPNpBm4FVzmvDS7UzavbBSUQjnGBqsQhUTjTkOVUZJK0yqaZYyy3AsLAZ3Os9YZGObUyVzwvLUSGIVS5SiKrGcGHIK9UWxMGcQ4HwpmxiBF4yoTLnSmhOCKy0IpZLpc2i4CZi9lUQZs-rdL_4-fQt7vcnzYDboD58uYb8Ea10l4RXUV8u1uYbd_GM1f1_e-HX-AoK7oSY
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2011+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29&rft.atitle=Extensions+of+recurrent+neural+network+language+model&rft.au=Mikolov%2C+Tomas&rft.au=Kombrink%2C+Stefan&rft.au=Burget%2C+Lukas&rft.au=Cernocky%2C+Jan+Honza&rft.date=2011-05-01&rft.pub=IEEE&rft.isbn=9781457705380&rft.issn=1520-6149&rft.spage=5528&rft.epage=5531&rft_id=info:doi/10.1109%2FICASSP.2011.5947611&rft.externalDocID=5947611
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1520-6149&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1520-6149&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1520-6149&client=summon