A Sparse Plus Low-Rank Exponential Language Model for Limited Resource Scenarios

This paper describes a new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the training data and one or more sparse matrices that learn exceptions (e.g., keywords). The low-rank matrices induce continuous-space representat...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE/ACM transactions on audio, speech, and language processing Vol. 23; no. 3; pp. 494 - 504
Main Authors:	Hutchinson, Brian, Ostendorf, Mari, Fazel, Maryam
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 01.03.2015 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Adaptation models Algorithms Data models exponential History Language model log bilinear low-hyphen Mathematical models Matrix decomposition Natural language processing Regularity Representations sparse Sparse matrices Speech Training
ISSN:	2329-9290, 2329-9304
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	This paper describes a new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the training data and one or more sparse matrices that learn exceptions (e.g., keywords). The low-rank matrices induce continuous-space representations of words and histories. The sparse matrices learn multi-word lexical items and topic/domain idiosyncrasies. This model generalizes the standard ℓ 1 -regularized exponential language model, and has an efficient accelerated first-order training algorithm. Language modeling experiments show that the approach is useful in scenarios with limited training data, including low resource languages and domain adaptation.
AbstractList	This paper describes a new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the training data and one or more sparse matrices that learn exceptions (e.g., keywords). The low-rank matrices induce continuous-space representations of words and histories. The sparse matrices learn multi-word lexical items and topic/domain idiosyncrasies. This model generalizes the standard ℓ 1 -regularized exponential language model, and has an efficient accelerated first-order training algorithm. Language modeling experiments show that the approach is useful in scenarios with limited training data, including low resource languages and domain adaptation. This paper describes a new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the training data and one or more sparse matrices that learn exceptions (e.g., keywords). The low-rank matrices induce continuous-space representations of words and histories. The sparse matrices learn multi-word lexical items and topic/domain idiosyncrasies. This model generalizes the standard [ell] 1 -regularized exponential language model, and has an efficient accelerated first-order training algorithm. Language modeling experiments show that the approach is useful in scenarios with limited training data, including low resource languages and domain adaptation. This paper describes a new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the training data and one or more sparse matrices that learn exceptions (e.g., keywords). The low-rank matrices induce continuous-space representations of words and histories. The sparse matrices learn multi-word lexical items and topic/domain idiosyncrasies. This model generalizes the standard [Formula Omitted]-regularized exponential language model, and has an efficient accelerated first-order training algorithm. Language modeling experiments show that the approach is useful in scenarios with limited training data, including low resource languages and domain adaptation.
Author	Fazel, Maryam Ostendorf, Mari Hutchinson, Brian
Author_xml	– sequence: 1 givenname: Brian surname: Hutchinson fullname: Hutchinson, Brian email: brian.hutchinson@wwu.edu organization: Comput. Sci. Dept., Western Washington Univ., Bellingham, WA, USA – sequence: 2 givenname: Mari surname: Ostendorf fullname: Ostendorf, Mari organization: Electr. Eng. Dept., Univ. of Washington, Seattle, WA, USA – sequence: 3 givenname: Maryam surname: Fazel fullname: Fazel, Maryam organization: Electr. Eng. Dept., Univ. of Washington, Seattle, WA, USA
BookMark	eNp9kD1PwzAQhi0EEp9_ABZLLCwpPjuJ47FCfElBVG2ZI8e5IENqFzsR8O8JFBgYmO6G93nv9OyTbecdEnIMbALA1PlyuihnE84gnXAhVabEFtnjgqtECZZu_-xcsV1yFOMTYwyYVEqme2Q2pYu1DhHprBsiLf1rMtfumV6-rccjrre6o6V2j4N-RHrnG-xo6wMt7cr22NA5Rj8Eg3Rh0OlgfTwkO63uIh59zwPycHW5vLhJyvvr24tpmRjBiz4B0YBIwYjUGCNroSQv8lYL2bBWGg2ImZYix1qDrNMMeM3qDHIjirpRMs_EATnb9K6Dfxkw9tXKRoNdpx36IVaQS6lA5LIYo6d_ok_j0278bkzlDADSr8JikzLBxxiwrYztdW-964O2XQWs-rRdfdmuPm1X37ZHlP9B18GudHj_HzrZQBYRfwHJMiaKTHwA5EuLgA
CODEN	ITASD8
CitedBy_id	crossref_primary_10_1109_TASLP_2015_2482118 crossref_primary_10_1016_j_specom_2019_03_004 crossref_primary_10_1109_TASLP_2015_2405131 crossref_primary_10_1162_tacl_a_00035
Cites_doi	10.1109/ICASSP.2007.367158 10.1162/jmlr.2003.3.4-5.993 10.1016/j.specom.2003.08.002 10.1109/ASRU.2011.6163937 10.1109/ASRU.2009.5373380 10.3115/1620754.1620822 10.1109/ICASSP.2002.1005858 10.1109/LSP.2011.2160850 10.21437/Interspeech.2010-343 10.21437/Interspeech.2010-519 10.21437/Interspeech.2011-243 10.1006/csla.1999.0128 10.1198/016214506000000302 10.1109/ICASSP.2011.5947608 10.1109/ICASSP.2011.5947611 10.1016/j.csl.2005.10.001 10.3115/1620754.1620820 10.1109/ICASSP.2011.5947609 10.1007/978-1-4419-8853-9 10.1109/ICASSP.2013.6639340 10.1137/080716542 10.1006/csla.1996.0011 10.21437/Interspeech.2012-459 10.1162/153244303322533223 10.1145/1273496.1273499 10.21437/Interspeech.2011-242 10.1109/ICASSP.1998.675356 10.1137/070697835 10.1006/csla.2001.0174 10.1109/ICASSP.1993.319375 10.3115/v1/D14-1162 10.21437/Interspeech.2010-341 10.1016/j.csl.2006.09.003 10.21437/Interspeech.2008-253 10.1145/1273496.1273577 10.1109/89.817454 10.21437/Eurospeech.1999-409
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Mar 2015
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Mar 2015
DBID	97E RIA RIE AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D
DOI	10.1109/TASLP.2014.2379593
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional
DatabaseTitleList	Computer and Information Systems Abstracts Computer and Information Systems Abstracts
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	2329-9304
EndPage	504
ExternalDocumentID	3611613931 10_1109_TASLP_2014_2379593 7050385
Genre	orig-research
GroupedDBID	0R~ 4.4 6IK 97E AAJGR AAKMM AALFJ AARMG AASAJ AAWTH AAWTV ABAZT ABQJQ ABVLG ACIWK ACM ADBCU AEBYY AEFXT AEJOY AENSD AFWIH AFWXC AGQYO AGSQL AHBIQ AIKLT AKJIK AKQYR AKRVB ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CCLIF EBS EJD GUFHI HGAVV IFIPE IPLJI JAVBF LHSKQ M43 OCL PQQKQ RIA RIE RNS ROL AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c328t-13d1341c34ccc7b397286fa37d0f7ca1ee5a736eba17b4512b0b516c38bd97653
IEDL.DBID	RIE
ISICitedReferencesCount	7
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000350876100008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2329-9290
IngestDate	Sat Sep 27 20:09:17 EDT 2025 Sun Nov 09 08:04:28 EST 2025 Tue Nov 18 19:41:25 EST 2025 Sat Nov 29 07:52:15 EST 2025 Tue Aug 26 16:39:04 EDT 2025
IsDoiOpenAccess	false
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	3
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c328t-13d1341c34ccc7b397286fa37d0f7ca1ee5a736eba17b4512b0b516c38bd97653
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23
PQID	1660111465
PQPubID	85426
PageCount	11
ParticipantIDs	proquest_journals_1660111465 proquest_miscellaneous_1677913678 ieee_primary_7050385 crossref_citationtrail_10_1109_TASLP_2014_2379593 crossref_primary_10_1109_TASLP_2014_2379593
PublicationCentury	2000
PublicationDate	2015-March 2015-3-00 20150301
PublicationDateYYYYMMDD	2015-03-01
PublicationDate_xml	– month: 03 year: 2015 text: 2015-March
PublicationDecade	2010
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE/ACM transactions on audio, speech, and language processing
PublicationTitleAbbrev	TASLP
PublicationYear	2015
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref12 ref15 ref14 mikolov (ref32) 2010 brown (ref2) 1992; 18 ref11 ref17 mikolov (ref35) 2011 ref16 huang (ref48) 2008 srebro (ref13) 2004 hutchinson (ref10) 2013 ref50 arisoy (ref31) 2012 ref46 fazel (ref18) 2002 ref45 ref47 ref41 ref44 ref43 stolcke (ref37) 2002 parikh (ref7) 2013; abs 1312 7077 ref9 ref3 ref6 ref5 ref40 chen (ref27) 2010 hutchinson (ref8) 2012 ref36 ref33 toh (ref19) 2010; 6 ref1 nesterov (ref20) 2004 nocedal (ref22) 2000 mikolov (ref34) 2011 siu (ref4) 2000; 8 adda (ref42) 1999 ref24 ref23 ref26 ref25 ref21 alume (ref38) 2010 zweig (ref29) 2011 ref28 bengio (ref30) 2001 wood (ref49) 2009; 12 graff (ref39) 2003
References_xml	– ident: ref47 doi: 10.1109/ICASSP.2007.367158 – ident: ref45 doi: 10.1162/jmlr.2003.3.4-5.993 – start-page: 901 year: 2002 ident: ref37 article-title: SRILM - an extensible language modeling toolkit publication-title: Proc ICSLP – year: 2000 ident: ref22 publication-title: Numerical Optimization – ident: ref43 doi: 10.1016/j.specom.2003.08.002 – year: 2003 ident: ref39 article-title: English Gigaword LDC2003T05 publication-title: Linguistic Data Consortium – ident: ref25 doi: 10.1109/ASRU.2011.6163937 – volume: 12 year: 2009 ident: ref49 article-title: A hierarchical nonparametric Bayesian approach to statistical language model domain adaptation publication-title: Proc AISTATS – ident: ref24 doi: 10.1109/ASRU.2009.5373380 – ident: ref23 doi: 10.3115/1620754.1620822 – ident: ref40 doi: 10.1109/ICASSP.2002.1005858 – ident: ref6 doi: 10.1109/LSP.2011.2160850 – start-page: 1045 year: 2010 ident: ref32 article-title: Recurrent neural network based language model publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2010-343 – start-page: 1820 year: 2010 ident: ref38 article-title: Efficient estimation of maximum entropy language models with n-gram features: An SRILM extension publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2010-519 – start-page: 609 year: 2011 ident: ref29 article-title: Personalizing model M for voice-search publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2011-243 – ident: ref1 doi: 10.1006/csla.1999.0128 – volume: abs 1312 7077 year: 2013 ident: ref7 article-title: Language modeling with power low rank ensembles publication-title: Proc CoRR – volume: 6 start-page: 615 year: 2010 ident: ref19 article-title: An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems publication-title: Pacific J Optimiz – ident: ref46 doi: 10.1198/016214506000000302 – ident: ref28 doi: 10.1109/ICASSP.2011.5947608 – ident: ref33 doi: 10.1109/ICASSP.2011.5947611 – year: 2002 ident: ref18 publication-title: Matrix Rank Minimization With Applications – ident: ref5 doi: 10.1016/j.csl.2005.10.001 – year: 2013 ident: ref10 publication-title: Rank and sparsity in language processing – ident: ref12 doi: 10.3115/1620754.1620820 – ident: ref26 doi: 10.1109/ICASSP.2011.5947609 – year: 2004 ident: ref20 publication-title: Introductory Lectures on Convex Optimization doi: 10.1007/978-1-4419-8853-9 – year: 2004 ident: ref13 publication-title: Learning with Matrix Factorization – ident: ref9 doi: 10.1109/ICASSP.2013.6639340 – ident: ref21 doi: 10.1137/080716542 – ident: ref11 doi: 10.1006/csla.1996.0011 – start-page: 196 year: 2011 ident: ref35 article-title: RNNLM recurrent neural network language modeling toolkit publication-title: Proc ASRU – year: 2012 ident: ref8 article-title: A sparse plus low rank maximum entropy language model publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2012-459 – ident: ref16 doi: 10.1162/153244303322533223 – ident: ref15 doi: 10.1145/1273496.1273499 – start-page: 605 year: 2011 ident: ref34 article-title: Empirical evaluation and combination of advanced language modeling techniques publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2011-242 – ident: ref44 doi: 10.1109/ICASSP.1998.675356 – ident: ref14 doi: 10.1137/070697835 – ident: ref3 doi: 10.1006/csla.2001.0174 – ident: ref41 doi: 10.1109/ICASSP.1993.319375 – ident: ref50 doi: 10.3115/v1/D14-1162 – start-page: 932 year: 2001 ident: ref30 article-title: A neural probabilistic language model publication-title: Proc NIPS – start-page: 1037 year: 2010 ident: ref27 article-title: Enhanced word classing for model M publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2010-341 – ident: ref17 doi: 10.1016/j.csl.2006.09.003 – start-page: 833 year: 2008 ident: ref48 article-title: Unsupervised language model adaptation based on topic and role information in multiparty meetings publication-title: Proc INTERSPEECH doi: 10.21437/Interspeech.2008-253 – volume: 18 start-page: 467 year: 1992 ident: ref2 article-title: Class-based n-gram models of natural language publication-title: Comput Linguist – ident: ref36 doi: 10.1145/1273496.1273577 – volume: 8 start-page: 63 year: 2000 ident: ref4 article-title: Variable n-grams and extensions for conversational speech language modeling publication-title: IEEE Trans Speech Audio Process doi: 10.1109/89.817454 – start-page: 1759 year: 1999 ident: ref42 article-title: Language modeling for broadcast news transcription publication-title: Proc EUROSPEECH doi: 10.21437/Eurospeech.1999-409 – start-page: 20 year: 2012 ident: ref31 article-title: Deep neural network language models publication-title: Proc NAACL-HLT Workshop Future Lang Model for HLT
SSID	ssj0001079974
Score	2.174668
Snippet	This paper describes a new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	494
SubjectTerms	Adaptation models Algorithms Data models exponential History Language model log bilinear low-hyphen Mathematical models Matrix decomposition Natural language processing Regularity Representations sparse Sparse matrices Speech Training
Title	A Sparse Plus Low-Rank Exponential Language Model for Limited Resource Scenarios
URI	https://ieeexplore.ieee.org/document/7050385 https://www.proquest.com/docview/1660111465 https://www.proquest.com/docview/1677913678
Volume	23
WOSCitedRecordID	wos000350876100008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2329-9304 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001079974 issn: 2329-9290 databaseCode: RIE dateStart: 20140101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8QwEB5UPOjBt7i-iOBNq2nTNu1xEcXDIour4K0k01kQl3ZxXfXnO8l2V0URvBWStCWTZB6Z-T6AY22Jj0VFQRajDOLUYpCniEFIUal13nfkVp5sQt_cZA8PeXcOTme1METkk8_ozD36u_yyxrELlZ1rD16SzMO81umkVuszniJ1nnvQZbYR8oC1vpzWyMj8_K7d63RdIld8FilHr62-6SFPrPLjNPYq5mr1fz-3BiuNKSnaE9mvwxxVG7D8BWBwE7pt0Ruy60qiOxiPRKd-C25N9SQu34d15fKEeHyniVgKR4s2EGzEiqbqSUxj-6KHVLFTXY-24P7q8u7iOmg4FAJUUeaY5ksH2YYqRkRt2fqIsrRvlC5lX6MJiRKjVUrWhNrGrP2ttEmYospsyZZKorZhoeI_2gERkpXI9lWiTBTryLCi13EqFRmFCmXSgnA6owU2AOOO52JQeEdD5oWXQuGkUDRSaMHJbMxwAq_xZ-9NN--zns2Ut2B_Krii2YGjIkzZ1XQl19x8NGvmveMuRExF9dj14eXoMOuy3d_fvAdL_P1kknO2Dwsvz2M6gEV8fXkcPR_6BfgBd7XVsQ
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3da9swED-6brD2ofvoRtN2nQZ729zKlm3Zj6G0tMwLYcmgb0Y6X2As2KFptv75PSlK1tFR2JtBkhE6Sfehu98P4KO2xNeioqhIUUZpbjEqc8QopqTRupw4citPNqEHg-LqqhxuwOd1LQwR-eQzOnaf_i2_6XDhQmUn2oOXZE_gqWPOCtVafyIqUpelh11mK6GMWO_LVZWMLE_G_VE1dKlc6XGiHMG2-ksTeWqVB_exVzLnL_5vei9hJxiTor-U_ivYoPY1bN-DGNyFYV-MZuy8khhOF3NRdb-jb6b9Kc5uZ13rMoV4fBVilsIRo00Fm7Ei1D2JVXRfjJBadqu7-Rv4fn42Pr2IAotChCopHNd840DbUKWIqC3bH0mRT4zSjZxoNDFRZrTKyZpY25T1v5U2i3NUhW3YVsnUW9hseUZ7IGKyEtnCypRJUp0YVvU6zaUio1ChzHoQr1a0xgAx7pguprV3NWRZeynUTgp1kEIPPq3HzJYAG4_23nXrvu4ZlrwHhyvB1eEMzus4Z2fTFV1z84d1M58e9yRiWuoWrg9vSIdaV-z_-8_v4fnF-GtVV5eDLwewxXPJlhloh7B5c72gd_AMf938mF8f-c14B8Se2Po
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Sparse+Plus+Low-Rank+Exponential+Language+Model+for+Limited+Resource+Scenarios&rft.jtitle=IEEE%2FACM+transactions+on+audio%2C+speech%2C+and+language+processing&rft.au=Hutchinson%2C+Brian&rft.au=Ostendorf%2C+Mari&rft.au=Fazel%2C+Maryam&rft.date=2015-03-01&rft.issn=2329-9290&rft.eissn=2329-9304&rft.volume=23&rft.issue=3&rft.spage=494&rft.epage=504&rft_id=info:doi/10.1109%2FTASLP.2014.2379593&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TASLP_2014_2379593
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2329-9290&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2329-9290&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2329-9290&client=summon