Sensitivity as a Complexity Measure for Sequence Classification Tasks
We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint su...
Saved in:
| Published in: | Transactions of the Association for Computational Linguistics Vol. 9; pp. 891 - 908 |
|---|---|
| Main Authors: | , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
One Rogers Street, Cambridge, MA 02142-1209, USA
MIT Press
18.08.2021
MIT Press Journals, The The MIT Press |
| Subjects: | |
| ISSN: | 2307-387X, 2307-387X |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders. |
|---|---|
| AbstractList | We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders. AbstractWe introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders. |
| Author | Jurafsky, Dan Hahn, Michael Futrell, Richard |
| Author_xml | – sequence: 1 givenname: Michael surname: Hahn fullname: Hahn, Michael email: mhahn2@stanford.edu organization: Stanford University, United States. mhahn2@stanford.edu – sequence: 2 givenname: Dan surname: Jurafsky fullname: Jurafsky, Dan email: jurafsky@stanford.edu organization: Stanford University, United States. jurafsky@stanford.edu – sequence: 3 givenname: Richard surname: Futrell fullname: Futrell, Richard email: rfutrell@uci.edu organization: University of California, Irvine, United States. rfutrell@uci.edu |
| BookMark | eNp1kE9vEzEQxS1UJErpjQ-wEhcOpIz_rNd74ICiQisVcWiRuFlj7xg5bNbBdqqWT8-mASmqymns0e-9eTMv2dGUJmLsNYczzrV4X9GPFi2AAvmMHQsJ3UKa7vvRwfsFOy1lBQDccANaHLPza5pKrPE21vsGS4PNMq03I93t_l8IyzZTE1JurunXliZPzXLEUmKIHmtMU3OD5Wd5xZ4HHAud_q0n7Nun85vlxeLq6-fL5cerhVetqgvtqNMuCBOUo-C4EdpxAcgV10pqBx76Xre9AyNpaMGTdr6XBg2C9hzkCbvc-w4JV3aT4xrzvU0Y7UMj5R8Wc41-JNu3hoJUqh3coHw3OCQUQ-elgSF0Qsxeb_Zem5zm1Uq1q7TN0xzfCjMPNart9EyJPeVzKiVTsD7Wh81rxjhaDnZ3fXt4_Vn07pHoX9T_4B_2-DoehNght72VSmveWwFCWphrb3_HzWP92yf0T476A-1aqn0 |
| CitedBy_id | crossref_primary_10_1007_s42421_024_00109_x crossref_primary_10_1016_j_jbi_2025_104881 |
| Cites_doi | 10.1162/neco.1997.9.8.1735 10.3115/v1/D14-1181 10.1145/1014052.1014073 10.18653/v1/2020.acl-main.24 10.1017/S1351324909990209 10.1162/tacl_a_00115 10.18653/v1/D17-1215 10.1017/CBO9781139814782 10.18653/v1/D18-1151 10.18653/v1/S18-2023 10.1109/TIT.1956.1056813 10.1109/SFCS.1988.21923 10.18653/v1/N19-1004 10.1016/0020-0190(96)00105-6 10.18653/v1/D16-1264 10.3115/1073336.1073357 10.18653/v1/2020.acl-main.768 10.3115/1219840.1219855 10.1016/j.neucom.2006.01.025 10.1137/0220062 10.18653/v1/W19-3901 10.18653/v1/D18-1546 10.1162/tacl_a_00306 10.18653/v1/P19-1449 10.18653/v1/N18-1101 10.18653/v1/N18-2017 10.18653/v1/P18-1031 10.3115/v1/D14-1162 10.1007/978-1-4757-3860-5 10.1007/s10579-005-7880-9 10.1007/978-3-030-36708-4_22 10.18653/v1/2020.emnlp-demos.6 10.3115/1218955.1218990 10.1162/tacl_a_00290 10.18653/v1/N18-1202 10.1016/S0010-0277(98)00034-1 10.18653/v1/2020.acl-demos.14 10.18653/v1/2020.findings-emnlp.117 10.18653/v1/W18-3012 10.18653/v1/2020.acl-demos.10 10.18653/v1/S17-2001 |
| ContentType | Journal Article |
| Copyright | 2021. This work is published under https://creativecommons.org/licenses/by/4.0/legalcode (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: 2021. This work is published under https://creativecommons.org/licenses/by/4.0/legalcode (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | AAYXX CITATION 7T9 8FE 8FG ABUWG AFKRA ALSLI ARAPS AZQEC BENPR BGLVJ CCPQU CPGLG CRLPW DWQXO GNUQQ HCIFZ JQ2 K7- P5Z P62 PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PRQQA DOA |
| DOI | 10.1162/tacl_a_00403 |
| DatabaseName | CrossRef Linguistics and Language Behavior Abstracts (LLBA) ProQuest SciTech Collection ProQuest Technology Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland Social Science Premium Collection Advanced Technologies & Computer Science Collection ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College Linguistics collection. Linguistics Database ProQuest Central ProQuest Central Student SciTech Premium Collection ProQuest Computer Science Collection Computer Science Database Advanced Technologies & Aerospace Database ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Premium ProQuest One Academic Publicly Available Content Database ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China ProQuest One Social Sciences DOAJ Directory of Open Access Journals |
| DatabaseTitle | CrossRef Publicly Available Content Database Computer Science Database ProQuest Central Student Technology Collection ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials ProQuest Computer Science Collection ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences Linguistics Collection ProQuest Central Korea ProQuest Central (New) Advanced Technologies & Aerospace Collection Social Science Premium Collection ProQuest One Social Sciences ProQuest One Academic Eastern Edition Linguistics and Language Behavior Abstracts (LLBA) ProQuest Technology Collection ProQuest SciTech Collection Advanced Technologies & Aerospace Database ProQuest One Academic UKI Edition Linguistics Database ProQuest One Academic ProQuest One Academic (New) |
| DatabaseTitleList | CrossRef Publicly Available Content Database |
| Database_xml | – sequence: 1 dbid: DOA name: DOAJ Directory of Open Access Journals url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: PIMPY name: Publicly Available Content Database url: http://search.proquest.com/publiccontent sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| EISSN | 2307-387X |
| EndPage | 908 |
| ExternalDocumentID | oai_doaj_org_article_958ef3445dbd4c7dbaea2d7c380df722 10_1162_tacl_a_00403 tacl_a_00403.pdf |
| GroupedDBID | AAFWJ AFPKN ALMA_UNASSIGNED_HOLDINGS EBS GROUPED_DOAJ JMNJE M~E OJV OK1 RMI AAYXX ABUWG AFFHD AFKRA ALSLI ARAPS BENPR BGLVJ CCPQU CITATION CPGLG CRLPW DWQXO HCIFZ K7- PHGZM PHGZT PIMPY PQGLB PRQQA 7T9 8FE 8FG AZQEC GNUQQ JQ2 P62 PKEHL PQEST PQQKQ PQUKI PRINS |
| ID | FETCH-LOGICAL-c454t-6be76bf28f4befb1826b120a1416436b0c099659b083ed50ce6bc938a8a06c103 |
| IEDL.DBID | BENPR |
| ISICitedReferencesCount | 10 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000751952200053&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2307-387X |
| IngestDate | Fri Oct 03 12:52:10 EDT 2025 Sat Nov 08 19:51:57 EST 2025 Sat Nov 29 05:34:28 EST 2025 Tue Nov 18 22:40:09 EST 2025 Sat Sep 30 12:10:40 EDT 2023 Sun Oct 01 15:05:32 EDT 2023 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c454t-6be76bf28f4befb1826b120a1416436b0c099659b083ed50ce6bc938a8a06c103 |
| Notes | 2021 ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| OpenAccessLink | https://www.proquest.com/docview/2893884576?pq-origsite=%requestingapplication% |
| PQID | 2893884576 |
| PQPubID | 6535866 |
| PageCount | 18 |
| ParticipantIDs | proquest_journals_2893884576 crossref_primary_10_1162_tacl_a_00403 doaj_primary_oai_doaj_org_article_958ef3445dbd4c7dbaea2d7c380df722 crossref_citationtrail_10_1162_tacl_a_00403 mit_journals_taclv9_346619_2023_09_29_zip_tacl_a_00403 mit_journals_10_1162_tacl_a_00403 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-08-18 |
| PublicationDateYYYYMMDD | 2021-08-18 |
| PublicationDate_xml | – month: 08 year: 2021 text: 2021-08-18 day: 18 |
| PublicationDecade | 2020 |
| PublicationPlace | One Rogers Street, Cambridge, MA 02142-1209, USA |
| PublicationPlace_xml | – name: One Rogers Street, Cambridge, MA 02142-1209, USA – name: Cambridge |
| PublicationTitle | Transactions of the Association for Computational Linguistics |
| PublicationYear | 2021 |
| Publisher | MIT Press MIT Press Journals, The The MIT Press |
| Publisher_xml | – name: MIT Press – name: MIT Press Journals, The – name: The MIT Press |
| References | O’Donnell (2021081914590098500_bib43) 2014 Howard (2021081914590098500_bib23) 2018 Cao (2021081914590098500_bib3) 2019 Poliak (2021081914590098500_bib51) 2018 Jeretic (2021081914590098500_bib26) 2020 Horne (2021081914590098500_bib22) 1994 Gauthier (2021081914590098500_bib15) 2020 Rajpurkar (2021081914590098500_bib56) 2016 Wang (2021081914590098500_bib63) 2019 Gururangan (2021081914590098500_bib17) 2018 Pennington (2021081914590098500_bib48) 2014 Radford (2021081914590098500_bib54) 2019; 1 Qi (2021081914590098500_bib52) 2018 Merrill (2021081914590098500_bib37) 2019 Qi (2021081914590098500_bib53) 2020 Peters (2021081914590098500_bib49) 2018 Kaushik (2021081914590098500_bib29) 2020 Nivre (2021081914590098500_bib41) 2016 Hatami (2021081914590098500_bib20) 2010; 4 Chomsky (2021081914590098500_bib6) 1956; 2 Szegedy (2021081914590098500_bib59) 2014 Dolan (2021081914590098500_bib10) 2005 Pang (2021081914590098500_bib47) 2005 Minsky (2021081914590098500_bib38) 1969 Levesque (2021081914590098500_bib32) 2012 Wiebe (2021081914590098500_bib65) 2005; 39 Hochreiter (2021081914590098500_bib21) 1997; 9 Hu (2021081914590098500_bib25) 2004 Datta (2021081914590098500_bib8) 2020 Hale (2021081914590098500_bib19) 2001 Ott (2021081914590098500_bib44) 2019 Valle-Perez (2021081914590098500_bib60) 2019 Hahn (2021081914590098500_bib18) 2020; 8 Novak (2021081914590098500_bib42) 2018 Franco (2021081914590098500_bib12) 2006; 70 Kim (2021081914590098500_bib31) 2014 Gibson (2021081914590098500_bib16) 1998; 68 Li (2021081914590098500_bib33) 1993 Silveira (2021081914590098500_bib57) 2014 Wolf (2021081914590098500_bib68) 2019 Devlin (2021081914590098500_bib9) 2019 Dagan (2021081914590098500_bib7) 2009; 15 Gardner (2021081914590098500_bib14) 2020 Wang (2021081914590098500_bib62) 2019 Jia (2021081914590098500_bib27) 2017 Warstadt (2021081914590098500_bib64) 2019; 7 Kaushik (2021081914590098500_bib30) 2018 Petrov (2021081914590098500_bib50) 2012 Wieting (2021081914590098500_bib66) 2016 Arora (2021081914590098500_bib1) 2017 Futrell (2021081914590098500_bib13) 2019 Marvin (2021081914590098500_bib36) 2018 Liao (2021081914590098500_bib34) 2020 Xu (2021081914590098500_bib69) 2019 Nisan (2021081914590098500_bib40) 1991; 20 Chelba (2021081914590098500_bib5) 2014 Vaswani (2021081914590098500_bib61) 2017 Williams (2021081914590098500_bib67) 2018 Nangia (2021081914590098500_bib39) 2019 Hu (2021081914590098500_bib24) 2020; 3 Palma (2021081914590098500_bib45) 2019 Yang (2021081914590098500_bib70) 2019 Bernasconi (2021081914590098500_bib2) 1996; 59 Cer (2021081914590098500_bib4) 2017 Rahaman (2021081914590098500_bib55) 2019 Kahn (2021081914590098500_bib28) 1988 Pang (2021081914590098500_bib46) 2004 Socher (2021081914590098500_bib58) 2013 Ethayarajh (2021081914590098500_bib11) 2018 Linzen (2021081914590098500_bib35) 2016; 4 |
| References_xml | – volume: 9 start-page: 1735 issue: 8 year: 1997 ident: 2021081914590098500_bib21 article-title: Long short-term memory publication-title: Neural Computation doi: 10.1162/neco.1997.9.8.1735 – start-page: 1746 volume-title: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) year: 2014 ident: 2021081914590098500_bib31 article-title: Convolutional neural networks for sentence classification doi: 10.3115/v1/D14-1181 – year: 2019 ident: 2021081914590098500_bib3 article-title: Towards understanding the spectral bias of deep learning publication-title: arXiv preprint arXiv:1912.01198 – start-page: 168 volume-title: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining year: 2004 ident: 2021081914590098500_bib25 article-title: Mining and summarizing customer reviews doi: 10.1145/1014052.1014073 – start-page: 263 volume-title: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics year: 2020 ident: 2021081914590098500_bib34 article-title: Probabilistically masked language model capable of autoregressive generation in arbitrary word order doi: 10.18653/v1/2020.acl-main.24 – volume: 15 issue: 4 year: 2009 ident: 2021081914590098500_bib7 article-title: Recognizing textual entailment: Rational, evaluation and approaches publication-title: Natural Language Engineering doi: 10.1017/S1351324909990209 – volume: 4 start-page: 521 year: 2016 ident: 2021081914590098500_bib35 article-title: Assessing the ability of LSTMs to learn syntax-sensitive dependencies publication-title: Transactions of the Association for Computational Linguistics doi: 10.1162/tacl_a_00115 – start-page: 2021 volume-title: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing year: 2017 ident: 2021081914590098500_bib27 article-title: Adversarial examples for evaluating reading comprehension systems doi: 10.18653/v1/D17-1215 – volume-title: Analysis of Boolean Functions year: 2014 ident: 2021081914590098500_bib43 doi: 10.1017/CBO9781139814782 – start-page: 552 volume-title: KR’12 Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning year: 2012 ident: 2021081914590098500_bib32 article-title: The winograd schema challenge – year: 2018 ident: 2021081914590098500_bib36 article-title: Targeted syntactic evaluation of language models publication-title: arXiv preprint arXiv:1808.09031 doi: 10.18653/v1/D18-1151 – volume-title: ICLR 2020: Eighth International Conference on Learning Representations year: 2020 ident: 2021081914590098500_bib29 article-title: Learning the difference that makes a difference with counterfactually-augmented data – start-page: 180 volume-title: Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics year: 2018 ident: 2021081914590098500_bib51 article-title: Hypothesis only baselines in natural language inference doi: 10.18653/v1/S18-2023 – volume: 4 start-page: 1 year: 2010 ident: 2021081914590098500_bib20 article-title: Variations on the sensitivity conjecture publication-title: Theory of Computing – start-page: 359 volume-title: Advances in Neural Information Processing Systems year: 1994 ident: 2021081914590098500_bib22 article-title: Bounds on the complexity of recurrent neural network implementations of finite state machines – volume: 2 start-page: 113 issue: 3 year: 1956 ident: 2021081914590098500_bib6 article-title: Three models for the description of language publication-title: IEEE Transactions on Information Theory doi: 10.1109/TIT.1956.1056813 – start-page: 160 volume-title: Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies year: 2018 ident: 2021081914590098500_bib52 article-title: Universal dependency parsing from scratch – volume-title: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 – May 3, 2018, Conference Track Proceedings year: 2018 ident: 2021081914590098500_bib42 article-title: Sensitivity and generalization in neural networks: an empirical study – volume-title: Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005 year: 2005 ident: 2021081914590098500_bib10 article-title: Automatically constructing a corpus of sentential paraphrases – volume-title: Tenth International Conference on Language Resources and Evaluation (LREC 2016) year: 2016 ident: 2021081914590098500_bib41 article-title: Universal dependencies v1: A multilingual treebank collection – volume-title: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, Conference Track Proceedings year: 2014 ident: 2021081914590098500_bib59 article-title: Intriguing properties of neural networks – start-page: 2635 volume-title: INTERSPEECH year: 2014 ident: 2021081914590098500_bib5 article-title: One billion word benchmark for measuring progress in statistical language modeling – start-page: 1962 volume-title: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8–14, 2019, Vancouver, BC, Canada year: 2019 ident: 2021081914590098500_bib45 article-title: Random deep neural networks are biased towards simple functions – start-page: 68 volume-title: [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science year: 1988 ident: 2021081914590098500_bib28 article-title: The influence of variables on Boolean functions doi: 10.1109/SFCS.1988.21923 – start-page: 32 volume-title: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) year: 2019 ident: 2021081914590098500_bib13 article-title: Neural language models as psycholinguistic subjects: Representations of syntactic state doi: 10.18653/v1/N19-1004 – volume-title: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings year: 2016 ident: 2021081914590098500_bib66 article-title: Towards universal paraphrastic sentence embeddings – volume: 59 start-page: 151 issue: 3 year: 1996 ident: 2021081914590098500_bib2 article-title: Sensitivity vs. block sensitivity (an average-case study) publication-title: Information Processing Letters doi: 10.1016/0020-0190(96)00105-6 – start-page: 2383 volume-title: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing year: 2016 ident: 2021081914590098500_bib56 article-title: Squad: 100,000+ questions for machine comprehension of text doi: 10.18653/v1/D16-1264 – start-page: 1 volume-title: Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics and Language Technologies year: 2001 ident: 2021081914590098500_bib19 article-title: A probabilistic Earley parser as a psycholinguistic model doi: 10.3115/1073336.1073357 – start-page: 8690 volume-title: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020 year: 2020 ident: 2021081914590098500_bib26 article-title: Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition doi: 10.18653/v1/2020.acl-main.768 – start-page: 115 volume-title: Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05) year: 2005 ident: 2021081914590098500_bib47 article-title: Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales doi: 10.3115/1219840.1219855 – start-page: 2897 volume-title: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) year: 2014 ident: 2021081914590098500_bib57 article-title: A gold standard dependency corpus for English – volume-title: ICLR 2017: International Conference on Learning Representations 2017 year: 2017 ident: 2021081914590098500_bib1 article-title: A simple but tough-to-beat baseline for sentence embeddings – volume: 70 start-page: 351 issue: 1 year: 2006 ident: 2021081914590098500_bib12 article-title: Generalization ability of boolean functions implemented in feedforward neural networks publication-title: Neurocomputing doi: 10.1016/j.neucom.2006.01.025 – volume: 1 start-page: 9 issue: 8 year: 2019 ident: 2021081914590098500_bib54 article-title: Language models are unsupervised multitask learners publication-title: OpenAI Blog – start-page: 5998 volume-title: Advances in Neural Information Processing Systems year: 2017 ident: 2021081914590098500_bib61 article-title: Attention is all you need – volume: 20 start-page: 999 issue: 6 year: 1991 ident: 2021081914590098500_bib40 article-title: CREW PRAMs and decision trees publication-title: SIAM Journal on Computing doi: 10.1137/0220062 – year: 2019 ident: 2021081914590098500_bib37 article-title: Sequential neural networks as automata publication-title: arXiv preprint arXiv:1906.01615 doi: 10.18653/v1/W19-3901 – start-page: 5010 volume-title: EMNLP 2018: 2018 Conference on Empirical Methods in Natural Language Processing year: 2018 ident: 2021081914590098500_bib30 article-title: How much reading does reading comprehension require? A critical investigation of popular benchmarks doi: 10.18653/v1/D18-1546 – volume-title: Perceptrons: An Introduction to Computational Geometry year: 1969 ident: 2021081914590098500_bib38 – start-page: 48 volume-title: NAACL-HLT 2019: Annual Conference of the North American Chapter of the Association for Computational Linguistics year: 2019 ident: 2021081914590098500_bib44 article-title: fairseq: A fast, extensible toolkit for sequence modeling – start-page: 4171 volume-title: NAACL-HLT 2019: Annual Conference of the North American Chapter of the Association for Computational Linguistics year: 2019 ident: 2021081914590098500_bib9 article-title: BERT: Pre-training of deep bidirectional transformers for language understanding – volume: 8 start-page: 156 year: 2020 ident: 2021081914590098500_bib18 article-title: Theoretical limitations of self-attention in neural sequence models publication-title: Transactions of the Association for Computational Linguistics doi: 10.1162/tacl_a_00306 – start-page: 4566 volume-title: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics year: 2019 ident: 2021081914590098500_bib39 article-title: Human vs. muppet: A conservative estimate of human performance on the glue benchmark doi: 10.18653/v1/P19-1449 – start-page: 1112 volume-title: NAACL HLT 2018: 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies year: 2018 ident: 2021081914590098500_bib67 article-title: A broad-coverage challenge corpus for sentence understanding through inference doi: 10.18653/v1/N18-1101 – volume: 3 start-page: 382 issue: 1 year: 2020 ident: 2021081914590098500_bib24 article-title: A closer look at the performance of neural language models on reflexive anaphor licensing publication-title: Proceedings of the Society for Computation in Linguistics – start-page: 107 volume-title: NAACL HLT 2018: 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies year: 2018 ident: 2021081914590098500_bib17 article-title: Annotation artifacts in natural language inference data doi: 10.18653/v1/N18-2017 – start-page: 328 volume-title: ACL 2018: 56th Annual Meeting of the Association for Computational Linguistics year: 2018 ident: 2021081914590098500_bib23 article-title: Universal language model fine-tuning for text classification doi: 10.18653/v1/P18-1031 – start-page: 2089 volume-title: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012) year: 2012 ident: 2021081914590098500_bib50 article-title: A universal part-of-speech tagset – start-page: 1532 volume-title: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) year: 2014 ident: 2021081914590098500_bib48 article-title: GloVe: Global vectors for word representation doi: 10.3115/v1/D14-1162 – volume-title: An Introduction to Kolmogorov Complexity and its Applications year: 1993 ident: 2021081914590098500_bib33 doi: 10.1007/978-1-4757-3860-5 – volume: 39 start-page: 165 issue: 2 year: 2005 ident: 2021081914590098500_bib65 article-title: Annotating expressions of opinions and emotions in language publication-title: Language Resources and Evaluation doi: 10.1007/s10579-005-7880-9 – start-page: 1631 volume-title: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing year: 2013 ident: 2021081914590098500_bib58 article-title: Recursive deep models for semantic compositionality over a sentiment treebank – start-page: 264 volume-title: International Conference on Neural Information Processing year: 2019 ident: 2021081914590098500_bib69 article-title: Training behavior of deep neural network in frequency domain doi: 10.1007/978-3-030-36708-4_22 – year: 2019 ident: 2021081914590098500_bib68 article-title: Huggingface’s transformers: State-of-the-art natural language processing publication-title: arXiv preprint arXiv:1910.03771 doi: 10.18653/v1/2020.emnlp-demos.6 – start-page: 5301 volume-title: ICML 2019: Thirty-sixth International Conference on Machine Learning year: 2019 ident: 2021081914590098500_bib55 article-title: On the spectral bias of neural networks – start-page: 271 volume-title: Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume year: 2004 ident: 2021081914590098500_bib46 article-title: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts doi: 10.3115/1218955.1218990 – volume: 7 start-page: 625 year: 2019 ident: 2021081914590098500_bib64 article-title: Neural network acceptability judgments publication-title: Transactions of the Association for Computational Linguistics doi: 10.1162/tacl_a_00290 – year: 2020 ident: 2021081914590098500_bib8 article-title: Geometry matters: Exploring language examples at the decision boundary publication-title: CoRR – volume-title: ICLR 2019: 7th International Conference on Learning Representations year: 2019 ident: 2021081914590098500_bib60 article-title: Deep learning generalizes because the parameter-function map is biased towards simple functions – start-page: 2227 volume-title: NAACL HLT 2018: 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies year: 2018 ident: 2021081914590098500_bib49 article-title: Deep contextualized word representations doi: 10.18653/v1/N18-1202 – volume: 68 start-page: 1 issue: 1 year: 1998 ident: 2021081914590098500_bib16 article-title: Linguistic complexity: Locality of syntactic dependencies publication-title: Cognition doi: 10.1016/S0010-0277(98)00034-1 – start-page: 101 volume-title: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5–10, 2020 year: 2020 ident: 2021081914590098500_bib53 article-title: Stanza: A Python natural language processing toolkit for many human languages doi: 10.18653/v1/2020.acl-demos.14 – start-page: 1307 volume-title: Findings of the Association for Computational Linguistics: EMNLP 2020 year: 2020 ident: 2021081914590098500_bib14 article-title: Evaluating NLP models via contrast sets doi: 10.18653/v1/2020.findings-emnlp.117 – start-page: 3266 volume-title: Advances in Neural Information Processing Systems year: 2019 ident: 2021081914590098500_bib62 article-title: SuperGLUE: A stickier benchmark for general-purpose language understanding systems – start-page: 5753 volume-title: NeurIPS 2019: Thirty-third Conference on Neural Information Processing Systems year: 2019 ident: 2021081914590098500_bib70 article-title: XLNet: Generalized autoregressive pretraining for language understanding – start-page: 91 volume-title: Proceedings of The Third Workshop on Representation Learning for NLP year: 2018 ident: 2021081914590098500_bib11 article-title: Unsupervised random walk sentence embeddings: A strong but simple baseline doi: 10.18653/v1/W18-3012 – volume-title: Proceedings of the Association for Computational Linguistics: System Demonstrations (ACL 2020) year: 2020 ident: 2021081914590098500_bib15 article-title: SyntaxGym: An online platform for targeted evaluation of language models doi: 10.18653/v1/2020.acl-demos.10 – volume-title: ICLR 2019: 7th International Conference on Learning Representations year: 2019 ident: 2021081914590098500_bib63 article-title: GLUE: A multi-task benchmark and analysis platform for natural language understanding doi: 10.1162/tacl_a_00290 – start-page: 1 volume-title: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) year: 2017 ident: 2021081914590098500_bib4 article-title: Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation doi: 10.18653/v1/S17-2001 |
| SSID | ssj0001818062 |
| Score | 2.348562 |
| Snippet | We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of... AbstractWe introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the... |
| SourceID | doaj proquest crossref mit |
| SourceType | Open Website Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 891 |
| SubjectTerms | Boolean functions Classification Classifiers Decoders Natural language processing Representations Sequences Subsets Task complexity Task performance |
| SummonAdditionalLinks | – databaseName: DOAJ Directory of Open Access Journals dbid: DOA link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV3NS8MwFA8yPHgRRcXplAh6kmKapGlyVNnwoEPwA28hnzCcU9a6g3-9SdrpVMSLp9L2pWnfS_LeC6-_HwCHlntFw9qbae5wRp0WmdAmzxjyjGrDCEl0QPeX5XDIHx7E9QLVV6wJa-CBG8WdiII7TygtrLbUlFYrp7AtDeHI-hKn1ReVYiGZSrsr8RdmhueV7gyf1MqMpZJx0JIvPihB9QfP8jSqf6zHyckM1sBqGx3C0-at1sGSm2yA_k0sMm9YHqCqoIJxEkcgy3B-1ezxwRB7wpu2LBompstYA5TUDm9V9VhtgrtB__b8ImvZDzJDC1pnTLuSaY-5p9p5HfMAnWOk8hBCUcI0MiG4Y4XQIYhytkDGMW0E4YorxEyOyBboTJ4nbhtAZYPmsSMkV5iWNlc-xwozbsKTlLeoC47n-pCmhQaPDBVjmVIEhuWi9rrg6EP6pYHE-EXuLKr2QyYCWacLwbyyNa_8y7xdcBAMI9uJVf3SEfsiE-_NhCQ0xB5CRn54icJRyLfRy7eGvbm9P1uHDJRwTkMetvMfH7ALVnAsg4kourwHOvX01e2BZTOrR9V0P43ad9oW81s priority: 102 providerName: Directory of Open Access Journals |
| Title | Sensitivity as a Complexity Measure for Sequence Classification Tasks |
| URI | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00403 https://www.proquest.com/docview/2893884576 https://doaj.org/article/958ef3445dbd4c7dbaea2d7c380df722 |
| Volume | 9 |
| WOSCitedRecordID | wos000751952200053&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAON databaseName: DOAJ Directory of Open Access Journals customDbUrl: eissn: 2307-387X dateEnd: 20241231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: DOA dateStart: 20130101 isFulltext: true titleUrlDefault: https://www.doaj.org/ providerName: Directory of Open Access Journals – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2307-387X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: M~E dateStart: 20130101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre – providerCode: PRVPQU databaseName: Advanced Technologies & Aerospace Database customDbUrl: eissn: 2307-387X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: P5Z dateStart: 20130101 isFulltext: true titleUrlDefault: https://search.proquest.com/hightechjournals providerName: ProQuest – providerCode: PRVPQU databaseName: Computer Science Database customDbUrl: eissn: 2307-387X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: K7- dateStart: 20130101 isFulltext: true titleUrlDefault: http://search.proquest.com/compscijour providerName: ProQuest – providerCode: PRVPQU databaseName: Linguistics Database customDbUrl: eissn: 2307-387X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: CRLPW dateStart: 20130101 isFulltext: true titleUrlDefault: https://search.proquest.com/linguistics providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest Central customDbUrl: eissn: 2307-387X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: BENPR dateStart: 20130101 isFulltext: true titleUrlDefault: https://www.proquest.com/central providerName: ProQuest – providerCode: PRVPQU databaseName: Publicly Available Content Database customDbUrl: eissn: 2307-387X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001818062 issn: 2307-387X databaseCode: PIMPY dateStart: 20130101 isFulltext: true titleUrlDefault: http://search.proquest.com/publiccontent providerName: ProQuest |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1Lb9QwELag5cCFggCxUKogwQlFdWzHcU5Vi7YCQVcRLWjhYvmVakXZXTZpD_31nfF6uxRULlwSJR7nYY_H48nk-wh57VVrBNje3KrAchFsndfWFbmkrRTWSc4jHdDXT9VopMbjukkBty6lVa5sYjTUfuYwRr4LCwOulAD3eG_-K0fWKPy6mig07pJNRCoDPd88GI6az-soC_7KHFlFMeEZgWTHq-x3yXZ748600ajI_Ma8FOH7Ybb5Oen_stFx4jnc-t9HfkgeJJcz21_qyCNyJ0wfk-ExZq4vqSMy02UmQ8uA6JhwfLQMHGbg0GbHKdc6i_SZmFgU-zI7Md2P7gn5cjg8efc-T5QKuROl6HNpQyVty1QrbGgtLi5swagpwC8TXFrqwGOUZW3BMwu-pC5I6-AtjDJUuoLyp2RjOpuGZyQzHrqTBc4Lw0TlC9MWzDCpHFzJtJ4OyNtVg2qX8MaR9uJMx3WHZPr35h-QN9fS8yXOxi1yB9g31zKIjh1PzBanOg02XZcqtFyI0lsvXOWtCYb5ynFFfVsxNiCvoGd1Gq3dLTeSN2Sw7KLWXIBDU2skndcU9rW-nMz_qLi90oZ17bUqPP938Qtyn2HWDILuqm2y0S_Ow0tyz130k26xk9R7J0YOYPuxymHblN-hpPlw1Hy7AiJKCDg |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1Lb9QwELZKiwQXHgLEQgEjUS4oqmM7jnPgAKVVq25XFV1gb65fQSvK7rIJRfCj-I3MZJMuBZVbD5yixGNbicczn53xfIQ8C7q0Emxv4nTkiYyuSArn00SxUknnlRANHdD7fj4Y6NGoOFwhP7uzMBhW2dnExlCHqcc98k1YGAitJcDjNoJyP37_Buuz6uXeGxjMDc53todbu0lLIZB4mck6US7mypVcl9LF0iGYdilnNgUcIoVyzANCUlnhAInEkDEflfPQkdWWKZ8yAe0-n31JkKUK_-a2lB1XyJpWRQ6GYG3rbf_ww3JXB49ONyymGGCNiWtHXbS94pu19SfGGpw44pwfbOgCwLt9Htd_-YTG0e3c_N8-0S1yo4XU9NViDtwmK3Fyh2wfYWT-ghqD2opaipYPs3_C_cFiY5QCYKdHbSw5behBMXCq0VU6tNWn6i55dykvc4-sTqaTeJ9QG0BdeRQitVzmIbVlyi1X2kNLtgysR150A2h8m08daT1OTLOuUtz8Ptw9snEmPVvkEblA7jXqwpkMZv9uHkznH01rTEyR6VgKKbPggvR5cDZaHnIvNAtlznmPPAVNMq01qi7oSJ2TwbLTwggJgK0wHKCcYXAtzI_x7I-K6532LWsvVe_Bv4ufkGu7w4O-6e8N9h-S6xwjhDDBsF4nq_X8a3xErvrTelzNH7dTi5Ljy9beXwjDX9E |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sensitivity+as+a+Complexity+Measure+for+Sequence+Classification+Tasks&rft.jtitle=Transactions+of+the+Association+for+Computational+Linguistics&rft.au=Hahn%2C+Michael&rft.au=Jurafsky%2C+Dan&rft.au=Futrell%2C+Richard&rft.date=2021-08-18&rft.pub=MIT+Press+Journals%2C+The&rft.issn=2307-387X&rft.eissn=2307-387X&rft.volume=9&rft.spage=891&rft_id=info:doi/10.1162%2Ftacl_a_00403 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2307-387X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2307-387X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2307-387X&client=summon |