Predictability and Causality in Spanish and English Natural Language Generation
In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new data-intensive methods introduce language-dependent disparities in NLG as the main training data sets are in English. Also, most neural NLG sy...
Saved in:
| Published in: | arXiv.org |
|---|---|
| Main Authors: | , , , |
| Format: | Paper |
| Language: | English |
| Published: |
Ithaca
Cornell University Library, arXiv.org
26.08.2024
|
| Subjects: | |
| ISSN: | 2331-8422 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new data-intensive methods introduce language-dependent disparities in NLG as the main training data sets are in English. Also, most neural NLG systems use decoder-only (causal) transformer language models, which work well for English, but were not designed with other languages in mind. In this work we depart from the hypothesis that they may introduce generation bias in target languages with less rigid word ordering, subject omission, or different attachment preferences for relative clauses, so that for these target languages other language generation strategies may be more desirable. This paper first compares causal and non-causal language modeling for English and Spanish, two languages with different grammatical structures and over 1.5 billion and 0.5 billion speakers, respectively. For this purpose, we define a novel metric of average causal and non-causal context-conditioned entropy of the grammatical category distribution for both languages as an information-theoretic a priori approach. The evaluation of natural text sources (such as training data) in both languages reveals lower average non-causal conditional entropy in Spanish and lower causal conditional entropy in English. According to this experiment, Spanish is more predictable than English given a non-causal context. Then, by applying a conditional relative entropy metric to text generation experiments, we obtain as insights that the best performance is respectively achieved with causal NLG in English, and with non-causal NLG in Spanish. These insights support further research in NLG in Spanish using bidirectional transformer language models. |
|---|---|
| AbstractList | In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new data-intensive methods introduce language-dependent disparities in NLG as the main training data sets are in English. Also, most neural NLG systems use decoder-only (causal) transformer language models, which work well for English, but were not designed with other languages in mind. In this work we depart from the hypothesis that they may introduce generation bias in target languages with less rigid word ordering, subject omission, or different attachment preferences for relative clauses, so that for these target languages other language generation strategies may be more desirable. This paper first compares causal and non-causal language modeling for English and Spanish, two languages with different grammatical structures and over 1.5 billion and 0.5 billion speakers, respectively. For this purpose, we define a novel metric of average causal and non-causal context-conditioned entropy of the grammatical category distribution for both languages as an information-theoretic a priori approach. The evaluation of natural text sources (such as training data) in both languages reveals lower average non-causal conditional entropy in Spanish and lower causal conditional entropy in English. According to this experiment, Spanish is more predictable than English given a non-causal context. Then, by applying a conditional relative entropy metric to text generation experiments, we obtain as insights that the best performance is respectively achieved with causal NLG in English, and with non-causal NLG in Spanish. These insights support further research in NLG in Spanish using bidirectional transformer language models. |
| Author | García-Méndez, Silvia González-Castaño, Francisco J Francisco de Arriba-Pérez Busto-Castiñeira, Andrea |
| Author_xml | – sequence: 1 givenname: Andrea surname: Busto-Castiñeira fullname: Busto-Castiñeira, Andrea – sequence: 2 givenname: Francisco surname: González-Castaño middlename: J fullname: González-Castaño, Francisco J – sequence: 3 givenname: Silvia surname: García-Méndez fullname: García-Méndez, Silvia – sequence: 4 fullname: Francisco de Arriba-Pérez |
| BookMark | eNotjV1LwzAYhYMoOOd-gHcFr1vTN0mTXsqYm1Cc4O7Hm-RtzSjp7Ifov3dOr85zeOCcG3YZu0iM3eU8k0Yp_oD9V_jMQHKT5RKMuGAzECJPjQS4ZothOHDOodCglJix7WtPPrgRbWjD-J1g9MkSpwHPLcTk7YgxDO9nsYpN-8svOE49tkmFsZmwoWRNkXocQxdv2VWN7UCL_5yz3dNqt9yk1Xb9vHysUlQAqQaJBRVYWOedrk0OdW2d9pa0htyT4KZ00hPw_CRL0FR7bq2vwZCzSos5u_-bPfbdx0TDuD90Ux9Pj3vBS10AcAniB_GYUyE |
| ContentType | Paper |
| Copyright | 2024. This work is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: 2024. This work is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS |
| DOI | 10.48550/arxiv.2408.14283 |
| DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni Edition) ProQuest Central UK/Ireland ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College ProQuest Central Korea SciTech Premium Collection ProQuest Engineering Collection Engineering Database ProQuest Central Premium ProQuest One Academic (New) Publicly Available Content Database ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
| DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest One Academic Middle East (New) ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest Central (New) ProQuest One Academic ProQuest One Academic (New) Engineering Collection |
| DatabaseTitleList | Publicly Available Content Database |
| Database_xml | – sequence: 1 dbid: PIMPY name: Publicly Available Content Database url: http://search.proquest.com/publiccontent sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Physics |
| EISSN | 2331-8422 |
| Genre | Working Paper/Pre-Print |
| GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS |
| ID | FETCH-LOGICAL-a522-724a6e6a6bcdc7f812ffbc7dbe7721de3089c4de201f81927efd0bbdf28ecb573 |
| IEDL.DBID | BENPR |
| IngestDate | Mon Jun 30 09:17:46 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a522-724a6e6a6bcdc7f812ffbc7dbe7721de3089c4de201f81927efd0bbdf28ecb573 |
| Notes | SourceType-Working Papers-1 ObjectType-Working Paper/Pre-Print-1 content type line 50 |
| OpenAccessLink | https://www.proquest.com/docview/3097622042?pq-origsite=%requestingapplication% |
| PQID | 3097622042 |
| PQPubID | 2050157 |
| ParticipantIDs | proquest_journals_3097622042 |
| PublicationCentury | 2000 |
| PublicationDate | 20240826 |
| PublicationDateYYYYMMDD | 2024-08-26 |
| PublicationDate_xml | – month: 08 year: 2024 text: 20240826 day: 26 |
| PublicationDecade | 2020 |
| PublicationPlace | Ithaca |
| PublicationPlace_xml | – name: Ithaca |
| PublicationTitle | arXiv.org |
| PublicationYear | 2024 |
| Publisher | Cornell University Library, arXiv.org |
| Publisher_xml | – name: Cornell University Library, arXiv.org |
| SSID | ssj0002672553 |
| Score | 1.880542 |
| SecondaryResourceType | preprint |
| Snippet | In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new... |
| SourceID | proquest |
| SourceType | Aggregation Database |
| SubjectTerms | Context English language Entropy Information theory Language Languages Natural language Speech recognition Transformers |
| Title | Predictability and Causality in Spanish and English Natural Language Generation |
| URI | https://www.proquest.com/docview/3097622042 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV07T8MwELagBYmJt3iUKgOr29Rx7HhCoioCCUJEO5SpOj8iuqQlaSv499huCgMSC6Plxbqz7z59d_4OoWtKCRcgYqwEKEwhklgSFmEgSlASxiRn2g-b4GmajMciqwm3qm6r3MREH6j1TDmOvBuFNnESYu_Yzfwdu6lRrrpaj9DYRk2nVEYbqHk7SLOXb5aFMG4xc7QuZ3rxri6UH9NVxyl7dbza2K8g7DPL3f5_z3SAmhnMTXmItkxxhHZ9R6eqjtFzVroazGKtw_0ZQKGDPiwrj7uDaREMbRyYVm9-o_7MG6TgZTiCx5rFDNaq1M55J2h0Nxj173E9PQGDxVSYEwrMMGBSacVzm8fzXCqupbF4uqdNFCZCUW0sAMidKBo3uQ6l1DlJjJIxj05Ro5gV5gwFkCgKVCrGqKAAsYxFLJOol3MNQmp-jlob80zqF1BNfmxz8ff2JdojFig4npawFmosyqW5QjtqtZhWZbt2aNv1ZA7tKnt4yl6_AMz8r-E |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V07T8MwED5BAcHEW7zxAGOgdRw7HhADD1FRSiU6wFSdHxFdAiRtgR_Ff8R2UxiQ2BiYLVk63_ne9x3AAWNUSJRJpCXqiGGsIkV5HCHVktF6QjNuwrIJ0W6n9_eyMwUfk1kY31Y50YlBUZsn7XPkx3HdGU5KnYydPr9EfmuUr65OVmiMxeLavr-6kK08aZ47_h5SennRPbuKqq0CETpfIxKUIbccudJGi8zZtyxTWhhlnZ_ZMDaup1IzY51hzDxYmLCZqStlMpparRIRu2unYcbRmKQ1mOk0bzoPX0kdyoVz0eNx9TRghR1j8dYfHXkgsaMAbvZD5wdDdrn4z55gyZGOz7ZYhimbr8Bc6FfV5SrcdgpfYRqMUcbfCeaGnOGwDFEF6efkzmm5fvkYDqpRZdLGADJCWlWOlowxt71orkH3L0hYh1r-lNsNIJhqhkxpzplkiIlKZKLSuJEJg1IZsQk7E270qv9d9r5ZsfX78T7MX3VvWr1Ws329DQvUuUQ-I035DtQGxdDuwqweDfplsVfJEoHeH7PuE0MSDKw |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Predictability+and+Causality+in+Spanish+and+English+Natural+Language+Generation&rft.jtitle=arXiv.org&rft.au=Busto-Casti%C3%B1eira%2C+Andrea&rft.au=Gonz%C3%A1lez-Casta%C3%B1o%2C+Francisco+J&rft.au=Garc%C3%ADa-M%C3%A9ndez%2C+Silvia&rft.au=Francisco+de+Arriba-P%C3%A9rez&rft.date=2024-08-26&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.2408.14283 |