A Characterization of All Single-Integral, Non-Kernel Divergence Estimators
Divergence measures have been used for a long time for different purposes in information theory and statistics. In particular, density-based minimum divergence estimation is a popular tool in the statistical literature. Given the sampled data and a parametric model, we estimate the model parameter b...
Saved in:
| Published in: | IEEE transactions on information theory Vol. 65; no. 12; pp. 7976 - 7984 |
|---|---|
| Main Authors: | , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
IEEE
01.12.2019
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 0018-9448, 1557-9654 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Divergence measures have been used for a long time for different purposes in information theory and statistics. In particular, density-based minimum divergence estimation is a popular tool in the statistical literature. Given the sampled data and a parametric model, we estimate the model parameter by choosing the member of the model family that is closest to the data distribution in terms of the given divergence. In the absolutely continuous set up, when the distributions from the model family and the unknown data generating distribution are assumed to have densities, the application of kernel based non-parametric smoothing is sometimes unavoidable to get an estimate of the true data density. The use of kernels (or other non-parametric smoothing techniques) makes the estimation process considerably more complex, as now one has to impose necessary conditions not just on the model but also on the kernel and its bandwidth. In higher dimensions the efficiency of the kernel density estimator (KDE) often becomes too low for the minimum divergence procedure to be practically useful. It can, therefore, lead to a significant advantage to have a divergence which allows minimum divergence estimation bypassing the use of non-parametric smoothing. For the same reason, characterizing the class of such divergences would be a notable achievement. In this work, we provide a characterization of the class of divergences that bypasses the use of non-parametric smoothing in the construction of divergences, providing a solution to this very important problem. |
|---|---|
| AbstractList | Divergence measures have been used for a long time for different purposes in information theory and statistics. In particular, density-based minimum divergence estimation is a popular tool in the statistical literature. Given the sampled data and a parametric model, we estimate the model parameter by choosing the member of the model family that is closest to the data distribution in terms of the given divergence. In the absolutely continuous set up, when the distributions from the model family and the unknown data generating distribution are assumed to have densities, the application of kernel based non-parametric smoothing is sometimes unavoidable to get an estimate of the true data density. The use of kernels (or other non-parametric smoothing techniques) makes the estimation process considerably more complex, as now one has to impose necessary conditions not just on the model but also on the kernel and its bandwidth. In higher dimensions the efficiency of the kernel density estimator (KDE) often becomes too low for the minimum divergence procedure to be practically useful. It can, therefore, lead to a significant advantage to have a divergence which allows minimum divergence estimation bypassing the use of non-parametric smoothing. For the same reason, characterizing the class of such divergences would be a notable achievement. In this work, we provide a characterization of the class of divergences that bypasses the use of non-parametric smoothing in the construction of divergences, providing a solution to this very important problem. |
| Author | Jana, Soham Basu, Ayanendranath |
| Author_xml | – sequence: 1 givenname: Soham orcidid: 0000-0001-7547-9333 surname: Jana fullname: Jana, Soham email: soham.jana@yale.edu organization: Department of Statistics and Data Science, Yale University, New Haven, CT, USA – sequence: 2 givenname: Ayanendranath orcidid: 0000-0003-1416-9109 surname: Basu fullname: Basu, Ayanendranath email: ayanbasu@isical.ac.in organization: Interdisciplinary Statistical Research Unit, Indian Statistical Institute, Kolkata, India |
| BookMark | eNp9kEFLAzEQRoNUsK3eBS8LXt2abHY22WOpVYtFD9ZzSLNJTVmTmqSC_nq3tnjw4Olj4HszzBugnvNOI3RO8IgQXF8vZotRgUk9KmrKoGBHqE8AWF5XUPZQH2PC87os-QkaxLjuxhJI0UcP42zyKoNUSQf7JZP1LvMmG7dt9mzdqtX5zCW9CrK9yh69yx90cLrNbuyHDivtlM6mMdk3mXyIp-jYyDbqs0MO0cvtdDG5z-dPd7PJeJ4rCizllalKvmwaLjEoQ7mRrDQECF02tGYlLCVZMtMAx6bB0GVDjAGASlVgqGJ0iC73ezfBv291TGLtt8F1J0VBSQUM04J3rWrfUsHHGLQRyqafB1OQthUEi5040YkTO3HiIK4D8R9wE7oPw-d_yMUesVrr3zrnhOKa0W_I8Xo- |
| CODEN | IETTAW |
| CitedBy_id | crossref_primary_10_1080_02331888_2022_2070622 crossref_primary_10_1007_s10463_022_00862_2 crossref_primary_10_1109_TIT_2024_3366538 crossref_primary_10_1007_s42519_020_00162_z crossref_primary_10_1109_TIT_2022_3210436 |
| Cites_doi | 10.1007/s00184-015-0556-6 10.1080/10485252.2011.599385 10.1109/TIT.2008.929943 10.1016/j.stamet.2016.01.003 10.1162/neco.2008.04-08-771 10.1093/biomet/88.3.865 10.1201/b10956 10.1111/j.2517-6161.1984.tb01318.x 10.1111/j.1467-842X.2010.00574.x 10.1017/CBO9780511804441 10.1214/aos/1176350820 10.1080/01621459.1986.10478264 10.1214/aos/1176325512 10.1080/14786440009463897 10.1111/j.2517-6161.1966.tb00626.x 10.1214/aoms/1177707038 10.1093/biomet/85.3.549 10.1109/TIT.2003.813509 10.1214/aoms/1177703732 10.1080/01621459.1987.10478501 10.1002/9780470316849 10.1016/j.jspi.2012.03.019 10.1214/aos/1176343842 10.1109/TIT.2011.2178139 10.1007/978-1-4899-3324-9 10.1201/b14876 10.1006/jmva.1997.1731 10.1016/j.jspi.2008.04.022 10.1214/11-AIHP425 10.1080/01621459.1989.10478744 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TIT.2019.2937527 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science |
| EISSN | 1557-9654 |
| EndPage | 7984 |
| ExternalDocumentID | 10_1109_TIT_2019_2937527 8813097 |
| Genre | orig-research |
| GroupedDBID | -~X .DC 0R~ 29I 3EH 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABFSI ABQJQ ABVLG ACGFO ACGFS ACGOD ACIWK AENEX AETEA AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 E.L EBS EJD F5P HZ~ H~9 IAAWW IBMZZ ICLAB IDIHD IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 VH1 VJK AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c357t-6f648bdd8a05cf38fa74f1513bd39745ba1b7fd580fd05d58d1ff5556c65f3c73 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 6 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000512370800020&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0018-9448 |
| IngestDate | Sun Sep 07 03:32:55 EDT 2025 Tue Nov 18 20:59:37 EST 2025 Sat Nov 29 03:31:42 EST 2025 Wed Aug 27 02:40:17 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 12 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c357t-6f648bdd8a05cf38fa74f1513bd39745ba1b7fd580fd05d58d1ff5556c65f3c73 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0003-1416-9109 0000-0001-7547-9333 |
| PQID | 2316570328 |
| PQPubID | 36024 |
| PageCount | 9 |
| ParticipantIDs | crossref_citationtrail_10_1109_TIT_2019_2937527 proquest_journals_2316570328 ieee_primary_8813097 crossref_primary_10_1109_TIT_2019_2937527 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-12-01 |
| PublicationDateYYYYMMDD | 2019-12-01 |
| PublicationDate_xml | – month: 12 year: 2019 text: 2019-12-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on information theory |
| PublicationTitleAbbrev | TIT |
| PublicationYear | 2019 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref35 ref13 banerjee (ref34) 2005; 6 ref12 ref37 ref15 ref36 ref14 csiszár (ref4) 1967; 2 ref30 ref11 ref32 ref10 ref2 ref1 ref39 vajda (ref17) 1989 ref38 ref19 devroye (ref20) 1985 liese (ref16) 1987; 95 ref24 itakura (ref31) 1968 ref23 ref26 ref25 pardo (ref18) 2006 ref21 ref28 mukherjee (ref33) 2018 ref27 ref29 ref8 ref7 ref9 ref6 ref5 csiszár (ref3) 1963; 8 scott (ref22) 1992 |
| References_xml | – ident: ref14 doi: 10.1007/s00184-015-0556-6 – ident: ref38 doi: 10.1080/10485252.2011.599385 – ident: ref36 doi: 10.1109/TIT.2008.929943 – ident: ref15 doi: 10.1016/j.stamet.2016.01.003 – ident: ref32 doi: 10.1162/neco.2008.04-08-771 – volume: 8 start-page: 85 year: 1963 ident: ref3 article-title: Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten publication-title: Publ Math Inst Hungar Acad Sci – ident: ref29 doi: 10.1093/biomet/88.3.865 – ident: ref19 doi: 10.1201/b10956 – start-page: 280 year: 1968 ident: ref31 article-title: Analysis synthesis telephony based on the maximum likelihood method publication-title: Proc 6th Int Congr Acoust – ident: ref12 doi: 10.1111/j.2517-6161.1984.tb01318.x – ident: ref27 doi: 10.1111/j.1467-842X.2010.00574.x – ident: ref37 doi: 10.1017/CBO9780511804441 – volume: 95 year: 1987 ident: ref16 publication-title: Convex Statistical Distances – ident: ref9 doi: 10.1214/aos/1176350820 – ident: ref8 doi: 10.1080/01621459.1986.10478264 – ident: ref13 doi: 10.1214/aos/1176325512 – year: 2006 ident: ref18 publication-title: Statistical Inference Based on Divergence Measures – ident: ref1 doi: 10.1080/14786440009463897 – year: 1989 ident: ref17 publication-title: Theory of Statistical Inference and Information – volume: 2 start-page: 229 year: 1967 ident: ref4 article-title: Information-type measures of difference of probability distributions and indirect observation publication-title: Stud Sci Math Hungarica – ident: ref5 doi: 10.1111/j.2517-6161.1966.tb00626.x – ident: ref2 doi: 10.1214/aoms/1177707038 – ident: ref25 doi: 10.1093/biomet/85.3.549 – ident: ref35 doi: 10.1109/TIT.2003.813509 – year: 2018 ident: ref33 article-title: The B-exponential divergence and its generalizations with applications to parametric estimation – ident: ref6 doi: 10.1214/aoms/1177703732 – ident: ref10 doi: 10.1080/01621459.1987.10478501 – year: 1992 ident: ref22 publication-title: Multivariate Density Estimation Theory Practice and Visualization doi: 10.1002/9780470316849 – ident: ref28 doi: 10.1016/j.jspi.2012.03.019 – ident: ref7 doi: 10.1214/aos/1176343842 – ident: ref30 doi: 10.1109/TIT.2011.2178139 – volume: 6 start-page: 1705 year: 2005 ident: ref34 article-title: Clustering with bregman divergences publication-title: J Mach Learn Res – year: 1985 ident: ref20 publication-title: Nonparametric Density Estimation The $L_ 1 $ View – ident: ref21 doi: 10.1007/978-1-4899-3324-9 – ident: ref23 doi: 10.1201/b14876 – ident: ref39 doi: 10.1006/jmva.1997.1731 – ident: ref26 doi: 10.1016/j.jspi.2008.04.022 – ident: ref24 doi: 10.1214/11-AIHP425 – ident: ref11 doi: 10.1080/01621459.1989.10478744 |
| SSID | ssj0014512 |
| Score | 2.3465714 |
| Snippet | Divergence measures have been used for a long time for different purposes in information theory and statistics. In particular, density-based minimum divergence... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 7976 |
| SubjectTerms | Bandwidth Bregman divergence characterization Data models Density Divergence Estimation Information theory Kernel Kernels Mathematical model non-kernel divergence Parameter estimation Robustness single-integral Smoothing Smoothing methods |
| Title | A Characterization of All Single-Integral, Non-Kernel Divergence Estimators |
| URI | https://ieeexplore.ieee.org/document/8813097 https://www.proquest.com/docview/2316570328 |
| Volume | 65 |
| WOSCitedRecordID | wos000512370800020&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1557-9654 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014512 issn: 0018-9448 databaseCode: RIE dateStart: 19630101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH5sw4MenG6K0yk5eBHWrW2avvQ45oZDGIITditNmoBQN9kP_36TtisDRfDUFpJS8jXvR_LyfQD3XLMQUaAjGSonQEwcoRQ6SRiip0zI4CU8F5vA2YwvFtFLDXrVWRilVF58pvr2Nt_LT1dyZ5fKBpwbixthHeqIYXFWq9oxCJhXMIN7ZgKbnGO_JelGg_l0bmu4or5xbcisfsyBC8o1VX4Y4ty7TJr_-64zOC2jSDIsYD-Hmlq2oLlXaCDlhG3ByQHdYBueh2RU8TMXxy_JSpNhlpFX0yBTzrQgj8h6ZGaVhdV6qTLyaEs3cs5OMjYG4cOm6ZsLeJuM56MnpxRTcCRluHVCHQZcpClPXCY15TrBQBt3T0VqQpKAicQTqFPGXZ26zFxTT2vGWChDpqlEegmN5WqproAkyvdE6vuSChqYByFopFiASSQ8ir7owGA_vrEsmcat4EUW5xmHG8UGkdgiEpeIdOCh6vFZsGz80bZtEajalYPfge4ewrichpvYBK-2tIf6_Pr3XjdwbN9d1Kd0obFd79QtHMmv7ftmfZf_Yd9-_M2V |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD7oFNQH7-J0ah58EezWNk1P-jim4tgcghN8K02agFA32cXfb9IbA0XwqS0ktORrziU5-T6Aa65ZiCjQkQyVEyAmjlAKnSQM0VMmZPASnotN4GjE396i5zW4rc_CKKXy4jPVtrf5Xn46lUu7VNbh3FjcCNdhwypnlae16j2DgHkFN7hnprDJOqpNSTfqjPtjW8UVtY1zQ2YVZFacUK6q8sMU5_7lYe9_X7YPu2UcSboF8AewpiaHsFdpNJByyh7Czgrh4BEMuqRXMzQXBzDJVJNulpEX0yBTTr-gj8huychqC6vZRGXkzhZv5Kyd5N6YhA-bqM-P4fXhftx7dEo5BUdShgsn1GHARZryxGVSU64TDLRx-FSkJigJmEg8gTpl3NWpy8w19bRmjIUyZJpKpCfQmEwn6hRIonxPpL4vqaCBeRCCRooFmETCo-iLJnSq8Y1lyTVuJS-yOM853Cg2iMQWkbhEpAk3dY_Pgmfjj7ZHFoG6XTn4TWhVEMblRJzHJny1xT3U52e_97qCrcfx0zAe9keDc9i27ymqVVrQWMyW6gI25dfifT67zP-2b18c0N4 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Characterization+of+All+Single-Integral%2C+Non-Kernel+Divergence+Estimators&rft.jtitle=IEEE+transactions+on+information+theory&rft.au=Jana%2C+Soham&rft.au=Basu%2C+Ayanendranath&rft.date=2019-12-01&rft.issn=0018-9448&rft.eissn=1557-9654&rft.volume=65&rft.issue=12&rft.spage=7976&rft.epage=7984&rft_id=info:doi/10.1109%2FTIT.2019.2937527&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TIT_2019_2937527 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9448&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9448&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9448&client=summon |