Estimating Renyi Entropy of Discrete Distributions

It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and H α (p). We determine...

Celý popis

Uložené v:

Podrobná bibliografia
Vydané v:	IEEE transactions on information theory Ročník 63; číslo 1; s. 38 - 56
Hlavní autori:	Acharya, Jayadev, Orlitsky, Alon, Suresh, Ananda Theertha, Tyagi, Himanshu
Médium:	Journal Article
Jazyk:	English
Vydavateľské údaje:	New York IEEE 01.01.2017 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:	Additives Approximation Complexity theory Electronic mail Entropy Entropy (Information theory) Entropy estimation Estimating techniques Estimation Functions (mathematics) Genetics Lower bounds Mathematical analysis minimax lower bounds Polynomials sample complexity sublinear algorithms Upper bound
ISSN:	0018-9448, 1557-9654
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!

Abstract	It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and H α (p). We determine the number of samples needed to estimate H α (p) for all α, showing that α <; 1 requires a super-linear, roughly k 1/α samples, noninteger α > 1 requires a near-linear k samples, but, perhaps surprisingly, integer α > 1 requires only Θ(k 1-1/α ) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form Σ x f (p x ), we reduce the sample complexity for noninteger values of α by a factor of log k compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Rényi entropies that are hard to distinguish.
AbstractList	It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and H α (p). We determine the number of samples needed to estimate H α (p) for all α, showing that α <; 1 requires a super-linear, roughly k 1/α samples, noninteger α > 1 requires a near-linear k samples, but, perhaps surprisingly, integer α > 1 requires only Θ(k 1-1/α ) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form Σ x f (p x ), we reduce the sample complexity for noninteger values of α by a factor of log k compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Rényi entropies that are hard to distinguish. It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and Hα(p). We determine the number of samples needed to estimate Hα(p) for all α, showing that α <; 1 requires a super-linear, roughly k1/α samples, noninteger α > 1 requires a near-linear k samples, but, perhaps surprisingly, integer α > 1 requires only Θ(k1-1/α) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form Σx f (px), we reduce the sample complexity for noninteger values of α by a factor of log k compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Rényi entropies that are hard to distinguish. It was shown recently that estimating the Shannon entropy H(p) of a discrete k -symbol distribution p requires ...(k/logk) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general R...nyi entropy of order a and Ha(p) . We determine the number of samples needed to estimate Ha(p) for all a , showing that a<1 requires a super-linear, roughly k... samples, noninteger a>1 requires a near-linear k samples, but, perhaps surprisingly, integer a>1 requires only ...(...) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form ...f(px) , we reduce the sample complexity for noninteger values of a by a factor of logk compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Renyi entropies that are hard to distinguish. (ProQuest: ... denotes formulae/symbols omitted.)
Author	Suresh, Ananda Theertha Tyagi, Himanshu Acharya, Jayadev Orlitsky, Alon
Author_xml	– sequence: 1 givenname: Jayadev surname: Acharya fullname: Acharya, Jayadev email: acharya@cornell.edu organization: Sch. of Electr. & Comput. Eng., Cornell Univ., Ithaca, NY, USA – sequence: 2 givenname: Alon surname: Orlitsky fullname: Orlitsky, Alon organization: Dept. of Electr. & Comput. Eng., Univ. of California, San Diego, La Jolla, CA, USA – sequence: 3 givenname: Ananda Theertha surname: Suresh fullname: Suresh, Ananda Theertha email: theertha@google.com organization: Google Res., New York, NY, USA – sequence: 4 givenname: Himanshu surname: Tyagi fullname: Tyagi, Himanshu email: htyagi@ece.iisc.ernet.in organization: Dept. of Electr. Commun. Eng., Indian Inst. of Sci., Bangalore, India
BookMark	eNp9kEtLAzEUhYNUsK3uBTcDrqfm_VhKbbVQEKSuwzRNJKUmNUkX_fdmaHHhoqt7D5xzH98IDEIMFoB7BCcIQfW0WqwmGCI-wRxDStgVGCLGRKs4owMwhBDJVlEqb8Ao522VlCE8BHiWi__uig9fzYcNR9_MQklxf2yia158NskW2zcl-fWh-BjyLbh23S7bu3Mdg8_5bDV9a5fvr4vp87I1BKnSdpIwYq1khAvlZFUCbYiU9VruoDFqbSjFlhIoN4RT5gQzzigGITOcc0vG4PE0d5_iz8HmorfxkEJdqTESlGAuqLrkQnW3wFhW7xjwk8ukmHOyThtfuv6dkjq_0wjqnqKuFHVPUZ8p1iD8F9ynyisdL0UeThFvrf2zC064qhR-AZULfAI
CODEN	IETTAW
CitedBy_id	crossref_primary_10_1109_TIT_2024_3440661 crossref_primary_10_1109_TIT_2024_3382037 crossref_primary_10_1016_j_eswa_2021_114748 crossref_primary_10_1016_j_peva_2023_102379 crossref_primary_10_1016_j_peva_2021_102247 crossref_primary_10_1007_s11042_018_6823_6 crossref_primary_10_1103_PhysRevA_108_012409 crossref_primary_10_3390_e25020185 crossref_primary_10_1007_s00034_020_01349_2 crossref_primary_10_1007_s40687_024_00483_6 crossref_primary_10_1080_00949655_2023_2298862 crossref_primary_10_1109_TIT_2019_2900048 crossref_primary_10_1109_TIT_2018_2883306 crossref_primary_10_4018_IJMCMC_2017070107 crossref_primary_10_3390_math11102298 crossref_primary_10_1109_TIT_2017_2733537 crossref_primary_10_1109_TIT_2018_2846245 crossref_primary_10_1007_s11135_022_01451_4
Cites_doi	10.1109/TIT.2004.833360 10.1002/ima.20079 10.1109/SFCS.1989.63486 10.1103/PhysRevE.69.056111 10.1109/FOCS.2008.76 10.1145/2432622.2432626 10.1145/1993636.1993727 10.1109/ISIT.1994.394764 10.1109/TIT.2008.928987 10.1109/TIT.2004.836665 10.1109/TIT.2013.2270273 10.1145/1374376.1374432 10.1109/TIT.2015.2412945 10.1145/1140103.1140295 10.1109/18.370121 10.1145/380752.380810 10.1162/089976603321780272 10.1109/TIT.2010.2090221 10.1214/aop/1176995801 10.1109/18.481781 10.1109/18.476316 10.1016/0375-9601(88)90193-4 10.1137/1.9781611973730.124 10.1002/rsa.10019 10.1007/978-1-4419-1570-2_2 10.1109/ICIP.2000.901000 10.1007/PL00003816 10.1109/18.42194 10.1007/978-1-4612-1880-7_29 10.1002/prot.340110408
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Jan 2017 Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2017
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Jan 2017 – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2017
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TIT.2016.2620435
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1557-9654
EndPage	56
ExternalDocumentID	4288742571 10_1109_TIT_2016_2620435 7636988
Genre	orig-research Feature
GrantInformation_xml	– fundername: MITEI-Shell Grant
GroupedDBID	-~X .DC 0R~ 29I 3EH 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABFSI ABQJQ ABVLG ACGFO ACGFS ACGOD ACIWK AENEX AETEA AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 E.L EBS EJD F5P HZ~ H~9 IAAWW IBMZZ ICLAB IDIHD IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 VH1 VJK AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c319t-a8353ee853679f883571d3881106f0cc9bc442e4308d3645f75cfc95005c666e3
IEDL.DBID	RIE
ISICitedReferencesCount	49
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000391740000003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	0018-9448
IngestDate	Sun Nov 09 07:47:30 EST 2025 Sun Nov 09 07:58:33 EST 2025 Sat Nov 29 03:31:37 EST 2025 Tue Nov 18 21:18:39 EST 2025 Tue Aug 26 16:43:24 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	1
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c319t-a8353ee853679f883571d3881106f0cc9bc442e4308d3645f75cfc95005c666e3
Notes	SourceType-Scholarly Journals-1 ObjectType-Feature-1 content type line 14 ObjectType-Article-1 ObjectType-Feature-2
ORCID	0000-0003-2950-706X
PQID	1853722874
PQPubID	36024
PageCount	19
ParticipantIDs	crossref_citationtrail_10_1109_TIT_2016_2620435 proquest_journals_2174326749 crossref_primary_10_1109_TIT_2016_2620435 proquest_journals_1853722874 ieee_primary_7636988
PublicationCentury	2000
PublicationDate	2017-Jan. 2017-1-00 20170101
PublicationDateYYYYMMDD	2017-01-01
PublicationDate_xml	– month: 01 year: 2017 text: 2017-Jan.
PublicationDecade	2010
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on information theory
PublicationTitleAbbrev	TIT
PublicationYear	2017
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref35 knuth (ref19) 1973; 3 ref34 ref12 jenssen (ref15) 2003 markov (ref23) 1892 ref37 ref14 rényi (ref36) 1961; 1 ref30 ref33 ref32 ref10 hardy (ref11) 1952 ref2 xu (ref44) 1998 ref38 ref16 orlitsky (ref31) 2004 acharya (ref1) 2015 tossavainen (ref40) 2006; 33 miller (ref25) 1955; 2 wu (ref43) 2014 ref46 ref24 ref45 ref26 ref20 ref42 ref41 ref21 ref28 ref27 ref29 timan (ref39) 1963 ref7 ili? (ref13) 0 källberg (ref18) 0 ref9 jiao (ref17) 0 ref4 ref3 mansour (ref22) 2012 ref6 ref5 goldreich (ref8) 2000; 7
References_xml	– ident: ref33 doi: 10.1109/TIT.2004.833360 – volume: 1 start-page: 547 year: 1961 ident: ref36 article-title: On measures of entropy and information publication-title: Proc 4th Berkeley Symp Math Statist Probab – ident: ref28 doi: 10.1002/ima.20079 – ident: ref14 doi: 10.1109/SFCS.1989.63486 – ident: ref29 doi: 10.1103/PhysRevE.69.056111 – ident: ref12 doi: 10.1109/FOCS.2008.76 – year: 0 ident: ref13 article-title: A unified characterization of generalized information and certainty measures – start-page: 426 year: 2004 ident: ref31 article-title: On modeling profiles instead of values publication-title: Proc 20th Conf Uncertainty Artif Intell – start-page: 523 year: 2003 ident: ref15 article-title: Clustering using Renyi's entropy publication-title: Proc Int Joint Conf Neural Netw – ident: ref5 doi: 10.1145/2432622.2432626 – ident: ref41 doi: 10.1145/1993636.1993727 – volume: 33 start-page: 47 year: 2006 ident: ref40 article-title: On the zeros of finite sums of exponential functions publication-title: Australian Mathematical Society Gazette – ident: ref24 doi: 10.1109/ISIT.1994.394764 – year: 2014 ident: ref43 article-title: Minimax rates of entropy estimation on large alphabets via best polynomial approximation publication-title: CoRR – year: 0 ident: ref18 article-title: Statistical inference for Rényi entropy functionals – ident: ref34 doi: 10.1109/TIT.2008.928987 – volume: 2 start-page: 95 year: 1955 ident: ref25 article-title: Note on the bias of information estimates publication-title: Proc Inf Theory Psychol Problems Methods – ident: ref35 doi: 10.1109/TIT.2004.836665 – ident: ref27 doi: 10.1109/TIT.2013.2270273 – volume: 7 start-page: 20 year: 2000 ident: ref8 article-title: On testing expansion in bounded-degree graphs publication-title: Proc Electron Colloq Comput Complex (ECCC) – ident: ref42 doi: 10.1145/1374376.1374432 – year: 1963 ident: ref39 publication-title: Theory of Approximation of Functions of a Real Variable – year: 1952 ident: ref11 publication-title: Inequalities – ident: ref16 doi: 10.1109/TIT.2015.2412945 – start-page: 218 year: 1892 ident: ref23 article-title: On functions deviating least from zero in a given interval publication-title: Proc Izdat Imp Akad Nauk – ident: ref20 doi: 10.1145/1140103.1140295 – ident: ref7 doi: 10.1109/18.370121 – ident: ref4 doi: 10.1145/380752.380810 – year: 1998 ident: ref44 article-title: Energy, entropy and information potential for neural computation – ident: ref32 doi: 10.1162/089976603321780272 – ident: ref10 doi: 10.1109/TIT.2010.2090221 – year: 0 ident: ref17 article-title: Maximum likelihood estimation of functionals of discrete distributions – ident: ref38 doi: 10.1214/aop/1176995801 – ident: ref3 doi: 10.1109/18.481781 – ident: ref6 doi: 10.1109/18.476316 – volume: 3 year: 1973 ident: ref19 publication-title: The Art of Computer Programming Sorting and Searching – ident: ref9 doi: 10.1016/0375-9601(88)90193-4 – start-page: 1855 year: 2015 ident: ref1 article-title: The complexity of estimating Rényi entropy publication-title: Proc Annu ACM-SIAM Symp Discrete Algorithms (SODA) doi: 10.1137/1.9781611973730.124 – ident: ref2 doi: 10.1002/rsa.10019 – ident: ref45 doi: 10.1007/978-1-4419-1570-2_2 – ident: ref21 doi: 10.1109/ICIP.2000.901000 – ident: ref30 doi: 10.1007/PL00003816 – ident: ref26 doi: 10.1109/18.42194 – ident: ref46 doi: 10.1007/978-1-4612-1880-7_29 – ident: ref37 doi: 10.1002/prot.340110408 – year: 2012 ident: ref22 article-title: Multiple source adaptation and the Rényi divergence publication-title: CoRR
SSID	ssj0014512
Score	2.5034225
Snippet	It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows... It was shown recently that estimating the Shannon entropy H(p) of a discrete k -symbol distribution p requires ...(k/logk) samples, a number that grows...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	38
SubjectTerms	Additives Approximation Complexity theory Electronic mail Entropy Entropy (Information theory) Entropy estimation Estimating techniques Estimation Functions (mathematics) Genetics Lower bounds Mathematical analysis minimax lower bounds Polynomials sample complexity sublinear algorithms Upper bound
Title	Estimating Renyi Entropy of Discrete Distributions
URI	https://ieeexplore.ieee.org/document/7636988 https://www.proquest.com/docview/1853722874 https://www.proquest.com/docview/2174326749
Volume	63
WOSCitedRecordID	wos000391740000003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Xplore customDbUrl: eissn: 1557-9654 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014512 issn: 0018-9448 databaseCode: RIE dateStart: 19630101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB3a4kEPVlvFapUcvAimTTZpdvco2qKXIlKht5BOJlKQtPRD6L93dptGQRG8JWTzwWzezpud3XkA14RBhknIQU6SCJf5f-Jq4ihFEDIdSQNBKrNiE3I4VOOxfq7AbbkXhojs4jPqmEOby09nuDZTZV3GQqSVqkJVSrndq1VmDMKev60M7jOA-Z27lKSnu6OnkVnDFXVs8XUr7Pblgqymyo-B2HqXQf1_33UEhwWLdO623X4MFcobUN8pNDgFYBtw8K3cYBNEn_FsGGr-5rxQvpk6fbNOfb5xZpnzMOUBhBm0OShVsJYn8Droj-4f3UIzwUUG08pNmFEFROyEI6kzxWfSTwOl2BBR5iHqCYahoDDwVGoykJnsYYa6x2BEjmQoOIVaPsvpDBxfYc_s4vEFcswhSJPyEpykofImmmlMC7o7M8ZYFBQ3uhbvsQ0sPB2z4WNj-LgwfAtuyjvm22Iaf7RtGkOX7Qobt6C966m4QNsyNpxDClO5_9fLNuoSkQz1-e8PvYB9Yby1nVlpQ221WNMl7OHHarpcXNn_7BP5xc1a
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB60CurBaqtYrZqDF8G0yea1exRtabEWkQq9hXQzkYKkpQ-h_97ZTRoFi-AtIZsH3-bb-WZndwbgBqWTyMglJyeKmEn6PzIFkpfCUJIciR2GPNHFJoJ-nw-H4mUL7oq9MIioF59hQx3qWH48kUs1VdYkLviC823Y8VyX2dlurSJm4Hp2lhvcJgrTW9dBSUs0B92BWsXlN3T6dV3a7dsI6aoqv4ZibV_a5f992REc5jrSuM86_hi2MK1AeV2jwcgpW4GDHwkHq8BaxGilUdN34xXT1dhoqZXq05UxSYzHMQ0hpKHVQVEHa34Cb-3W4KFj5lUTTEl0WpgRaSoHkcywH4iE01lgxw7nBISfWFKKkSTw0HUsHqsYZBJ4MpHCIzpK8mXQOYVSOknxDAybS0_t47GZJK-DoUBuRXIUu9waCRIyNWiuYQxlnlJcVbb4CLVrYYmQgA8V8GEOfA1uizumWTqNP9pWFdBFuxzjGtTXPRXmfJuHSnUETOXu33hZ-13MD1xxvvmh17DXGTz3wl63_3QB-0zZbj3PUofSYrbES9iVn4vxfHal_7kv1Z7QoQ
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Estimating+Renyi+Entropy+of+Discrete+Distributions&rft.jtitle=IEEE+transactions+on+information+theory&rft.au=Acharya%2C+Jayadev&rft.au=Orlitsky%2C+Alon&rft.au=Suresh%2C+Ananda+Theertha&rft.au=Tyagi%2C+Himanshu&rft.date=2017-01-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=0018-9448&rft.eissn=1557-9654&rft.volume=63&rft.issue=1&rft.spage=38&rft_id=info:doi/10.1109%2FTIT.2016.2620435&rft.externalDBID=NO_FULL_TEXT&rft.externalDocID=4288742571
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9448&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9448&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9448&client=summon