Estimating Renyi Entropy of Discrete Distributions

It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and H α (p). We determine...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on information theory Ročník 63; číslo 1; s. 38 - 56
Hlavní autori: Acharya, Jayadev, Orlitsky, Alon, Suresh, Ananda Theertha, Tyagi, Himanshu
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York IEEE 01.01.2017
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Additives
Approximation
Complexity theory
Electronic mail
Entropy
Entropy (Information theory)
Entropy estimation
Estimating techniques
Estimation
Functions (mathematics)
Genetics
Lower bounds
Mathematical analysis
minimax lower bounds
Polynomials
sample complexity
sublinear algorithms
Upper bound
ISSN:0018-9448, 1557-9654
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and H α (p). We determine the number of samples needed to estimate H α (p) for all α, showing that α <; 1 requires a super-linear, roughly k 1/α samples, noninteger α > 1 requires a near-linear k samples, but, perhaps surprisingly, integer α > 1 requires only Θ(k 1-1/α ) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form Σ x f (p x ), we reduce the sample complexity for noninteger values of α by a factor of log k compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Rényi entropies that are hard to distinguish.
Bibliografia:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
ISSN:0018-9448
1557-9654
DOI:10.1109/TIT.2016.2620435