Posted on Sat 07 March 2015 in Statistics
In the last two parts (part one, part two) we looked at the problem of entropy estimation and several popular estimators.
In this final article we will take a look at two Bayesian approaches to the problem.
The first Bayesian approach to entropy …
Posted on Sat 21 February 2015 in Statistics
In the last part we have looked at the basic problem of discrete entropy estimation. In this article we will see a number of proposals of improved estimators.
In 1955 George Miller proposed a simple correction to the naive plugin estimator \(\hat{H}_N\) by adding the constant …
Posted on Sat 07 February 2015 in Statistics
Estimation of the entropy of a random variable is an important problem that has many applications. If you can estimate entropy accurately, you can also estimate mutual information, which allows you to find dependent random variables in large data sets. There are numerous applications.
The setting of discrete entropy estimation …