4
$\begingroup$

Let $X$ be a random variable with geometric distribution, ie $P(X=k)=p(1-p)^k$. If I calculated it correctly, $X$ has mean $E(X)=\frac{1-p}p$ and entropy $H(X)=-\log p - \frac{1-p}p\log{(1-p)}$ (edited because I want to start at $0$). Now I'd like to show there is no non-negative integer valued random variable Y with $E(Y)=E(X)$ and $H(Y)>H(X)$.

There is something on Wikipedia but I can't quite follow it. Is there an elementary way to show a maximum entropy distribution must be of the form $Cr^k$? How does the maximality of the geometric distributon follow?

/edit: Just realised it's not geometric but almost: $k$ instead of $k-1$ in the exponent.

$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Take your geometric random variable distributed as $p_k=\mathbb P (X=k)=p(1-p)^{k}$ for $k\in\{0,1,2,\dots\}$. Note that your entropy is: $$ H(X)=-\log p-\frac{1-p}{p}\log(1-p). $$

And consider another random variable $Y$ distributed as $q_k=\mathbb P (Y=k)$ such that: $$ \frac{1-p}{p}=\mathbb E(X)=\mathbb E(Y)=\sum_{i=0}^\infty i q_i. $$ We use following lemma, which is nothing but positiveness of KL-divergence:

Lemma For the random variables $X$ and $Y$ distributed as $p_k\neq 0$ and $q_k$ on $k\in\{0,1,2,\dots\}$, we have $$ \mathbb D(Y|X)=\sum_0^\infty q_i\log \frac{q_i}{p_i}\geq 0. $$

The proof of lemma is really simple and it is omitted. Now use the lemma to see that :

$$ H(Y)=-\sum q_i\log q_i\leq -\sum q_i\log p_i\\ =-\sum q_i (\log p+i\log(1-p))=-\log p-\log(1-p)\mathbb E(Y)\\ =-\log p-\frac{1-p}{p}\log(1-p)=H(X). $$

$\endgroup$
2
  • $\begingroup$ Awesome! Instead of the lemma we could also use Gibbs' inequality, couldn't we? $\endgroup$ Commented Feb 2, 2016 at 0:40
  • 1
    $\begingroup$ Gibbs' inequality is the lemma; I just said it in terms of KL-divergence. So yes, you can use it of course, it is the same thing. $\endgroup$ Commented Feb 2, 2016 at 0:48
0
$\begingroup$

Not a solution to your question, but just to let you know that for a geometric random variable $X$, $$P(X=k)=p(1-p)^{k-1}$$ From the definition of expectation or by looking at the moment generating function, you should have that $$E(X)=\dfrac{1}{p}$$ The entropy $H(X)$ is then given by $$H(X) = \sum_{k=1}^{\infty} -(1-p)^{k-1}p \cdot \log_{2}{((1-p)^{k-1}p)}$$ $$\quad =- \log_{2}{(p)} - \frac{(1-p)\log_{2}{(1-p)}}{p}$$

$\endgroup$
5
  • $\begingroup$ Oops, you made me realise my distribution function is not geometric but it's geometric shifted by 1. Given the distribution function I specified my calculation results are correct, aren't they? $\endgroup$ Commented Feb 1, 2016 at 23:57
  • $\begingroup$ No. @akkarin is correct. $\endgroup$ Commented Feb 1, 2016 at 23:58
  • $\begingroup$ @John I agree of course for the computation of H(X), but for the computation of E(X), it depends if Akkarin starts his geometric distribution at $k=0$ or $k=1$. $\endgroup$ Commented Feb 2, 2016 at 0:00
  • $\begingroup$ I guess He is starting in 0, cause the form and mean of the distribution $\endgroup$ Commented Feb 2, 2016 at 0:03
  • $\begingroup$ I'm starting at $0$ but does it really matter for $E(X)$? The first term is zero anyway. For $H(X)$ it seems I started at $1$ but I actually want to start at $0$ so I think I agree with @John's result. $\endgroup$ Commented Feb 2, 2016 at 0:20

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.