Skip to main content
Cross Validated

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

4
  • 3
    $\begingroup$ I think you have to be more specific about what you are looking for: entropy is after all as "statistical" a measure as variance etc. so the maximum entropy distribution maximises entropy is a perfectly good statistical description. So it seems to me you have to go outside statistics to come up with a "justification" $\endgroup$ Commented Aug 1, 2013 at 9:48
  • 1
    $\begingroup$ Seanv: I agree that entropy, as a statistical functional, is just as "statistical" as variance, expected value, skew etc. However, using mean and standard deviation as examples, these have purely probabilistic interpretations via Markov's and Chebyshev's theorems and ultimately in one of a number of central limit theorems and also intuitively as long run sums (for the mean) and RMS error (for the standard deviation). I should perhaps repharase my question to read "Probabilistic interpretation of maximum entropy distributions". $\endgroup$ Commented Aug 1, 2013 at 12:24
  • 1
    $\begingroup$ Annika, maximum entropy distribution has the following interpretation: If $X_1,X_2,\dots$ are i.i.d. random variables, then the conditional probalitity $P(\cdot|X_1+\dots+X_n=na)\to P^*(\cdot)$ as $n\to \infty$ where $P^*$ is the maximum entropy distribution from the set $\{P:\mathbb{E}_PX=a\}$. See also ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1056374&tag=1 $\endgroup$ Commented Aug 1, 2013 at 12:45
  • 2
    $\begingroup$ Thanks Ashok. Ill take a look at that paper in more detail. This seems like a specific case of maximizing entropy for a given mean, but I am still curious as to what the operation of maximizing the Shanon entropy is doing mathematically such that the above result holds? Is it effectively minimizing the maximum density or average concentration of the probability measure? $\endgroup$ Commented Aug 1, 2013 at 15:55