Questions tagged [entropy]
This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.
1,690 questions
0 votes
1 answer
46 views
Can mutual information be defined between a random variable and a conditional distribution
Quantities like mutual Information $I$, entropy $H$,etc. are typically defined as taking random variables as input. However, they are actually just functions on probability distributions - e.g. the ...
- 35
5 votes
0 answers
81 views
How can we rigorously bound the performance of the Fano code?
There are estimates on the expected codeworth length of the Fano symbol code (not to be confused with the Shannon code), but I don't know where they come from. Some definitions: Let $\mathcal{X}$ be a ...
- 48.2k
1 vote
1 answer
67 views
Interpret the halving of entropy
I am trying to understand and explain entropy intuitively. I think of entropy as ambiguity, and also, as the expected value of the knowledge gained. The doubling of entropy is straightforward to ...
2 votes
0 answers
42 views
Is conditional mutual information $I(X;Y\mid Z)$ lower semicontinuous for general $X,Y,Z$?
Assume $(X_n,Y_n,Z_n)\Rightarrow (X,Y,Z)$ weakly on standard Borel spaces. Is it always true that $$I(X;Y\mid Z)\ \le\ \liminf_{n\to\infty} I(X_n;Y_n\mid Z_n)?$$ It is classical that relative entropy $...
1 vote
1 answer
82 views
Regarding an axiom of the entropy function
I'm currently attending an introductive course on information theory given by a very famous mathematician who is undeniably an expert in the field. When explaining the axioms of said entropy function, ...
- 1,087
0 votes
0 answers
38 views
Entropy of observed counts given expected probability
Suppose I have a set of symbols with expected probabilities for each, and a set of n observed sequences of these symbols, each of length m. From simply looking over the array of observed counts of X ...
- 133
12 votes
2 answers
419 views
Why does this process produce the binary entropy distribution? Random point from randomly chosen interval
Draw two independent random variables $A$ and $B$ each from ${\rm Uniform}(0,1)$. Then draw a sample $x$ from ${\rm Uniform}(A,B)$ or ${\rm Uniform}(B,A)$, depending on which of $A$ and $B$ is larger. ...
- 25.7k
0 votes
0 answers
32 views
Conditional Entropy on Whole vs. Split Timeseries Dataset
I'm a novice at statistics and I don't fully grasp what I'm doing mathematically so this question isn't asking a discrete question. Only support to help intuit the math. Suppose I have a simple time-...
- 1
1 vote
0 answers
30 views
Product Theorem for Topological Entropy
I’ve been reading Goodwyn’s paper “The Product Theorem for Topological Entropy”. Theorem (Goodwyn): If $X, Y$ are compact Hausdorff spaces and $T: X \to X$, $S: Y \to Y$ are continuous, then $$ h(T \...
- 161
1 vote
0 answers
37 views
Small-time Asymptotics of Nash Entropy for Heat Kernel on Noncompact Manifolds of Non-Negative Ricci Curvature
I've been attempting to prove a statement of Lei Ni regarding the Nash entropy on a noncompact manifold of non-negative Ricci curvature ,say $(M, g)$, and have been having some difficulty. A full ...
- 152
3 votes
2 answers
114 views
Variational representation of entropy
For a positive random variable $X$, the entropy is defined as $H(X) = \mathbb{E}(X \log X) - \mathbb{E}(X) \log (\mathbb{E}(X) )$. I want to prove following variational representation: \begin{align*} ...
- 2,316
0 votes
0 answers
35 views
Distance from a uniform vector using KL divergence on arbitrary non-negative large values
Motivated by the desideratum to prove that the uniform probability mass function maximizes Shannon entropy, I formulated the following convex optimization problem $$ \arg \max_{\bf x} - \sum_i x_i \...
- 393
2 votes
0 answers
55 views
Help Understanding Edge Case of Sanov's Theorem
I was reading the 2nd Edition of Foundations of Machine Learning by Mohri, Rostamizadeh, and Talwalkar, and I am confused about an aspect of Sanov's Theorem (i.e., Theorem D.3), which is stated below. ...
0 votes
1 answer
60 views
A question about entropy
I am reading Lemma 1 of this note: it says $C$ is a random variable over a finite set $S$. And $X$ is a Bernoulli random variable satisfying $$Pr(X=1\mid C=s)=p_s.$$ And then it uses $$H(X\mid C) =\...
- 2,498
2 votes
1 answer
62 views
Max. ent. distribution of a random variable under prescribing its $n$ lower-order moments?
It is classical that the real positive random variable of max. entropy with prescribed mean value has exponential pdf and the one with prescribed mean value and variance has a truncated Gaussian pdf. ...
- 133