2
$\begingroup$

Cross-entropy (loss), $-\sum y_i\;\log(\hat{p_i})$, estimates the amount of information needed to encode $y$ using Huffman encoding based on the estimated probabilities $\hat{p}$. Therefore one could claim it should be considered to measure the amount of information, for example a number of bits.

Depending on the base of $\log$, these can be binary bits or digits, but typically are Euler-bits since $\ln$ is mostly used. Is there a popular or official name or unit for these so called Euler-bits? Is it OK to consider the unit of cross-entropy loss this way?

Obviously, loss is used for optimization and most don't care about the exact unit, but I'm curious and would like to use the correct terminology when explaining ML to others.

$\endgroup$

1 Answer 1

2
$\begingroup$

Basically, the first one who defines such amount of information was Shanon thanks to his definition of entropy: https://en.wikipedia.org/wiki/Entropy_(information_theory)

using bits of Shanon.

It describes the level of "interest" or "surprise" of information and it is also based on log 2. But I don't know if it is applicable to cross entropy.

$\endgroup$
1
  • 1
    $\begingroup$ So they are natural units or nats. Thanks. $\endgroup$ Commented Jun 22, 2021 at 14:09

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.