7
$\begingroup$

Let $P=\{p_1,\dots,p_n\},Q=\{q_1,\dots,q_n\}$ are discrete probability distribution. It is well known that $D_\text{KL}(P\|Q)=\sum_{i=1}^np_i(\log p_i-\log q_i)$ is the Kullback–Leibler divergence from $Q$ to $P$, but what about the quantity $\sum_{i=1}^n(p_i-q_i)\log p_i$ ? Does it have a name or appear in any paper?

Motivation: That quantity appear when I calculate $\frac{d}{dx}\rvert_{x=0}H((1-x)P+xQ)$.

$\endgroup$
0

1 Answer 1

8
$\begingroup$

I've seen and used this quantity (the difference between the entropy of $P$ and the cross-entropy of $P$ from $Q$) informally as a gauge of over-/under-confidence in the context of forecasting: If an expert submits the forecast $\vec{p} = (p_1,\ldots,p_n)$ for the categorical distribution with true distribution $\vec{q} = (q_1,\ldots,q_n)$, then $\sum_i(p_i-q_i)\log p_i$ is the difference between the expert's subjective expectation (under $\vec{p}$) and the true objective expectation (under $\vec{q}$) of the forecast's log-score, so the sign of this quantity can be used to define notions of overconfidence and underconfidence. This is done for example in Section 5.5 of Eric Neyman's (2024) PhD thesis (written in a general form to work with any proper scoring rule).

$\endgroup$

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.