17
$\begingroup$

Question:

Might there be a natural geometric interpretation of the exponential of entropy in Classical and Quantum Information theory? This question occurred to me recently via a geometric inequality concerning the exponential of the Shannon entropy.

Original motivation:

The weighted AM-GM inequality states that if $\{a_i\}_{i=1}^n,\{\lambda_i\}_{i=1}^n \in \mathbb{R}_+^n$ and $\sum_{i=1}^n \lambda_i = 1$, then:

\begin{equation} \prod_{i=1}^n a_i^{\lambda_i} \leq \sum_{i=1}^n \lambda_i \cdot a_i \tag{1} \end{equation}

As an application, we find that if $H(\vec{p})$ denotes the Shannon entropy of a discrete probability distribution $\vec{p} = \{p_i\}_{i=1}^n$ and $r_p^2 = \lVert \vec{p} \rVert^2 $ is the $l_2$ norm of $\vec{p}$ then:

\begin{equation} e^{H(\vec{p})} \geq \frac{1}{r_p^2} \tag{2} \end{equation}

This result follows from the observation that if $a_i = p_i$ and $\lambda_i = p_i$,

\begin{equation} e^{-H(\vec{p})} = e^{\sum_i p_i \ln p_i} = \prod_{i=1}^n p_i^{p_i} \tag{3} \end{equation}

\begin{equation} \sum_{i=1}^n p_i^2 = \lVert \vec{p} \rVert^2 \tag{4} \end{equation}

and using (1), we may deduce (2) where equality is obtained when the Shannon entropy is maximised by the uniform distribution i.e. $\forall i, p_i = \frac{1}{n}$.

A remark on appropriate geometric embeddings:

If we consider that the Shannon entropy measures the quantity of hidden information in a stochastic system at the state $\vec{p} \in [0,1]^n$, we may define the level sets $\mathcal{L}_q$ in terms of the typical probability $q \in (0,1)$:

\begin{equation} \mathcal{L}_q = \{\vec{p} \in [0,1]^n: e^{H(\vec{p})} = e^{- \ln q} \} \tag{5} \end{equation}

which allows us to define an equivalence relation over states $\vec{p} \in [0,1]^n$. Such a model is appropriate for events which may have $n$ distinct outcomes.

Now, we'll note that $e^{H(\vec{p})}$ has a natural interpretation as a measure of hidden information while $e^{-H(\vec{p})}$ may be interpreted as the typical probability of the state $\vec{p}$. Given (5), a natural relation between these measures may be found using the Hyperbolic identities:

\begin{equation} \cosh^2(-\ln q) - \sinh^2(-\ln q) = 1 \tag{6} \end{equation}

\begin{equation} \cosh(-\ln q) - \sinh(-\ln q) = q \tag{7} \end{equation}

where $2 \cdot \cosh(-\ln q)$ is the sum of these two measures and $2 \cdot \sinh(-\ln q)$ may be understood as their difference. This suggests that the level sets $\mathcal{L}_q$ have a natural Hyperbolic embedding in terms of Hyperbolic functions.

References:

  1. Olivier Rioul. This is IT: A Primer on Shannon’s Entropy and Information. Séminaire Poincaré. 2018.

  2. David J.C. MacKay. Information Theory, Inference and Learning Algorithms. Cambridge University Press 2003.

  3. John C. Baez, Tobias Fritz, Tom Leinster. A Characterization of Entropy in Terms of Information Loss. Arxiv. 2011.

Improve this question
$\endgroup$
1

4 Answers 4

Reset to default
29
$\begingroup$

With apologies for promoting my own work, there's a whole book on the mathematics of the exponentials of various entropies:

Tom Leinster, Entropy and Diversity: The Axiomatic Approach. Cambridge University Press, 2021.

You can download a free copy by clicking, although persons of taste will naturally want to grace their bookshelves with the bound work.

The direct answer to your literal question is that I don't know of a compelling geometric interpretation of the exponential of entropy. But the spirit of your question is more open, so I'll explain (1) a non-geometric interpretation of the exponential of entropy, and (2) a geometric interpretation of the exponential of maximum entropy.

Diversity as the exponential of entropy

As Carlo Beenakker says, the exponential of entropy (Shannon or more generally Rényi) has long been used by ecologists to quantify biological diversity. One takes a community with $n$ species and writes $\mathbf{p} = (p_1, \ldots, p_n)$ for their relative abundances, so that $\sum p_i = 1$. Then $D_q(\mathbf{p})$, the exponential of the Rényi entropy of $\mathbf{p}$ of order $q \in [0, \infty]$, is a measure of the diversity of the community, or "effective number of species" in the community.

Ecologists call $D_q$ the Hill number of order $q$, after the ecologist Mark Hill, who introduced them in 1973 (acknowledging the prior work of Rényi). There is a precise mathematical sense in which the Hill numbers are the only well-behaved measures of diversity, at least if one is modelling an ecological community in this crude way. That's Theorem 7.4.3 of my book. I won't talk about that here.

Explicitly, for $q \in [0, \infty]$ $$ D_q(\mathbf{p}) = \biggl( \sum_{i:\,p_i \neq 0} p_i^q \biggr)^{1/(1 - q)} $$ ($q \neq 1, \infty$). The two exceptional cases are defined by taking limits in $q$, which gives $$ D_1(\mathbf{p}) = \prod_{i:\, p_i \neq 0} p_i^{-p_i} $$ (the exponential of Shannon entropy) and $$ D_\infty(\mathbf{p}) = 1/\max_{i:\, p_i \neq 0} p_i. $$

Rather than picking one $q$ to work with, it's best to consider all of them. So, given an ecological community and its abundance distribution $\mathbf{p}$, we graph $D_q(\mathbf{p})$ against $q$. This is called the diversity profile of the community, and is quite informative. As Carlo says, different values of the parameter $q$ tell you different things about the community. Specifically, low values of $q$ pay close attention to rare species, and high values of $q$ ignore them.

For example, here's the diversity profile for the global community of great apes:

ape diversity profile

(from Figure 4.3 of my book). What does it tell us? At least two things:

  • The value at $q = 0$ is $8$, because there are $8$ species of great ape present on Earth. $D_0$ measures only presence or absence, so that a nearly extinct species contributes as much as a common one.

  • The graph drops very quickly to $1$ — or rather, imperceptibly more than $1$. This is because 99.9% of ape individuals are of a single species (humans, of course: we "outcompeted" the rest, to put it diplomatically). It's only the very smallest values of $q$ that are affected by extremely rare species. Non-small $q$s barely notice such rare species, so from their point of view, there is essentially only $1$ species. That's why $D_q(\mathbf{p}) \approx 1$ for most $q$.

Maximum diversity as a geometric invariant

A major drawback of the Hill numbers is that they pay no attention to how similar or dissimilar the species may be. "Diversity" should depend on the degree of variation between the species, not just their abundances. Christina Cobbold and I found a natural generalization of the Hill numbers that factors this in — similarity-sensitive diversity measures.

I won't give the definition (see that last link or Chapter 6 of the book), but mathematically, this is basically a definition of the entropy or diversity of a probability distribution on a metric space. (As before, entropy is the log of diversity.) When all the distances are $\infty$, it reduces to the Rényi entropies/Hill numbers.

And there's some serious geometric content here.

Let's think about maximum diversity. Given a list of species of known similarities to one another — or mathematically, given a metric space — one can ask what the maximum possible value of the diversity is, maximizing over all possible species distributions $\mathbf{p}$. In other words, what's the value of $$ \sup_{\mathbf{p}} D_q(\mathbf{p}), $$ where $D_q$ now denotes the similarity-sensitive (or metric-sensitive) diversity? Diversity is not usually maximized by the uniform distribution (e.g. see Example 6.3.1 in the book), so the question is not trivial.

In principle, the answer depends on $q$. But magically, it doesn't! Mark Meckes and I proved this. So $$ D_{\text{max}}(X) := \sup_{\mathbf{p}} D_q(\mathbf{p}) $$ is a well-defined real invariant of finite metric spaces $X$, independent of the choice of $q \in [0, \infty]$.

All this can be extended to compact metric spaces, as Emily Roff and I showed. So every compact metric space has a maximum diversity, which is a nonnegative real number.

What on earth is this invariant? There's a lot we don't yet know, but we do know that maximum diversity is closely related to some classical geometric invariants.

For instance, when $X \subseteq \mathbb{R}^n$ is compact, $$ \text{Vol}(X) = n! \omega_n \lim_{t \to \infty} \frac{D_{\text{max}}(tX)}{t^n}, $$ where $\omega_n$ is the volume of the unit $n$-ball and $tX$ is $X$ scaled by a factor of $t$. This is Proposition 9.7 of my paper with Roff and follows from work of Juan Antonio Barceló and Tony Carbery. In short: maximum diversity determines volume.

Another example: Mark Meckes showed that the Minkowski dimension of a compact space $X \subseteq \mathbb{R}^n$ is given by $$ \dim_{\text{Mink}}(X) = \lim_{t \to \infty} \frac{D_{\text{max}}(tX)}{\log t} $$ (Theorem 7.1 here). So, maximum diversity determines Minkowski dimension too.

There's much more to say about the geometric aspects of maximum diversity. Maximum diversity is closely related to another recent invariant of metric spaces, magnitude. Mark and I wrote a survey paper on the more geometric and analytic aspects of magnitude, and you can find more on all this in Chapter 6 of my book.

Postscript

Although diversity is closely related to entropy, the diversity viewpoint really opens up new mathematical questions that you don't see from a purely information-theoretic standpoint. The mathematics of diversity is a rich, fertile and underexplored area, waiting for mathematicians to come along and explore it.

Improve this answer
$\endgroup$
7
$\begingroup$

If I may broaden the question somewhat to include other interpretations of the exponential of entropy, it is commonly used in ecology to measure how many different species there are in a community. In that context one considers the generalized diversity index (or Hill number) $${}^q\!D=\left ( {\sum_{i} p_i^q} \right )^{1/(1-q)},$$ with the exponential of entropy being the $q\rightarrow 1$ limit,$^\ast$ $$\lim_{q\rightarrow 1} {}^q\!D=\exp\left(-\sum_{i} p_i\ln p_i\right).$$ The quantity ${}^1\!D=e^S$ is called the Shannon diversity.

Different values of $q$ correspond to different weights given to different species $i=1,2,\ldots$, dependent on their relative abundance $p_i$.

Increasing the value of $q$ increases the weight given to the most abundant species, thereby reducing the effective number of species, hence $${}^q\!D\geq {}^{q'}\!D\;\;\text{if}\;\;q<q'$$ The inequality in the OP is the special case $q=1$, $q'=2$.

$\ast$ $$\lim_{q\rightarrow 1}=\left ( {\sum_{i} p_i^q} \right )^{1/(1-q)}=\lim_{q\rightarrow 1}\exp\left( -(1-q)^{-1}\left(1-{\sum_{i} p_i^q}\right) \right )=\lim_{q\rightarrow 1}\exp\left( \sum_i p_i\frac{1-p_i^{q-1}}{q-1} \right )$$ $$=\exp\left(-\sum_i p_i\ln p_i\right).$$ I have used $\sum_i p_i=1$ and the two identies $\lim_{x\rightarrow 0}(1-xa)^{1/x}=e^{-a}$ and $\lim_{x\rightarrow 0}(a^x-1)/x=\ln a$.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ I’d never heard of the diversity index before, but its log is also known as Renyi entropy. $\endgroup$ Commented Jul 24, 2021 at 11:32
  • 1
    $\begingroup$ @მამუკაჯიბლაძე That's almost right, except you want to take the reciprocal of your right-hand side. $\endgroup$ Commented Jul 24, 2021 at 12:14
  • 2
    $\begingroup$ @DeaneYang Yes. There are many diversity indices. This family was introduced by the ecologist Mark Hill in 1973, who explicitly acknowledged that they are the exponentials of the Rényi entropies. But for the purposes of quantifying diversity, the exponentials are much better: see e.g. Section 2.4 of arxiv.org/abs/2012.02113 $\endgroup$ Commented Jul 24, 2021 at 12:17
  • 1
    $\begingroup$ @TomLeinster, isn’t it true that the exponential of entropy often is more useful or gives cleaner formulas? The entropy itself seems to arise from the counting arguments that motivate the definition of entropy. I have co-authored papers studying the continuous version of Renyi entropy, but in fact everything is done using its exponential. $\endgroup$ Commented Jul 24, 2021 at 14:39
  • 2
    $\begingroup$ @DeaneYang In my experience I think the exponential does usualy give cleaner formulas. Other people's experiences may be different! And the logarithmic form certainly seems more intuitive when talking about coding. Also, the exponential has the advantage of not depending on a choice of base for logarithms: $\prod p_i^{-p_i}$ just is what it is, whereas for $-\sum p_i \log(p_i)$ you have to make a choice. $\endgroup$ Commented Jul 24, 2021 at 15:02
3
$\begingroup$

Not sure what you consider to be "geometric" in this context, but the exponential of the Shannon entropy is simply a measure of the "effective size of support" (ESS) of the underlying probability distribution. In classical mechanics, it is a measure of the hyper-volume of the reasonably probable part of phase space, and is independent of the choice of canonical phase space coordinates. In high energy physics, this is also closely related to the notion of the "emittance" of a particle beam in an accelerator or storage ring. In quantum mechanics, the exponential of the von Neumann entropy is, roughly speaking, equal to the number of accessible microstates, and is independent of the choice of basis. The logarithm is taken to (i) tame otherwise huge numbers, and more fundamentally, (ii) make the quantity additive over independent sub-systems.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Might you have particular references for what you call the Effective Size of Support?As for your remark on its application in classical mechanics, I think an ergodic assumption may be necessary? $\endgroup$ Commented Aug 11, 2021 at 10:42
  • 1
    $\begingroup$ effective size of support (ess), although not with that name, realy goes back to the beginnings of Shannon information theory --- see the discussion of the "asymptotic equipartition theorem" in Cover and Thomas or other standard textbooks. For other, more recent references, see for example researchgate.net/publication/… $\endgroup$ Commented Aug 16, 2021 at 2:08
  • $\begingroup$ As for identifying the exponential of the entropy with the hyper-volume of the "accessible" or "reasonably likely"part of phase-space, this really goes back to Boltzmann, Einstein, and other pioneers of statistical mechanics. As emphasized by E.T. Jaynes, it does not actually require any assumption of ergodicity to establish or justify this identification. $\endgroup$ Commented Aug 16, 2021 at 2:16
  • 1
    $\begingroup$ If the phase-space distribution function were uniform over some subset of phase space, then the exponential of the entropy (in suitable units) is equal to the hypervolume of the support of this distribution. The Shannon-Gibbs entropy generalizes this (in essentially the only fully consistent way) to allow for non-uniform weighting $\endgroup$ Commented Aug 16, 2021 at 2:16
3
$\begingroup$

With apologies also for promoting my own work, a colleague (Tim Hosgood) and I have recently published an article in Entropy, where we give a geometric interpretation for the exponentiated entropy of empirical distributions, namely as the length $L$ of a certain rectangle. What makes this rectangle interesting is that its width $W$ and area $A$ (from which you can calculate its length as $L=\frac{A}{W}$ and hence entropy as $H=\log(L)$) are additively and multiplicatively homomorphic in the original distribution.

It's well known that the entropy of a product of distributions is the sum of their entropies, so the exponentiated entropy (sometimes called the perplexity) of a product is the product of the exponentiated entropies. Thus one says that exponentiated entropy is multiplicatively homomorphic in the distribution. What we're saying in this short note is that if you use the whole rectangle (its area and width) instead of just its length, you can get not only a multiplicative but also an additive homomorphism; that is you get a rig homomorphism from empirical distributions to rectangles.

Improve this answer
$\endgroup$

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.