The meaning of entropy: Shannon's theorems

Bricogne, G.

doi:10.1107/97809553602060000690

International
Tables for
Crystallography
Volume F
Crystallography of biological macromolecules
Edited by M. G. Rossmann and E. Arnold

pdf | chapter contents | chapter index | related articles

International Tables for Crystallography (2006). Vol. F. ch. 16.2, p. 346 | 1 | 2 |

Section 16.2.2.2. The meaning of entropy: Shannon's theorems

G. Bricogne^a ^*

^aLaboratory of Molecular Biology, Medical Research Council, Cambridge CB2 2QH, England
Correspondence e-mail: gb10@mrc-lmb.cam.ac.uk

16.2.2.2. The meaning of entropy: Shannon's theorems

| top | pdf |

Two important theorems [Shannon & Weaver (1949), Appendix 3] provide a more intuitive grasp of the meaning and importance of entropy:

(1) H is approximately the logarithm of the reciprocal probability of a typical long message, divided by the number of symbols in the message; and
(2) H gives the rate of growth, with increasing message length, of the logarithm of the number of reasonably probable messages, regardless of the precise meaning given to the criterion of being `reasonably probable'.

The entropy H of a source is thus a direct measure of the strength of the restrictions placed on the permissible messages by the distribution of probabilities over the symbols, lower entropy being synonymous with greater restrictions. In the two cases above, the maximum values of the entropy $[H_{\max} = \log n]$ and $[H_{\max} = \log \mu ({\cal A})]$ are reached when all the symbols are equally probable, i.e. when q is a uniform probability distribution over the symbols. When this distribution is not uniform, the usage of the different symbols is biased away from this maximum freedom, and the entropy of the source is lower; by Shannon's theorem (2), the number of `reasonably probable' messages of a given length emanating from the source decreases accordingly.

The quantity that measures most directly the strength of the restrictions introduced by the non-uniformity of q is the difference $[H(q) - H_{\max}]$ , since the proportion of N-atom random structures which remain `reasonably probable' in the ensemble of the corresponding source is $[\exp \{N[H(q) - H_{\max}]\}]$ . This difference may be written (using continuous rather than discrete distributions) $[H(q) - H_{\max} = - \textstyle\int\limits_{{\cal A}}\displaystyle q({\bf s}) \log [q({\bf s})/m({\bf s})] \; \hbox{d}\mu ({\bf s}), \eqno(16.2.2.3)]$ where m(s) is the uniform distribution which is such that $[H(m) = H_{\max} = \ \log \mu ({\cal A})]$ .

References

Shannon, C. E. & Weaver, W. (1949). The mathematical theory of communication. Urbana: University of Illinois Press.Google Scholar

International Tables for Crystallography (2006). Vol. F. ch. 16.2, p. 346