International
Tables for Crystallography Volume F Crystallography of biological macromolecules Edited by M. G. Rossmann and E. Arnold © International Union of Crystallography 2006 |
International Tables for Crystallography (2006). Vol. F. ch. 16.2, p. 346
Section 16.2.2.2. The meaning of entropy: Shannon's theorems
aLaboratory of Molecular Biology, Medical Research Council, Cambridge CB2 2QH, England |
Two important theorems [Shannon & Weaver (1949), Appendix 3] provide a more intuitive grasp of the meaning and importance of entropy:
The entropy H of a source is thus a direct measure of the strength of the restrictions placed on the permissible messages by the distribution of probabilities over the symbols, lower entropy being synonymous with greater restrictions. In the two cases above, the maximum values of the entropy and
are reached when all the symbols are equally probable, i.e. when q is a uniform probability distribution over the symbols. When this distribution is not uniform, the usage of the different symbols is biased away from this maximum freedom, and the entropy of the source is lower; by Shannon's theorem (2), the number of `reasonably probable' messages of a given length emanating from the source decreases accordingly.
The quantity that measures most directly the strength of the restrictions introduced by the non-uniformity of q is the difference , since the proportion of N-atom random structures which remain `reasonably probable' in the ensemble of the corresponding source is
. This difference may be written (using continuous rather than discrete distributions)
where m(s) is the uniform distribution which is such that
.
References
