International
Tables for Crystallography Volume F Crystallography of biological macromolecules Edited by M. G. Rossmann and E. Arnold © International Union of Crystallography 2006 |
International Tables for Crystallography (2006). Vol. F. ch. 16.2, pp. 347-348
Section 16.2.2.4. Jaynes' maximum-entropy formalism
aLaboratory of Molecular Biology, Medical Research Council, Cambridge CB2 2QH, England |
Jaynes (1957) solved the problem of explicitly determining such maximum-entropy distributions in the case of general linear constraints, using an analytical apparatus first exploited by Gibbs in statistical mechanics.
The maximum-entropy distribution , under the prior prejudice m(s), satisfying the linear constraint equations
where the
are linear constraint functionals defined by given constraint functions
, and the
are given constraint values, is obtained by maximizing with respect to q the relative entropy defined by equation (16.2.2.4)
. An extra constraint is the normalization condition
to which it is convenient to give the label
, so that it can be handled together with the others by putting
,
.
By a standard variational argument, this constrained maximization is equivalent to the unconstrained maximization of the functional where the
are Lagrange multipliers whose values may be determined from the constraints. This new variational problem is readily solved: if q(s) is varied to
, the resulting variations in the functionals
and
will be
respectively. If the variation of the functional (16.2.2.7)
is to vanish for arbitrary variations
, the integrand in the expression for that variation from (16.2.2.8)
must vanish identically. Therefore the maximum-entropy density distribution
satisfies the relation
and hence
It is convenient now to separate the multiplier associated with the normalization constraint by putting
where Z is a function of the other multipliers
. The final expression for
is thus
The values of Z and of
may now be determined by solving the initial constraint equations. The normalization condition demands that
The generic constraint equations (16.2.2.5)
determine
by the conditions that
for
. But, by Leibniz's rule of differentiation under the integral sign, these equations may be written in the compact form
Equations (ME1), (ME2) and (ME3) constitute the maximum-entropy equations.
The maximal value attained by the entropy is readily found: i.e. using the constraint equations
The latter expression may be rewritten, by means of equations (ME3), as
which shows that, in their dependence on the λ's, the entropy and log Z are related by Legendre duality.
Jaynes' theory relates this maximal value of the entropy to the prior probability of the vector c of simultaneous constraint values, i.e. to the size of the sub-ensemble of messages of length N that fulfil the constraints embodied in (16.2.2.5)
, relative to the size of the ensemble of messages of the same length when the source operates with the symbol probability distribution given by the prior prejudice m. Indeed, it is a straightforward consequence of Shannon's second theorem (Section 16.2.2)
as expressed in equation (16.2.2.3)
that
where
is the total entropy for N symbols.
References
