Chapter 7.5. Statistical fluctuations

Intensities of diffraction have two distinct probability distributions: (1) the a priori probability that an arbitrarily chosen reflection of a particular substance will have a particular `true' intensity (R in the notation used below), and (2) the probability that a `true' R will have an observed value . The distributions of the first type depend on the symmetry and composition of the material, and are treated in Chapter 2.1 of Volume B of International Tables for Crystallography (Shmueli, 2001). The distributions of fluctuations of the second type, variations of the values observed for a particular reflection, are treated here.

The crude counts or counting rates are rarely used directly in crystallographic calculations; they are subjected to some form of data processing to provide `intensities of reflection' in a form suitable for the determination of crystal structures, electron densities, line profiles for the study of defects, etc. The resulting intensity for the jth reflection, , will be directly proportional to the corresponding ; when no confusion can arise, one of the subscripts will be omitted. The value of the proportionality factor in will be different for different reflections, since they will occur at different Bragg angles, have different absorption corrections, etc.

Whatever the radiation, in both single-crystal and powder diffractometry, the integrated intensity of a reflection is obtained as a difference between a counting rate averaged over a volume of reciprocal space intended to include the reflected intensity and a counting rate averaged over a neighbouring volume of reciprocal space intended to include only background. If these intentions are not effectively realized, there will be a systematic error in the measured intensity, but in any case there will be statistical fluctuations in the counting rates. The two basic modes (Parrish, 1956) are fixed-time counting and fixed-count timing. In the first, counts are accumulated for a pre-determined time interval, and the variance of the observed counting rate is proportional to the true (mean) counting rate. In the second, on the other hand, the counting is continued until a pre-determined number of counts is reached, and the variance of the observed counting rate is proportional to the square of the true counting rate. Put otherwise, the relative error in the intensities goes down inversely as the square root of the intensity for fixed-time counting, whereas it is independent of the intensity for fixed-count timing. Each mode has advantages, depending on the purpose of the measurements, and numerous modifications and compromises have been proposed in order to increase the efficiency of the use of the available time. References to some of the many papers are given in Section 7.5.7.

In principle, probability distributions can be determined for any postulated counting mode. In practice, they become complicated for all but the simplest modes; this is true even for the single measurement of the total counting rate or the background counting rate, but is even more pronounced for the distribution of their difference (the reflection-only rate). For most crystallographic purposes, however, it is only necessary to know the mean (to correct for bias, if present) and the variance (for the estimation of weights in refinement processes, see Part 8 ) of the distribution function.

In the absence of disturbing influences (mains-voltage fluctuations, unrectified or unsmoothed high-tension supplies, `dead time' of the counter or counter circuits, etc.), the number of counts recorded during the predetermined time interval used in the fixed-time mode will fluctuate in accordance with the Poisson probability distribution. If the `true' number of counts to be expected in the interval is N, the probability that the observed number will be is given by where all quantities appearing are necessarily non-negative. Both the mean and the variance of are N. If the `true' number of counts to be expected when the diffractometer is set to receive a reflection is T, and the `true' number when it is set to receive the immediate background is B, the `true' intensity of the reflection is provided that the time interval used for the reflection is equal to the time interval used for the background. In practice, the `observed' values of and fluctuate with probabilities given by (7.5.3.1) with T or B replacing N, so that the observed value, will also fluctuate, and can, occasionally, take on negative values. The treatment of `measured-as-negative' intensities is discussed in Section 7.5.6.

Although the sum of two Poisson-distributed variables is also Poisson, the difference is not, and the probability of given by (7.5.3.3) has been shown to be (Skellam, 1946; Wilson, 1978) where is the hyperbolic Bessel function of the first kind. The mean and variance of are T − B and T + B, respectively. If the times used for reflection and background are not equal, but are in the ratio of k : 1, the mean and variance of are T − kB and , respectively. The distribution of then involves a generalized Bessel function¹ depending on k: for the properties of the generalized Bessel function, see Wright (1933) and Olkha & Rathie (1971). The probability distributions (7.5.3.4) and (7.5.3.6) seem to be very close to a normal distribution with the same mean and variance for near R, but the probability of moderate deviations from R is less than normal and for large deviations is greater than normal.

In applications, it is usual to work with intensities expressed as counting rates, rather than as numbers of counts, even when fixed-time counting is used. The total intensity τ expressed as a counting rate is where t is the time devoted to the measurement, and the variance of the counting rate is Similar expressions apply for the background, with B for the count, b for the time, and β for the counting rate. For the reflection count, the corresponding expressions are To avoid confusion, upper-case italic letters are used for numbers of counts, lower-case italic for counting times, and the corresponding lower-case Greek letters for the corresponding counting rates. In accordance with common practice, however, will be used for the intensity of the jth reflection, the context making it clear whether I is a number of counts or a counting rate.

The probability of a time t being required to accumulate N counts when the true counting rate is ν is given by a Γ distribution (Abramowitz & Stegun, 1964, p. 255): The ratio N/t is a slightly biased estimate of the counting rate ν; the unbiased estimate is (N − 1)/t. The variance of this estimate is , or, nearly enough for most purposes, . The differences introduced by the corrections −1 and −2 are generally negligible, but would not be so for counts as low as those proposed by Killean (1967). If such corrections are important, it should be noticed that there is an ambiguity concerning N, depending on how the timing is triggered. It may be triggered by a count that is counted, or by a count that is not counted, or may simply be begun, independently of the incidence of a count. Equation (7.5.4.1) assumes the first of these.

Equation (7.5.4.1) may be inverted to give the probability distribution of the observed counting rate ν_o instead of the probability distribution of the time t: There does not seem to be any special name for the distribution (7.5.4.2). Only its first (N − 1) moments exist, and the integral expressing the probability distribution of the difference of the reflection and the background rates is intractable (Wilson, 1980).

It has been customary in crystallographic computations, but without theoretical justification, to omit all reflections with intensities less than two or three times their standard uncertainties. Hirshfeld & Rabinovich (1973) asserted that the failure to use all reflections, even those for which the subtraction of background has resulted in a negative net intensity, at their measured values will lead to a bias in the parameters resulting from a least-squares refinement. This is, however, inconsistent with the Gauss–Markov theorem (see Section 8.1.2 ), which shows that least-squares estimates are unbiased, independent of the weights used, if the observations are unbiased estimates of quantities predicted by a model. Giving some observations zero weight therefore cannot introduce bias. Provided the set of included observations is sufficient to give a nonsingular normal equations matrix, parameter estimates will be unbiased, but inclusion of as many well determined observations as possible will yield the most precise estimates. Requiring that the net intensity be greater than 2σ assures that the value of |F| will be well determined. Furthermore, Prince & Nicholson (1985) showed that, if the net intensity, I, or |F|² is used as the observed quantity, weak reflections have very little leverage (see Section 8.4.4 ), and therefore omitting them cannot have a significant effect on the precision of parameter estimates.

The use of negative values of I or |F|² is also inconsistent with Bayes's theorem, which implies that a negative value cannot be an unbiased estimate of an inherently non-negative quantity. There are statistical methods for estimating the positive value of |F| that led to a negative value of I. The best known approach is the Bayesian method of French & Wilson (1978), who observe that "Instead of thanking the data for the information that certain structure factor moduli are small, we accuse them of assuming `impossible' negative values. What we should do is combine our knowledge of the non-negativity of the true intensities with the information concerning their magnitudes contained in the data."

There have been many papers on optimizing counting times for achieving different purposes, and all optimization procedures require some knowledge of the distributions of counts or counting rates; often only the mean and variance of the distribution are required. It is also necessary to know the functional relationship between the quantity of interest and the counts (counting rates, intensities) entering into its measurement. Typically, the object is to minimize the variance of some function of the measured intensities, say . General statistical theory gives the usual approximation where cov(I_i, I_j) is the covariance of I_i and I_j if , and is the variance of , , if i = j. There is very little correlation² between successive intensity measurements in diffractometry, so that cov(I_i, I_j) is negligible for . Equation (7.5.7.1) becomes These equations are strictly accurate only if F is a linear function of the I's, a condition satisfied for the integrated intensity, but for few other quantities of interest. In most applications in diffractometry, however, the contribution of each I_j is sufficiently small in comparison with the total to make the application of equations (7.5.7.1) and (7.5.7.2) plausible. Any proportionality factors (Section 7.5.1) can be absorbed into the functional relationship between F and the I_j's.

The object is to minimize σ²(F) by varying the time spent on each observation, subject to a fixed total time It is simplest to regard the total intensity and the background intensity as separate observations, so that in (7.5.7.2) the sum is over n `background' and n `total' observations. With expressed as a counting rate, its variance is [equation (7.5.3.8)], so that (7.5.7.2) becomes where for brevity G has been written for . The variance of F will be a minimum if, for any small variations of the counting times , subject to the constancy of the total time T. There is thus the constraint These equations are consistent if for all j where k is a constant determined by the total time T: The minimum variance is thus achieved if each observation is given a time proportional to the square root of its intensity. A little manipulation now gives for the desired minimum variance The minimum variance is found to be a perfect square, and the standard uncertainty takes a simple form.

Here, the optimization has been treated as a modification of fixed-time counting. However, the same final expression is obtained if the optimization is treated as a modification of fixed-count timing (Wilson, Thomsen & Yap, 1965).

Space does not permit detailed discussion of the numerous papers on various aspects of optimization. If the time required to move the diffractometer from one observation position to another is appreciable, the optimization problem is affected (Shoemaker & Hamilton, 1972, and references cited therein). There is some dependence on the radiation (X-ray versus neutron) (Shoemaker, 1968; Werner, 1972a,b). A few other papers of historical or other interest are included in the list of references, without detailed mention in the text: Grant (1973); Killean (1972, 1973); Mack & Spielberg (1958); Mackenzie & Williams (1973); Szabó (1978); Thomsen & Yap (1968); Zevin, Umanskii, Kheiker & Panchenko (1961).