Least-squares plane based on uncorrelated, isotropic weights

Marsh, R. E.; Schomaker, V.

doi:10.1107/97809553602060000560

International
Tables for
Crystallography
Volume B
Reciprocal space
Edited by U. Shmueli

pdf | chapter contents | chapter index | related articles

International Tables for Crystallography (2006). Vol. B. ch. 3.2, pp. 353-355 | 1 | 2 |

Section 3.2.2. Least-squares plane based on uncorrelated, isotropic weights

R. E. Marsh^a ^* and V. Schomaker^b ^‡

^aThe Beckman Institute–139–74, California Institute of Technology, 1201 East California Blvd, Pasadena, California 91125, USA, and ^bDepartment of Chemistry, University of Washington, Seattle, Washington 98195, USA
Correspondence e-mail: rem@xray.caltech.edu

3.2.2. Least-squares plane based on uncorrelated, isotropic weights

| top | pdf |

This is surely the most common situation; it is not often that one will wish to take the trouble, or be presumptive enough, to assign anisotropic or correlated weights to the various atoms. And one will sometimes, perhaps even often, not be genuinely interested in the hypothesis that the atoms actually are rigorously coplanar; for instance, one might be interested in examining the best plane through such a patently non-planar molecule as cyclohexane. Moreover, the calculation is simple enough, given the availability of computers and programs, as to be a practical realization of the off-the-cuff treatment suggested in our opening paragraph. The problem of deriving the plane's coefficients is intrinsically nonlinear in the way first discussed by Schomaker et al. (1959; SWMB). Any formulation other than as an eigenvalue–eigenvector problem (SWMB), as far as we can tell, will sometimes go astray. As to the propagation of errors, numerous treatments have been given, but none that we have seen is altogether satisfactory.

We refer all vectors and matrices to Cartesian axes, because that is the most convenient in calculation. However, a more elegant formulation can be written in terms of general axes [e.g., as in Shmueli (1981)].

The notation is troublesome. Indices are needed for atom number and Cartesian direction, and the exponent 2 is needed as well, which is difficult if there are superscript indices. The best way seems to be to write all the indices as subscripts and distinguish among them by context – i, j, 1, 2, 3 for directions; k, l, p (and sometimes K, …) for atoms. In any case, atom first then direction if there are two subscripts; direction, if only one index for a vector component, but atom (in this section at least) if for a weight or a vector. And $[\sigma_{d_{1}}]$ , e.g., for the standard uncertainty of the distance of atom 1 from a plane. For simplicity in practice, we use Cartesian coordinates throughout.

The first task is to find the plane, which we write as $[0 = {\bf m}\cdot {\bf r} - d\equiv {\sf m}^{T}{\sf r} - d,]$ where r is here the vector from the origin to any point on the plane (but usually represents the measured position of an atom), m is a unit vector parallel to the normal from the origin to the plane, d is the length of the normal, and $[{\sf m}]$ and $[{\sf r}]$ are the column representations of m and r. The least-squares condition is to find the stationary values of $[S\equiv [w_{k}({\sf m}^{T}{\sf r}_{k} - d)^{2}]]$ subject to $[{\sf m}^{T}{\sf m} = 1]$ , with $[{\sf r}_{k}]$ , $[k = 1,\ldots, n]$ , the vector from the origin to atom k and with weights, $[w_{k}]$ , isotropic and without interatomic correlations for the n atoms of the plane. We also write S as $[S\equiv [w({\sf m}^{T}{\sf r} - d)^{2}]]$ , the subscript for atom number being implicit in the Gaussian summations $[([\ldots])]$ over all atoms, as it is also in the angle-bracket notation for the weighted average over all atoms, for example in $[\langle{\sf r} \rangle]$ – the weighted centroid of the groups of atoms – just below.

First solve for d, the origin-to-plane distance. $[\eqalign{ 0 &= - {1\over 2} {\partial S\over \partial d} = [w({\sf m}^{T}{\sf r} - d)] = 0,\cr d &= [w{\sf m}^{T}{\sf r}]/[w]\equiv {\sf m}^{T}\langle {\sf r} \rangle.}]$ Then $[\sspecialfonts\eqalign{ S&\equiv [w({\sf m}^{T}{\sf r} - d)^{2}] = [w\{{\sf m}^{T}({\sf r} - \langle {\sf r} \rangle)\}^{2}]\cr &\equiv [w({\sf m}^{T}{\bsf s})^{2}]\equiv {\sf m}^{T}[w\hbox{ss}^{T}]{\sf m}\equiv {\sf m}^{T}{\bsf A}{\sf m}.}]$ Here $[\sspecialfonts{\bsf s}_{k}\equiv {\sf r}_{k} - \langle {\sf r} \rangle]$ is the vector from the centroid to atom k. Then solve for m. This is the eigenvalue problem – to diagonalize $[\sspecialfonts{\bsf A}]$ (bear in mind that $[\sspecialfonts{\bsf A}_{ij}]$ is just $[[ws_{i}s_{j}]]$ ) by rotating the coordinate axes, i.e., to find the $[3 \times 3]$ arrays $[\sspecialfonts{\bsf M}]$ and $[\sspecialfonts{\bsf L}]$ , $[\sspecialfonts{\bsf L}]$ diagonal, to satisfy $[\sspecialfonts{\bsf M}^{T}{\bsf AM} = {\bsf L},\qquad {\bsf M}^{T}{\bsf M} = {\bsf I}.]$ $[\sspecialfonts{\bsf A}]$ and $[\sspecialfonts{\bsf M}]$ are symmetric; the columns $[\sspecialfonts\sf m]$ of $[\sspecialfonts{\bsf M}]$ are the direction cosines of, and the diagonal elements of $[\sspecialfonts{\bsf L}]$ are the sums of weighted squares of residuals from, the best, worst and intermediate planes, as discussed by SWMB.

3.2.2.1. Error propagation

| top | pdf |

Waser et al. (1973; WMC) carefully discussed how the random errors of measurement of the atom positions propagate into the derived quantities in the foregoing determination of a least-squares plane. This section presents an extension of their discussion. To begin, however, we first show how standard first-order perturbation theory conveniently describes the propagation of error into $[\sspecialfonts{\bsf M}]$ and $[\sspecialfonts{\bsf L}]$ when the positions $[{\sf r}_{k}]$ of the atoms are incremented by the amounts $[\delta{\sf r}_{k} \equiv \xi_{k}]$ and the corresponding quantities $[\sspecialfonts{\bsf s}_{k} \equiv {\sf r}_{k} - \langle {\sf r}\rangle]$ (the vectors from the centroid to the atoms) by the amounts $[\sspecialfonts\eta_{k}, ({\bsf s} \rightarrow {\bsf s} + \eta),]$ $[\eta_{k}\equiv\xi_{k} - \langle \xi \rangle]$ . (The need to account for the variation in position of the centroid, i.e. to distinguish between $[\eta_{k}]$ and $[\xi_{k}]$ , was overlooked by WMC.) The consequent increments in $[\sspecialfonts{\bsf A}, {\bsf M}]$ and $[\sspecialfonts{\bsf L}]$ are $[\sspecialfonts\eqalign{ \delta {\bsf A} &= [w \eta {\bsf s}^{T}] + [w {\bsf s} \eta^{T}] \equiv \alpha,\cr \delta {\bsf M} &= {\bsf M} \mu,\cr \delta {\bsf L} &\equiv \lambda.}]$ Here the columns of $[\mu]$ are expressed as linear combinations of the columns of $[\sspecialfonts{\bsf M}]$ . Note also that both perturbations, $[\mu]$ and $[\lambda]$ , which are the adjustments to the orientations and associated eigenvalues of the principal planes, will depend on the reduced coordinates $[\sspecialfonts{\bsf s}]$ and the perturbing influences $[\xi]$ by way of $[\alpha]$ , which in turn depends only on the reduced coordinates and the reduced shifts $[\eta_{k}]$ . In contrast, $[\delta {\sf d} = \delta ({\sf m}^{T} \langle {\sf r} \rangle) = (\delta {\sf m}^{T}) \langle {\sf r} \rangle + {\sf m}^{T} \langle \xi \rangle\hbox{;}]$ the change in the origin-to-plane distance for the plane defined by the column vectors m of $[\sspecialfonts{\bsf M}]$ , depends on the $[\langle {\sf r} \rangle]$ and $[\langle \xi \rangle]$ directly as well as on the $[\sspecialfonts{\bsf s}]$ and $[\eta]$ by way of the $[\delta {\bf m}.]$

The first-order results arising from the standard conditions, $[\sspecialfonts{\bsf M}^{T}{\bsf M} = {\bsf I},{\bsf L}]$ diagonal, and $[\sspecialfonts{\bsf M}^{T}{\bsf AM} = {\bsf L}]$ , are $[\mu^{T} + \mu = 0, \lambda \hbox{ diagonal},]$ and $[\sspecialfonts\mu^{T} {\bsf M}^{T} {\bsf AM} + {\bsf M}^{T} \alpha {\bsf M} + {\bsf M}^{T} {\bsf AM}\mu = \mu^{T} {\bsf L} + {\bsf L}\mu + {\bsf M}^{T} \alpha {\bsf M} = \lambda.]$ Stated in terms of the matrix components $[\lambda_{ij}]$ and $[\mu_{ij}]$ , the first condition is $[\mu_{ij} = - \mu_{ji}]$ , hence $[\mu_{ii} = 0,\ i,j = 1, 2, 3]$ , and the second is $[\lambda_{ij} = 0,\ i \neq j]$ . With these results the third condition then reads $[\sspecialfonts\let\normalbaselines\relax\openup3pt\matrix{ \lambda_{jj} = ({\bsf M}^{T} \alpha {\bsf M})_{jj},\hfill &j = 1, 2, 3\hfill\cr \mu_{ij} = ({\bsf M}^{T} \alpha {\bsf M})_{ij}/(L_{jj} - L_{ii}),\hfill & i \neq j,\quad i,j = 1, 2, 3.\hfill}]$ All this is analogous to the usual first-order perturbation theory, as, for example, in elementary quantum mechanics.

Now rotate to the coordinates defined by WMC, with axes parallel to the original eigenvectors $[\sspecialfonts[{\bsf M} = {\bsf I},\ A_{ij} = L_{ij}\delta_{ij},]$ $[\sspecialfonts({\bsf M}^{T} \alpha {\bsf M})_{ij}= \alpha_{ij}]]$ , restrict attention to the best plane $[(M_{13} \equiv m_{1} = 0, M_{23} \equiv m_{2} = 0, M_{33} \equiv m_{3} = 1)]$ , and define $[\varepsilon^{T}]$ as $[(\delta m_{1}, \delta m_{2}, \delta d_{c})]$ , keeping in mind $[\delta m_{3} = \mu_{33} = 0]$ ; $[d_{c}]$ itself, the original plane-to-centroid distance, of course vanishes. One then finds $[\let\normalbaselines\relax\openup3pt\matrix{ \delta m_{i} \equiv \varepsilon_{i} = \alpha_{i3}/(L_{33} - L_{ii})\hfill&\cr \phantom{\delta m_{i} \equiv \varepsilon_{i}} = [w(s_{i}\eta_{3} + s_{3} \eta_{i})]/([ws_{3}^{2}] - [ws_{i}^{2}]),\hfill & i = 1, 2,\hfill\cr \delta d_{c} \equiv \varepsilon_{3} = [w \xi_{3}]/[w] \equiv \langle \xi_{3} \rangle,\hfill &\xi_{k} \equiv \delta r_{k},\hfill}]$ and also $[\delta d = \varepsilon_{1} \langle r_{1} \rangle + \varepsilon_{2} \langle r_{2} \rangle + \varepsilon_{3}.]$ These results have simple interpretations. The changes in direction of the plane normal (the $[\delta m_{i}]$ ) are rotations, described by $[\varepsilon_{1}]$ and $[\varepsilon_{2}]$ , in response to changes in moments acting against effective torsion force constants. For $[\varepsilon_{2}]$ , for example, the contribution of atom k to the total relevant moment, about direction 1, is $[-w_{k}s_{k3}s_{k2}]$ ( $[w_{k}s_{k3}]$ the `force' and $[s_{k2}]$ the lever arm), and its nominally first-order change has two parts, $[-w_{k}s_{k2}\eta_{3}]$ from the change in force and $[-w_{k}s_{k3}\eta_{2}]$ from the change in lever arm; the resisting torsion constant is $[[ws_{2}^{2}] - [ws_{3}^{2}]]$ , which, reflection will show, is qualitatively reasonable if not quantitatively obvious. The perpendicular displacement of the plane from the original centroid $[\langle r \rangle]$ is $[\varepsilon_{3}]$ , but there are two further contributions to $[\delta d]$ , the change in distance from origin to plane along the plane normal, that arise from the two components of out-of-plane rotation of the plane about its centroid. Note that $[\varepsilon_{3}]$ is not given by $[[w\eta_{3}]/[w] = [w(\xi_{3} - \langle \xi_{3} \rangle)]/[w]]$ , which vanishes identically.

There is a further, somewhat delicate point: If the group of atoms is indeed essentially coplanar, the $[s_{k3}]$ are of the same order of magnitude as the $[\eta_{ki}]$ , unlike the $[s_{ki}]$ , $[i \neq 3]$ , which are in general about as big as the lateral extent of the group. It is then appropriate to drop all terms in $[\eta_{i}]$ or $[\xi_{i},\ i \neq 3]$ , and, in the denominators, the terms in $[s_{k3}^{2}]$ .

The covariances of the derived quantities (by covariances we mean here both variances and covariances) can now be written out rather compactly by extending the implicit designation of atom numbers to double sums, the first of each of two similar factors referring to the first atom index and the second to the second, e.g., $[{\textstyle\sum_{kl}} ww(s_{i}s_{j})\ldots \equiv {\textstyle\sum_{kl}} w_{k}w_{l}(s_{ki}s_{ij})\ldots]$ . Note that the various covariances, i.e. the averages over the presumed population of random errors of replicated measurements, are indicated by overlines, angle brackets having been pre-empted for averages over sets of atoms. $[\displaylines{\hbox{cov}(m_{i}, m_{j}) \equiv \overline{\varepsilon_{i} \varepsilon_{j}}\hfill \cr\quad= {{\textstyle\sum_{kl}} ww(s_{i}s_{j} \overline{\eta_{3} \eta_{3}} + s_{3}s_{3} \overline{\eta_{i} \eta_{j}} + s_{i}s_{3} \overline{\eta_{3} \eta_{j}} + s_{3}s_{j} \overline{\eta_{i} \eta_{3}})\over \{[w(s_{3}^{2} - s_{i}^{2})]\} \{[w(s_{3}^{2} - s_{j}^{2})]\}}, {i, j = 1, 2} \cr \hbox{cov} (m_{i}, d_{c}) \equiv \overline{\varepsilon_{i} \varepsilon_{3}} = {{\textstyle\sum_{kl}} ww(s_{ki} \overline{\eta_{3} \xi_{3}} + s_{k3} \overline{\eta_{i} \xi_{3}})\over \{[w(s_{3}^{2} - s_{i}^{2})]\} [w]},\quad i,j = 1, 2\cr \sigma^{2} (d_{c}) \equiv \overline{\varepsilon_{3}^{2}} = {{\textstyle\sum_{kl}} ww \overline{\xi_{3} \xi_{3}}\over [w]^{2}}\cr \eqalign{\sigma^{2} (d) \equiv \langle (\delta d)^{2}\rangle &= \langle r_{1}\rangle^{2} \overline{\varepsilon_{1}^{2}} + \langle r_{2}\rangle^{2} \overline{\varepsilon_{2}^{2}} + \overline{\varepsilon_{3}^{2}} + 2\langle r_{1} \rangle \langle r_{2} \rangle \overline{\varepsilon_{1} \varepsilon_{2}} \cr &\quad+ 2 \langle r_{1} \rangle \overline{\varepsilon_{1} \varepsilon_{3}} + 2 \langle r_{2} \rangle \overline{\varepsilon_{2} \varepsilon_{3}}.}}]$

Interatomic covariance (e.g., $[\overline{\eta_{k3} \eta_{l3}},\ k \neq l]$ ) thus presents no formal difficulty, although actual computation may be tedious. Nonzero covariance for the $[\eta]$ 's may arise explicitly from interatomic covariance (e.g., $[\overline{\xi_{ki} \xi_{lj}},\ k \neq l]$ ) of the errors in the atomic positions $[{\sf r}_{k}]$ , and it will always arise implicitly because $[\langle\xi\rangle]$ in $[\eta_{k} = \xi_{k} - \langle\xi\rangle]$ includes all the $[\xi_{k}]$ and therefore has nonzero covariance with all of them and with itself, even if there is no interatomic covariance among the $[\xi_{i}]$ 's.

If both types of interatomic covariance (explicit and implicit) are negligible, the $[\varepsilon]$ covariances simplify a great deal, the double summations reducing to single summations. [The formal expression for $[\sigma^{2} (d)]$ does not change, so it will not be repeated.] $[\eqalignno{\hbox{cov} (m_{i}, m_{j}) &\equiv \overline{\varepsilon_{i} \varepsilon_{j}}\cr&= {[w^{2} (s_{i}s_{j} \overline{\eta_{3}^{2}} + s_{3}^{2} \overline{\eta_{i} \eta_{j}} + s_{i}s_{3} \overline{\eta_{3} \eta_{j}} + s_{3} s_{j} \overline{\eta_{i} \eta_{3}}]\over \{[w (s_{3}^{2} - s_{i}^{2})]\} \{[w (s_{3}^{2} - s_{j}^{2})]\}}, i, j = 1, 2\cr \hbox{cov} (m_{i}, d_{c}) & \equiv \overline{\varepsilon_{i} \varepsilon_{3}} = {[w^{2} (s_{i} \overline{\eta_{3} \xi_{3}} + s_{3} \overline{\eta_{i} \xi_{3}})]\over \{[w (s_{3}^{2} - s_{i}^{2})]\} [w]},\quad i, j = 1, 2\cr \sigma^{2} (d_{c}) &\equiv \overline{\varepsilon_{3}^{2}} = {[w^{2} \overline{\xi_{3}^{2}}]\over [w]^{2}}.}]$

When the variances are the same for $[\xi]$ as for $[\eta]$ (i.e. $[\overline{\xi_{i} \xi_{j}} = \overline{\eta_{i} \eta_{j}}]$ , all i, j) and the covariances all vanish $[(\overline{\xi_{i} \xi_{j}} = 0,\ i \neq j)]$ , the $[\overline{\varepsilon_{i} \varepsilon_{j}}]$ simplify further. If the variances are also isotropic $[(\overline{\xi_{i}^{2}} = \overline{\xi_{j}^{2}} = \sigma^{2}]$ , all i, j), there is still further simplification to $[\eqalign{ \sigma^{2} (m_{i}) &\equiv \overline{\varepsilon_{i}^{2}} = {[w^{2} \sigma^{2} (s_{i}^{2} + s_{3}^{2})]\over \{[w (s_{3}^{2} - s_{i}^{2})]\}^{2}},\quad i = 1, 2\cr \sigma^{2} (d_{c}) &\equiv \overline{\varepsilon_{3}^{2}} = [w^{2} \sigma^{2}]/[w]^{2}\cr \hbox{cov} (m_{1}, m_{2}) &\equiv \overline{\varepsilon_{1} \varepsilon_{2}} = {[w^{2} \sigma^{2} s_{1} s_{2}]\over \{[w (s_{3}^{2} - s_{1}^{2})]\}\{[w (s_{3}^{2} - s_{2}^{2})]\}}\cr \hbox{cov} (m_{i}, d_{c}) &\equiv \overline{\varepsilon_{i} \varepsilon_{3}} = {[w^{2} \sigma^{2} s_{i}]\over \{[w (s_{3}^{2} - s_{i}^{2})]\}[w]},\quad i = 1, 2.}]$ If allowance is made for the difference in definition between $[\varepsilon_{3}]$ and $[\delta d]$ , these expressions are equivalent to the ones (equations 7–9) given by WMC, who, however, do not appear to have been aware of the distinction between $[\eta]$ and $[\xi]$ and the possible consequences thereof.

If, finally, $[w^{-1}]$ for each atom is taken equal to its $[\overline{\eta_{j}^{2}} = \sigma^{2}]$ , all j, there is still further simplification. $[\eqalign{ \sigma^{2} (m_{i}) &\equiv \overline{\varepsilon_{i}^{2}} = {[w (s_{3}^{2} + s_{i}^{2})]\over \{[w (s_{3}^{2} - s_{i}^{2})]\}^{2}},\quad i = 1, 2\cr \sigma^{2} (d_{c}) &\equiv \overline{\varepsilon_{3}^{2}} = [w]/[w]^{2} = 1/[w]\cr \hbox{cov} (m_{1}, m_{2}) &\equiv \overline{\varepsilon_{1} \varepsilon_{2}} = {[ws_{1} s_{2}]\over \{[w (s_{3}^{2} - s_{1}^{2})]\} \{[w (s_{3}^{2} - s_{2}^{2})]\}}\cr \hbox{cov} (m_{i}, d_{c}) &\equiv \overline{\varepsilon_{i} \varepsilon_{3}} = {[w s_{i}]\over \{[w (s_{3}^{2} - s_{i}^{2})]\}[w]},\quad i = 1, 2.}]$

For the earlier, more general expressions for the components of $[\overline{\varepsilon \varepsilon^{T}}]$ it is still necessary to find $[\overline{\eta_{ki} \eta_{lj}}]$ and $[\overline{\eta_{ki} \xi_{l3}}]$ in terms of $[\overline{\xi_{ki} \xi_{lj}}]$ , with $[\eta_{ki} \equiv \delta s_{ki} = \delta (r_{ki} - \langle r_{ki}\rangle ) = \xi_{ki} - \langle\xi_{i}\rangle = {\textstyle\sum_{l}} w_{l} (\xi_{ki} - \xi_{li})/[w]]$ . $[\eqalign{ \overline{\eta_{ki} \eta_{pj}} &= {\textstyle\sum\limits_{l, q}} w_{l} w_{q} \overline{(\xi_{ki} - \xi_{li}) (\xi_{pj} - \xi_{qj})}/[w]^{2}\cr &= \overline{(\xi_{ki} - \langle\xi_{i}\rangle) (\xi_{pj} - \langle\xi_{j}\rangle)}\cr \overline{\eta_{ki} \xi_{p3}} &= {\textstyle\sum\limits_{l}} w_{l} \overline{(\xi_{ki} - \xi_{li}) \xi_{p3}}/[w] = \overline{\xi_{ki} \xi_{p3}} - \overline{\langle\xi_{i}\rangle \xi_{p3}}.}]$

In the isotropic, `no-correlation' case, for example, these reduce to $[\eqalign{ \overline{\eta_{ki} \eta_{pi}} &= - w_{k} \overline{\xi_{ki}^{2}}/[w] - w_{p} \overline{\xi_{pi}^{2}}/[w] \cr &\quad + [w^{2} \overline{\xi_{i}^{2}}]/[w]^{2},\quad k \neq p; i = 1, 2\cr \overline{\eta_{ki}^{2}} &= (1 - 2w_{k}/[w]) \overline{\xi_{ki}^{2}} + [w^{2} \overline{\xi_{i}^{2}}]/[w]^{2},\quad i = 1, 2\cr \overline{\eta_{k3} \eta_{p3}} &= - w_{p} \overline{\xi_{p3}^{2}}/[w],}]$ and $[\overline{\eta_{k3}^{2}} = \overline{\xi_{k3}^{2}} - w_{k} \overline{\xi_{k3}^{2}}/[w] = \overline{\xi_{k3}^{2}} (1 - w_{k}/[w]).]$ Here the difference between the correct covariance values and the values obtained on ignoring the variation in $[\langle r\rangle]$ may be important if the number of defining atoms is small, say, 5 or 4 or, in the extreme, 3.

3.2.2.2. The standard uncertainty of the distance from an atom to the plane

| top | pdf |

There are two cases, as has been pointed out, e.g., by Ito (1982).

(1) The atom (atom K) was not included in the specification of the plane. $[\eqalign{ d_{K} &= {\sf m}^{T} ({\sf r}_{K} - \langle {\sf r}\rangle) = r_{k3} - \langle r_{3}\rangle\cr \delta d_{K} &= \xi_{K3} + s_{K1} \varepsilon_{1} + s_{K2} \varepsilon_{2} - \varepsilon_{3}\cr \sigma^{2}_{d_{K}} &= \overline{\xi_{K3}^{2}} + s_{K1}^{2} \overline{\varepsilon_{1}^{2}} + s_{K2}^{2} \overline{\varepsilon_{2}^{2}} + \overline{\varepsilon_{3}^{2}}\cr &\quad + 2 s_{K1} s_{K2} \overline{\varepsilon_{1} \varepsilon_{2}} - 2 s_{K1} \overline{\varepsilon_{1} \varepsilon_{3}} - 2 s_{K2} \overline{\varepsilon_{2} \varepsilon_{3}}\cr &\quad + 2 s_{K1} \overline{\xi_{K3} \varepsilon_{1}} + 2 s_{K2} \overline{\xi_{K3} \varepsilon_{2}} - 2 \overline{\xi_{K3} \varepsilon_{3}}.}]$ In the isotropic, `no-correlation' case the last three terms, i.e. the terms in $[\overline{\varepsilon_{i} \xi_{K3}}]$ , are all negligible or zero.

In either case the value for $[\overline{\xi_{K3}^{2}}]$ and the appropriate $[\overline{\varepsilon_{i} \varepsilon_{j}}]$ values from the least-squares-plane calculation need to be inserted.
(2) Atom K was included in the specification of the plane. The expression for $[\sigma_{d_{K}}^{2}]$ remains the same, but the averages in it may be importantly different.

For example, consider a plane defined by only three atoms, one of overwhelmingly great w at (0, 0, 0), one at (1, 0, 0) and one at (0, 1, 0). The centroid is at (0, 0, 0) and we take [K = 2] , i.e. $[\sigma_{d_{2}}]$ is the item of interest. Of course, it is obvious without calculation that the standard uncertainties vanish for the distances of the three atoms from the plane they alone define; the purpose here is only to show, in one case for one of the atoms, that the calculation gives the same result, partly, it will be seen, because the change in orientation of the plane is taken into account. If the only variation in the atom positions is described by $[\overline{\xi_{23}^{2}} = \sigma^{2}]$ , one has $[{s}_{21} = 1, \varepsilon_{3} = \varepsilon_{2} = 0,]$ $[\varepsilon_{1} = - \xi_{23}]$ , and $[\overline{\xi_{K3} \varepsilon_{1}} = \sigma^{2}]$ . The non-vanishing terms in the desired variance are then $[\eqalign{ \sigma_{d_{2}}^{2} &= \overline{\xi_{23}^{2}} + s_{21}^{2} \overline{\varepsilon_{1}^{2}} + 2 s_{21} \overline{\xi_{23} \varepsilon_{1}}\cr &= (1 + 1 - 2) \sigma^{2} = 0.}]$ If, however, the problem concerns the same plane and a fourth atom at position $[(1, 0, r_{43})]$ , not included in the specification of the plane and uncertain only in respect to $[r_{43}]$ (which is arbitrary) with $[\overline{\xi_{43}^{2}} = \sigma^{2}]$ (the same mean-square variation in direction 3 as for atom 2) and $[\overline{\xi_{43} \xi_{23}} = 0]$ , the calculation for $[\sigma_{d_{4}}^{2}]$ runs the same as before, except for the third term: $[\sigma_{d_{4}}^{2} = (1 + 1 - 0) \sigma^{2} = 2\sigma^{2}.]$

Extreme examples of this kind show clearly enough that variation in the direction of the plane normal or in the normal component of the centroid position will sometimes be important, the remarks to the contrary by Shmueli (1981) and, for the centroid, the omission by WMC notwithstanding. If only a few atoms are used to define the plane (e.g., three or, as is often the case, a very few more), both the covariance with the centroid position and uncertainty in the direction of the normal are likely to be important. The uncertainty in the normal may still be important, even if a goodly number of atoms are used to define the plane, whenever the test atom lies near or beyond the edge of the lateral domain defined by the other atoms.

References

Ito, T. (1982). On the estimated standard deviation of the atom-to-plane distance. Acta Cryst. A38, 869–870.Google Scholar

Schomaker, V., Waser, J., Marsh, R. E. & Bergman, G. (1959). To fit a plane or a line to a set of points by least squares. Acta Cryst. 12, 600–604.Google Scholar

Shmueli, U. (1981). On the statistics of atomic deviations from the `best' molecular plane. Acta Cryst. A37, 249–251.Google Scholar

Waser, J., Marsh, R. E. & Cordes, A. W. (1973). Variances and covariances for best-plane parameters including dihedral angles. Acta Cryst. B29, 2703–2708.Google Scholar

International Tables for Crystallography (2006). Vol. B. ch. 3.2, pp. 353-355