Tables for
Volume F
Crystallography of biological macromolecules
Edited by M. G. Rossmann and E. Arnold

International Tables for Crystallography (2006). Vol. F, ch. 19.5, pp. 446-447   | 1 | 2 |

Section 19.5.6. Data processing

R. Chandrasekarana* and G. Stubbsb

aWhistler Center for Carbohydrate Research, Purdue University, West Lafayette, IN 47907, USA, and  bDepartment of Molecular Biology, Vanderbilt University, Nashville, TN 37235, USA
Correspondence e-mail:

19.5.6. Data processing

| top | pdf | Coordinate transformation

| top | pdf |

Data must be transformed from detector space into reciprocal space (Fraser et al., 1976[link]). Transformation of coordinates requires determination of the origin of the diffraction pattern in detector space, the fibre tilt angle β, the twist angle (often called in-plane tilt, the inclination of the projection of Z along the beam to the detector coordinate system) and the specimen-to-detector distance. It may also require determination of the detector mis-setting angles (the deviation of the normal to the detector plane from the beam). All of these parameters can be determined by comparing equivalent reflections in the diffraction pattern.

Most data-processing programs determine the transformation parameters by some form of minimization of the deviation from equivalence in the positions of well resolved equivalent reflections. The tilt was traditionally determined by comparing the apparent Z values of equivalent reflections, but the apparent value of R for near-meridional reflections is much more sensitive to tilt. The minimization set should therefore include some near-meridional reflections if the tilt value is to be determined accurately. The helical repeat distance and, for polycrystalline fibres, the unit-cell parameters must also be determined at this time, but helical repeat distance and specimen-to-detector distance are so highly correlated that it is not often practical to refine both. Intensity correction

| top | pdf |

Data intensities must be corrected for geometric and polarization effects (Fraser et al., 1976[link]; Millane & Arnott, 1986[link]). The geometric correction has two components: a factor due to the geometry of the intersection between the diffraction pattern in reciprocal space and the sphere of reflection, and a factor due to the angle of incidence of the diffracted beam on the detector. The first factor is analogous to the Lorentz factor in crystallography, which arises because of the time taken for a reflection from a moving sample to pass through the Ewald sphere. The geometric correction can be applied to each data point as a single correction (Fraser et al., 1976[link]); this is the simpler procedure for diffraction from noncrystalline fibres. For crystalline fibres, it is often convenient to apply Lorentz and polarization corrections to each data point, to integrate the intensities within each reflection, and then to apply the remaining geometric corrections (Millane & Arnott, 1986[link]). The Lorentz correction is [{1/L = 2 \pi \sin \theta [\cos^{2}\theta \cos^{2}\beta - (\cos \sigma - \sin \theta \sin \beta)^{2}]^{1/2},}\hfill\!\!\!\! \eqno(] where θ is the Bragg angle and [\tan \sigma = R/Z] (Millane & Arnott, 1986[link]). The polarization correction is [p = (1 + \cos^{2} 2\theta)/2. \eqno(] Intensities should be divided by Lp. Intensities may also be corrected for nonlinearity of detector response and for absorption by the specimen and by detector components. Background subtraction

| top | pdf |

The background can be very high in fibre-diffraction data because of long exposure times and scattering from amorphous material. Because of specimen disorientation, fibre-diffraction data often contain large regions where there is no space between layer lines, so local-background-fitting methods are rarely useful. The background may be determined by fitting an analytical function to intensities at points between reflections (Millane & Arnott, 1985[link]; Lorenz & Holmes, 1993[link]), or by fitting a function that includes both signal and background components to the reflection data. This type of profile fitting has been described for individual reflections (Fraser et al., 1976[link]), for data in concentric rings about the centre of the diffraction pattern (Makowski, 1978[link]) and for entire data sets (Yamashita et al., 1995[link]; Ivanova & Makowski, 1998[link]). Integration of crystalline fibre data

| top | pdf |

The variation of reflection shape in detector space can be determined using a few sharp reflections and taking into account parameters related to crystallite size and disorientation in the specimen (Millane & Arnott, 1986[link]). This allows the integration boundary of a reflection to be determined. Sometimes, the boundary encompasses two or more reflections too close to separate; such reflections are considered to constitute a composite reflection. Integration of continuous data

| top | pdf |

In diffraction from noncrystalline fibres, intensity is a function of R on each layer line. Angular deconvolution (Makowski, 1978[link]; Namba & Stubbs, 1985[link]; Yamashita et al., 1995[link]) or profile fitting (Millane & Arnott, 1986[link]) corrects for disorientation and overlap between adjacent layer lines and may also incorporate background subtraction. The intensity determined in this way should be corrected for geometric and other effects if this has not been done previously (Section[link]; Namba & Stubbs, 1985[link]; Millane & Arnott, 1986[link]).


Fraser, R. D. B., MacRae, T. P., Miller, A. & Rowlands, R. J. (1976). Digital processing of fibre diffraction patterns. J. Appl. Cryst. 9, 81–94.Google Scholar
Ivanova, M. I. & Makowski, L. (1998). Iterative low-pass filtering for estimation of the background in fiber diffraction patterns. Acta Cryst. A54, 626–631.Google Scholar
Lorenz, M. & Holmes, K. C. (1993). Computer processing and analysis of X-ray fiber diffraction data. J. Appl. Cryst. 26, 82–91.Google Scholar
Makowski, L. (1978). Processing of X-ray diffraction data from partially oriented specimens. J. Appl. Cryst. 11, 273–283.Google Scholar
Millane, R. P. & Arnott, S. (1985). Background removal in X-ray fiber diffraction patterns. J. Appl. Cryst. 18, 419–423.Google Scholar
Millane, R. P. & Arnott, S. (1986). Digital processing of X-ray diffraction patterns from oriented fibers. J. Macromol. Sci. Phys. B24, 193–227.Google Scholar
Namba, K. & Stubbs, G. (1985). Solving the phase problem in fiber diffraction. Application to tobacco mosaic virus at 3.6 Å resolution. Acta Cryst. A41, 252–262.Google Scholar
Yamashita, I., Vonderviszt, F., Mimori, Y., Suzuki, H., Oosawa, K. & Namba, K. (1995). Radial mass analysis of the flagellar filament of Salmonella: implications for the subunit folding. J. Mol. Biol. 253, 547–558.Google Scholar

to end of page
to top of page