Section 2.5.5. Image reconstruction ⁵

In many fields of physical measurements, instrumental and informative techniques, including electron microscopy and computational or analogue methods for processing and transforming signals from objects investigated, find a wide application in obtaining the most accurate structural data. The signal may be radiation from an object, or radiation transmitted through the object, or reflected by it, which is transformed and recorded by a detector.

The image is the two-dimensional signal on the observation plane recorded from the whole three-dimensional volume of the object, or from its surface, which provides information on its structure. In an object this information may change owing to transformation of the scattered wave inside an instrument. The real image is composed of and noise from signal disturbances:

Image-reconstruction methods are aimed at obtaining the most accurate information on the structure of the object; they are subdivided into two types (Picture Processing and Digital Filtering, 1975; Rozenfeld, 1969):

These two methods may be used separately or in combination.

The image should be represented in the form convenient for perception and analysis, e.g. in digital form, in lines of equal density, in points of different density, in half-tones or colour form and using, if necessary, a change or reversal of contrast.

Reconstructed images may be used for the three-dimensional reconstruction of the spatial structure of an object, e.g. of the density distribution in it (see Section 2.5.6).

This section is connected with an application of the methods of image processing in transmission electron microscopy (TEM). In TEM (see Section 2.5.2), the source-emitted electrons are transmitted through an object and, with the aid of a system of lenses, form a two-dimensional image subject to processing.

Another possibility for obtaining information on the structure of an object is structural analysis with the aid of electron diffraction – EDSA. This method makes use of information in reciprocal space – observation and measurement of electron-diffraction patterns and calculation from them of a two-dimensional projection or three-dimensional structure of an object using the Fourier synthesis. To do this, one has to find the relative phases of the scattered beams.

The wavefunction of an electron-microscopic image is written as Here is the incident plane wave. When the wave is transmitted through an object, it interacts with the electrostatic potential [ is the three-dimensional vector in the space of the object]; this process is described by the Schrödinger equation (Section 2.5.2.1). As a result, on the exit surface of an object the wave takes the form where q is the transmission function and x is the two-dimensional vector . The diffraction of the wave is described by the two-dimensional Fourier operator:

Here, we assume the initial wave amplitude to be equal to unity and the initial phase to be zero, so that , which defines, in this case, the wavefunction in the back focal plane of an objective lens with the reciprocal-space coordinates . The function Q is modified in reciprocal space by the lens transfer function . The scattered wave transformation into an image is described by the inverse Fourier operator .

The process of the diffraction , as seen from (2.5.5.1), is the same in both TEM and EDSA. Thus, in TEM under the lens actions the image formation from a diffraction pattern takes place with an account of the phases, but these phases are modified by the objective-lens transfer function. In EDSA, on the other hand, there is no distorting action of the transfer function and the `image' is obtained by computing the operation .

The computation of projections, images and Fourier transformation is made by discretization of two-dimensional functions on a two-dimensional network of points – pixels in real space and in reciprocal space .

The intensity distribution of an electron wave in the image plane depends not only on the coherent and inelastic scattering, but also on the instrumental functions. The electron wave transmitted through an object interacts with the electrostatic potential which is produced by the nuclei charges and the electronic shells of the atoms. The scattering and absorption of electrons depend on the structure and thickness of a specimen, and the atomic numbers of the atoms of which it is composed. If an object with the three-dimensional distribution of potential is sufficiently thin, then the interaction of a plane electron wave with it can be described as the interaction with a two-dimensional distribution of potential projection , where b is the specimen thickness. It should be noted that, unlike the three-dimensional function of potential with dimension , the two-dimensional function of potential projection has the potential-length dimension which, formally, coincides with the charge dimension. The transmission function, in the general case, has the form (2.5.2.42), and for weak phase objects the approximation is valid.

In the back focal plane of the objective lens the wave has the form where is the Scherzer phase function (Scherzer, 1949) of an objective lens (Fig. 2.5.5.1), is the aperture function, the spherical aberration coefficient, and Δf the defocus value [(2.5.2.32) –(2.5.2.35)].

Figure 2.5.5.1| top | pdf | The χ function and two components of the Scherzer phase function sin and cos .

The bright-field image intensity (in object coordinates) is where . The phase function (2.5.5.7) depends on defocus, and for a weak phase object (Cowley, 1981) where , which includes only an imaginary part of function (2.5.5.6). While selecting defocus in such a way that under the Scherzer defocus conditions [(2.5.2.44), (2.5.2.45)] , one could obtain In this very simple case the image reflects directly the structure of the object – the two-dimensional distribution of the projection of the potential convoluted with the spread function . In this case, no image restoration is necessary. Contrast reversal may be achieved by a change of defocus.

At high resolution, this method enables one to obtain an image of projections of the atomic structure of crystals and defects in the atomic arrangement – vacancies, replacements by foreign atoms, amorphous structures and so on; at resolution worse than atomic one obtains images of dislocations as continuous lines, inserted phases, inclusions etc. (Cowley, 1981). It is also possible to obtain images of thin biological crystals, individual molecules, biological macromolecules and their associations.

Image restoration. In the case just considered (2.5.5.10), the projection of potential , convoluted with the spread function, can be directly observed. In the general case (2.5.5.9), when the aperture becomes larger, the contribution to image formation is made by large values of spatial frequencies U, in which the function sin χ oscillates, changing its sign. Naturally, this distorts the image just in the region of appropriate high resolution. However, if one knows the form of the function sin χ (2.5.5.7), the true function can be restored.

This could be carried out experimentally if one were to place in the back focal plane of an objective lens a zone plate transmitting only one-sign regions of sin χ (Hoppe, 1971). In this case, the information on is partly lost, but not distorted. To perform such a filtration in an electron microscope is a rather complicated task.

Another method is used (Erickson & Klug, 1971). It consists of a Fourier transformation of the measured intensity distribution TQ (2.5.5.6) and division of this transform, according to (2.5.5.7a,b), by the phase function sin χ. This gives Then, the new Fourier transformation yields (in the weak-phase-object approximation) the true distribution The function sin χ depending on defocus Δf should be known to perform this procedure. The transfer function can also be found from an electron micrograph (Thon, 1966). It manifests itself in a circular image intensity modulation of an amorphous substrate or, if the specimen is crystalline, in the `noise' component of the image. The analogue method (optical Fourier transformation for obtaining the image ) can be used (optical diffraction, see below); digitization and Fourier transformation can also be applied (Hoppe et al., 1973).

The thin crystalline specimen implies that in the back focal objective lens plane the discrete kinematic amplitudes are arranged and, by the above method, they are corrected and released from phase distortions introduced by the function sin χ (see below) (Unwin & Henderson, 1975).

For the three-dimensional reconstruction (see Section 2.5.6) it is necessary to have the projections of potential of the specimen tilted at different angles α to the beam direction (normal beam incidence corresponds to ). In this case, the defocus Δf changes linearly with increase of the distance l of specimen points from the rotation axis . Following the above procedure for passing on to reciprocal space and correction of sin χ, one can find (Henderson & Unwin, 1975).

Elastic interaction of an incident wave with a weak phase object is defined on its exit surface by the distribution of potential projection ; however, in the general case, the electron scattering amplitude is a complex one (Glauber & Schomaker, 1953). In such a way, the image itself has the phase and amplitude contrast. This may be taken into account if one considers not only the potential projection , but also the `imaginary potential' which describes phenomenologically the absorption in thin specimens. Then, instead of (2.5.5.5), the wave on the exit surface of a specimen can be written as and in the back focal plane if and Usually, μ is small, but it can, nevertheless, make a certain contribution to an image. In a sufficiently good linear approximation, it may be assumed that the real part cos χ of the phase function (2.5.5.7a) affects , while , as we know, is under the action of the imaginary part sin χ.

Thus, instead of (2.5.5.6), one can write and as the result, instead of (2.5.5.10),

The functions and can be separated by object imaging using the through-focus series method. In this case, using the Fourier transformation, one passes from the intensity distribution (2.5.5.15) in real space to reciprocal space. Now, at two different defocus values and [(2.5.5.6), (2.5.5.7a,b)] the values and can be found from the two linear equations (2.5.5.14). Using the inverse Fourier transformation, one can pass on again to real space which gives and (Schiske, 1968). In practice, it is possible to use several through-focus series and to solve a set of equations by the least-squares method.

Another method for processing takes into account the simultaneous presence of noise and transfer function zeros (Kirkland et al., 1980). In this method the space frequencies corresponding to small values of the transfer function modulus are suppressed, while the regions where such a modulus is large are found to be reinforced.

When the specimen thickness exceeds a certain critical value (50–100 Å), the kinematic approximation does not hold true and the scattering is dynamic. This means that on the exit surface of a specimen the wave is not defined as yet by the projection of potential (2.5.5.3), but one has to take into account the interaction of the incident wave and of all the secondary waves arising in the whole volume of a specimen.

The dynamic scattering calculation can be made by various methods. One is the multi-slice (or phase-grating) method based on a recurrent application of formulae (2.5.5.3) for n thin layers thick, and successive construction of the transmission functions (2.5.5.4), phase functions , and propagation function (Cowley & Moodie, 1957).

Another method – the scattering matrix method – is based on the solution of equations of the dynamic theory (Chapter 5.2 ). The emerging wave on the exit surface of a crystal is then found to diffract and experience the transfer function action [(2.5.5.6), (2.5.5.7a,b)].

The dynamic scattering in crystals may be interpreted using Bloch waves: It turns out that only a few (bound and valence Bloch waves) have strong excitation amplitudes. Depending on the thickness of a crystal, only one of these waves or their linear combinations (Kambe, 1982) emerges on the exit surface. An electron-microscopic image can be interpreted, at certain thicknesses, as an image of one of these waves [with a correction for the transfer function action (2.5.5.6), (2.5.5.7a,b)]; in this case, the identical images repeat with increasing thickness, while, at a certain thickness, the contrast reversal can be observed. Only the first Bloch wave which arises at small thickness, and also repeats with increasing thickness, corresponds to the projection of potential , i.e. the atom projection distribution in a thin crystal layer.

An image of other Bloch waves is defined by the function , but their maxima or minima do not coincide, in the general case, with the atomic positions and cannot be interpreted as the projection of potential. It is difficult to reconstruct from these images, especially when the crystal is not ideal and contains imperfections. In these cases one resorts to computer modelling of images at different thicknesses and defocus values, and to comparison with an experimentally observed pattern.

The imaging can be performed directly in an electron microscope not by a photo plate, but using fast-response detectors with digitized intensity output on line. The computer contains the necessary algorithms for Fourier transformation, image calculation, transfer function computing, averaging, and correction for the observed and calculated data. This makes possible the interpretation of the pattern observed directly in experiment (Herrmann et al., 1980).

The real electron-microscope image is subdivided into two components: The main of these, , is a two-dimensional image of the `ideal' object obtained in an electron microscope with instrumental functions inherent to it. However, in the process of object imaging and transfer of this information to the detector there are various sources of noise. In an electron microscope, these arise owing to emission-current and accelerating-voltage fluctuations, lens-supplying current (temporal fluctuations), or mechanical instabilities in a device, specimen or detector (spatial shifts). The two-dimensional detector (e.g. a photographic plate) has structural inhomogeneities affecting a response to the signal. In addition, the specimen is also unstable; during preparation or imaging it may change owing to chemical or some other transformations in its structure, thermal effects and so on. Biological specimens scatter electrons very weakly and their natural state is moist, while in the electron-microscope column they are under vacuum conditions. The methods of staining (negative or positive), e.g. of introducing into specimens substances containing heavy atoms, as well as the freeze-etching method, somewhat distort the structure of a specimen. Another source of structure perturbation is radiation damage, which can be eliminated at small radiation doses or by using the cryogenic technique. The structure of stained specimens is affected by stain graininess. We assume that all the deviations of a specimen image from the `ideal' image are included in the noise term . The substrate may also be inhomogeneous. All kinds of perturbations cannot be separated and they appear on an electron microscope image as the full noise content .

The image enhancement involves maximum noise suppression and hence the most accurate separation of a useful signal from the real image (2.5.5.1). At the signal/noise ratio such a separation appears to be rather complicated. But in some cases the real image reflects the structure sufficiently well, e.g. during the atomic structure imaging of some crystals . In other cases, especially of biological specimen imaging, the noise N distorts substantially the image, . Here one should use the methods of enhancement. This problem is usually solved by the methods of statistical processing of sets of images . If one assumes that the informative signal is always the same, then the noise error may be reduced.

The image enhancement methods are subdivided into two classes:

These methods can be used separately or in combination. The enhancement can be applied to both the original and the restored images; there are also methods of simultaneous restoration and enhancement.

The image can be enhanced by analogue (mainly optical and photographic) methods or by computational methods for processing digitized functions in real and reciprocal space.

The cases where the image has translational symmetry, rotational symmetry, and where the image is asymmetric will be considered.

Periodic images. An image of the crystal structure with atomic or molecular resolution may be brought to self-alignment by a shift by a and b periods in a structure projection. This can be performed photographically by printing the shifted image on the same photographic paper or, vice versa, by shifting the paper (McLachlan, 1958).

The Fourier filtration method for a periodic image with noise N is based on the fact that in Fourier space the components and are separated. Let us carry out the Fourier transformation of the periodic signal with the periods a, b and noise N: The left part of (2.5.5.18) represents the Fourier coefficients distributed discretely with periods and in the plane . This is the two-dimensional reciprocal lattice. The right-hand side of (2.5.5.18) is the Fourier transform distributed continuously in the plane. Thus these parts are separated. Let us `cut out' from distribution (2.5.5.18) only values using the `window' function . The window should match each of the real peaks which, owing to the finite dimensions of the initial periodic image, are not points, as this is written in an idealized form in (2.5.5.18) with the aid of δ functions. In reality, the `windows' may be squares of about , in size, or a circle. Performing the Fourier transformation of product (2.5.5.18) without , and set of windows , we obtain: the periodic component without the background, . The zero coefficient in (2.5.5.19) should be decreased, since it is due, in part, to the noise. When the window w is sufficiently small, in (2.5.5.19) represents the periodic distribution (average over all the unit cells of the projection) included in (2.5.5.18). Nevertheless, some error from noise in an image does exist, since with we also introduced into the inverse Fourier transformation the background transform values which are within the `windows'.

This approach is realized by an analogue method [optical diffraction and filtering of electron micrographs in a laser beam (Klug & Berger, 1964)] and can also be carried out by computing.

As an example, Fig. 2.5.5.2(b) shows an electron micrograph of the periodic structure of a two-dimensional protein crystal, while Fig. 2.5.5.2(c) represents optical diffraction from this layer. In order to dissect the aperiodic component in a diffraction plane, according to the scheme in Fig. 2.5.5.2(a), one places a mask with windows covering reciprocal-lattice points. After such a filtration, only the component makes a contribution during the image formation by means of a lens, while the component diffracted by the background is delayed. As a result, an optical pattern of the periodic structure is obtained (Fig. 2.5.5.2d).

Figure 2.5.5.2| top | pdf | (a) Diagram of an optical diffractometer. D is the object (an electron micrograph), is the diffraction plane and a mask that transmits only , is the plane of the (filtered) image; (b) an electron micrograph of a crystalline layer of the protein phosphorylase b; (c) its optical diffraction pattern (the circles correspond to the windows in the mask that transmits only the diffracted beams from the periodic component of the image); (d) the filtered image. Parts (b)–(d) are based on the article by Kiselev et al. (1971).

Optical diffractometry also assists in determining the parameters of a two-dimensional lattice and its symmetry.

Using the same method, one can separate the superimposed images of two-dimensional structures with different periodicity and in different orientation, the images of the `near' and `far' sides of tubular periodic structures with monomolecular walls (Klug & DeRosier, 1966; Kiselev et al., 1971), and so on.

Computer filtering involves measuring the image optical density , digitization, and Fourier transformation (Crowther & Amos, 1971). The sampling distance usually corresponds to one-third of the image resolution. When periodic weak phase objects are investigated, the transformation (2.5.5.18) yields the Fourier coefficients. If necessary, we can immediately make corrections in them using the microscope transfer function according to (2.5.5.6), (2.5.5.7a,b) and (2.5.5.11a), and thereby obtain the true kinematic amplitudes . The inverse transformation (2.5.5.16) gives a projection of the structure (Unwin & Henderson, 1975; Henderson & Unwin, 1975).

Sometimes, an observed image is `noised' by the to a great extent. Then, one may combine data on real and reciprocal space to construct a sufficiently accurate image. In this case, the electron-diffraction pattern is measured and structure-factor moduli from diffraction reflection intensities are obtained: At the same time, the structure factors are calculated from the processed structure projection image by means of the Fourier transformation. However, owing to poor image quality we take from these data only the values of phases since they are less sensitive to scattering density distortions than the moduli, and construct the Fourier synthesis

Here the possibilities of combining various methods open up, e.g. for obtaining the structure-factor moduli from X-ray diffraction, and phases from electron microscopy, and so on (Gurskaya et al., 1971).

Images with point symmetry. If a projection of an object (and consequently, the object itself) has a rotational N-fold axis of symmetry, the structure coincides with itself on rotation through the angle . If the image is rotated through arbitrary angles and is aligned photographically with the initial image, then the best density coincidence will take place at a rotation through which defines N. The pattern averaging over all the rotations will give the enhanced structure image with an times reduced background (Markham et al., 1963).

Rotational filtering can be performed on the basis of the Fourier expansion of an image in polar coordinates over the angles (Crowther & Amos, 1971). The integral over the radius from azimuthal components gives their power where a is the maximum radius of the particle. A set forms a spectrum, the least common multiple N of strong peaks defining the N-fold symmetry. The two-dimensional reconstructed image of a particle with rotational symmetry is defined by the synthesis (2.5.5.24) with .

Asymmetric images. In this case, a set of images is processed by computational or analogue methods. The initial selection of images involves the fulfillment of the maximum similarity condition.

The averaging of n images in real space gives The signal/noise ratio on an average image is times enhanced.

The degree of similarity and accuracy of superposition of two images with an account both of translational and angular shifts is estimated by a cross-correlation function ³ of two selected images and (Frank, 1975, 1980). The value is the measure of image similarity, the x′ coordinate of the maximum indicates the shift of the images relative to each other. The first term of the resultant expression (2.5.5.26) is the cross-correlation function of noise-corrected images being compared, the second and third terms are approximately equal to zero, since the noise does not correlate with the signal; the last term is the autocorrelation function of the noise (Cramér, 1954; Frank, 1975, 1980).

The calculation of a correlation function is performed by means of Fourier transformation on the basis of the convolution theorem, since the Fourier transformation of the product of the Fourier transform of function and the conjugated Fourier transform function gives the cross-correlation function of the initial functions:

The probability density of samples for images has the form

Here J is the tentative image (as such, a certain `best' image can first be selected, while at the repeated cycle an average image is obtained), is the image investigated, σ is the standard deviation of the normal distribution of noises and the relative shift of the image. This function is called a likelihood function; it has maxima relative to the parameters , . The average image and dispersion are This method is called the maximum-likelihood method (Cramér, 1954; Kosykh et al., 1983).

It is convenient to carry out the image alignment, in turn, with respect to translational and angular coordinates. If we start with an angular alignment we first use autocorrelation functions or power spectra, which have the maximum and the symmetry centre at the origin of the coordinates. The angular correlation maximum gives the mutual angle of rotation of two images.

Then we carry out the translational alignment of rotationally aligned images using the translational correlation function (2.5.5.26) (Langer et al., 1970).

In the iteration alignment method, the images are first translationally aligned and then an angular shift is determined in image space in polar coordinates with the centre at the point of the best translational alignment. After the angular alignment the whole procedure may be repeated (Steinkilberg & Schramm, 1980).

The average image obtained may have false high-frequency components. They can be excluded by multiplying its Fourier components by some function and suppressing high-space frequencies, for instance by an `artificial temperature factor' .

For a set of similar images the Fourier filtration method can also be used (Ottensmeyer et al., 1977). To do this, one should prepare from these images an artificial `two-dimensional crystal', i.e. place them in the same orientation at the points of the two-dimensional lattice with periods a, b. The processing is then performed according to (2.5.5.18), (2.5.5.19); as a result one obtains with reduced background. Some translational and angular errors in the arrangement of the images at the artificial lattice points act as an artificial temperature factor. The method can be realized by computing or by optical diffraction.