Almost all of the information contained in the CSD has been abstracted from the published literature. Over 800 primary literature sources are cited and the earliest reference is from 1930. Much of the data has been re-keyboarded from the original literature and from hard-copy supplementary deposition documents. The CSD now acts as the official depository for some 40 major international journals. Today, an increasing proportion (around 75% in mid-2000) of the numerical information is received directly in electronic form. The switch from hard-copy input to electronic deposition has been catalysed by the development of the exchange format for crystallographic data, the crystallographic information fileCrystallographic information file (CIF)CIF (crystallographic information file) or CIF (Hall et al., 1991). The CIF has been adopted as the standard for the subject by the International Union of Crystallography, and is now output by nearly all of the major software packages for structure determination and refinement. Development of the CIF has also led to an increase in direct private depositions of structural data to the CSD, data that, for various reasons, are unlikely to be published through formal mechanisms.

Each individual structure in the CSD is referred to as an entry and each entry is identified by a reference code (refcode) containing six alphabetic characters, which characterize a specific chemical compound, and a further two numeric characters which trace the publication history of the structure. The information content of a typical CSD entry is illustrated schematically in Fig. 24.3.2.1. Individual data items can be categorized into three different groupings which are most conveniently described in terms of their dimensionality.

The one-dimensional data for each entry comprise chemical and bibliographic text strings, together with certain individual numerical items, viz chemical compound name and any common synonym(s), chemical formula, authors names, journal name and literature citation, text comment reflecting any special experimental details (non-room-temperature study, absolute configuration determined, neutron study etc.). The cell parameters, crystal data, space group and precision indicators also fall into this category.

The formal two-dimensional chemical structural diagram for each entry (Fig. 24.3.2.2) is encoded in the form of a compact connection table. Chemical connectivity is recorded in terms of a set of atom and bond properties. The atom properties recorded are: atom number, element type, number of connected non-H atoms, number of terminal H atoms and the formal atomic charge. Bond properties are encoded as a pair of atom numbers and the formal chemical bond type that connects those atoms. Bond types employed in the CSD connectivity descriptions are: single, double, triple, quadruple (metal–metal), aromatic, delocalized double and π bonds. Bond types are (automatically) coded negative if the bond forms part of a cyclic system.

The three-dimensional data consist of the fractional coordinates and symmetry operators for each entry. This information, together with the cell dimensions, is used to establish a crystallographic connectivity using standard covalent radii. The chemical and crystallographic connectivities are then mapped onto one another, using graph-theoretic algorithms, so that the chemical atom and bond properties are associated with the three-dimensional structure for search purposes. The CSD always records coordinates for complete molecules. Thus, if a molecule adopts a special position in the assigned space group, i.e. the asymmetric unit is some fraction of the total number of atoms in the molecule, then the CSD system also records those symmetry-generated atoms that complete the chemical entity. This speeds up the search process and also makes the data more accessible to non-crystallographers.

Derived data are calculated directly from the evaluated raw data and stored in the master archive for search purposes. Numerical items such as Z′, the number of chemical entities in the asymmetric unit, is a typical (real) numerical data item in this category. However, by far the most useful of the derived data items are a set of 682 individual pieces of yes/no information which are encoded as a bitmap, referred to as the screen record. The first 155 of these bits record information about (a) the elemental constitution of the compound, (b) results of the data-validation procedure and (c) summary information about the data content of the entry. These bits can be accessed directly by the user as search keys. The most important parts of the bitmap contain codified yes/no information about the presence/absence of specific features in the complete 2D or 3D structures held in the CSD. When a chemical substructure is entered as a query, its constitution is analysed in the same way to produce a bitmap for the query. Logical comparison of the query bitmap with the bitmap stored for each full CSD entry is computationally rapid, and quickly eliminates those entries that do not contain the requested features. Only those entries that pass this initial screening process need enter the detailed and computationally intensive atom-by-atom, bond-by-bond connectivity mapping that finally confirms (or not) the presence of the required query substructure.

All data entering the CSD are subject to stringent check and evaluation procedures. Some of these are visual, but the majority are automated within the CSD program PreQuestCambridge Structural Database (CSD)PreQuestPreQuest. The checks ensure that the 1D and 2D information fields abstracted by CCDC staff are accurately encoded, and that the 3D crystallographic coordinates are consistent with both the chemical description of the structure and with the geometrical description supplied by the authors. Most typographical errors in original papers can be corrected by the CCDC but, in the case of serious discrepancies, the original authors are consulted.

CSD-Use is a database of scientific research papers in which the CSD was used as the principal or sole source of experimental information. The database comprises more than 700 literature citations classified according to the type of systematic study undertaken. Each CSD-Use entry also contains a short summary of the major findings of the research. The database is growing rapidly over time, and is expected to be a valuable resource in the future, since it contains a fully retrospective overview of the data-mining methods and research applications of the CSD.

The CSD is supplied with a suite of fully interactive graphical software modules which provides users with facilities to: (a) interrogate all of the 1D, 2D and 3D information fields; (b) display entries graphically in a variety of styles; (c) retrieve relevant data for search hits, including geometrical parameters derived from the stored coordinates; and (d) display the derived numerical information, e.g. as histograms, scattergrams etc., generate descriptive statistics and perform more complex numerical analyses. More recently, software has been added that permits users to transform their own in-house structural data to CSD formats for inclusion in these processes. A summary of the overall CSD software system is given in Fig. 24.3.3.1 which shows the functional relationships between the four major applications programs.

PreQuest is a data-validation and data-conversion program which is used to create high-quality structural data files in CSD format from, e.g., raw input data from a CIF. PreQuest is used routinely by CCDC's scientific editors to create and validate entries for inclusion in the master CSD archive, hence the program is constantly being maintained and upgraded. The released version enables users to build a private CSD-format database of their own structures which can then be searched independently of, or in conjunction with, the master CSD files using the database access programs described below.

Quest3D has been the main search engine and information-retrieval program for the CSD since the late 1980s. Its main features are summarized below. However, since 1997, the CCDC has been developing its successor, the ConQuest program, which was first released as part of the CSD system in April 2000. During an interim period, perhaps two years, ConQuest and Quest3D will both form part of the released CSD system on certain computing platforms while the functionality of the new program is being fully developed. Further details of ConQuest are provided in Section 24.3.3.5, indicating in particular how it differs from, and improves upon, the facilities available in Quest3D.

Quest3D is the main search engine and information-retrieval program for the CSD. It permits interrogation of all information fields: (a) 19 text fields, (b) 38 individual numerical fields, (c) element symbols and element counts, (d) full or partial molecular formulae, (e) direct access to over 150 bit screens, (f) extensive 2D chemical substructure search capabilities, and (g) 3D substructure searching at the molecular level or at the extended crystal-structure level. A search of a specific information field is termed a test of that field, and is constructed graphically via the menu system; menu components correspond to the categories of searches identified above. A complete query is then constructed by combining a number of separate test components using Boolean logic.

Substructure searching is the most important and frequently used facility. At the molecular level, the substructure (chemical fragment) query is entered graphically and is defined using the formal covalent bond types present in the 2D chemical connectivity tables of the CSD. The process can be extended to locate non-bonded contacts in the complete crystal structure. Here, the individual atoms or chemical groups involved in the contact must be specified, and a limiting non-bonded contact distance must be provided, along with any other geometrical criteria required to define the contact more precisely.

All substructure searches begin with the user drawing the required chemical unit(s) via the BUILD menu. Chemical variability and precision are controlled through (a) the PERIODIC TABLE sub-menu, which allows for specification of variable element types at specific atomic sites, (b) the 2D-CONSTRAIN menu, which allows further chemical restrictions to be specified, such as cyclicity/acyclicity of bonds, exact hydrogen-atom counts, total coordination numbers for atoms etc., and (c) the 3D-CONSTRAIN menu, which permits the user to specify a list of geometrical parameters to be calculated by the program for each instance of the fragment located in the CSD; any of these geometrical parameters may be used as criteria to limit the scope of the search, especially at the intermolecular level. A file of calculated geometrical information is output by Quest3D and may be read by Vista, or by external data analysis software. Other Quest3D output files allow CSD search results to be communicated rapidly to proprietary modelling software.

The overall aim of the ConQuest project is to replace Quest3D with graphical search software that makes best use of modern computing environments. The primary objective has been to create an interface that is both simple and intuitive to use, so as to encourage use of the CSD by a broader spectrum of scientists. Thus, ConQuest provides: (a) text and numeric searches via pop-up windows, (b) a new sketcher window within which to encode 2D and 3D substructure searches, pharmacophore searches, and searches for non-bonded contacts in crystal structures, and (c) the immediate viewing of hits with facilities for backward and forward scrolling within hit lists. ConQuest is provided with full documentation and tutorials, both online and in printed form, and with context-dependent help facilities. Version 1.0, released in April 2000, contains most of the functionality available within the Quest3D program, and it is expected that Quest3D capabilities will soon be exceeded by the new program.

A most important feature of ConQuest is its availability on PC-Windows platforms, as well as its implementation under Unix/Linux. Initially, ConQuest and the CSD will be the only parts of the full CSD system available under PC-Windows, but Vista (or Vista-like facilities, see Section 24.3.3.6), a new visualizer and provision of the CCDC's knowledge bases (IsoStar and Mogul) will follow as planned developments in the PC area.

Vista reads geometrical table(s) generated by Quest3D and provides extensive facilities for the graphical representation and statistical analysis of the numerical data. Graphical facilities include histograms and scattergrams referred to Cartesian or polar axes, with a hyperlink back to the original CSD entries to permit immediate investigation of, e.g., outlying observations. The contents of plots can be edited interactively, and all illustrations can be output in PostScript format for inclusion in reports and publications. Additionally, Vista will generate descriptive statistics for a distribution, carry out simple linear regressions and perform principal-component analyses.

Pluto is used to visualize crystal and molecular structures in a variety of styles, including stick diagrams and ball-and-spoke and space-filling representations of individual molecules or extended crystal structures.

The preceding sections can only give a flavour of the extensive search, analysis and visualization capabilities of Quest3D, ConQuest, Vista and Pluto, which are fully documented in manuals available online via the web address given below, or in printed form from the CCDC.

In this section, we illustrate the application of the CSD system to one specific example: a CSD-based analysis to examine the O—H···O hydrogen-bonding ability of the keto oxygen of Fig. 24.3.3.2. This example illustrates a number of key features of the software system. The example is constructed in terms of Quest3D terminology, but identical facilities are available in the ConQuest program.

Searches can be performed interactively or allowed to run to completion without further intervention from the user. In interactive mode, Quest3D presents each hit as it is located, as illustrated in Fig. 24.3.3.3, and can then display the 1D bibliographic information, a 2D structural diagram, the 3D molecular structure, or a 3D packing diagram by toggling between display options. For an intermolecular search, as exemplified here, the non-bonded contact that triggered the hit is clearly identified. For the example described above, a file of the four user-defined geometrical parameters (DOH, AH, THETA, PHI) for each hit is created for use by Vista.

As illustrated in earlier sections, the CSD represents a collection of primary data resulting from diffraction experiments on crystals of small molecules – in particular the fractional coordinates, space group and cell dimensions that define the 3D crystal and molecular structure. However, the user of the system is usually interested in structural knowledge – in the form of bond lengths, angles, intermolecular contact distances and other parameters – that can be synthesized from the raw data by the use of the CSD system software. Thus, each detailed analysis carried out using the CSD system represents an experiment in data mining, and considerable operational and intellectual effort is employed in performing such analyses.

At the present state of development of the field, three facts are apparent: (a) many data-mining activities centre around a set of standard geometrical data types that are essential for major applications, particularly in structural chemistry, molecular modelling and rational drug design; (b) the expertise required to carry out data-mining experiments is not inconsiderable and the time required can be lengthy; and (c) as the size of the CSD is increasing rapidly and any compilations of structural knowledge should be updated on a regular basis, the increasing database size makes this operation very time consuming for individual users.

These considerations indicate that access to CSD information should be at two levels, the raw-data level and the structural-knowledge level, and, since 1995, the CCDC has started to derive libraries of structural knowledge from the raw data content of the CSD. The first of these libraries – IsoStarCambridge Structural Database (CSD)IsoStarIsoStar: a library of information on intermolecular interactions (Bruno et al., 1997) – is briefly summarized below. A second library, MogulCambridge Structural Database (CSD)MogulMogul, containing bond lengths, valence angles and torsional distributions, is currently under development in mid-2000. Such a knowledge base has obvious applications in crystallography, structural chemistry and molecular biology, not least in providing precise geometrical parameters which can be used in 3D model building, structure refinement and reality checking of developing and refined structures. The scientific applications of non-bonded contact geometries and conformational (torsional) information are more fully discussed in Chapter The relevance of the Cambridge Structural Database in protein crystallographyF. H. AllenJ. C. ColeM. L. Verdonk22.4 .

IsoStar (Bruno et al., 1997) is based on experimental data, not only from the CSD but also from the PDB, and contains some theoretical results calculated using the ab initio intermolecular perturbation theoryIntermolecular perturbation theory (IMPT) method of Hayes & Stone (1984). The experimental data in the CSD and the PDB have been used to display interaction geometries involving central groups (A) and contact groups (B). CSD search results of the type exemplified above are transformed into an easily visualized form by overlaying the A moieties. This results in a 3D distribution (scatterplot) showing the experimental distribution of B around A. A web-browser front end permits rapid access to these scatterplots, which can be viewed in RasMol (Sayle, 1996), interrogated interactively, converted into contoured surfaces etc.

Version 1.1 of IsoStar, released in October 1998, contains information on non-bonded interactions formed between 310 central groups and 45 contact groups. Version 1.1 contains over 12 000 scatterplots: 9 000 from the CSD and 3 000 from the PDB. IsoStar also reports results for 867 theoretical potential-energy minima calculated using the IMPT procedure. The library will be updated on a regular basis using automated software procedures developed at the CCDC.

Chapter The relevance of the Cambridge Structural Database in protein crystallographyF. H. AllenJ. C. ColeM. L. Verdonk22.4 contains illustrative examples from IsoStar, together with a more complete description of the knowledge base and its applications.

The CSD system, comprising the CSD and CSD-Use databases, and all of the applications software described above, is available on CD-ROM for Unix and DEC-VMS platforms and for PCs operating under Linux. At the time of writing (mid-2000), ConQuest alone is available for PC-Windows platforms, but the full availability of other components of the CSD system is currently being addressed. The CSD is released twice yearly, in April and October, as an indexed sequential binary file, with full installation instructions contained within the CD. Versions of the CSD have been reformatted for use with proprietary software systems: the MACCS3D/Isis system from Molecular Design Limited, and the Sybyl-UNITY system from Tripos Associates.

Subscribers in academic and other not-for-profit institutions may obtain the CSD system through their local National Affiliated Centre (NAC). The names, addresses and other coordinates of these centres are contained in the CCDC's web pages (see below). Users in countries not covered by NAC arrangements, or users from for-profit companies and organizations, should contact the CCDC directly.

The IsoStar library, for Unix systems only, is released after each library update, currently planned to occur on an annual basis. IsoStar forms part of the distributed CSD system, and CDs are available through the same mechanisms as the main system.

The CCDC maintains an extensive set of information on the web site http://www.ccdc.cam.ac.uk . The site describes the CSD system, IsoStar, and the associated research and development activities of the CCDC. These pages also provide access to CSD system documentation, provide lists of contact details for the National Affiliated Centres that service not-for-profit users worldwide, and give up-to-date information on how to contact the CCDC directly.