<Copyright statement>= (U->)
"""
1.This Software copyright � Australian Synchrotron Research Program Inc, ("ASRP").
2.Subject to ensuring that this copyright notice and licence terms
appear on all copies and all modified versions, of PyCIFRW computer
code ("this Software"), a royalty-free non-exclusive licence is hereby
given (i) to use, copy and modify this Software including the use of
reasonable portions of it in other software and (ii) to publish,
bundle and otherwise re-distribute this Software or modified versions
of this Software to third parties, provided that this copyright notice
and terms are clearly shown as applying to all parts of software
derived from this Software on each occasion it is published, bundled
or re-distributed. You are encouraged to communicate useful
modifications to ASRP for inclusion for future versions.
3.No part of this Software may be sold as a standalone package.
4.If any part of this Software is bundled with Software that is sold,
a free copy of the relevant version of this Software must be made
available through the same distribution channel (be that web server,
tape, CD or otherwise).
5.It is a term of exercise of any of the above royalty free licence
rights that ASRP gives no warranty, undertaking or representation
whatsoever whether express or implied by statute, common law, custom
or otherwise, in respect of this Software or any part of it. Without
limiting the generality of the preceding sentence, ASRP will not be
liable for any injury, loss or damage (including consequential loss or
damage) or other loss, loss of profits, costs, charges or expenses
however caused which may be suffered, incurred or arise directly or
indirectly in respect of this Software.
6. This Software is not licenced for use in medical applications.
"""
This file implements a general CIF reading/writing utility. A class
is initialised with either no arguments (a new CIF file) or with the
name of an already existing CIF file. Data items are accessed/changed/
added using the python mapping type ie to get dataitem you would
type value = cf[blockname][dataitem].
The methods available for the CifFile type are:
ReadCif(filename): (re)initialise using Cif file filename.
NewBlock(blockname,[block contents]): add new block to this object. Returns new
block name, which will not be the requested name if the Cif object
already has the requested name. If blockcontents is provided, it
must be a CifBlock object (see below).
WriteOut(comment): return the contents of the current file as a
CIF-conformant string, with optional comment at the beginning.
The methods available for the CifBlock type are:
GetCifItem(itemname): return the value of itemname in
current block (equivalent to using [])
AddCifItem(data): add data to the current block. data is
a tuple containing either an array of itemnames and an array of arrays of
data, or else an itemname and a data value for that item. This method
is called when setting data using []
RemoveCifItem(dataname): remove the given Cif dataname from the
current block. Same as typing 'del block[item]'
GetLoop(dataname): for looped data item dataname, get a list
of all itemnames and values co-occurring in this loop. Returns an error
if dataname is not in a loop.
AddLoop(dataname,data): add data to the loop containing
dataname. If dataname is not a looped item, an error is returned.
If (data) has the wrong length, an error is returned.
Note also that a CifFile object can be accessed as a mapping type, ie using square brackets. Most mapping operations have been implemented (see below).
We import type objects at the module level, as required by later versions of Python.
<*>= <Copyright statement> from types import * <CifFile class> <CifBlock class> <Define an error class>
If we are passed a filename, we open it and read it in, assuming that it is a conformant Cif file. A CifFile object is a dictionary of CifBlock objects, accessed by block name. As the maximum line length is subject to change, we allow the length to be specified, with the current default set at 2048 characters (Cif 1.1). For reading in files, we only flag a length error if the parameter "strict" is true, in which case we use parameter maxinlength as our maximum line length on input. Parameter maxoutlength sets the maximum line size for output. If maxoutlength is not specified, it defaults to the maximum input length.
Note that this applies to the input only. For changing output length, you can provide an optional parameter in the WriteOut method.
<CifFile class>= (<-U)
class CifFile:
<Initialise data structures>
<Check data name lengths>
<CifFile emulation of mapping type>
<Read in a CIF file>
<Add a new data section>
<Write out to string representation>
<Initialise data structures>= (<-U)
def __init__(self,datasource=None,strict=1,maxinlength=2048,maxoutlength=0):
self.dictionary = {}
self.maxinlength = maxinlength
if maxoutlength == 0:
self.maxoutlength = maxinlength
else:
self.maxoutlength = maxoutlength
self.strict = strict
if isinstance(datasource,DictType):
for (key,value) in datasource.items():
self.__setitem__(key,value)
elif type(datasource) is StringType:
self.ReadCif(datasource,strict,maxinlength)
elif isinstance(datasource,CifFile):
self.dictionary = datasource.dictionary.copy()
if strict:
self.checklengths()
Checking item name lengths. Although the new 1.1 standard allows very long lines (2048 characters), data names are still restricted to be no more than 75 characters in length. We hard code this in.
<Check data name lengths>= (<-U)
def checklengths(self):
blocks = self.dictionary.items()
for name,block in blocks:
toolong = len(filter(lambda a:len(a)>75, block.keys()))
if toolong:
print 'Warning: block ' + name + ' has ' + `toolong` + ' overlength data names'
Reading in a file. We now use the Yapps2-generated YappsCifParser module to provide grammar
services. The structure returned from parsing is an array of blocks,
which we convert to a dictionary and place in self.dictionary. We do a preliminary
check of line length, to satisfy the standard.
<Read in a CIF file>= (<-U)
def ReadCif(self,filename,strict,maxlength):
import YappsCifParser,string
stream = open(filename,'r')
text = stream.read()
stream.close()
if not text: # empty file, return empty block
return
split = string.split(text,'\n')
if strict:
toolong = filter(lambda a:len(a)>maxlength, split)
if toolong:
pos = split.index(toolong[0])
raise CifError, 'Line %d contains more than %d characters' % (pos+1,maxlength)
context = {"loops":[],"latest":{}}
try:
parser = YappsCifParser.CifParser(YappsCifParser.CifParserScanner(text))
filecontents = getattr(parser,"input")()
except YappsCifParser.SyntaxError:
errorstring = 'Syntax error in input file: last value parsed was %s' % YappsCifParser.lastval
errorstring = errorstring + '\nParser status: %s' % `parser._scanner`
raise CifError, errorstring
else:
if not filecontents: # comments only, return empty
return
for block in filecontents.keys():
self.dictionary.update({block:CifBlock(filecontents[block])})
<CifFile emulation of mapping type>= (<-U)
def __str__(self):
return self.WriteOut()
def __setitem__(self,key,value):
if isinstance(value,CifBlock):
self.NewBlock(key,value)
else: raise TypeError
def __getitem__(self,key):
return self.dictionary[key]
def __delitem__(self,key):
del self.dictionary[key]
def __len__(self):
return len(self.dictionary)
def keys(self):
return self.dictionary.keys()
def has_key(self,key):
return self.dictionary.has_key(key)
def get(self,key,default=None):
if self.dictionary.has_key(key):
return self.dictionary[key]
else:
return default
def clear(self):
self.dictionary.clear()
def copy(self):
newcopy = self.dictionary.copy()
return CifFile('',newcopy)
def update(self,adict):
for key in adict.keys():
self.dictionary[key] = adict[key]
Adding a new block. A new block is just a new dictionary, so we add a new dictionary to the current list of dictionaries, and set the count of the number of loops to zero. We return the new block name in case we have changed it, so the calling routine can refer to it later. Also, there is a limit of 75 characters for the block name length, which we enforce here.
<Add a new data section>= (<-U)
def NewBlock(self,blockname,blockcontents=()):
import re
if not blockcontents:
blockcontents = CifBlock()
newblockname = re.sub('\W','_',blockname)
blocknames = self.dictionary.keys()
i = 0
while blocknames.count(newblockname):
i = i + 1
newblockname = newblockname+`i`
if len(newblockname) > 75:
raise CifError, 'Cif block name too long:' + newblockname
if blockcontents.__class__.__name__[-8:] != 'CifBlock':
raise CifError, 'Cif block initialised with non CifBlock object %s' % blockcontents.__class__.__name__[-8:]
self.dictionary.update({newblockname:blockcontents})
return newblockname
Writing all this stuff out to a string. We loop over each of the individual sections, getting their string representation. We implement this using the cStringIO module for faster work. Note that the default output comment specifies a CIF 1.1 standard file.
<Write out to string representation>= (<-U)
def WriteOut(self,comment=''):
import cStringIO
if not comment:
comment = \
"""#\\#CIF1.1
##########################################################################
# Crystallographic Information Format file
# Produced by PyCifRW module
#
# This is a CIF file. CIF has been adopted by the International
# Union of Crystallography as the standard for data archiving and
# transmission.
#
# For information on this file format, follow the CIF links at
# http://www.iucr.org
##########################################################################
"""
outstring = cStringIO.StringIO()
outstring.write(comment)
for datablock in self.dictionary.keys():
outstring.write('\ndata_'+datablock+'\n')
outstring.write(str(self.dictionary[datablock]))
returnstring = outstring.getvalue()
outstring.close()
return returnstring
A Cif Block is essentially a dictionary with a few extra items for our own internal reference. These are used to handle loop management, so that data belonging within one loop are able to be handled together when printing out. When initialising, we allow either a set of tuples, suitable for input to AddCifItem, or a complete block, suitable when copying.
<CifBlock class>= (<-U)
class CifBlock:
<Initialise Cif Block>
<Add emulation of a mapping type>
<Return value of Cif item>
<Remove a data item>
<Add a data item>
<Check data name for CIF conformance>
<Check data item for CIF conformance>
<Regularise data values>
<Get complete looped data>
<Add to looped data>
<Print a data block>
<Format a string>
If given non-zero data to initialise the block with, we either copy (if it is a dictionary) or else initialise each key-value pair separately (if tuples). We take care to include our special "loop" key if it is not in the supplied dictionary, but apart from this we make no check of the actual conformance of the dictionary items. If given a maximum output line length, we save that for use in writing out lines. We use the wraplength parameter to try to wrap nicely; however, if we are given very long data, we will continue on until we get to maxoutlength characters.
<Initialise Cif Block>= (<-U)
def __init__(self,data = (),maxoutlength=2048,wraplength=80):
self.block = {"loops":[]}
self.maxoutlength = maxoutlength
self.wraplength = wraplength
if type(data) is DictType: #direct placement
self.block = data
if not self.block.has_key("loops"):
self.block.update({"loops":[]})
elif type(data) is TupleType:
for item in data:
self.AddCifItem(item)
else: raise TypeError
Adding emulation of a mapping type. We add any of the other
functions we'd like to emulate. __len__ returns the number
of CIF items in this block, either in a loop or not. So it is
not the simple length of the dictionary.
<Add emulation of a mapping type>= (<-U)
def __str__(self):
return self.printsection()
def __setitem__(self,key,value):
self.AddCifItem((key,value))
def __getitem__(self,key):
return self.GetCifItem(key)
def __delitem__(self,key):
self.RemoveCifItem(key)
def __len__(self):
blen = len(self.block) - 1 #non-looped items
for aloop in self.block["loops"]:
blen = blen + len(aloop.keys())
return blen
def __nonzero__(self):
if len(self.block) == 1 and len(self.block["loops"]) == 0:
return 0
return 1
def keys(self):
thesekeys = self.block.keys()
for aloop in self.block["loops"]:
thesekeys.extend(aloop.keys())
return thesekeys
def has_key(self,key):
if self.block.has_key(key):
return 1
for aloop in self.block["loops"]:
if aloop.has_key(key):
return 1
return 0
def get(self,key,default=None):
if self.has_key(key):
retval = self.GetCifItem(key)
else:
retval = default
return retval
def clear(self):
self.block = self.NewBlock()
def copy(self):
newcopy = self.block.copy()
newcopy["loops"] = []
for aloop in self.block["loops"]: # do a deeper copy
newcopy["loops"].append(aloop.copy())
return CifBlock(newcopy)
def update(self,adict):
loopdone = []
if not isinstance(adict,CifBlock):
raise TypeError
for key in adict.block.keys():
if key!="loops":
self.AddCifItem((key,adict[key]))
else:
for aloop in adict.block["loops"]:
self.AddCifItem((aloop.keys(),aloop.values()))
Returning a Cif item value. Note that a looped block has little
meaning without all the items in the loop. Routine GetLoop is
better in this case.
<Return value of Cif item>= (<-U)
def GetCifItem(self,itemname):
if self.block.has_key(itemname):
return self.block[itemname]
else:
for aloop in self.block["loops"]:
if aloop.has_key(itemname):
return aloop[itemname]
raise KeyError, 'Item not in Cif block'
This function is required if we wish a list of all items in a data block.
We would use this if we needed to match things up properly. We give the name
of one item, and all items from that loop, and their values, are returned.
As we are using the built-in items method of dictionaries, the returned
values are a series of key,value pairs.
Also, it would be nice if the return value is in a form that can be fed right back into the file. So if we are asked for an item which is part of a loop, we reconstruct all of the data before returning it.
<Get complete looped data>= (<-U)
def GetLoop(self,itemname):
for aloop in self.block["loops"]:
if aloop.has_key(itemname):
return aloop.items()
# not a looped item
if self.block.has_key(itemname):
raise TypeError, 'Non-looped item'
raise KeyError, 'Item not in loop'
Adding to a loop. We find the loop number containing the dataname that
we've been passed, and then append all of the (key,values) pairs that we
are passed in data, which is a dictionary. We expect that the data
have been sorted out for us, unlike when data are passed in AddCifItem,
when there can be both unlooped and looped data in one set. The dataname
passed to this routine is simply a convenient way to refer to the
loop, and has no other significance.
<Add to looped data>= (<-U)
def AddToLoop(self,dataname,loopdata):
found = 0
for aloop in self.block["loops"]:
if aloop.has_key(dataname):
found = 1
for itemname in loopdata.keys():
if len(loopdata[itemname])!= len(aloop[dataname]):
raise CifError, 'Datalength mismatch adding to loop: %s and %s' % (itemname, dataname)
aloop.update({itemname:loopdata[itemname]})
if not found:
raise KeyError, 'No such looped item name: %s' % dataname
Removing a data item. We delete the item, and if it is looped, and nothing is left in the loop, we remove that element of the list.
<Remove a data item>= (<-U)
def RemoveCifItem(self,itemname):
if self.block.has_key(itemname):
del self.block[itemname]
return
for aloop in self.block["loops"]:
if aloop.has_key(itemname):
del aloop[itemname]
self.block["loops"] = filter(None, self.block["loops"])
Adding a data item. We are passed a tuple with the (set) of data names at the beginning, and a (set) of values for them following. For efficiency, we want to group looped items together; so in the stored dictionary, all data items in a loop belong in one dictionary in an array of dictionaries keyed by the word "loops".
If an item is already stored, it will be silently replaced; if it earlier lived in a loop, it will be removed from the loop, and the loop itself removed if it is now zero length.
We check the length of the name, and give an error if the name is greater than 75 characters, which is the CIF 1.1 maximum length.
We also check for consistency, by making sure the new item is not in the block already. If it is, we replace it (consistent with the meaning of square brackets). If it is in a loop, we replace the looped value and all other items in that loop block. This means that when adding loops, we must add them all at once if we call this routine directly.
We typecheck the data items. They can be tuples, strings or lists. If we have a list of values for a single item, the item name should also occur in a single member tuple.
<Add a data item>= (<-U)
def AddCifItem(self,data):
# we accept only tuples, strings and lists!!
if not (isinstance(data[0],StringType) or isinstance(data[0],TupleType)
or isinstance(data[0],ListType)):
raise TypeError, 'Cif datanames are either a string, tuple or list'
# now put into the dictionary properly...
if isinstance(data[0],StringType): # a single name
self.check_data_name(data[0]) # make sure no nasty characters
# now make sure the data is OK
regval = self.regularise_data(data[1])
self.check_item_value(regval)
self.block.update({data[0]:regval}) # trust the data is OK
for aloop in self.block["loops"]:
if aloop.has_key(data[0]):
del aloop[data[0]]
self.block["loops"] = filter(len,self.block["loops"])
else: # we loop
if(len(data[0])!=len(data[1])):
raise TypeError, 'Length mismatch between itemnames and values'
dellist = []
map (self.check_data_name,data[0])
for itemname in data[0]:
self.block["loops"] = filter(lambda a,b=itemname:b not in a.keys(),self.block["loops"])
newdict = {}
proper_vals = map(self.regularise_data,data[1])
map(self.check_item_value,proper_vals)
map(lambda a,b,c=newdict:c.update({a:b}),data[0],proper_vals)
self.block["loops"].append(newdict)
return
Checking the data names. The CIF 1.1 standard restricts characters in a data name to ASCII 33-126 and there should be a leading underscore. Items are allowed to have the blank characters as well, i.e. ascii 09,10,13 and 32. Data items may be lists, which we need to detect before checking. We assume that the item has been regularised before this check is called.
<Check data name for CIF conformance>= (<-U)
def check_data_name(self,dataname):
if len(dataname) > 75:
raise CifError, 'Dataname ' + dataname + ' too long.'
if dataname[0]!='_':
raise CifError, 'Dataname ' + dataname + ' does not begin with _'
if len (filter (lambda a: ord(a) < 33 or ord(a) > 126, dataname)) > 0:
raise CifError, 'Dataname ' + dataname + ' contains forbidden characters'
<Check data item for CIF conformance>= (<-U)
def check_item_value(self,item):
test_item = item
if type(item) != TupleType and type(item) != ListType:
test_item = [item] #single item list
def check_one (it):
if type(it) == StringType:
if len (filter (lambda a:ord(a) != 9 and ord(a) != 10 and ord(a) != 13
and (ord(a) < 32 or ord(a) > 126), it)) > 0:
raise CifError, 'Data item ' + it + '... contains forbidden characters'
map(check_one,test_item)
Regularising data. We want the copy.deepcopy operation to work, so we can't have any arrays passed into the master dictionary. We make sure everything goes in either as a single item or as a list/tuple.
<Regularise data values>= (<-U)
def regularise_data(self,dataitem):
alrighttypes = [IntType, LongType,
FloatType, StringType]
okmappingtypes = [TupleType, ListType]
thistype = type(dataitem)
if thistype in alrighttypes or thistype in okmappingtypes:
return dataitem
# so try to make into a list
try:
regval = list(dataitem)
except TypeError, value:
raise CifError, str(dataitem) + ' is wrong type for data value\n'
return regval
Printing a section. We allow an optional order list to be given, in case the caller wants to order things in some nice way. This is overridden if any item in the order list occurs within a loop, in which case it will be packaged together with everything else within that same loop.
Note that we must be careful to add spaces between data items, especially when formatting string loop data, where our string addition could get quite hairy. As we are doing so much concatenation, we use a stringIO buffer to speed it up.
<Print a data block>= (<-U)
def printsection(self,order=[]):
import cStringIO
import string
# first make an ordering
if not order:
order = self.block.keys()
order.sort()
# now prune that ordering...
order = filter(lambda a,b=self.block:b.has_key(a),order)
order.remove('loops')
# now do it...
outstring = cStringIO.StringIO() # the returned string
for itemname in order:
itemvalue = self.block[itemname]
if isinstance(itemvalue,StringType):
thisstring = self._formatstring(itemvalue)
if len(thisstring) + len(itemname) < (self.wraplength-2):
outstring.write('%s %s\n' % (itemname,thisstring))
else:
outstring.write('%s\n %s\n' % (itemname, thisstring))
else:
if len(str(itemvalue)) + len(itemname) < (self.wraplength-2):
outstring.write('%s %s\n' % (itemname, itemvalue))
else:
outstring.write('%s\n %s\n' % (itemname, itemvalue))
continue
#do the loops
for aloop in self.block["loops"]:
outstring.write('\n loop_\n')
loopnames = aloop.keys()
loopnames.sort()
numdata = len(aloop[loopnames[0]])
for name in loopnames:
outstring.write(' %-75s\n' % name)
if len(aloop[name]) != numdata:
raise CifError,'Loop data mismatch for ' + name + ':output aborted'
curstring = ''
# when adding to outstring, make sure to add spaces
for position in range(numdata):
for name in loopnames:
# at each point, get the next data value
datapoint = aloop[name][position]
if isinstance(datapoint,StringType):
thisstring = '%s' % (self._formatstring(datapoint)) #no spaces yet
if '\n' in thisstring:
# we try to wrap if the first <eol> is bigger than our wrap length
if len(curstring) + string.find(thisstring,'\n') > (self.wraplength):
outstring.write(' ' + curstring + '\n' + thisstring)
else:
outstring.write(' ' + curstring + ' ' + thisstring) #a space
curstring = ''
continue
else:
thisstring = ' %s ' % datapoint
if len(curstring) + len(thisstring)> self.wraplength-2: #past end of line with space
outstring.write(' ' + curstring+'\n') #add the space
curstring = ''
curstring = curstring + ' ' + thisstring
outstring.write(' ' + curstring + '\n') #last time through
curstring = ''
returnstring = outstring.getvalue()
outstring.close()
return returnstring
Formatting a string. We make sure that the length of the item value
is less than self.maxoutlength, or else we should split them, and so on. We check the
value for terminators and impossible apostrophes and length, before
deciding whether to print it and the item on a single line. We try to
respect carriage returns in the string, if the caller has tried to do
the formatting for us. If we are not putting apostrophes around a
string, we make the first character a space, to avoid problems if the
first character of a line is a semicolon.
The CIF specification states that embedded quotes are allowed so long as they are not followed by a space. So if we find any quotes followed by spaces we output a semicolon-terminated string to avoid to much messing around. This routine is called very often and could be improved.
<Format a string>= (<-U)
def _formatstring(self,instring):
import re, string
if len(instring)< (self.maxoutlength-2) and '\n' not in instring and not ('"' in instring and '\'' in instring):
if not ' ' in instring and not '\t' in instring and not '\v' \
in instring: # no blanks
return ' %s ' % (instring)
if not "'" in instring: #use apostrophes
return "'%s'" % (instring)
elif not "\"" in instring:
return '"%s"' % (instring)
# is a long one or one that needs semicolons due to carriage returns
outstring = "\n;\n"
# if there are returns in the string, try to work with them
while 1:
retin = string.find(instring,'\n')+1
if retin < self.maxoutlength and retin > 0: # honour this break
outstring = outstring + instring[:retin]
instring = instring[retin:]
elif len(instring)<self.maxoutlength: # finished
outstring = outstring + instring + '\n;\n'
break
else: # find a space
for letter in range(self.maxoutlength-1,40,-1):
if instring[letter] in ' \t\f': break
outstring = outstring + instring[:letter+1]
outstring = outstring + '\n'
instring = instring[letter+1:]
return outstring
Defining an error class: we simply derive a 'nothing' class from the root Python class
<Define an error class>= (<-U)
class CifError(Exception):
def __init__(self,value):
self.value = value
def __str__(self):
print '\nCif Format error: '+ self.value