<>= """ 1.This Software copyright © Australian Synchrotron Research Program Inc, ("ASRP"). 2.Subject to ensuring that this copyright notice and licence terms appear on all copies and all modified versions, of PyCIFRW computer code ("this Software"), a royalty-free non-exclusive licence is hereby given (i) to use, copy and modify this Software including the use of reasonable portions of it in other software and (ii) to publish, bundle and otherwise re-distribute this Software or modified versions of this Software to third parties, provided that this copyright notice and terms are clearly shown as applying to all parts of software derived from this Software on each occasion it is published, bundled or re-distributed. You are encouraged to communicate useful modifications to ASRP for inclusion for future versions. 3.No part of this Software may be sold as a standalone package. 4.If any part of this Software is bundled with Software that is sold, a free copy of the relevant version of this Software must be made available through the same distribution channel (be that web server, tape, CD or otherwise). 5.It is a term of exercise of any of the above royalty free licence rights that ASRP gives no warranty, undertaking or representation whatsoever whether express or implied by statute, common law, custom or otherwise, in respect of this Software or any part of it. Without limiting the generality of the preceding sentence, ASRP will not be liable for any injury, loss or damage (including consequential loss or damage) or other loss, loss of profits, costs, charges or expenses however caused which may be suffered, incurred or arise directly or indirectly in respect of this Software. 6. This Software is not licenced for use in medical applications. """ @ \section{Introduction} This file implements a general CIF reading/writing utility. A class is initialised with either no arguments (a new CIF file) or with the name of an already existing CIF file. Data items are accessed/changed/ added using the python mapping type ie to get [[dataitem]] you would type [[value = cf[blockname][dataitem]]]. The methods available for the CifFile type are: \begin{enumerate} \item [[ReadCif(filename)]]: (re)initialise using Cif file [[filename]]. \item [[NewBlock(blockname,[block contents])]]: add new block to this object. Returns new block name, which will not be the requested name if the Cif object already has the requested name. If [[blockcontents]] is provided, it must be a CifBlock object (see below). \item [[WriteOut(comment)]]: return the contents of the current file as a CIF-conformant string, with optional [[comment]] at the beginning. \end{enumerate} The methods available for the CifBlock type are: \begin{enumerate} \item [[GetCifItem(itemname)]]: return the value of [[itemname]] in current block (equivalent to using []) \item [[AddCifItem(data)]]: add [[data]] to the current block. [[data]] is a tuple containing either an array of itemnames and an array of arrays of data, or else an itemname and a data value for that item. This method is called when setting data using [] \item [[RemoveCifItem(dataname)]]: remove the given Cif dataname from the current block. Same as typing 'del block[item]' \item [[GetLoop(dataname)]]: for looped data item [[dataname]], get a list of all itemnames and values co-occurring in this loop. Returns an error if [[dataname]] is not in a loop. \item [[AddLoop(dataname,data)]]: add [[data]] to the loop containing [[dataname]]. If dataname is not a looped item, an error is returned. If (data) has the wrong length, an error is returned. \end{enumerate} Note also that a CifFile object can be accessed as a mapping type, ie using square brackets. Most mapping operations have been implemented (see below). We import type objects at the module level, as required by later versions of Python. <<*>>= <> from types import * <> <> <> @ \section{CifFile} If we are passed a filename, we open it and read it in, assuming that it is a conformant Cif file. A CifFile object is a dictionary of CifBlock objects, accessed by block name. As the maximum line length is subject to change, we allow the length to be specified, with the current default set at 2048 characters (Cif 1.1). For reading in files, we only flag a length error if the parameter "strict" is true, in which case we use parameter maxinlength as our maximum line length on input. Parameter maxoutlength sets the maximum line size for output. If maxoutlength is not specified, it defaults to the maximum input length. Note that this applies to the input only. For changing output length, you can provide an optional parameter in the WriteOut method. <>= class CifFile: <> <> <> <> <> <> <>= def __init__(self,datasource=None,strict=1,maxinlength=2048,maxoutlength=0): self.dictionary = {} self.maxinlength = maxinlength if maxoutlength == 0: self.maxoutlength = maxinlength else: self.maxoutlength = maxoutlength self.strict = strict if isinstance(datasource,DictType): for (key,value) in datasource.items(): self.__setitem__(key,value) elif type(datasource) is StringType: self.ReadCif(datasource,strict,maxinlength) elif isinstance(datasource,CifFile): self.dictionary = datasource.dictionary.copy() if strict: self.checklengths() @ Checking item name lengths. Although the new 1.1 standard allows very long lines (2048 characters), data names are still restricted to be no more than 75 characters in length. We hard code this in. <>= def checklengths(self): blocks = self.dictionary.items() for name,block in blocks: toolong = len(filter(lambda a:len(a)>75, block.keys())) if toolong: print 'Warning: block ' + name + ' has ' + `toolong` + ' overlength data names' @ Reading in a file. We now use the Yapps2-generated YappsCifParser module to provide grammar services. The structure returned from parsing is an array of blocks, which we convert to a dictionary and place in [[self.dictionary]]. We do a preliminary check of line length, to satisfy the standard. <>= def ReadCif(self,filename,strict,maxlength): import YappsCifParser,string stream = open(filename,'r') text = stream.read() stream.close() if not text: # empty file, return empty block return split = string.split(text,'\n') if strict: toolong = filter(lambda a:len(a)>maxlength, split) if toolong: pos = split.index(toolong[0]) raise CifError, 'Line %d contains more than %d characters' % (pos+1,maxlength) context = {"loops":[],"latest":{}} try: parser = YappsCifParser.CifParser(YappsCifParser.CifParserScanner(text)) filecontents = getattr(parser,"input")() except YappsCifParser.SyntaxError: errorstring = 'Syntax error in input file: last value parsed was %s' % YappsCifParser.lastval errorstring = errorstring + '\nParser status: %s' % `parser._scanner` raise CifError, errorstring else: if not filecontents: # comments only, return empty return for block in filecontents.keys(): self.dictionary.update({block:CifBlock(filecontents[block])}) @ Emulation of a mapping type. <>= def __str__(self): return self.WriteOut() def __setitem__(self,key,value): if isinstance(value,CifBlock): self.NewBlock(key,value) else: raise TypeError def __getitem__(self,key): return self.dictionary[key] def __delitem__(self,key): del self.dictionary[key] def __len__(self): return len(self.dictionary) def keys(self): return self.dictionary.keys() def has_key(self,key): return self.dictionary.has_key(key) def get(self,key,default=None): if self.dictionary.has_key(key): return self.dictionary[key] else: return default def clear(self): self.dictionary.clear() def copy(self): newcopy = self.dictionary.copy() return CifFile('',newcopy) def update(self,adict): for key in adict.keys(): self.dictionary[key] = adict[key] @ Adding a new block. A new block is just a new dictionary, so we add a new dictionary to the current list of dictionaries, and set the count of the number of loops to zero. We return the new block name in case we have changed it, so the calling routine can refer to it later. Also, there is a limit of 75 characters for the block name length, which we enforce here. <>= def NewBlock(self,blockname,blockcontents=()): import re if not blockcontents: blockcontents = CifBlock() newblockname = re.sub('\W','_',blockname) blocknames = self.dictionary.keys() i = 0 while blocknames.count(newblockname): i = i + 1 newblockname = newblockname+`i` if len(newblockname) > 75: raise CifError, 'Cif block name too long:' + newblockname if blockcontents.__class__.__name__[-8:] != 'CifBlock': raise CifError, 'Cif block initialised with non CifBlock object %s' % blockcontents.__class__.__name__[-8:] self.dictionary.update({newblockname:blockcontents}) return newblockname @ Writing all this stuff out to a string. We loop over each of the individual sections, getting their string representation. We implement this using the cStringIO module for faster work. Note that the default output comment specifies a CIF 1.1 standard file. <>= def WriteOut(self,comment=''): import cStringIO if not comment: comment = \ """#\\#CIF1.1 ########################################################################## # Crystallographic Information Format file # Produced by PyCifRW module # # This is a CIF file. CIF has been adopted by the International # Union of Crystallography as the standard for data archiving and # transmission. # # For information on this file format, follow the CIF links at # http://www.iucr.org ########################################################################## """ outstring = cStringIO.StringIO() outstring.write(comment) for datablock in self.dictionary.keys(): outstring.write('\ndata_'+datablock+'\n') outstring.write(str(self.dictionary[datablock])) returnstring = outstring.getvalue() outstring.close() return returnstring @ \section{Cif Block class} A Cif Block is essentially a dictionary with a few extra items for our own internal reference. These are used to handle loop management, so that data belonging within one loop are able to be handled together when printing out. When initialising, we allow either a set of tuples, suitable for input to AddCifItem, or a complete block, suitable when copying. <>= class CifBlock: <> <> <> <> <> <> <> <> <> <> <> <> @ If given non-zero data to initialise the block with, we either copy (if it is a dictionary) or else initialise each key-value pair separately (if tuples). We take care to include our special "loop" key if it is not in the supplied dictionary, but apart from this we make no check of the actual conformance of the dictionary items. If given a maximum output line length, we save that for use in writing out lines. We use the wraplength parameter to try to wrap nicely; however, if we are given very long data, we will continue on until we get to maxoutlength characters. <>= def __init__(self,data = (),maxoutlength=2048,wraplength=80): self.block = {"loops":[]} self.maxoutlength = maxoutlength self.wraplength = wraplength if type(data) is DictType: #direct placement self.block = data if not self.block.has_key("loops"): self.block.update({"loops":[]}) elif type(data) is TupleType: for item in data: self.AddCifItem(item) else: raise TypeError @ Adding emulation of a mapping type. We add any of the other functions we'd like to emulate. [[__len__]] returns the number of CIF items in this block, either in a loop or not. So it is not the simple length of the dictionary. <>= def __str__(self): return self.printsection() def __setitem__(self,key,value): self.AddCifItem((key,value)) def __getitem__(self,key): return self.GetCifItem(key) def __delitem__(self,key): self.RemoveCifItem(key) def __len__(self): blen = len(self.block) - 1 #non-looped items for aloop in self.block["loops"]: blen = blen + len(aloop.keys()) return blen def __nonzero__(self): if len(self.block) == 1 and len(self.block["loops"]) == 0: return 0 return 1 def keys(self): thesekeys = self.block.keys() for aloop in self.block["loops"]: thesekeys.extend(aloop.keys()) return thesekeys def has_key(self,key): if self.block.has_key(key): return 1 for aloop in self.block["loops"]: if aloop.has_key(key): return 1 return 0 def get(self,key,default=None): if self.has_key(key): retval = self.GetCifItem(key) else: retval = default return retval def clear(self): self.block = self.NewBlock() def copy(self): newcopy = self.block.copy() newcopy["loops"] = [] for aloop in self.block["loops"]: # do a deeper copy newcopy["loops"].append(aloop.copy()) return CifBlock(newcopy) def update(self,adict): loopdone = [] if not isinstance(adict,CifBlock): raise TypeError for key in adict.block.keys(): if key!="loops": self.AddCifItem((key,adict[key])) else: for aloop in adict.block["loops"]: self.AddCifItem((aloop.keys(),aloop.values())) @ Returning a Cif item value. Note that a looped block has little meaning without all the items in the loop. Routine [[GetLoop]] is better in this case. <>= def GetCifItem(self,itemname): if self.block.has_key(itemname): return self.block[itemname] else: for aloop in self.block["loops"]: if aloop.has_key(itemname): return aloop[itemname] raise KeyError, 'Item not in Cif block' @ This function is required if we wish a list of all items in a data block. We would use this if we needed to match things up properly. We give the name of one item, and all items from that loop, and their values, are returned. As we are using the built-in [[items]] method of dictionaries, the returned values are a series of key,value pairs. Also, it would be nice if the return value is in a form that can be fed right back into the file. So if we are asked for an item which is part of a loop, we reconstruct all of the data before returning it. <>= def GetLoop(self,itemname): for aloop in self.block["loops"]: if aloop.has_key(itemname): return aloop.items() # not a looped item if self.block.has_key(itemname): raise TypeError, 'Non-looped item' raise KeyError, 'Item not in loop' @ Adding to a loop. We find the loop number containing the dataname that we've been passed, and then append all of the (key,values) pairs that we are passed in [[data]], which is a dictionary. We expect that the data have been sorted out for us, unlike when data are passed in [[AddCifItem]], when there can be both unlooped and looped data in one set. The dataname passed to this routine is simply a convenient way to refer to the loop, and has no other significance. <>= def AddToLoop(self,dataname,loopdata): found = 0 for aloop in self.block["loops"]: if aloop.has_key(dataname): found = 1 for itemname in loopdata.keys(): if len(loopdata[itemname])!= len(aloop[dataname]): raise CifError, 'Datalength mismatch adding to loop: %s and %s' % (itemname, dataname) aloop.update({itemname:loopdata[itemname]}) if not found: raise KeyError, 'No such looped item name: %s' % dataname @ Removing a data item. We delete the item, and if it is looped, and nothing is left in the loop, we remove that element of the list. <>= def RemoveCifItem(self,itemname): if self.block.has_key(itemname): del self.block[itemname] return for aloop in self.block["loops"]: if aloop.has_key(itemname): del aloop[itemname] self.block["loops"] = filter(None, self.block["loops"]) @ Adding a data item. We are passed a tuple with the (set) of data names at the beginning, and a (set) of values for them following. For efficiency, we want to group looped items together; so in the stored dictionary, all data items in a loop belong in one dictionary in an array of dictionaries keyed by the word "loops". If an item is already stored, it will be silently replaced; if it earlier lived in a loop, it will be removed from the loop, and the loop itself removed if it is now zero length. We check the length of the name, and give an error if the name is greater than 75 characters, which is the CIF 1.1 maximum length. We also check for consistency, by making sure the new item is not in the block already. If it is, we replace it (consistent with the meaning of square brackets). If it is in a loop, we replace the looped value and all other items in that loop block. This means that when adding loops, we must add them all at once if we call this routine directly. We typecheck the data items. They can be tuples, strings or lists. If we have a list of values for a single item, the item name should also occur in a single member tuple. <>= def AddCifItem(self,data): # we accept only tuples, strings and lists!! if not (isinstance(data[0],StringType) or isinstance(data[0],TupleType) or isinstance(data[0],ListType)): raise TypeError, 'Cif datanames are either a string, tuple or list' # now put into the dictionary properly... if isinstance(data[0],StringType): # a single name self.check_data_name(data[0]) # make sure no nasty characters # now make sure the data is OK regval = self.regularise_data(data[1]) self.check_item_value(regval) self.block.update({data[0]:regval}) # trust the data is OK for aloop in self.block["loops"]: if aloop.has_key(data[0]): del aloop[data[0]] self.block["loops"] = filter(len,self.block["loops"]) else: # we loop if(len(data[0])!=len(data[1])): raise TypeError, 'Length mismatch between itemnames and values' dellist = [] map (self.check_data_name,data[0]) for itemname in data[0]: self.block["loops"] = filter(lambda a,b=itemname:b not in a.keys(),self.block["loops"]) newdict = {} proper_vals = map(self.regularise_data,data[1]) map(self.check_item_value,proper_vals) map(lambda a,b,c=newdict:c.update({a:b}),data[0],proper_vals) self.block["loops"].append(newdict) return @ Checking the data names. The CIF 1.1 standard restricts characters in a data name to ASCII 33-126 and there should be a leading underscore. Items are allowed to have the blank characters as well, i.e. ascii 09,10,13 and 32. Data items may be lists, which we need to detect before checking. We assume that the item has been regularised before this check is called. <>= def check_data_name(self,dataname): if len(dataname) > 75: raise CifError, 'Dataname ' + dataname + ' too long.' if dataname[0]!='_': raise CifError, 'Dataname ' + dataname + ' does not begin with _' if len (filter (lambda a: ord(a) < 33 or ord(a) > 126, dataname)) > 0: raise CifError, 'Dataname ' + dataname + ' contains forbidden characters' <>= def check_item_value(self,item): test_item = item if type(item) != TupleType and type(item) != ListType: test_item = [item] #single item list def check_one (it): if type(it) == StringType: if len (filter (lambda a:ord(a) != 9 and ord(a) != 10 and ord(a) != 13 and (ord(a) < 32 or ord(a) > 126), it)) > 0: raise CifError, 'Data item ' + it + '... contains forbidden characters' map(check_one,test_item) @ Regularising data. We want the copy.deepcopy operation to work, so we can't have any arrays passed into the master dictionary. We make sure everything goes in either as a single item or as a list/tuple. <>= def regularise_data(self,dataitem): alrighttypes = [IntType, LongType, FloatType, StringType] okmappingtypes = [TupleType, ListType] thistype = type(dataitem) if thistype in alrighttypes or thistype in okmappingtypes: return dataitem # so try to make into a list try: regval = list(dataitem) except TypeError, value: raise CifError, str(dataitem) + ' is wrong type for data value\n' return regval @ Printing a section. We allow an optional order list to be given, in case the caller wants to order things in some nice way. This is overridden if any item in the order list occurs within a loop, in which case it will be packaged together with everything else within that same loop. Note that we must be careful to add spaces between data items, especially when formatting string loop data, where our string addition could get quite hairy. As we are doing so much concatenation, we use a stringIO buffer to speed it up. <>= def printsection(self,order=[]): import cStringIO import string # first make an ordering if not order: order = self.block.keys() order.sort() # now prune that ordering... order = filter(lambda a,b=self.block:b.has_key(a),order) order.remove('loops') # now do it... outstring = cStringIO.StringIO() # the returned string for itemname in order: itemvalue = self.block[itemname] if isinstance(itemvalue,StringType): thisstring = self._formatstring(itemvalue) if len(thisstring) + len(itemname) < (self.wraplength-2): outstring.write('%s %s\n' % (itemname,thisstring)) else: outstring.write('%s\n %s\n' % (itemname, thisstring)) else: if len(str(itemvalue)) + len(itemname) < (self.wraplength-2): outstring.write('%s %s\n' % (itemname, itemvalue)) else: outstring.write('%s\n %s\n' % (itemname, itemvalue)) continue #do the loops for aloop in self.block["loops"]: outstring.write('\n loop_\n') loopnames = aloop.keys() loopnames.sort() numdata = len(aloop[loopnames[0]]) for name in loopnames: outstring.write(' %-75s\n' % name) if len(aloop[name]) != numdata: raise CifError,'Loop data mismatch for ' + name + ':output aborted' curstring = '' # when adding to outstring, make sure to add spaces for position in range(numdata): for name in loopnames: # at each point, get the next data value datapoint = aloop[name][position] if isinstance(datapoint,StringType): thisstring = '%s' % (self._formatstring(datapoint)) #no spaces yet if '\n' in thisstring: # we try to wrap if the first is bigger than our wrap length if len(curstring) + string.find(thisstring,'\n') > (self.wraplength): outstring.write(' ' + curstring + '\n' + thisstring) else: outstring.write(' ' + curstring + ' ' + thisstring) #a space curstring = '' continue else: thisstring = ' %s ' % datapoint if len(curstring) + len(thisstring)> self.wraplength-2: #past end of line with space outstring.write(' ' + curstring+'\n') #add the space curstring = '' curstring = curstring + ' ' + thisstring outstring.write(' ' + curstring + '\n') #last time through curstring = '' returnstring = outstring.getvalue() outstring.close() return returnstring @ Formatting a string. We make sure that the length of the item value is less than [[self.maxoutlength]], or else we should split them, and so on. We check the value for terminators and impossible apostrophes and length, before deciding whether to print it and the item on a single line. We try to respect carriage returns in the string, if the caller has tried to do the formatting for us. If we are not putting apostrophes around a string, we make the first character a space, to avoid problems if the first character of a line is a semicolon. The CIF specification states that embedded quotes are allowed so long as they are not followed by a space. So if we find any quotes followed by spaces we output a semicolon-terminated string to avoid to much messing around. This routine is called very often and could be improved. <>= def _formatstring(self,instring): import re, string if len(instring)< (self.maxoutlength-2) and '\n' not in instring and not ('"' in instring and '\'' in instring): if not ' ' in instring and not '\t' in instring and not '\v' \ in instring: # no blanks return ' %s ' % (instring) if not "'" in instring: #use apostrophes return "'%s'" % (instring) elif not "\"" in instring: return '"%s"' % (instring) # is a long one or one that needs semicolons due to carriage returns outstring = "\n;\n" # if there are returns in the string, try to work with them while 1: retin = string.find(instring,'\n')+1 if retin < self.maxoutlength and retin > 0: # honour this break outstring = outstring + instring[:retin] instring = instring[retin:] elif len(instring)>= class CifError(Exception): def __init__(self,value): self.value = value def __str__(self): print '\nCif Format error: '+ self.value