Changeset 3814 for trunk/GSASIIfiles.py


Ignore:
Timestamp:
Feb 10, 2019 4:40:06 PM (3 years ago)
Author:
toby
Message:

refactor to move some IO-only routines; add initial image support to scriptable

File:
1 edited

Legend:

Unmodified
Added
Removed
  • trunk/GSASIIfiles.py

    r3216 r3814  
    423423                fp.close()
    424424    return exporterlist
     425
     426def readColMetadata(imagefile):
     427    '''Reads image metadata from a column-oriented metadata table
     428    (1-ID style .par file). Called by :func:`GetColumnMetadata`
     429   
     430    The .par file has any number of columns separated by spaces.
     431    The directory for the file must be specified in
     432    Config variable ``Column_Metadata_directory``.
     433    As an index to the .par file a second "label file" must be specified with the
     434    same file root name as the .par file but the extension must be .XXX_lbls (where
     435    .XXX is the extension of the image) or if that is not present extension
     436    .lbls.
     437
     438    :param str imagefile: the full name of the image file (with extension, directory optional)
     439
     440    :returns: a dict with parameter values. Named parameters will have the type based on
     441       the specified Python function, named columns will be character strings
     442   
     443    The contents of the label file will look like this::
     444   
     445        # define keywords
     446        filename:lambda x,y: "{}_{:0>6}".format(x,y)|33,34
     447        distance: float | 75
     448        wavelength:lambda keV: 12.398425/float(keV)|9
     449        pixelSize:lambda x: [74.8, 74.8]|0
     450        ISOlikeDate: lambda dow,m,d,t,y:"{}-{}-{}T{} ({})".format(y,m,d,t,dow)|0,1,2,3,4
     451        Temperature: float|53
     452        FreePrm2: int | 34 | Free Parm2 Label
     453        # define other variables
     454        0:day
     455        1:month
     456        2:date
     457        3:time
     458        4:year
     459        7:I_ring
     460
     461    This file contains three types of lines in any order.
     462     * Named parameters are evaluated with user-supplied Python code (see
     463       subsequent information). Specific named parameters are used
     464       to determine values that are used for image interpretation (see table,
     465       below). Any others are copied to the Comments subsection of the Image
     466       tree item.
     467     * Column labels are defined with a column number (integer) followed by
     468       a colon (:) and a label to be assigned to that column. All labeled
     469       columns are copied to the Image's Comments subsection.
     470     * Comments are any line that does not contain a colon.
     471
     472    Note that columns are numbered starting at zero.
     473
     474    Any named parameter may be defined provided it is not a valid integer,
     475    but the named parameters in the table have special meanings, as descibed.
     476    The parameter name is followed by a colon. After the colon, specify
     477    Python code that defines or specifies a function that will be called to
     478    generate a value for that parameter.
     479
     480    Note that several keywords, if defined in the Comments, will be found and
     481    placed in the appropriate section of the powder histogram(s)'s Sample
     482    Parameters after an integration: ``Temperature``,``Pressure``,``Time``,
     483    ``FreePrm1``,``FreePrm2``,``FreePrm3``,``Omega``,``Chi``, and ``Phi``.
     484
     485    After the Python code, supply a vertical bar (|) and then a list of one
     486    more more columns that will be supplied as arguments to that function.
     487
     488    Note that the labels for the three FreePrm items can be changed by
     489    including that label as a third item with an additional vertical bar. Labels
     490    will be ignored for any other named parameters.
     491   
     492    The examples above are discussed here:
     493
     494    ``filename:lambda x,y: "{}_{:0>6}".format(x,y)|33,34``
     495        Here the function to be used is defined with a lambda statement::
     496       
     497          lambda x,y: "{}_{:0>6}".format(x,y)
     498
     499        This function will use the format function to create a file name from the
     500        contents of columns 33 and 34. The first parameter (x, col. 33) is inserted directly into
     501        the file name, followed by a underscore (_), followed by the second parameter (y, col. 34),
     502        which will be left-padded with zeros to six characters (format directive ``:0>6``).
     503
     504        When there will be more than one image generated per line in the .par file, an alternate way to
     505        generate list of file names takes into account the number of images generated::
     506
     507          lambda x,y,z: ["{}_{:0>6}".format(x,int(y)+i) for i in range(int(z))]
     508
     509        Here a third parameter is used to specify the number of images generated, where
     510        the image number is incremented for each image.
     511         
     512    ``distance: float | 75``
     513        Here the contents of column 75 will be converted to a floating point number
     514        by calling float on it. Note that the spaces here are ignored.
     515       
     516    ``wavelength:lambda keV: 12.398425/float(keV)|9``
     517        Here we define an algebraic expression to convert an energy in keV to a
     518        wavelength and pass the contents of column 9 as that input energy
     519       
     520    ``pixelSize:lambda x: [74.8, 74.8]|0``
     521        In this case the pixel size is a constant (a list of two numbers). The first
     522        column is passed as an argument as at least one argument is required, but that
     523        value is not used in the expression.
     524
     525    ``ISOlikeDate: lambda dow,m,d,t,y:"{}-{}-{}T{} ({})".format(y,m,d,t,dow)|0,1,2,3,4``
     526        This example defines a parameter that takes items in the first five columns
     527        and formats them in a different way. This parameter is not one of the pre-defined
     528        parameter names below. Some external code could be used to change the month string
     529        (argument ``m``) to a integer from 1 to 12.
     530       
     531    ``FreePrm2: int | 34 | Free Parm2 Label``
     532        In this example, the contents of column 34 will be converted to an integer and
     533        placed as the second free-named parameter in the Sample Parameters after an
     534        integration. The label for this parameter will be changed to "Free Parm2 Label".
     535           
     536    **Pre-defined parameter names**
     537   
     538    =============  =========  ========  =====================================================
     539     keyword       required    type      Description
     540    =============  =========  ========  =====================================================
     541       filename    yes         str or   generates the file name prefix for the matching image
     542                               list     file (MyImage001 for file /tmp/MyImage001.tif) or
     543                                        a list of file names.
     544     polarization  no         float     generates the polarization expected based on the
     545                                        monochromator angle, defaults to 0.99.
     546       center      no         list of   generates the approximate beam center on the detector
     547                              2 floats  in mm, such as [204.8, 204.8].
     548       distance    yes        float     generates the distance from the sample to the detector
     549                                        in mm
     550       pixelSize   no         list of   generates the size of the pixels in microns such as
     551                              2 floats  [200.0, 200.0].
     552       wavelength  yes        float     generates the wavelength in Angstroms
     553    =============  =========  ========  =====================================================
     554   
     555    '''
     556    dir,fil = os.path.split(os.path.abspath(imagefile))
     557    imageName,ext = os.path.splitext(fil)
     558    if not GSASIIpath.GetConfigValue('Column_Metadata_directory'): return
     559    parfiles = glob.glob(os.path.join(GSASIIpath.GetConfigValue('Column_Metadata_directory'),'*.par'))
     560    if len(parfiles) == 0:
     561        print('Sorry, No Column metadata (.par) file found in '+
     562              GSASIIpath.GetConfigValue('Column_Metadata_directory'))
     563        return {}
     564    for parFil in parfiles: # loop over all .par files (hope just 1) in image dir until image is found
     565        parRoot = os.path.splitext(parFil)[0]
     566        for e in (ext+'_lbls','.lbls'):
     567            if os.path.exists(parRoot+e):
     568                lblFil = parRoot+e
     569                break
     570        else:
     571            print('Warning: No labels definitions found for '+parFil)
     572            continue
     573        labels,lbldict,keyCols,keyExp,errors = readColMetadataLabels(lblFil)
     574        if errors:
     575            print('Errors in labels file '+lblFil)
     576            for i in errors: print('  '+i)
     577            continue
     578        else:
     579            print('Read '+lblFil)
     580        # scan through each line in this .par file, looking for the matching image rootname
     581        fp = open(parFil,'Ur')
     582        for iline,line in enumerate(fp):
     583            items = line.strip().split(' ')
     584            nameList = keyExp['filename'](*[items[j] for j in keyCols['filename']])
     585            if type(nameList) is str:
     586                if nameList != imageName: continue
     587                name = nameList
     588            else:
     589                for name in nameList:
     590                    if name == imageName: break # got a match
     591                else:
     592                    continue
     593            # parse the line and finish
     594            metadata = evalColMetadataDicts(items,labels,lbldict,keyCols,keyExp)
     595            metadata['par file'] = parFil
     596            metadata['lbls file'] = lblFil
     597            print("Metadata read from {} line {}".format(parFil,iline+1))
     598            fp.close()
     599            return metadata
     600        else:
     601            print("Image {} not found in {}".format(imageName,parFil))
     602            fp.close()
     603            continue
     604        fp.close()
     605    else:
     606        print("Warning: No .par metadata for image {}".format(imageName))
     607        return {}
     608
     609def readColMetadataLabels(lblFil):
     610    '''Read the .*lbls file and setup for metadata assignments
     611    '''
     612    lbldict = {}
     613    keyExp = {}
     614    keyCols = {}
     615    labels = {}
     616    errors = []
     617    fp = open(lblFil,'Ur')         # read column labels
     618    for iline,line in enumerate(fp): # read label definitions
     619        line = line.strip()
     620        if not line or line[0] == '#': continue # comments
     621        items = line.split(':')
     622        if len(items) < 2: continue # lines with no colon are also comments
     623        # does this line a definition for a named parameter?
     624        key = items[0]
     625        try:
     626            int(key)
     627        except ValueError: # try as named parameter since not a valid number
     628            items = line.split(':',1)[1].split('|')
     629            try:
     630                f = eval(items[0]) # compile the expression
     631                if not callable(f):
     632                    errors += ['Expression "{}" for key {} is not a function (line {})'.
     633                           format(items[0],key,iline)]
     634                    continue
     635                keyExp[key] = f
     636            except Exception as msg:
     637                errors += ['Expression "{}" for key {} is not valid (line {})'.
     638                           format(items[0],key,iline)]
     639                errors += [str(msg)]
     640                continue
     641            keyCols[key] = [int(i) for i in items[1].strip().split(',')]
     642            if key.lower().startswith('freeprm') and len(items) > 2:
     643                labels[key] = items[2]
     644            continue
     645        if len(items) == 2: # simple column definition
     646            lbldict[int(items[0])] = items[1]
     647    fp.close()
     648    if 'filename' not in keyExp:
     649        errors += ["File {} is invalid. No valid filename expression.".format(lblFil)]
     650    return labels,lbldict,keyCols,keyExp,errors
     651
     652def evalColMetadataDicts(items,labels,lbldict,keyCols,keyExp,ShowError=False):
     653    '''Evaluate the metadata for a line in the .par file
     654    '''
     655    metadata = {lbldict[j]:items[j] for j in lbldict}
     656    named = {}
     657    for key in keyExp:
     658        try:
     659            res = keyExp[key](*[items[j] for j in keyCols[key]])
     660        except:
     661            if ShowError:
     662                res = "*** error ***"
     663            else:
     664                continue
     665        named[key] = res
     666    metadata.update(named)
     667    for lbl in labels: # add labels for FreePrm's
     668        metadata['label_'+lbl[4:].lower()] = labels[lbl]
     669    return metadata
     670
     671def GetColumnMetadata(reader):
     672    '''Add metadata to an image from a column-type metadata file
     673    using :func:`readColMetadata`
     674   
     675    :param reader: a reader object from reading an image
     676   
     677    '''
     678    if not GSASIIpath.GetConfigValue('Column_Metadata_directory'): return
     679    parParms = readColMetadata(reader.readfilename)
     680    if not parParms: return # check for read failure
     681    specialKeys = ('filename',"polarization", "center", "distance", "pixelSize", "wavelength",)
     682    reader.Comments = ['Metadata from {} assigned by {}'.format(parParms['par file'],parParms['lbls file'])]
     683    for key in parParms:
     684        if key in specialKeys+('par file','lbls file'): continue
     685        reader.Comments += ["{} = {}".format(key,parParms[key])]
     686    if "polarization" in parParms:
     687        reader.Data['PolaVal'][0] = parParms["polarization"]
     688    else:
     689        reader.Data['PolaVal'][0] = 0.99
     690    if "center" in parParms:
     691        reader.Data['center'] = parParms["center"]
     692    if "pixelSize" in parParms:
     693        reader.Data['pixelSize'] = parParms["pixelSize"]
     694    if "wavelength" in parParms:
     695        reader.Data['wavelength'] = parParms['wavelength']
     696    else:
     697        print('Error: wavelength not defined in {}'.format(parParms['lbls file']))
     698    if "distance" in parParms:
     699        reader.Data['distance'] = parParms['distance']
     700        reader.Data['setdist'] = parParms['distance']
     701    else:
     702        print('Error: distance not defined in {}'.format(parParms['lbls file']))
     703
     704def LoadControls(Slines,data):
     705    'Read values from a .imctrl (Image Controls) file'
     706    cntlList = ['color','wavelength','distance','tilt','invert_x','invert_y','type','Oblique',
     707        'fullIntegrate','outChannels','outAzimuths','LRazimuth','IOtth','azmthOff','DetDepth',
     708        'calibskip','pixLimit','cutoff','calibdmin','Flat Bkg','varyList','setdist',
     709        'PolaVal','SampleAbs','dark image','background image','twoth']
     710    save = {}
     711    for S in Slines:
     712        if S[0] == '#':
     713            continue
     714        [key,val] = S.strip().split(':',1)
     715        if key in ['type','calibrant','binType','SampleShape','color',]:    #strings
     716            save[key] = val
     717        elif key in ['varyList',]:
     718            save[key] = eval(val)   #dictionary
     719        elif key in ['rotation']:
     720            save[key] = float(val)
     721        elif key in ['center',]:
     722            if ',' in val:
     723                save[key] = eval(val)
     724            else:
     725                vals = val.strip('[] ').split()
     726                save[key] = [float(vals[0]),float(vals[1])]
     727        elif key in cntlList:
     728            save[key] = eval(val)
     729    data.update(save)
     730
     731def WriteControls(filename,data):
     732    'Write current values to a .imctrl (Image Controls) file'
     733    File = open(filename,'w')
     734    keys = ['type','color','wavelength','calibrant','distance','center','Oblique',
     735            'tilt','rotation','azmthOff','fullIntegrate','LRazimuth','setdist',
     736            'IOtth','outChannels','outAzimuths','invert_x','invert_y','DetDepth',
     737            'calibskip','pixLimit','cutoff','calibdmin','Flat Bkg','varyList',
     738            'binType','SampleShape','PolaVal','SampleAbs','dark image','background image',
     739            'twoth']
     740    for key in keys:
     741        if key not in data:     #uncalibrated!
     742            continue
     743        File.write(key+':'+str(data[key])+'\n')
     744    File.close()
     745   
Note: See TracChangeset for help on using the changeset viewer.