source: trunk/GSASIIobj.py

Last change on this file was 5307, checked in by vondreele, 3 months ago

add micro-ED support. New electron scattering tables, new histogram type ('SEC'), etc., etc.
Fix sign of ZERO for RMCProfile MakeInst? fxn.
Add calc of Rvalues to Error Analysis for PWDR & HKLF histograms; results are printed om console.
Add a stub for Cluster analysis in Controls GUI

  • Property svn:eol-style set to native
  • Property svn:keywords set to Date Author Revision URL Id
File size: 172.3 KB
Line 
1# -*- coding: utf-8 -*-
2#GSASIIobj - data objects for GSAS-II
3########### SVN repository information ###################
4# $Date: 2022-07-07 17:02:04 +0000 (Thu, 07 Jul 2022) $
5# $Author: vondreele $
6# $Revision: 5307 $
7# $URL: trunk/GSASIIobj.py $
8# $Id: GSASIIobj.py 5307 2022-07-07 17:02:04Z vondreele $
9########### SVN repository information ###################
10
11'''
12*GSASIIobj: Data objects*
13=========================
14
15This module defines and/or documents the data structures used in GSAS-II, as well
16as provides misc. support routines.
17
18.. Next command allows \\AA to be used in HTML
19
20.. only:: html
21
22   :math:`\\require{mediawiki-texvc}`
23
24
25.. _VarNames_table:
26
27.. index::
28   single: Parameter names
29   single: GSAS-II variable naming
30
31Variable names in GSAS-II
32----------------------------
33
34Parameter are named using the following pattern,
35``p:h:<var>:n``, where ``<var>`` is a variable name, as shown in the following table. Also,
36``p`` is the phase number, ``h`` is the histogram number,
37and ``n`` is the atom parameter number
38If a parameter does not depend on a histogram, phase or atom, ``h``, ``p`` and/or ``n`` will be omitted,
39so ``p::<var>:n``, ``:h:<var>`` and ``p:h:<var>`` are all valid names.
40
41.. include:: vars.rst
42
43.. _Constraints_table:
44
45.. index::
46   single: Constraints object description
47   single: Data object descriptions; Constraints
48
49Constraints Tree Item
50----------------------
51
52Constraints are stored in a dict, separated into groups.
53Note that parameter are named in the following pattern,
54p:h:<var>:n, where p is the phase number, h is the histogram number
55<var> is a variable name and n is the parameter number.
56If a parameter does not depend on a histogram or phase or is unnumbered, that
57number is omitted.
58Note that the contents of each dict item is a List where each element in the
59list is a :ref:`constraint definition objects <Constraint_definitions_table>`.
60The constraints in this form are converted in
61:func:`GSASIImapvars.ProcessConstraints` to the form used in :mod:`GSASIImapvars`
62
63The keys in the Constraints dict are:
64
65.. tabularcolumns:: |l|p{4.5in}|
66
67==========  ====================================================
68  key         explanation
69==========  ====================================================
70Hist        This specifies a list of constraints on
71            histogram-related parameters,
72            which will be of form :h:<var>:n.
73HAP         This specifies a list of constraints on parameters
74            that are defined for every histogram in each phase
75            and are of form p:h:<var>:n.
76Phase       This specifies a list of constraints on phase
77            parameters,
78            which will be of form p::<var>:n.
79Global      This specifies a list of constraints on parameters
80            that are not tied to a histogram or phase and
81            are of form ::<var>:n
82==========  ====================================================
83
84.. _Constraint_definitions_table:
85
86.. index::
87   single: Constraint definition object description
88   single: Data object descriptions; Constraint Definition
89
90Each constraint is defined as an item in a list. Each constraint is of form::
91
92[[<mult1>, <var1>], [<mult2>, <var2>],..., <fixedval>, <varyflag>, <constype>]
93
94Where the variable pair list item containing two values [<mult>, <var>], where:
95
96  * <mult> is a multiplier for the constraint (float)
97  * <var> a :class:`G2VarObj` object. (Note that in very old .gpx files this might be a str with a variable name of form 'p:h:name[:at]')
98
99Note that the last three items in the list play a special role:
100
101 * <fixedval> is the fixed value for a `constant equation` (``constype=c``)
102   constraint or is None. For a `New variable` (``constype=f``) constraint,
103   a variable name can be specified as a str (used for externally
104   generated constraints)
105 * <varyflag> is True or False for `New variable` (``constype=f``) constraints
106   or is None. This indicates if this variable should be refined.
107 * <constype> is one of four letters, 'e', 'c', 'h', 'f' that determines the type of constraint:
108
109    * 'e' defines a set of equivalent variables. Only the first variable is refined (if the
110      appropriate refine flag is set) and and all other equivalent variables in the list
111      are generated from that variable, using the appropriate multipliers.
112    * 'c' defines a constraint equation of form,
113      :math:`m_1 \\times var_1 + m_2 \\times var_2 + ... = c`
114    * 'h' defines a variable to hold (not vary). Any variable on this list is not varied,
115      even if its refinement flag is set. Only one [mult,var] pair is allowed in a hold
116      constraint and the mult value is ignored.
117      This is of particular value when needing to hold one or more variables where a
118      single flag controls a set of variables such as, coordinates,
119      the reciprocal metric tensor or anisotropic displacement parameter.
120    * 'f' defines a new variable (function) according to relationship
121      :math:`newvar = m_1 \\times var_1 + m_2 \\times var_2 + ...`
122
123.. _Covariance_table:
124
125.. index::
126   single: Covariance description
127   single: Data object descriptions; Covariance
128
129Covariance Tree Item
130--------------------
131
132The Covariance tree item has results from the last least-squares run. They
133are stored in a dict with these keys:
134
135.. tabularcolumns:: |l|l|p{4in}|
136
137=============  ===============  ====================================================
138  key            sub-key        explanation
139=============  ===============  ====================================================
140newCellDict    \\                (dict) ith lattice parameters computed by
141                                :func:`GSASIIstrMath.GetNewCellParms`
142title          \\                (str) Name of gpx file(?)
143variables      \\                (list) Values for all N refined variables
144                                (list of float values, length N,
145                                ordered to match varyList)
146sig            \\                (list) Uncertainty values for all N refined variables
147                                (list of float values, length N,
148                                ordered to match varyList)
149varyList       \\                (list of str values, length N) List of directly refined variables
150                               
151newAtomDict    \\                (dict) atom position values computed in
152                                :func:`GSASIIstrMath.ApplyXYZshifts`
153Rvals          \\                (dict) R-factors, GOF, Marquardt value for last
154                                refinement cycle
155\\              Nobs             (int) Number of observed data points
156\\              Rwp              (float) overall weighted profile R-factor (%)
157\\              chisq            (float) :math:`\\sum w*(I_{obs}-I_{calc})^2`                               
158                                for all data.
159                                Note: this is not the reduced :math:`\\chi^2`.
160\\              lamMax           (float) Marquardt value applied to Hessian diagonal
161\\              GOF              (float) The goodness-of-fit, aka square root of
162                                the reduced chi squared.
163covMatrix      \\                (np.array) The (NxN) covVariance matrix
164=============  ===============  ====================================================
165
166.. _Phase_table:
167
168.. index::
169   single: Phase object description
170   single: Data object descriptions; Phase
171
172Phase Tree Items
173----------------
174
175Phase information is stored in the GSAS-II data tree as children of the
176Phases item in a dict with keys:
177
178.. tabularcolumns:: |l|l|p{4in}|
179
180==========  ===============     =====================================================================================================
181  key         sub-key           explanation
182==========  ===============     =====================================================================================================
183General         \\               (dict) Overall information for the phase
184  \\         3Dproj              (list of str) projections for 3D pole distribution plots
185  \\         AngleRadii          (list of floats) Default radius for each atom used to compute
186                                interatomic angles
187  \\         AtomMass            (list of floats) Masses for atoms
188  \\         AtomPtrs            (list of int) four locations (cx,ct,cs & cu) to use to pull info
189                                from the atom records
190  \\         AtomTypes           (llist of str) Atom types
191  \\         BondRadii           (list of floats) Default radius for each atom used to compute
192                                interatomic distances
193  \\         Cell                Unit cell parameters & ref. flag
194                                (list with 8 items. All but first item are float.)
195
196                                 0: cell refinement flag (True/False),
197
198                                 1-3: a, b, c, (:math:`\\AA`)
199
200                                 4-6: alpha, beta & gamma, (degrees)
201
202                                 7: volume (:math:`\\AA^3`)
203
204  \\         Color               (list of (r,b,g) triplets) Colors for atoms
205  \\         Compare             (dict) Polygon comparison parameters
206  \\         Data plot type      (str) data plot type ('Mustrain', 'Size' or
207                                'Preferred orientation') for powder data
208  \\         DisAglCtls          (dDict) with distance/angle search controls,
209                                which has keys 'Name', 'AtomTypes',
210                                'BondRadii', 'AngleRadii' which are as above
211                                except are possibly edited. Also contains
212                                'Factors', which is a 2 element list with
213                                a multiplier for bond and angle search range
214                                [typically (0.85,0.85)].
215  \\         F000X               (float) x-ray F(000) intensity
216  \\         F000N               (float) neutron F(000) intensity
217  \\         Flip                (dict) Charge flip controls
218  \\         HydIds              (dict) geometrically generated hydrogen atoms
219  \\         Isotope             (dict) Isotopes for each atom type
220  \\         Isotopes            (dict) Scattering lengths for each isotope
221                                combination for each element in phase
222  \\         MCSA controls       (dict) Monte Carlo-Simulated Annealing controls
223  \\         Map                 (dict) Map parameters
224  \\         Mass                (float) Mass of unit cell contents in g/mol
225  \\         Modulated           (bool) True if phase modulated
226  \\         Mydir               (str) Directory of current .gpx file
227  \\         Name                (str) Phase name
228  \\         NoAtoms             (dict) Number of atoms per unit cell of each type
229  \\         POhkl               (list) March-Dollase preferred orientation direction
230  \\         Pawley dmin         (float) maximum Q (as d-space) to use for Pawley extraction
231  \\         Pawley dmax         (float) minimum Q (as d-space) to use for Pawley extraction
232  \\         Pawley neg wt       (float) Restraint value for negative Pawley intensities
233  \\         SGData              (object) Space group details as a
234                                :ref:`space group (SGData) <SGData_table>`
235                                object, as defined in :func:`GSASIIspc.SpcGroup`.
236  \\         SH Texture          (dict) Spherical harmonic preferred orientation parameters
237  \\         Super               (int) dimension of super group (0,1 only)
238  \\         Type                (str) phase type (e.g. 'nuclear')
239  \\         Z                   (dict) Atomic numbers for each atom type
240  \\         doDysnomia          (bool) flag for max ent map modification via Dysnomia
241  \\         doPawley            (bool) Flag for Pawley intensity extraction
242  \\         vdWRadii            (dict) Van der Waals radii for each atom type
243ranId           \\               (int) unique random number Id for phase
244pId             \\               (int) Phase Id number for current project.
245Atoms           \\               (list of lists) Atoms in phase as a list of lists. The outer list
246                                is for each atom, the inner list contains varying
247                                items depending on the type of phase, see
248                                the :ref:`Atom Records <Atoms_table>` description.
249Drawing         \\               (dict) Display parameters
250\\           Atoms               (list of lists) with an entry for each atom that is drawn
251\\           Plane               (list) Controls for contour density plane display
252\\           Quaternion          (4 element np.array) Viewing quaternion
253\\           Zclip               (float) clipping distance in :math:`\\AA`
254\\           Zstep               (float) Step to de/increase Z-clip
255\\           atomPtrs            (list) positions of x, type, site sym, ADP flag in Draw Atoms
256\\           backColor           (list) background for plot as and R,G,B triplet
257                                (default = [0, 0, 0], black).
258\\           ballScale           (float) Radius of spheres in ball-and-stick display
259\\           bondList            (dict) Bonds
260\\           bondRadius          (float) Radius of binds in :math:`\\AA`
261\\           cameraPos           (float) Viewing position in :math:`\\AA` for plot
262\\           contourLevel        (float) map contour level in :math:`e/\\AA^3`
263\\           contourMax          (float) map contour maximum
264\\           depthFog            (bool) True if use depthFog on plot - set currently as False
265\\           ellipseProb         (float) Probability limit for display of thermal
266                                ellipsoids in % .
267\\           magMult             (float) multiplier for magnetic moment arrows
268\\           mapSize             (float) x & y dimensions of contourmap (fixed internally)
269\\           modelView           (4,4 array) from openGL drawing transofmation matrix
270\\           oldxy               (list with two floats) previous view point
271\\           radiusFactor        (float) Distance ratio for searching for bonds. Bonds
272                                are located that are within r(Ra+Rb) and (Ra+Rb)/r
273                                where Ra and Rb are the atomic radii.
274\\           selectedAtoms       (list of int values) List of selected atoms
275\\           showABC             (bool) Flag to show view point triplet. True=show.
276\\           showHydrogen        (bool) Flag to control plotting of H atoms.
277\\           showRigidBodies     (bool) Flag to highlight rigid body placement
278\\           showSlice           (bool) flag to show contour map
279\\           sizeH               (float) Size ratio for H atoms
280\\           unitCellBox         (bool) Flag to control display of the unit cell.
281\\           vdwScale            (float) Multiplier of van der Waals radius for display of vdW spheres.
282\\           viewDir             (np.array with three floats) cartesian viewing direction
283\\           viewPoint           (list of lists) First item in list is [x,y,z]
284                                in fractional coordinates for the center of
285                                the plot. Second item list of previous & current
286                                atom number viewed (may be [0,0])
287ISODISTORT      \\               (dict) contains controls for running ISODISTORT and results from it
288\\           ISOmethod           (int) ISODISTORT method (currently 1 or 4; 2 & 3 not implemented in GSAS-II)
289\\           ParentCIF           (str) parent cif file name for ISODISTORT method 4
290\\           ChildCIF            (str) child cif file name for ISODISTORT method 4
291\\           SGselect            (dict) selection list for lattice types in radio result from ISODISTORT method 1
292\\           selection           (int) chosen selection from radio
293\\           radio               (list) results from ISODISTORT method 1
294\\           ChildMatrix         (3x3 array) transformation matrix for method 3 (not currently used)
295\\           ChildSprGp          (str) child space group for method 3 (not currently used)
296\\           ChildCell           (str) cell ordering for nonstandard orthorhombic ChildSprGrp in method 3 (not currently used)
297\\           G2ModeList          (list) ISODISTORT mode names
298\\           modeDispl           (list) distortion mode values; refinable parameters
299\\           ISOmodeDispl        (list) distortion mode values as determined in method 4 by ISODISTORT
300\\           NormList            (list) ISODISTORT normalization values; to convert mode value to fractional coordinate dsplacement
301\\           G2parentCoords      (list) full set of parent structure coordinates transformed to child structure; starting basis for mode displacements
302\\           G2VarList           (list)
303\\           IsoVarList          (list)
304\\           G2coordOffset       (list) only adjustible set of parent structure coordinates
305\\           G2OccVarList        (list)
306\\           Var2ModeMatrix      (array) atom variable to distortion mode transformation
307\\           Mode2VarMatrix      (array) distortion mode to atom variable transformation
308\\           rundata             (dict) saved input information for use by ISODISTORT method 1
309
310RBModels        \\               Rigid body assignments (note Rigid body definitions
311                                are stored in their own main top-level tree entry.)
312RMC             \\               (dict) RMCProfile, PDFfit & fullrmc controls
313Pawley ref      \\               (list) Pawley reflections
314Histograms      \\               (dict of dicts) The key for the outer dict is
315                                the histograms tied to this phase. The inner
316                                dict contains the combined phase/histogram
317                                parameters for items such as scale factors,
318                                size and strain parameters. The following are the
319                                keys to the inner dict. (dict)
320\\           Babinet             (dict) For protein crystallography. Dictionary with two
321                                entries, 'BabA', 'BabU'
322\\           Extinction          (list of float, bool) Extinction parameter
323\\           Flack               (list of [float, bool]) Flack parameter & refine flag
324\\           HStrain             (list of two lists) Hydrostatic strain. The first is
325                                a list of the HStrain parameters (1, 2, 3, 4, or 6
326                                depending on unit cell), the second is a list of boolean
327                                refinement parameters (same length)
328\\           Histogram           (str) The name of the associated histogram
329\\           Layer Disp          (list of [float, bool]) Layer displacement in beam direction & refine flag
330\\           LeBail              (bool) Flag for LeBail extraction
331\\           Mustrain            (list) Microstrain parameters, in order:
332   
333                                0. Type, one of  u'isotropic', u'uniaxial', u'generalized'
334                                1. Isotropic/uniaxial parameters - list of 3 floats
335                                2. Refinement flags - list of 3 bools
336                                3. Microstrain axis - list of 3 ints, [h, k, l]
337                                4. Generalized mustrain parameters - list of 2-6 floats, depending on space group
338                                5. Generalized refinement flags - list of bools, corresponding to the parameters of (4)
339\\           Pref.Ori.           (list) Preferred Orientation. List of eight parameters.
340                                Items marked SH are only used for Spherical Harmonics.
341                               
342                                0. (str) Type, 'MD' for March-Dollase or 'SH' for Spherical Harmonics
343                                1. (float) Value
344                                2. (bool) Refinement flag
345                                3. (list) Preferred direction, list of ints, [h, k, l]
346                                4. (int) SH - number of terms
347                                5. (dict) SH - 
348                                6. (list) SH
349                                7. (float) SH
350\\           Scale               (list of [float, bool]) Phase fraction & refine flag
351\\           Size                List of crystallite size parameters, in order:
352
353                                0. (str) Type, one of  u'isotropic', u'uniaxial', u'ellipsoidal'
354                                1. (list) Isotropic/uniaxial parameters - list of 3 floats
355                                2. (list) Refinement flags - list of 3 bools
356                                3. (list) Size axis - list of 3 ints, [h, k, l]
357                                4. (list) Ellipsoidal size parameters - list of 6 floats
358                                5. (list) Ellipsoidal refinement flags - list of bools, corresponding to the parameters of (4)
359\\           Use                 (bool) True if this histogram is to be used in refinement
360MCSA            \\               (dict) Monte-Carlo simulated annealing parameters
361==========  ===============     =====================================================================================================
362
363.. _RBData_table:
364
365.. index::
366   single: Rigid Body Data description
367   single: Data object descriptions; Rigid Body Data
368
369Rigid Body Objects
370------------------
371
372Rigid body descriptions are available for two types of rigid bodies: 'Vector'
373and 'Residue'. Vector rigid bodies are developed by a sequence of translations each
374with a refinable magnitude and Residue rigid bodies are described as Cartesian coordinates
375with defined refinable torsion angles.
376
377.. tabularcolumns:: |l|l|p{4in}|
378
379==========  ===============     ====================================================
380  key         sub-key           explanation
381==========  ===============     ====================================================
382Vector      RBId                (dict of dict) vector rigid bodies
383\\           AtInfo              (dict) Drad, Color: atom drawing radius & color for each atom type
384\\           RBname              (str) Name assigned by user to rigid body
385\\           VectMag             (list) vector magnitudes in :math:`\\AA`
386\\           rbXYZ               (list of 3 float Cartesian coordinates for Vector rigid body )
387\\           rbRef               (list of 3 int & 1 bool) 3 assigned reference atom nos. in rigid body for origin
388                                definition, use center of atoms flag
389\\           VectRef             (list of bool refinement flags for VectMag values )
390\\           rbTypes             (list of str) Atom types for each atom in rigid body
391\\           rbVect              (list of lists) Cartesian vectors for each translation used to build rigid body
392\\           useCount            (int) Number of times rigid body is used in any structure
393Residue     RBId                (dict of dict) residue rigid bodies
394\\           AtInfo              (dict) Drad, Color: atom drawing radius & color for each atom type
395\\           RBname              (str) Name assigned by user to rigid body
396\\           rbXYZ               (list of 3 float) Cartesian coordinates for Residue rigid body
397\\           rbTypes             (list of str) Atom types for each atom in rigid body
398\\           atNames             (list of str) Names of each atom in rigid body (e.g. C1,N2...)
399\\           rbRef               (list of 3 int & 1 bool) 3 assigned reference atom nos. in rigid body for origin
400                                definition, use center of atoms flag
401\\           rbSeq               (list) Orig,Piv,angle,Riding : definition of internal rigid body
402                                torsion; origin atom (int), pivot atom (int), torsion angle (float),
403                                riding atoms (list of int)
404\\           SelSeq              (int,int) used by SeqSizer to identify objects
405\\           useCount            (int)Number of times rigid body is used in any structure
406RBIds           \\               (dict) unique Ids generated upon creation of each rigid body
407\\           Vector              (list) Ids for each Vector rigid body
408\\           Residue             (list) Ids for each Residue rigid body
409==========  ===============     ====================================================
410
411.. _SGData_table:
412
413.. index::
414   single: Space Group Data description
415   single: Data object descriptions; Space Group Data
416
417Space Group Objects
418-------------------
419
420Space groups are interpreted by :func:`GSASIIspc.SpcGroup`
421and the information is placed in a SGdata object
422which is a dict with these keys. Magnetic ones are marked "mag"
423
424.. tabularcolumns:: |l|p{4.5in}|
425
426==========  ========================================================================================
427  key         explanation
428==========  ========================================================================================
429BNSlattsym  mag - (str) BNS magnetic space group symbol and centering vector
430GenFlg      mag - (list) symmetry generators indices
431GenSym      mag - (list) names for each generator
432MagMom      mag - (list) "time reversals" for each magnetic operator
433MagPtGp     mag - (str) Magnetic point group symbol
434MagSpGrp    mag - (str) Magnetic space group symbol
435OprNames    mag - (list) names for each space group operation
436SGCen       (np.array) Symmetry cell centering vectors. A (n,3) np.array
437            of centers. Will always have at least one row: ``np.array([[0, 0, 0]])``
438SGFixed     (bool) Only True if phase mported from a magnetic cif file
439            then the space group can not be changed by the user because
440            operator set from cif may be nonstandard
441SGGen       (list) generators
442SGGray      (bool) True if space group is a gray group (incommensurate magnetic structures)
443SGInv       (bool) True if centrosymmetric, False if not
444SGLatt      (str)Lattice centering type. Will be one of
445            P, A, B, C, I, F, R
446SGLaue      (str) one of the following 14 Laue classes:
447            -1, 2/m, mmm, 4/m, 4/mmm, 3R,
448            3mR, 3, 3m1, 31m, 6/m, 6/mmm, m3, m3m
449SGOps       (list) symmetry operations as a list of form
450            ``[[M1,T1], [M2,T2],...]``
451            where :math:`M_n` is a 3x3 np.array
452            and :math:`T_n` is a length 3 np.array.
453            Atom coordinates are transformed where the
454            Asymmetric unit coordinates [X is (x,y,z)]
455            are transformed using
456            :math:`X^\\prime = M_n*X+T_n`
457SGPolax     (str) Axes for space group polarity. Will be one of
458            '', 'x', 'y', 'x y', 'z', 'x z', 'y z',
459            'xyz'. In the case where axes are arbitrary
460            '111' is used (P 1, and ?).
461SGPtGrp     (str) Point group of the space group
462SGUniq      unique axis if monoclinic. Will be
463            a, b, or c for monoclinic space groups.
464            Will be blank for non-monoclinic.
465SGSpin      mag - (list) of spin flip operatiors (+1 or -1) for the space group operations
466SGSys       (str) symmetry unit cell: type one of
467            'triclinic', 'monoclinic', 'orthorhombic',
468            'tetragonal', 'rhombohedral', 'trigonal',
469            'hexagonal', 'cubic'
470SSGK1       (list) Superspace multipliers
471SpGrp       (str) space group symbol
472SpnFlp      mag - (list) Magnetic spin flips for every magnetic space group operator
473==========  ========================================================================================
474
475.. _SSGData_table:
476
477.. index::
478   single: Superspace Group Data description
479   single: Data object descriptions; Superspace Group Data
480
481Superspace groups [3+1] are interpreted by :func:`GSASIIspc.SSpcGroup`
482and the information is placed in a SSGdata object
483which is a dict with these keys:
484
485.. tabularcolumns:: |l|p{4.5in}|
486
487==========  ====================================================
488  key         explanation
489==========  ====================================================
490SSGCen      (list) 4D cell centering vectors [0,0,0,0] at least
491SSGK1       (list) Superspace multipliers
492SSGOps      (list) 4D symmetry operations as [M,T] so that M*x+T = x'
493SSpGrp      (str) superspace group symbol extension to space group
494            symbol, accidental spaces removed
495modQ        (list) modulation/propagation vector
496modSymb     (list of str) Modulation symbols
497==========  ====================================================
498
499
500Phase Information
501--------------------
502
503.. index::
504   single: Phase information record description
505
506Phase information is placed in one of the following keys:
507
508.. tabularcolumns:: |l|p{4.5in}|
509
510==========  ==============================================================
511  key         explanation
512==========  ==============================================================
513General       Overall information about a phase
514Histograms    Information about each histogram linked to the
515              current phase as well as parameters that
516              are defined for each histogram and phase
517              (such as sample peak widths and preferred
518              orientation parameters.
519Atoms         Contains a list of atoms, as described in the
520              :ref:`Atom Records <Atoms_table>` description.
521Drawing       Parameters that determine how the phase is
522              displayed, including a list of atoms to be
523              included, as described in the
524              :ref:`Drawing Atom Records <Drawing_atoms_table>`
525              description
526MCSA          Monte-Carlo simulated annealing parameters
527pId           The index of each phase in the project, numbered
528              starting at 0
529ranId         An int value with a unique value for each phase
530RBModels      A list of dicts with parameters for each
531              rigid body inserted into the current phase,
532              as defined in the
533              :ref:`Rigid Body Insertions <Rigid_Body_Insertions>`.
534              Note that the rigid bodies are defined as
535              :ref:`Rigid Body Objects <RBData_table>`
536RMC           PDF modeling parameters
537Pawley ref    Pawley refinement parameters
538
539==========  ==============================================================
540
541.. _Atoms_table:
542
543.. index::
544   single: Atoms record description
545   single: Data object descriptions; Atoms record
546
547--------------------
548Atom Records
549--------------------
550
551If ``phasedict`` points to the phase information in the data tree, then
552atoms are contained in a list of atom records (list) in
553``phasedict['Atoms']``. Also needed to read atom information
554are four pointers, ``cx,ct,cs,cia = phasedict['General']['AtomPtrs']``,
555which define locations in the atom record, as shown below. Items shown are
556always present; additional ones for macromolecular phases are marked 'mm',
557and those for magnetic structures are marked 'mg'
558
559.. tabularcolumns:: |l|p{4.5in}|
560
561==============      ====================================================
562location            explanation
563==============      ====================================================
564ct-4                mm - (str) residue number
565ct-3                mm - (str) residue name (e.g. ALA)
566ct-2                mm - (str) chain label
567ct-1                (str) atom label
568ct                  (str) atom type
569ct+1                (str) refinement flags; combination of 'F', 'X', 'U', 'M'
570cx,cx+1,cx+2        (3 floats) the x,y and z coordinates
571cx+3                (float) site occupancy
572cx+4,cx+5,cx+6      mg - (list) atom magnetic moment along a,b,c in Bohr magnetons
573cs                  (str) site symmetry
574cs+1                (int) site multiplicity
575cia                 (str) ADP flag: Isotropic ('I') or Anisotropic ('A')
576cia+1               (float) Uiso
577cia+2...cia+7       (6 floats) U11, U22, U33, U12, U13, U23
578atom[cia+8]         (int) unique atom identifier
579
580==============      ====================================================
581
582.. _Drawing_atoms_table:
583
584.. index::
585   single: Drawing atoms record description
586   single: Data object descriptions; Drawing atoms record
587
588----------------------------
589Drawing Atom Records
590----------------------------
591
592If ``phasedict`` points to the phase information in the data tree, then
593drawing atoms are contained in a list of drawing atom records (list) in
594``phasedict['Drawing']['Atoms']``. Also needed to read atom information
595are four pointers, ``cx,ct,cs,ci = phasedict['Drawing']['AtomPtrs']``,
596which define locations in the atom record, as shown below. Items shown are
597always present; additional ones for macromolecular phases are marked 'mm',
598and those for magnetic structures are marked 'mg'
599
600.. tabularcolumns:: |l|p{4.5in}|
601
602==============   ===================================================================================
603location            explanation
604==============   ===================================================================================
605ct-4                mm - (str) residue number
606ct-3                mm - (str) residue name (e.g. ALA)
607ct-2                mm - (str) chain label
608ct-1                (str) atom label
609ct                  (str) atom type
610cx,cx+1,cx+2        (3 floats) the x,y and z coordinates
611cx+3,cx+4,cx+5      mg - (3 floats) atom magnetic moment along a,b,c in Bohr magnetons
612cs-1                (str) Sym Op symbol; sym. op number + unit cell id (e.g. '1,0,-1')
613cs                  (str) atom drawing style; e.g. 'balls & sticks'
614cs+1                (str) atom label style (e.g. 'name')
615cs+2                (int) atom color (RBG triplet)
616cs+3                (str) ADP flag: Isotropic ('I') or Anisotropic ('A')
617cs+4                (float) Uiso
618cs+5...cs+11        (6 floats) U11, U22, U33, U12, U13, U23
619ci                  (int) unique atom identifier; matches source atom Id in Atom Records
620==============   ===================================================================================
621
622.. _Rigid_Body_Insertions:
623
624----------------------------
625Rigid Body Insertions
626----------------------------
627
628If ``phasedict`` points to the phase information in the data tree, then
629rigid body information is contained in list(s) in
630``phasedict['RBModels']['Residue']`` and/or ``phasedict['RBModels']['Vector']``
631for each rigid body inserted into the current phase.
632
633.. tabularcolumns:: |l|p{4.5in}|
634
635==============   ===================================================================================
636key              explanation
637==============   ===================================================================================
638fixOrig           Should the origin be fixed (when editing, not the refinement flag)
639Ids               Ids for assignment of atoms in the rigid body
640numChain          Chain number for macromolecular fits
641Orient            Orientation of the RB as a quaternion and a refinement flag (' ', 'A' or 'AV')
642OrientVec         Orientation of the RB expressed as a vector and azimuthal rotation angle
643Orig              Origin of the RB in fractional coordinates and refinement flag (bool)
644RBId              References the unique ID of a rigid body in the
645                  :ref:`Rigid Body Objects <RBData_table>`
646RBname            The name for the rigid body (str)
647AtomFrac          The atom fractions for the rigid body
648ThermalMotion     The thermal motion description for the rigid body, which includes a choice for
649                  the model and can include TLS parameters or an overall Uiso value.
650Torsions          Defines the torsion angle and refinement flag for each torsion defined in
651                  the :ref:`Rigid Body Object <RBData_table>`
652==============   ===================================================================================
653
654.. _Powder_table:
655
656.. index::
657   single: Powder data object description
658   single: Data object descriptions; Powder Data
659
660Powder Diffraction Tree Items
661-----------------------------
662
663Every powder diffraction histogram is stored in the GSAS-II data tree
664with a top-level entry named beginning with the string "PWDR ". The
665diffraction data for that information are directly associated with
666that tree item and there are a series of children to that item. The
667routines :func:`GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
668and :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` will
669load this information into a dictionary where the child tree name is
670used as a key, and the information in the main entry is assigned
671a key of ``Data``, as outlined below.
672
673.. tabularcolumns:: |p{1in}|p{1in}|p{4in}|
674
675======================     ===============  ===========================================================
676  key                       sub-key          explanation
677======================     ===============  ===========================================================
678Comments                    \\               (list of str) Text strings extracted from the original powder
679                                            data header. These cannot be changed by the user;
680                                            it may be empty.
681Limits                      \\               (list) two two element lists, as [[Ld,Hd],[L,H]]
682                                            where L and Ld are the current and default lowest
683                                            two-theta value to be used and
684                                            where H and Hd are the current and default highest
685                                            two-theta value to be used.
686Reflection Lists            \\               (dict of dicts) with an entry for each phase in the
687                                            histogram. The contents of each dict item
688                                            is a dict containing reflections, as described in
689                                            the :ref:`Powder Reflections <PowderRefl_table>`
690                                            description.
691Instrument Parameters       \\               (dict) The instrument parameters uses different dicts
692                                            for the constant wavelength (CW) and time-of-flight (TOF)
693                                            cases. See below for the descriptions of each.
694wtFactor                    \\               (float) A weighting factor to increase or decrease
695                                            the leverage of data in the histogram .
696                                            A value of 1.0 weights the data with their
697                                            standard uncertainties and a larger value
698                                            increases the weighting of the data (equivalent
699                                            to decreasing the uncertainties).
700Sample Parameters           \\               (dict) Parameters that describe how
701                                            the data were collected, as listed
702                                            below. Refinable parameters are a list containing
703                                            a float and a bool, where the second value
704                                            specifies if the value is refined, otherwise
705                                            the value is a float unless otherwise noted.
706\\                           Scale           The histogram scale factor (refinable)
707\\                           Absorption      The sample absorption coefficient as
708                                            :math:`\\mu r` where r is the radius
709                                            (refinable). Only valid for Debye-Scherrer geometry.
710\\                           SurfaceRoughA   Surface roughness parameter A as defined by
711                                            Surotti, *J. Appl. Cryst*, **5**, 325-331, 1972.
712                                            (refinable - only valid for Bragg-Brentano geometry)
713\\                           SurfaceRoughB   Surface roughness parameter B (refinable -
714                                            only valid for Bragg-Brentano geometry)
715\\                           DisplaceX,      Sample displacement from goniometer center
716                            DisplaceY       where Y is along the beam direction and
717                                            X is perpendicular. Units are :math:`\\mu m`
718                                            (refinable).
719\\                           Phi, Chi,       Goniometer sample setting angles, in degrees.
720                            Omega
721\\                           Gonio. radius   Radius of the diffractometer in mm
722\\                           InstrName       (str) A name for the instrument, used in preparing
723                                            a CIF .
724\\                           Force,          Variables that describe how the measurement
725                            Temperature,    was performed. Not used directly in
726                            Humidity,       any computations.
727                            Pressure,
728                            Voltage
729\\                           ranId           (int) The random-number Id for the histogram
730                                            (same value as where top-level key is ranId)
731\\                           Type            (str) Type of diffraction data, may be 'Debye-Scherrer'
732                                            or 'Bragg-Brentano' .
733hId                         \\               (int) The number assigned to the histogram when
734                                            the project is loaded or edited (can change)
735ranId                       \\               (int) A random number id for the histogram
736                                            that does not change
737Background                  \\               (list) The background is stored as a list with where
738                                            the first item in the list is list and the second
739                                            item is a dict. The list contains the background
740                                            function and its coefficients; the dict contains
741                                            Debye diffuse terms and background peaks.
742                                            (TODO: this needs to be expanded.)
743Data                        \\               (list) The data consist of a list of 6 np.arrays
744                                            containing in order:
745
746                                            0. the x-postions (two-theta in degrees),
747                                            1. the intensity values (Yobs),
748                                            2. the weights for each Yobs value
749                                            3. the computed intensity values (Ycalc)
750                                            4. the background values
751                                            5. Yobs-Ycalc
752======================     ===============  ===========================================================
753
754.. _CWPowder_table:
755
756.. index::
757   single: Powder data CW Instrument Parameters
758
759-----------------------------
760CW Instrument Parameters
761-----------------------------
762
763Instrument Parameters are placed in a list of two dicts,
764where the keys in the first dict are listed below. Note that the dict contents are different for
765constant wavelength (CW) vs. time-of-flight (TOF) histograms.
766The value for each item is a list containing three values: the initial value, the current value
767and a refinement flag which can have a value of True, False or 0 where 0 indicates a value that
768cannot be refined. The first and second values are floats unless otherwise noted.
769Items not refined are noted as [*]
770
771.. tabularcolumns:: |l|p{1in}|p{4in}|
772
773========================    ===============  ===========================================================
774  key                       sub-key           explanation
775========================    ===============  ===========================================================
776Instrument Parameters[0]    Type [*]            (str) Histogram type:
777                                                * 'PXC' for constant wavelength x-ray
778                                                * 'PNC' for constant wavelength neutron
779\\                           Bank [*]            (int) Data set number in a multidata file (usually 1)
780\\                           Lam                 (float) Specifies a wavelength in :math:`\\AA`
781\\                           Lam1 [*]            (float) Specifies the primary wavelength in
782                                                :math:`\\AA`, used in place of Lam
783                                                when an :math:`\\alpha_1, \\alpha_2`
784                                                source is used.
785\\                           Lam2 [*]            (float) Specifies the secondary wavelength in
786                                                :math:`\\AA`, used with Lam1
787\\                           I(L2)/I(L1)         (float) Ratio of Lam2 to Lam1, used with Lam1
788\\                           Zero                (float) Two-theta zero correction in *degrees*
789\\                           Azimuth [*]         (float) Azimuthal setting angle for data recorded with differing setting angles
790\\                           U, V, W             (float) Cagliotti profile coefficients
791                                                for Gaussian instrumental broadening, where the
792                                                FWHM goes as
793                                                :math:`U \\tan^2\\theta + V \\tan\\theta + W`
794\\                           X, Y, Z             (float) Cauchy (Lorentzian) instrumental broadening coefficients
795\\                           SH/L                (float) Variant of the Finger-Cox-Jephcoat asymmetric
796                                                peak broadening ratio. Note that this is the
797                                                sum of S/L and H/L where S is
798                                                sample height, H is the slit height and
799                                                L is the goniometer diameter.
800\\                           Polariz.            (float) Polarization coefficient.
801Instrument Parameters[1]                        (empty dict)
802========================    ===============  ===========================================================
803
804.. _TOFPowder_table:
805
806.. index::
807   single: Powder data TOF Instrument Parameters
808
809-----------------------------
810TOF Instrument Parameters
811-----------------------------
812
813Instrument Parameters are also placed in a list of two dicts,
814where the keys in each dict listed below, but here for
815time-of-flight (TOF) histograms.
816The value for each item is a list containing three values: the initial value, the current value
817and a refinement flag which can have a value of True, False or 0 where 0 indicates a value that
818cannot be refined. The first and second values are floats unless otherwise noted.
819Items not refined are noted as [*]
820
821.. tabularcolumns:: |l|p{1.5in}|p{4in}|
822
823========================    ===============  ===========================================================
824  key                        sub-key          explanation
825========================    ===============  ===========================================================
826Instrument Parameters[0]    Type [*]            (str) Histogram type:
827                                                * 'PNT' for time of flight neutron
828\\                           Bank                (int) Data set number in a multidata file
829\\                           2-theta [*]         (float) Nominal scattering angle for the detector
830\\                           fltPath [*]         (float) Total flight path source-sample-detector
831\\                           Azimuth [*]         (float) Azimuth angle for detector right hand rotation
832                                                from horizontal away from source
833\\                           difC,difA,          (float) Diffractometer constants for conversion of d-spacing to TOF
834                            difB                in microseconds
835\\                           Zero                (float) Zero point offset (microseconds)
836\\                           alpha               (float) Exponential rise profile coefficients
837\\                           beta-0              (float) Exponential decay profile coefficients
838                            beta-1
839                            beta-q
840\\                           sig-0               (float) Gaussian profile coefficients
841                            sig-1
842                            sig-2
843                            sig-q   
844\\                           X,Y,Z               (float) Lorentzian profile coefficients
845Instrument Parameters[1]    Pdabc               (list of 4 float lists) Originally created for use in gsas as optional tables
846                                                of d, alp, bet, d-true; for a reflection alpha & beta are obtained via interpolation
847                                                from the d-spacing and these tables. The d-true column is apparently unused.
848========================    ===============  ===========================================================
849
850
851.. _PowderRefl_table:
852
853.. index::
854   single: Powder reflection object description
855   single: Data object descriptions; Powder Reflections
856
857Powder Reflection Data Structure
858--------------------------------
859
860For every phase in a histogram, the ``Reflection Lists`` value is a dict
861one element of which is `'RefList'`, which is a np.array containing
862reflections. The columns in that array are documented below.
863
864==========  ====================================================
865  index         explanation
866==========  ====================================================
867 0,1,2          h,k,l (float)
868 3              (int) multiplicity
869 4              (float) d-space, :math:`\\AA`
870 5              (float) pos, two-theta
871 6              (float) sig, Gaussian width
872 7              (float) gam, Lorenzian width
873 8              (float) :math:`F_{obs}^2`
874 9              (float) :math:`F_{calc}^2`
875 10             (float) reflection phase, in degrees
876 11             (float) intensity correction for reflection, this times
877                :math:`F_{obs}^2` or :math:`F_{calc}^2` gives Iobs or Icalc
878 12             (float) Preferred orientation correction
879 13             (float) Transmission (absorption correction)
880 14             (float) Extinction correction
881==========  ====================================================
882
883.. _Xtal_table:
884
885.. index::
886   single: Single Crystal data object description
887   single: Data object descriptions; Single crystal data
888
889Single Crystal Tree Items
890-------------------------
891
892Every single crystal diffraction histogram is stored in the GSAS-II data tree
893with a top-level entry named beginning with the string "HKLF ". The
894diffraction data for that information are directly associated with
895that tree item and there are a series of children to that item. The
896routines :func:`GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
897and :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` will
898load this information into a dictionary where the child tree name is
899used as a key, and the information in the main entry is assigned
900a key of ``Data``, as outlined below.
901
902.. tabularcolumns:: |l|l|p{4in}|
903
904======================  ===============     ====================================================
905  key                      sub-key          explanation
906======================  ===============     ====================================================
907Data                        \\               (dict) that contains the
908                                            reflection table,
909                                            as described in the
910                                            :ref:`Single Crystal Reflections
911                                            <XtalRefl_table>`
912                                            description.
913
914Instrument Parameters       \\               (list) containing two dicts where the possible
915                                            keys in each dict are listed below. The value
916                                            for most items is a list containing two values:
917                                            the initial value, the current value.
918                                            The first and second
919                                            values are floats unless otherwise noted.
920\\                           Lam             (two floats) Specifies a wavelength in :math:`\\AA`
921\\                           Type            (two str values) Histogram type :
922                                            * 'SXC' for constant wavelength x-ray
923                                            * 'SNC' for constant wavelength neutron
924                                            * 'SNT' for time of flight neutron
925                                            * 'SEC' for constant wavelength electrons (e.g. micro-ED)
926\\                           InstrName       (str) A name for the instrument, used in preparing a CIF
927wtFactor                    \\               (float) A weighting factor to increase or decrease
928                                            the leverage of data in the histogram.
929                                            A value of 1.0 weights the data with their
930                                            standard uncertainties and a larger value
931                                            increases the weighting of the data (equivalent
932                                            to decreasing the uncertainties).
933
934hId                         \\               (int) The number assigned to the histogram when
935                                            the project is loaded or edited (can change)
936ranId                       \\               (int) A random number id for the histogram
937                                            that does not change
938======================  ===============     ====================================================
939
940.. _XtalRefl_table:
941
942.. index::
943   single: Single Crystal reflection object description
944   single: Data object descriptions; Single Crystal Reflections
945
946Single Crystal Reflection Data Structure
947----------------------------------------
948
949For every single crystal a histogram, the ``'Data'`` item contains
950the structure factors as an np.array in item `'RefList'`.
951The columns in that array are documented below.
952
953.. tabularcolumns:: |l|p{4in}|
954
955==========  ====================================================
956  index         explanation
957==========  ====================================================
958 0,1,2          (float) h,k,l
959 3              (int) multiplicity
960 4              (float) d-space, :math:`\\AA`
961 5              (float) :math:`F_{obs}^2`
962 6              (float) :math:`\\sigma(F_{obs}^2)`
963 7              (float) :math:`F_{calc}^2`
964 8              (float) :math:`F_{obs}^2T`
965 9              (float) :math:`F_{calc}^2T`
966 10             (float) reflection phase, in degrees
967 11             (float) intensity correction for reflection, this times
968                :math:`F_{obs}^2` or :math:`F_{calc}^2`
969                gives Iobs or Icalc
970==========  ====================================================
971
972.. _Image_table:
973
974.. index::
975   image: Image data object description
976   image: Image object descriptions
977
978Image Data Structure
979--------------------
980
981Every 2-dimensional image is stored in the GSAS-II data tree
982with a top-level entry named beginning with the string "IMG ". The
983image data are directly associated with that tree item and there
984are a series of children to that item. The routines :func:`GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
985and :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` will
986load this information into a dictionary where the child tree name is
987used as a key, and the information in the main entry is assigned
988a key of ``Data``, as outlined below.
989
990.. tabularcolumns:: |l|l|p{4in}|
991
992======================  ======================  ====================================================
993  key                      sub-key              explanation
994======================  ======================  ====================================================
995Comments                    \\                   (list of str) Text strings extracted from the original image data
996                                                header or a metafile. These cannot be changed by
997                                                the user; it may be empty.
998Image Controls              azmthOff            (float) The offset to be applied to an azimuthal
999                                                value. Accomodates
1000                                                detector orientations other than with the detector
1001                                                X-axis
1002                                                horizontal.
1003\\                           background image    (list:str,float) The name of a tree item ("IMG ...") that is to be subtracted
1004                                                during image integration multiplied by value. It must have the same size/shape as
1005                                                the integrated image. NB: value < 0 for subtraction.
1006\\                           calibrant           (str) The material used for determining the position/orientation
1007                                                of the image. The data is obtained from :func:`ImageCalibrants`
1008                                                and UserCalibrants.py (supplied by user).
1009\\                           calibdmin           (float) The minimum d-spacing used during the last calibration run.
1010\\                           calibskip           (int) The number of expected diffraction lines skipped during the last
1011                                                calibration run.
1012\\                           center              (list:floats) The [X,Y] point in detector coordinates (mm) where the direct beam
1013                                                strikes the detector plane as determined by calibration. This point
1014                                                does not have to be within the limits of the detector boundaries.
1015\\                           centerAzm           (bool) If True then the azimuth reported for the integrated slice
1016                                                of the image is at the center line otherwise it is at the leading edge.
1017\\                           color               (str) The name of the colormap used to display the image. Default = 'Paired'.
1018\\                           cutoff              (float) The minimum value of I/Ib for a point selected in a diffraction ring for
1019                                                calibration calculations. See pixLimit for details as how point is found.
1020\\                           DetDepth            (float) Coefficient for penetration correction to distance; accounts for diffraction
1021                                                ring offset at higher angles. Optionally determined by calibration.
1022\\                           DetDepthRef         (bool) If True then refine DetDepth during calibration/recalibration calculation.
1023\\                           distance            (float) The distance (mm) from sample to detector plane.
1024\\                           ellipses            (list:lists) Each object in ellipses is a list [center,phi,radii,color] where
1025                                                center (list) is location (mm) of the ellipse center on the detector plane, phi is the
1026                                                rotation of the ellipse minor axis from the x-axis, and radii are the minor & major
1027                                                radii of the ellipse. If radii[0] is negative then parameters describe a hyperbola. Color
1028                                                is the selected drawing color (one of 'b', 'g' ,'r') for the ellipse/hyperbola.
1029\\                           edgemin             (float) Not used;  parameter in EdgeFinder code.
1030\\                           fullIntegrate       (bool) If True then integrate over full 360 deg azimuthal range.
1031\\                           GonioAngles         (list:floats) The 'Omega','Chi','Phi' goniometer angles used for this image.
1032                                                Required for texture calculations.
1033\\                           invert_x            (bool) If True display the image with the x-axis inverted.
1034\\                           invert_y            (bool) If True display the image with the y-axis inverted.
1035\\                           IOtth               (list:floats) The minimum and maximum 2-theta values to be used for integration.
1036\\                           LRazimuth           (list:floats) The minimum and maximum azimuth values to be used for integration.
1037\\                           Oblique             (list:float,bool) If True apply a detector absorption correction using the value to the
1038                                                intensities obtained during integration.
1039\\                           outAzimuths         (int) The number of azimuth pie slices.
1040\\                           outChannels         (int) The number of 2-theta steps.
1041\\                           pixelSize           (list:ints) The X,Y dimensions (microns) of each pixel.
1042\\                           pixLimit            (int) A box in the image with 2*pixLimit+1 edges is searched to find the maximum.
1043                                                This value (I) along with the minimum (Ib) in the box is reported by :func:`GSASIIimage.ImageLocalMax`
1044                                                and subject to cutoff in :func:`GSASIIimage.makeRing`.
1045                                                Locations are used to construct rings of points for calibration calcualtions.
1046\\                           PolaVal             (list:float,bool) If type='SASD' and if True, apply polarization correction to intensities from
1047                                                integration using value.
1048\\                           rings               (list:lists) Each entry is [X,Y,dsp] where X & Y are lists of x,y coordinates around a
1049                                                diffraction ring with the same d-spacing (dsp)
1050\\                           ring                (list) The x,y coordinates of the >5 points on an inner ring
1051                                                selected by the user,
1052\\                           Range               (list) The minimum & maximum values of the image
1053\\                           rotation            (float) The angle between the x-axis and the vector about which the
1054                                                detector is tilted. Constrained to -180 to 180 deg.
1055\\                           SampleShape         (str) Currently only 'Cylinder'. Sample shape for Debye-Scherrer experiments; used for absorption
1056                                                calculations.
1057\\                           SampleAbs           (list: float,bool) Value of absorption coefficient for Debye-Scherrer experimnents, flag if True
1058                                                to cause correction to be applied.
1059\\                           setDefault          (bool) If True the use the image controls values for all new images to be read. (might be removed)
1060\\                           setRings            (bool) If True then display all the selected x,y ring positions (vida supra rings) used in the calibration.
1061\\                           showLines           (bool) If True then isplay the integration limits to be used.
1062\\                           size                (list:int) The number of pixels on the image x & y axes
1063\\                           type                (str) One of 'PWDR', 'SASD' or 'REFL' for powder, small angle or reflectometry data, respectively.
1064\\                           tilt                (float) The angle the detector normal makes with the incident beam; range -90 to 90.
1065\\                           wavelength          (float) The radiation wavelength (:math:`\\AA`) as entered by the user
1066                                                (or someday obtained from the image header).
1067Masks                       Arcs                (list: lists) Each entry [2-theta,[azimuth[0],azimuth[1]],thickness] describes an arc mask
1068                                                to be excluded from integration
1069\\                           Frames              (list:lists) Each entry describes the x,y points (3 or more - mm) that describe a frame outside
1070                                                of which is excluded from recalibration and integration. Only one frame is allowed.
1071\\                           Points              (list:lists) Each entry [x,y,radius] (mm) describes an excluded spot on the image to be excluded
1072                                                from integration.
1073\\                           Polygons            (list:lists) Each entry is a list of 3+ [x,y] points (mm) that describe a polygon on the image
1074                                                to be excluded from integration.
1075\\                           Rings               (list: lists) Each entry [2-theta,thickness] describes a ring mask
1076                                                to be excluded from integration.
1077\\                           Thresholds          (list:[tuple,list]) [(Imin,Imax),[Imin,Imax]] This gives lower and upper limits for points on the image to be included
1078                                                in integrsation. The tuple is the image intensity limits and the list are those set by the user.
1079\\                           SpotMask            (dict: int & array)
1080                                                'esdMul'(int) number of standard deviations above mean ring intensity to mask
1081                                                'spotMask' (bool array) the spot mask for every pixel in image         
1082
1083Stress/Strain               Sample phi          (float) Sample rotation about vertical axis.
1084\\                           Sample z            (float) Sample translation from the calibration sample position (for Sample phi = 0)
1085                                                These will be restricted by space group symmetry; result of strain fit refinement.
1086\\                           Type                (str) 'True' or 'Conventional': The strain model used for the calculation.
1087\\                           d-zero              (list:dict) Each item is for a diffraction ring on the image; all items are from the same phase
1088                                                and are used to determine the strain tensor.
1089                                                The dictionary items are:
1090                                                'Dset': (float) True d-spacing for the diffraction ring; entered by the user.
1091                                                'Dcalc': (float) Average calculated d-spacing determined from strain coeff.
1092                                                'Emat': (list: float) The strain tensor elements e11, e12 & e22 (e21=e12, rest are 0)
1093                                                'Esig': (list: float) Esds for Emat from fitting.
1094                                                'pixLimit': (int) Search range to find highest point on ring for each data point
1095                                                'cutoff': (float) I/Ib cutoff for searching.
1096                                                'ImxyObs': (list: lists) [[X],[Y]] observed points to be used for strain calculations.
1097                                                'ImtaObs': (list: lists) [[d],[azm]] transformed via detector calibration from ImxyObs.
1098                                                'ImtaCalc': (list: lists [[d],[azm]] calculated d-spacing & azimuth from fit.
1099
1100======================  ======================  ====================================================
1101
1102.. _parmDict_table:
1103
1104.. index::
1105   single: Parameter dictionary
1106
1107Parameter Dictionary
1108-------------------------
1109
1110The parameter dictionary contains all of the variable parameters for the refinement.
1111The dictionary keys are the name of the parameter (<phase>:<hist>:<name>:<atom>).
1112It is prepared in two ways. When loaded from the tree
1113(in :meth:`GSASIIdataGUI.GSASII.MakeLSParmDict` and
1114:meth:`GSASIIIO.ExportBaseclass.loadParmDict`),
1115the values are lists with two elements: ``[value, refine flag]``
1116
1117When loaded from the GPX file (in
1118:func:`GSASIIstrMain.Refine` and :func:`GSASIIstrMain.SeqRefine`), the value in the
1119dict is the actual parameter value (usually a float, but sometimes a
1120letter or string flag value (such as I or A for iso/anisotropic).
1121
1122Texture implementation
1123------------------------------
1124
1125There are two different places where texture can be treated in GSAS-II.
1126One is for mitigating the effects of texture in a structural refinement.
1127The other is for texture characterization.
1128
1129For reducing the effect of texture in a structural refinement
1130there are entries labeled preferred orientation in each phase's
1131data tab. Two different approaches can be used for this, the March-Dollase
1132model and spherical harmonics.
1133
1134For the March-Dollase model, one axis in reciprocal space is designated as
1135unique (defaulting to the 001 axis) and reflections are corrected
1136according to the angle they make with this axis depending on
1137the March-Dollase ratio. (If unity, no correction is made).
1138The ratio can be greater than one or less than one depending on if
1139crystallites oriented along the designated axis are
1140overrepresented or underrepresented. For most crystal systems there is an
1141obvious choice for the direction of the unique axis and then only a single
1142term needs to be refined. If the number is close to 1, then the correction
1143is not needed.
1144
1145The second method for reducing the effect of texture in a structural
1146refinement is to create a crystallite orientation probability surface as an
1147expansion in terms spherical harmonic functions. Only functions consistent with
1148cylindrical diffraction suymmetry and having texture symmetry
1149consistent with the Laue class of phase are used and are allowed,
1150so the higher the symmetry the fewer terms that are available for a given spherical harmonics order.
1151To use this correction, select the lowest order that provides
1152refinable terms and perform a refinement. If the texture index remains close to
1153one, then the correction is not needed. If a significant improvement is
1154noted in the profile Rwp, one may wish to see if a higher order expansion
1155gives an even larger improvement.
1156
1157To characterize texture in a material, generally one needs data collected with the
1158sample at multiple orientations or, for TOF, with detectors at multiple
1159locations around the sample. In this case the detector orientation is given in
1160each histogram's Sample Parameters and the sample's orientation is described
1161with the Euler angles specifed on the phase's Texture tab, which is also
1162where the texture type (cylindrical, rolling,...) and the spherical
1163harmonic order is selected. This should not be used with a single dataset and
1164should not be used if the preferred orientations corrections are used.
1165
1166The coordinate system used for texture characterization is defined where
1167the sample coordinates (Psi, gamma) are defined with an instrument coordinate
1168system (I, J, K) such that K is normal to the diffraction plane and J is coincident with the
1169direction of the incident radiation beam toward the source. We further define
1170a standard set of right-handed goniometer eulerian angles (Omega, Chi, Phi) so that Omega and Phi are
1171rotations about K and Chi is a rotation about J when Omega = 0. Finally, as the sample
1172may be mounted so that the sample coordinate system (Is, Js, Ks) does not coincide with
1173the instrument coordinate system (I, J, K), we define three eulerian sample rotation angles
1174(Omega-s, Chi-s, Phi-s) that describe the rotation from (Is, Js, Ks) to (I, J, K). The sample rotation
1175angles are defined so that with the goniometer angles at zero Omega-s and Phi-s are rotations
1176about K and Chi-s is a rotation about J.
1177
1178Three typical examples:
1179
1180    1) Bragg-Brentano laboratory diffractometer: Chi=0
1181    2) Debye-Scherrer counter detector; sample capillary axis perpendicular to diffraction plane: Chi=90
1182    3) Debye-Scherrer 2D area detector positioned directly behind sample; sample capillary axis horizontal; Chi=0
1183
1184NB: The area detector azimuthal angle will equal 0 in horizontal plane to right as viewed from x-ray source and will equal
118590 at vertical "up" direction.
1186           
1187ISODISTORT implementation
1188------------------------------
1189
1190CIFs prepared with the ISODISTORT web site
1191https://stokes.byu.edu/iso/isodistort_version5.6.1/isodistort.php
1192[B. J. Campbell, H. T. Stokes, D. E. Tanner, and D. M. Hatch, "ISODISPLACE: An Internet Tool for Exploring Structural Distortions."
1193J. Appl. Cryst. 39, 607-614 (2006).] can be read into GSAS-II using import CIF. This will cause constraints to be established for
1194structural distortion modes read from the CIF. At present, of the five types of modes  only displacive(``_iso_displacivemode``...)
1195and occupancy (``_iso_occupancymode``...) are processed. Not yet processed: ``_iso_magneticmode``...,
1196``_iso_rotationalmode``... & ``_iso_strainmode``...
1197
1198The CIF importer :mod:`G2phase_CIF` implements class :class:`G2phase_CIF.CIFPhaseReader` which offers two methods associated
1199with ISODISTORT (ID) input. Method :meth:`G2phase_CIF.CIFPhaseReader.ISODISTORT_test` checks to see if a CIF block contains
1200the loops with ``_iso_displacivemode_label`` or  ``_iso_occupancymode_label`` items. If so, method
1201:meth:`G2phase_CIF.CIFPhaseReader.ISODISTORT_proc` is called to read and interpret them. The results are placed into the
1202reader object's ``.Phase`` class variable as a dict item with key ``'ISODISTORT'``.
1203
1204Note that each mode ID has a long label with a name such as  Pm-3m[1/2,1/2,1/2]R5+(a,a,0)[La:b:dsp]T1u(a). Function
1205:func:`G2phase_CIF.ISODISTORT_shortLbl` is used to create a short name for this, such as R5_T1u(a) which is made unique
1206by addition of _n if the short name is duplicated. As each mode is processed, a constraint corresponding to that mode is
1207created and is added to list in the reader object's ``.Constraints`` class variable. Items placed into that list can either
1208be a list, which corresponds to a function (new var) type :ref:`constraint definition <Constraints_table>` entry, or an item
1209can be a dict, which provides help information for each constraint.
1210
1211------------------------------
1212Displacive modes
1213------------------------------
1214
1215The coordinate variables, as named by ISODISTORT, are placed in ``.Phase['ISODISTORT']['IsoVarList']`` and the
1216corresponding :class:`GSASIIobj.G2VarObj` objects for each are placed in ``.Phase['ISODISTORT']['G2VarList']``.
1217The mode variables, as named by ISODISTORT, are placed in ``.Phase['ISODISTORT']['IsoModeList']`` and the
1218corresponding :class:`GSASIIobj.G2VarObj` objects for each are placed in ``.Phase['ISODISTORT']['G2ModeList']``.
1219[Use ``str(G2VarObj)`` to get the variable name from the G2VarObj object, but note that the phase number, *n*, for the prefix
1220"*n*::" cannot be determined as the phase number is not yet assigned.]
1221
1222Displacive modes are a bit complex in that they relate to delta displacements, relative to an offset value for each coordinate,
1223and because the modes are normalized. While GSAS-II also uses displacements,  these are added to the coordinates after
1224each refinement cycle and then the delta values are set to zero.
1225ISODISTORT uses fixed offsets (subtracted from the actual position
1226to obtain the delta values) that are taken from the parent structure coordinate and the initial offset value
1227(in ``_iso_deltacoordinate_value``) and these are placed in
1228``.Phase['ISODISTORT']['G2coordOffset']`` in the same order as ``.Phase['ISODISTORT']['G2ModeList']``,
1229``.Phase['ISODISTORT']['IsoVarList']`` and ''.Phase[ISODISTORT']['G2parentCoords']''.'
1230
1231The normalization factors (which the delta values are divided by)
1232are taken from ``_iso_displacivemodenorm_value`` and are placed in ``.Phase['ISODISTORT']['NormList']`` in the same
1233order as as ``...['IsoModeList']`` and ``...['G2ModeList']``.
1234
1235The CIF contains a sparse matrix, from the ``loop_`` containing ``_iso_displacivemodematrix_value`` which provides the equations
1236for determining the mode values from the coordinates, that matrix is placed in ``.Phase['ISODISTORT']['Mode2VarMatrix']``.
1237The matrix is inverted to produce ``.Phase['ISODISTORT']['Var2ModeMatrix']``, which determines how to compute the
1238mode values from the delta coordinate values. These values are used for the in :func:`GSASIIconstrGUI.ShowIsoDistortCalc`,
1239which shows coordinate and mode values, the latter with s.u. values.
1240
1241------------------------------
1242Occupancy modes
1243------------------------------
1244
1245
1246The delta occupancy variables, as named by ISODISTORT, are placed in
1247``.Phase['ISODISTORT']['OccVarList']`` and the corresponding :class:`GSASIIobj.G2VarObj` objects for each are placed
1248in ``.Phase['ISODISTORT']['G2OccVarList']``. The mode variables, as named by ISODISTORT, are placed in
1249``.Phase['ISODISTORT']['OccModeList']`` and the corresponding :class:`GSASIIobj.G2VarObj` objects for each are placed
1250in ``.Phase['ISODISTORT']['G2OccModeList']``.
1251
1252Occupancy modes, like Displacive modes, are also refined as delta values.  However, GSAS-II directly refines the fractional
1253occupancies. Offset values for each atom, are taken from ``_iso_occupancy_formula`` and are placed in
1254``.Phase['ISODISTORT']['ParentOcc]``. (Offset values are subtracted from the actual position to obtain the delta values.)
1255Modes are normalized (where the mode values are divided by the normalization factor) are taken from ``_iso_occupancymodenorm_value``
1256and are placed in ``.Phase['ISODISTORT']['OccNormList']`` in the same order as as ``...['OccModeList']`` and
1257``...['G2OccModeList']``.
1258
1259The CIF contains a sparse matrix, from the ``loop_`` containing ``_iso_occupancymodematrix_value``, which provides the
1260equations for determining the mode values from the coordinates. That matrix is placed in ``.Phase['ISODISTORT']['Occ2VarMatrix']``.
1261The matrix is inverted to produce ``.Phase['ISODISTORT']['Var2OccMatrix']``, which determines how to compute the
1262mode values from the delta coordinate values.
1263
1264
1265------------------------------
1266Mode Computations
1267------------------------------
1268
1269Constraints are processed after the CIF has been read in :meth:`GSASIIdataGUI.GSASII.OnImportPhase` or 
1270:meth:`GSASIIscriptable.G2Project.add_phase` by moving them from the reader object's ``.Constraints``
1271class variable to the Constraints tree entry's ['Phase'] list (for list items defining constraints) or
1272the Constraints tree entry's ['_Explain'] dict (for dict items defining constraint help information)
1273
1274The information in ``.Phase['ISODISTORT']`` is used in :func:`GSASIIconstrGUI.ShowIsoDistortCalc` which shows coordinate and mode
1275values, the latter with s.u. values. This can be called from the Constraints and Phase/Atoms tree items.
1276
1277Before each refinement, constraints are processed as :ref:`described elsewhere <Constraints_processing>`. After a refinement
1278is complete, :func:`GSASIIstrIO.PrintIndependentVars` shows the shifts and s.u.'s on the refined modes,
1279using GSAS-II values, but :func:`GSASIIstrIO.PrintISOmodes` prints the ISODISTORT modes as computed in the web site.
1280
1281
1282.. _ParameterLimits:
1283
1284.. index::
1285   single: Parameter limits
1286
1287Parameter Limits
1288------------------------------
1289
1290One of the most often requested "enhancements" for GSAS-II would be the inclusion
1291of constraints to force parameters such as occupancies or Uiso values to stay within
1292expected ranges. While it is possible for users to supply their own restraints that would
1293perform this by supplying an appropriate expression with the "General" restraints, the
1294GSAS-II authors do not feel that use of restraints or constraints are a good solution for
1295this common problem where parameters refine to non-physical values. This is because when
1296this occurs, most likely one of the following cases is occurring:
1297
1298#. there is a significant problem
1299   with the model, for example for an x-ray fit if an O atom is placed where a S is actually
1300   present, the Uiso will refine artificially small or the occupancy much larger than unity
1301   to try to compensate for the missing electrons; or
1302 
1303#. the data are simply insensitive
1304   to the parameter or combination of parameters, for example unless very high-Q data
1305   are included, the effects of a occupancy and Uiso value can have compensating effects,
1306   so an assumption must be made; likewise, with neutron data natural-abundance V atoms
1307   are nearly invisible due to weak coherent scattering. No parameters can be fit for a
1308   V atom with neutrons.
1309
1310#. the parameter is non-physical (such as a negative Uiso value) but within
1311   two sigma (sigma = standard uncertainty, aka e.s.d.) of a reasonable value,
1312   in which case the
1313   value is not problematic as it is experimentally indistinguishable from an
1314   expected value. 
1315
1316#. there is a systematic problem with the data (experimental error)
1317
1318In all these cases, this situation needs to be reviewed by a crystallographer to decide
1319how to best determine a structural model for these data. An implementation with a constraint
1320or restraint is likely to simply hide the problem from the user, making it more probable
1321that a poor model choice is obtained.
1322
1323What GSAS-II does implement is to allow users to specify ranges for parameters
1324that works by disabling
1325refinement of parameters that refine beyond either a lower limit or an upper limit, where
1326either or both may be optionally specified. Parameters limits are specified in the Controls
1327tree entry in dicts named as ``Controls['parmMaxDict']`` and ``Controls['parmMinDict']``, where
1328the keys are :class:`G2VarObj` objects corresponding to standard GSAS-II variable
1329(see :func:`getVarDescr` and :func:`CompileVarDesc`) names, where a
1330wildcard ('*') may optionally be used for histogram number or atom number
1331(phase number is intentionally not  allowed as a wildcard as it makes little sense
1332to group the same parameter together different phases). Note
1333that :func:`prmLookup` is used to see if a name matches a wildcard. The upper or lower limit
1334is placed into these dicts as a float value. These values can be edited using the window
1335created by the Calculate/"View LS parms" menu command or in scripting with the
1336:meth:`GSASIIscriptable.G2Project.set_Controls` function.
1337In the GUI, a checkbox labeled "match all histograms/atoms" is used to insert a wildcard
1338into the appropriate part of the variable name.
1339
1340When a refinement is conducted, routine :func:`GSASIIstrMain.dropOOBvars` is used to
1341find parameters that have refined to values outside their limits. If this occurs, the parameter
1342is set to the limiting value and the variable name is added to a list of frozen variables
1343(as a :class:`G2VarObj` objects) kept in a list in the
1344``Controls['parmFrozen']`` dict. In a sequential refinement, this is kept separate for
1345each histogram as a list in
1346``Controls['parmFrozen'][histogram]`` (where the key is the histogram name) or as a list in
1347``Controls['parmFrozen']['FrozenList']`` for a non-sequential fit.
1348This allows different variables
1349to be frozen in each section of a sequential fit.
1350Frozen parameters are not included in refinements through removal from the
1351list of parameters to be refined (``varyList``) in :func:`GSASIIstrMain.Refine` or
1352:func:`GSASIIstrMain.SeqRefine`.
1353The data window for the Controls tree item shows the number of Frozen variables and
1354the individual variables can be viewed with the Calculate/"View LS parms" menu window or
1355obtained with :meth:`GSASIIscriptable.G2Project.get_Frozen`.
1356Once a variable is frozen, it will not be refined in any
1357future refinements unless the the variable is removed (manually) from the list. This can also
1358be done with the Calculate/"View LS parms" menu window or
1359:meth:`GSASIIscriptable.G2Project.set_Frozen`.
1360
1361
1362.. seealso::
1363  :class:`G2VarObj`
1364  :func:`getVarDescr`
1365  :func:`CompileVarDesc`
1366  :func:`prmLookup`
1367  :class:`GSASIIctrlGUI.ShowLSParms`
1368  :class:`GSASIIctrlGUI.VirtualVarBox`
1369  :func:`GSASIIstrIO.SetUsedHistogramsAndPhases`
1370  :func:`GSASIIstrIO.SaveUpdatedHistogramsAndPhases`
1371  :func:`GSASIIstrIO.SetSeqResult`
1372  :func:`GSASIIstrMain.dropOOBvars`
1373  :meth:`GSASIIscriptable.G2Project.set_Controls`
1374  :meth:`GSASIIscriptable.G2Project.get_Frozen`
1375  :meth:`GSASIIscriptable.G2Project.set_Frozen`
1376
1377*Classes and routines*
1378----------------------
1379
1380'''
1381from __future__ import division, print_function
1382import platform
1383import re
1384import random as ran
1385import sys
1386import os.path
1387if '2' in platform.python_version_tuple()[0]:
1388    import cPickle
1389else:
1390    import pickle as cPickle
1391import GSASIIpath
1392import GSASIImath as G2mth
1393import GSASIIspc as G2spc
1394import numpy as np
1395
1396GSASIIpath.SetVersionNumber("$Revision: 5307 $")
1397
1398DefaultControls = {
1399    'deriv type':'analytic Hessian',
1400    'min dM/M':0.001,'shift factor':1.,'max cyc':3,'F**2':False,'SVDtol':1.e-6,
1401    'UsrReject':{'minF/sig':0.,'MinExt':0.01,'MaxDF/F':100.,'MaxD':500.,'MinD':0.05},
1402    'Copy2Next':False,'Reverse Seq':False,'HatomFix':False,
1403    'Author':'no name','newLeBail':False,
1404    'FreePrm1':'Sample humidity (%)',
1405    'FreePrm2':'Sample voltage (V)',
1406    'FreePrm3':'Applied load (MN)',
1407    'ShowCell':False,
1408    }
1409'''Values to be used as defaults for the initial contents of the ``Controls``
1410data tree item.
1411'''
1412def StripUnicode(string,subs='.'):
1413    '''Strip non-ASCII characters from strings
1414
1415    :param str string: string to strip Unicode characters from
1416    :param str subs: character(s) to place into string in place of each
1417      Unicode character. Defaults to '.'
1418
1419    :returns: a new string with only ASCII characters
1420    '''
1421    s = ''
1422    for c in string:
1423        if ord(c) < 128:
1424            s += c
1425        else:
1426            s += subs
1427    return s
1428#    return s.encode('ascii','replace')
1429
1430def MakeUniqueLabel(lbl,labellist):
1431    '''Make sure that every a label is unique against a list by adding
1432    digits at the end until it is not found in list.
1433
1434    :param str lbl: the input label
1435    :param list labellist: the labels that have already been encountered
1436    :returns: lbl if not found in labellist or lbl with ``_1-9`` (or
1437      ``_10-99``, etc.) appended at the end
1438    '''
1439    lbl = StripUnicode(lbl.strip(),'_')
1440    if not lbl: # deal with a blank label
1441        lbl = '_1'
1442    if lbl not in labellist:
1443        labellist.append(lbl)
1444        return lbl
1445    i = 1
1446    prefix = lbl
1447    if '_' in lbl:
1448        prefix = lbl[:lbl.rfind('_')]
1449        suffix = lbl[lbl.rfind('_')+1:]
1450        try:
1451            i = int(suffix)+1
1452        except: # suffix could not be parsed
1453            i = 1
1454            prefix = lbl
1455    while prefix+'_'+str(i) in labellist:
1456        i += 1
1457    else:
1458        lbl = prefix+'_'+str(i)
1459        labellist.append(lbl)
1460    return lbl
1461
1462PhaseIdLookup = {}
1463'''dict listing phase name and random Id keyed by sequential phase index as a str;
1464best to access this using :func:`LookupPhaseName`
1465'''
1466PhaseRanIdLookup = {}
1467'''dict listing phase sequential index keyed by phase random Id;
1468best to access this using :func:`LookupPhaseId`
1469'''
1470HistIdLookup = {}
1471'''dict listing histogram name and random Id, keyed by sequential histogram index as a str;
1472best to access this using :func:`LookupHistName`
1473'''
1474HistRanIdLookup = {}
1475'''dict listing histogram sequential index keyed by histogram random Id;
1476best to access this using :func:`LookupHistId`
1477'''
1478AtomIdLookup = {}
1479'''dict listing for each phase index as a str, the atom label and atom random Id,
1480keyed by atom sequential index as a str;
1481best to access this using :func:`LookupAtomLabel`
1482'''
1483AtomRanIdLookup = {}
1484'''dict listing for each phase the atom sequential index keyed by atom random Id;
1485best to access this using :func:`LookupAtomId`
1486'''
1487ShortPhaseNames = {}
1488'''a dict containing a possibly shortened and when non-unique numbered
1489version of the phase name. Keyed by the phase sequential index.
1490'''
1491ShortHistNames = {}
1492'''a dict containing a possibly shortened and when non-unique numbered
1493version of the histogram name. Keyed by the histogram sequential index.
1494'''
1495
1496#VarDesc = {}  # removed 1/30/19 BHT as no longer needed (I think)
1497#''' This dictionary lists descriptions for GSAS-II variables,
1498#as set in :func:`CompileVarDesc`. See that function for a description
1499#for how keys and values are written.
1500#'''
1501
1502reVarDesc = {}
1503''' This dictionary lists descriptions for GSAS-II variables where
1504keys are compiled regular expressions that will match the name portion
1505of a parameter name. Initialized in :func:`CompileVarDesc`.
1506'''
1507
1508reVarStep = {}
1509''' This dictionary lists the preferred step size for numerical
1510derivative computation w/r to a GSAS-II variable. Keys are compiled
1511regular expressions and values are the step size for that parameter.
1512Initialized in :func:`CompileVarDesc`.
1513'''
1514# create a default space group object for P1; N.B. fails when building documentation
1515try:
1516    P1SGData = G2spc.SpcGroup('P 1')[1] # data structure for default space group
1517except:
1518    pass
1519
1520def GetPhaseNames(fl):
1521    ''' Returns a list of phase names found under 'Phases' in GSASII gpx file
1522    NB: there is another one of these in GSASIIstrIO.py that uses the gpx filename
1523
1524    :param file fl: opened .gpx file
1525    :return: list of phase names
1526    '''
1527    PhaseNames = []
1528    while True:
1529        try:
1530            data = cPickle.load(fl)
1531        except EOFError:
1532            break
1533        datum = data[0]
1534        if 'Phases' == datum[0]:
1535            for datus in data[1:]:
1536                PhaseNames.append(datus[0])
1537    fl.seek(0)          #reposition file
1538    return PhaseNames
1539
1540def SetNewPhase(Name='New Phase',SGData=None,cell=None,Super=None):
1541    '''Create a new phase dict with default values for various parameters
1542
1543    :param str Name: Name for new Phase
1544
1545    :param dict SGData: space group data from :func:`GSASIIspc:SpcGroup`;
1546      defaults to data for P 1
1547
1548    :param list cell: unit cell parameter list; defaults to
1549      [1.0,1.0,1.0,90.,90,90.,1.]
1550
1551    '''
1552    if SGData is None: SGData = P1SGData
1553    if cell is None: cell=[1.0,1.0,1.0,90.,90.,90.,1.]
1554    phaseData = {
1555        'ranId':ran.randint(0,sys.maxsize),
1556        'General':{
1557            'Name':Name,
1558            'Type':'nuclear',
1559            'Modulated':False,
1560            'AtomPtrs':[3,1,7,9],
1561            'SGData':SGData,
1562            'Cell':[False,]+cell,
1563            'Pawley dmin':1.0,
1564            'Data plot type':'None',
1565            'SH Texture':{
1566                'Order':0,
1567                'Model':'cylindrical',
1568                'Sample omega':[False,0.0],
1569                'Sample chi':[False,0.0],
1570                'Sample phi':[False,0.0],
1571                'SH Coeff':[False,{}],
1572                'SHShow':False,
1573                'PFhkl':[0,0,1],
1574                'PFxyz':[0,0,1],
1575                'PlotType':'Pole figure',
1576                'Penalty':[['',],0.1,False,1.0]}},
1577        'Atoms':[],
1578        'Drawing':{},
1579        'Histograms':{},
1580        'Pawley ref':[],
1581        'RBModels':{},
1582        }
1583    if Super and Super.get('Use',False):
1584        phaseData['General'].update({'Modulated':True,'Super':True,'SuperSg':Super['ssSymb']})
1585        phaseData['General']['SSGData'] = G2spc.SSpcGroup(SGData,Super['ssSymb'])[1]
1586        phaseData['General']['SuperVec'] = [Super['ModVec'],False,Super['maxH']]
1587
1588    return phaseData
1589
1590def ReadCIF(URLorFile):
1591    '''Open a CIF, which may be specified as a file name or as a URL using PyCifRW
1592    (from James Hester).
1593    The open routine gets confused with DOS names that begin with a letter and colon
1594    "C:\\dir\" so this routine will try to open the passed name as a file and if that
1595    fails, try it as a URL
1596
1597    :param str URLorFile: string containing a URL or a file name. Code will try first
1598      to open it as a file and then as a URL.
1599
1600    :returns: a PyCifRW CIF object.
1601    '''
1602    import CifFile as cif # PyCifRW from James Hester
1603
1604    # alternate approach:
1605    #import urllib
1606    #ciffile = 'file:'+urllib.pathname2url(filename)
1607
1608    try:
1609        fp = open(URLorFile,'r')
1610        cf = cif.ReadCif(fp)
1611        fp.close()
1612        return cf
1613    except IOError:
1614        return cif.ReadCif(URLorFile)
1615
1616def TestIndexAll():
1617    '''Test if :func:`IndexAllIds` has been called to index all phases and
1618    histograms (this is needed before :func:`G2VarObj` can be used.
1619
1620    :returns: Returns True if indexing is needed.
1621    '''
1622    if PhaseIdLookup or AtomIdLookup or HistIdLookup:
1623        return False
1624    return True
1625       
1626def IndexAllIds(Histograms,Phases):
1627    '''Scan through the used phases & histograms and create an index
1628    to the random numbers of phases, histograms and atoms. While doing this,
1629    confirm that assigned random numbers are unique -- just in case lightning
1630    strikes twice in the same place.
1631
1632    Note: this code assumes that the atom random Id (ranId) is the last
1633    element each atom record.
1634
1635    This is called when phases & histograms are looked up
1636    in these places (only):
1637       
1638     * :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` (which loads the histograms and phases from a GPX file),
1639     * :meth:`~GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree` (which does the same thing but from the data tree.)
1640     * :meth:`~GSASIIdataGUI.GSASII.OnFileClose` (clears out an old project)
1641   
1642    Note that globals :data:`PhaseIdLookup` and :data:`PhaseRanIdLookup` are
1643    also set in :func:`AddPhase2Index` to temporarily assign a phase number
1644    as a phase is being imported.
1645 
1646    TODO: do we need a lookup for rigid body variables?
1647    '''
1648    # process phases and atoms
1649    PhaseIdLookup.clear()
1650    PhaseRanIdLookup.clear()
1651    AtomIdLookup.clear()
1652    AtomRanIdLookup.clear()
1653    ShortPhaseNames.clear()
1654    for ph in Phases:
1655        cx,ct,cs,cia = Phases[ph]['General']['AtomPtrs']
1656        ranId = Phases[ph]['ranId']
1657        while ranId in PhaseRanIdLookup:
1658            # Found duplicate random Id! note and reassign
1659            print ("\n\n*** Phase "+str(ph)+" has repeated ranId. Fixing.\n")
1660            Phases[ph]['ranId'] = ranId = ran.randint(0,sys.maxsize)
1661        pId = str(Phases[ph]['pId'])
1662        PhaseIdLookup[pId] = (ph,ranId)
1663        PhaseRanIdLookup[ranId] = pId
1664        shortname = ph  #[:10]
1665        while shortname in ShortPhaseNames.values():
1666            shortname = ph[:8] + ' ('+ pId + ')'
1667        ShortPhaseNames[pId] = shortname
1668        AtomIdLookup[pId] = {}
1669        AtomRanIdLookup[pId] = {}
1670        for iatm,at in enumerate(Phases[ph]['Atoms']):
1671            ranId = at[cia+8]
1672            while ranId in AtomRanIdLookup[pId]: # check for dups
1673                print ("\n\n*** Phase "+str(ph)+" atom "+str(iatm)+" has repeated ranId. Fixing.\n")
1674                at[cia+8] = ranId = ran.randint(0,sys.maxsize)
1675            AtomRanIdLookup[pId][ranId] = str(iatm)
1676            if Phases[ph]['General']['Type'] == 'macromolecular':
1677                label = '%s_%s_%s_%s'%(at[ct-1],at[ct-3],at[ct-4],at[ct-2])
1678            else:
1679                label = at[ct-1]
1680            AtomIdLookup[pId][str(iatm)] = (label,ranId)
1681    # process histograms
1682    HistIdLookup.clear()
1683    HistRanIdLookup.clear()
1684    ShortHistNames.clear()
1685    for hist in Histograms:
1686        ranId = Histograms[hist]['ranId']
1687        while ranId in HistRanIdLookup:
1688            # Found duplicate random Id! note and reassign
1689            print ("\n\n*** Histogram "+str(hist)+" has repeated ranId. Fixing.\n")
1690            Histograms[hist]['ranId'] = ranId = ran.randint(0,sys.maxsize)
1691        hId = str(Histograms[hist]['hId'])
1692        HistIdLookup[hId] = (hist,ranId)
1693        HistRanIdLookup[ranId] = hId
1694        shortname = hist[:15]
1695        while shortname in ShortHistNames.values():
1696            shortname = hist[:11] + ' ('+ hId + ')'
1697        ShortHistNames[hId] = shortname
1698
1699def AddPhase2Index(rdObj,filename):
1700    '''Add a phase to the index during reading
1701    Used where constraints are generated during import (ISODISTORT CIFs)       
1702    '''
1703    ranId = rdObj.Phase['ranId']
1704    ph = 'from  '+filename #  phase is not named yet
1705    if ranId in PhaseRanIdLookup: return
1706    maxph = -1
1707    for r in PhaseRanIdLookup:
1708        maxph = max(maxph,int(PhaseRanIdLookup[r]))
1709    PhaseRanIdLookup[ranId] = pId = str(maxph + 1)
1710    PhaseIdLookup[pId] = (ph,ranId)
1711    shortname = 'from '+ os.path.splitext((os.path.split(filename))[1])[0]
1712    while shortname in ShortPhaseNames.values():
1713        shortname = ph[:8] + ' ('+ pId + ')'
1714    ShortPhaseNames[pId] = shortname
1715    AtomIdLookup[pId] = {}
1716    AtomRanIdLookup[pId] = {}
1717    for iatm,at in enumerate(rdObj.Phase['Atoms']):
1718        ranId = at[-1]
1719        while ranId in AtomRanIdLookup[pId]: # check for dups
1720            print ("\n\n*** Phase "+str(ph)+" atom "+str(iatm)+" has repeated ranId. Fixing.\n")
1721            at[-1] = ranId = ran.randint(0,sys.maxsize)
1722        AtomRanIdLookup[pId][ranId] = str(iatm)
1723        #if Phases[ph]['General']['Type'] == 'macromolecular':
1724        #    label = '%s_%s_%s_%s'%(at[ct-1],at[ct-3],at[ct-4],at[ct-2])
1725        #else:
1726        #    label = at[ct-1]
1727        label = at[0]
1728        AtomIdLookup[pId][str(iatm)] = (label,ranId)
1729
1730def LookupAtomId(pId,ranId):
1731    '''Get the atom number from a phase and atom random Id
1732
1733    :param int/str pId: the sequential number of the phase
1734    :param int ranId: the random Id assigned to an atom
1735
1736    :returns: the index number of the atom (str)
1737    '''
1738    if not AtomRanIdLookup:
1739        raise Exception('Error: LookupAtomId called before IndexAllIds was run')
1740    if pId is None or pId == '':
1741        raise KeyError('Error: phase is invalid (None or blank)')
1742    pId = str(pId)
1743    if pId not in AtomRanIdLookup:
1744        raise KeyError('Error: LookupAtomId does not have phase '+pId)
1745    if ranId not in AtomRanIdLookup[pId]:
1746        raise KeyError('Error: LookupAtomId, ranId '+str(ranId)+' not in AtomRanIdLookup['+pId+']')
1747    return AtomRanIdLookup[pId][ranId]
1748
1749def LookupAtomLabel(pId,index):
1750    '''Get the atom label from a phase and atom index number
1751
1752    :param int/str pId: the sequential number of the phase
1753    :param int index: the index of the atom in the list of atoms
1754
1755    :returns: the label for the atom (str) and the random Id of the atom (int)
1756    '''
1757    if not AtomIdLookup:
1758        raise Exception('Error: LookupAtomLabel called before IndexAllIds was run')
1759    if pId is None or pId == '':
1760        raise KeyError('Error: phase is invalid (None or blank)')
1761    pId = str(pId)
1762    if pId not in AtomIdLookup:
1763        raise KeyError('Error: LookupAtomLabel does not have phase '+pId)
1764    if index not in AtomIdLookup[pId]:
1765        raise KeyError('Error: LookupAtomLabel, ranId '+str(index)+' not in AtomRanIdLookup['+pId+']')
1766    return AtomIdLookup[pId][index]
1767
1768def LookupPhaseId(ranId):
1769    '''Get the phase number and name from a phase random Id
1770
1771    :param int ranId: the random Id assigned to a phase
1772    :returns: the sequential Id (pId) number for the phase (str)
1773    '''
1774    if not PhaseRanIdLookup:
1775        raise Exception('Error: LookupPhaseId called before IndexAllIds was run')
1776    if ranId not in PhaseRanIdLookup:
1777        raise KeyError('Error: LookupPhaseId does not have ranId '+str(ranId))
1778    return PhaseRanIdLookup[ranId]
1779
1780def LookupPhaseName(pId):
1781    '''Get the phase number and name from a phase Id
1782
1783    :param int/str pId: the sequential assigned to a phase
1784    :returns:  (phase,ranId) where phase is the name of the phase (str)
1785      and ranId is the random # id for the phase (int)
1786    '''
1787    if not PhaseIdLookup:
1788        raise Exception('Error: LookupPhaseName called before IndexAllIds was run')
1789    if pId is None or pId == '':
1790        raise KeyError('Error: phase is invalid (None or blank)')
1791    pId = str(pId)
1792    if pId not in PhaseIdLookup:
1793        raise KeyError('Error: LookupPhaseName does not have index '+pId)
1794    return PhaseIdLookup[pId]
1795
1796def LookupHistId(ranId):
1797    '''Get the histogram number and name from a histogram random Id
1798
1799    :param int ranId: the random Id assigned to a histogram
1800    :returns: the sequential Id (hId) number for the histogram (str)
1801    '''
1802    if not HistRanIdLookup:
1803        raise Exception('Error: LookupHistId called before IndexAllIds was run')
1804    if ranId not in HistRanIdLookup:
1805        raise KeyError('Error: LookupHistId does not have ranId '+str(ranId))
1806    return HistRanIdLookup[ranId]
1807
1808def LookupHistName(hId):
1809    '''Get the histogram number and name from a histogram Id
1810
1811    :param int/str hId: the sequential assigned to a histogram
1812    :returns:  (hist,ranId) where hist is the name of the histogram (str)
1813      and ranId is the random # id for the histogram (int)
1814    '''
1815    if not HistIdLookup:
1816        raise Exception('Error: LookupHistName called before IndexAllIds was run')
1817    if hId is None or hId == '':
1818        raise KeyError('Error: histogram is invalid (None or blank)')
1819    hId = str(hId)
1820    if hId not in HistIdLookup:
1821        raise KeyError('Error: LookupHistName does not have index '+hId)
1822    return HistIdLookup[hId]
1823
1824def fmtVarDescr(varname):
1825    '''Return a string with a more complete description for a GSAS-II variable
1826
1827    :param str varname: A full G2 variable name with 2 or 3 or 4
1828       colons (<p>:<h>:name[:<a>] or <p>::RBname:<r>:<t>])
1829
1830    :returns: a string with the description
1831    '''
1832    s,l = VarDescr(varname)
1833    return s+": "+l
1834
1835def VarDescr(varname):
1836    '''Return two strings with a more complete description for a GSAS-II variable
1837
1838    :param str name: A full G2 variable name with 2 or 3 or 4
1839       colons (<p>:<h>:name[:<a>] or <p>::RBname:<r>:<t>])
1840
1841    :returns: (loc,meaning) where loc describes what item the variable is mapped
1842      (phase, histogram, etc.) and meaning describes what the variable does.
1843    '''
1844
1845    # special handling for parameter names without a colon
1846    # for now, assume self-defining
1847    if varname.find(':') == -1:
1848        return "Global",varname
1849
1850    l = getVarDescr(varname)
1851    if not l:
1852        return ("invalid variable name ("+str(varname)+")!"),""
1853#        return "invalid variable name!",""
1854
1855    if not l[-1]:
1856        l[-1] = "(variable needs a definition! Set it in CompileVarDesc)"
1857
1858    if len(l) == 3:         #SASD variable name!
1859        s = 'component:'+l[1]
1860        return s,l[-1]
1861    s = ""
1862    if l[0] is not None and l[1] is not None: # HAP: keep short
1863        if l[2] == "Scale": # fix up ambigous name
1864            l[5] = "Phase fraction"
1865        if l[0] == '*':
1866            lbl = 'Seq. ref.'
1867        else:
1868            lbl = ShortPhaseNames.get(l[0],'? #'+str(l[0]))
1869        if l[1] == '*':
1870            hlbl = 'Seq. ref.'
1871        else:
1872            hlbl = ShortHistNames.get(l[1],'? #'+str(l[1]))
1873        if hlbl[:4] == 'HKLF':
1874            hlbl = 'Xtl='+hlbl[5:]
1875        elif hlbl[:4] == 'PWDR':
1876            hlbl = 'Pwd='+hlbl[5:]
1877        else:
1878            hlbl = 'Hist='+hlbl
1879        s = "Ph="+str(lbl)+" * "+str(hlbl)
1880    else:
1881        if l[2] == "Scale": # fix up ambigous name: must be scale factor, since not HAP
1882            l[5] = "Scale factor"
1883        if l[2] == 'Back': # background parameters are "special", alas
1884            s = 'Hist='+ShortHistNames.get(l[1],'? #'+str(l[1]))
1885            l[-1] += ' #'+str(l[3])
1886        elif l[4] is not None: # rigid body parameter or modulation parm
1887            lbl = ShortPhaseNames.get(l[0],'phase?')
1888            if 'RB' in l[2]:    #rigid body parm
1889                s = "RB body #"+str(l[3])+" (type "+str(l[4])+") in "+str(lbl) + ','
1890            else: #modulation parm
1891                s = 'Atom %s wave %s in %s'%(LookupAtomLabel(l[0],l[3])[0],l[4],lbl)
1892        elif l[3] is not None: # atom parameter,
1893            lbl = ShortPhaseNames.get(l[0],'phase?')
1894            try:
1895                albl = LookupAtomLabel(l[0],l[3])[0]
1896            except KeyError:
1897                albl = 'Atom?'
1898            s = "Atom "+str(albl)+" in "+str(lbl)
1899        elif l[0] == '*':
1900            s = "All phases "
1901        elif l[0] is not None:
1902            lbl = ShortPhaseNames.get(l[0],'phase?')
1903            s = "Phase "+str(lbl)
1904        elif l[1] == '*':
1905            s = 'All hists'
1906        elif l[1] is not None:
1907            hlbl = ShortHistNames.get(l[1],'? #'+str(l[1]))
1908            if hlbl[:4] == 'HKLF':
1909                hlbl = 'Xtl='+hlbl[5:]
1910            elif hlbl[:4] == 'PWDR':
1911                hlbl = 'Pwd='+hlbl[5:]
1912            else:
1913                hlbl = 'Hist='+hlbl
1914            s = str(hlbl)
1915    if not s:
1916        s = 'Global'
1917    return s,l[-1]
1918
1919def getVarDescr(varname):
1920    '''Return a short description for a GSAS-II variable
1921
1922    :param str name: A full G2 variable name with 2 or 3 or 4
1923       colons (<p>:<h>:name[:<a1>][:<a2>])
1924
1925    :returns: a six element list as [`p`,`h`,`name`,`a1`,`a2`,`description`],
1926      where `p`, `h`, `a1`, `a2` are str values or `None`, for the phase number,
1927      the histogram number and the atom number; `name` will always be
1928      a str; and `description` is str or `None`.
1929      If the variable name is incorrectly formed (for example, wrong
1930      number of colons), `None` is returned instead of a list.
1931    '''
1932    l = varname.split(':')
1933    if len(l) == 2:     #SASD parameter name
1934        return varname,l[0],getDescr(l[1])
1935    if len(l) == 3:
1936        l += [None,None]
1937    elif len(l) == 4:
1938        l += [None]
1939    elif len(l) != 5:
1940        return None
1941    for i in (0,1,3,4):
1942        if l[i] == "":
1943            l[i] = None
1944    l += [getDescr(l[2])]
1945    return l
1946
1947def CompileVarDesc():
1948    '''Set the values in the variable lookup tables
1949    (:attr:`reVarDesc` and :attr:`reVarStep`).
1950    This is called in :func:`getDescr` and :func:`getVarStep` so this
1951    initialization is always done before use.
1952
1953    Note that keys may contain regular expressions, where '[xyz]'
1954    matches 'x' 'y' or 'z' (equivalently '[x-z]' describes this as range
1955    of values). '.*' matches any string. For example::
1956
1957    'AUiso':'Atomic isotropic displacement parameter',
1958
1959    will match variable ``'p::AUiso:a'``.
1960    If parentheses are used in the key, the contents of those parentheses can be
1961    used in the value, such as::
1962
1963    'AU([123][123])':'Atomic anisotropic displacement parameter U\\1',
1964
1965    will match ``AU11``, ``AU23``,... and `U11`, `U23` etc will be displayed
1966    in the value when used.
1967
1968    '''
1969    if reVarDesc: return # already done
1970    for key,value in {
1971        # derived or other sequential vars
1972        '([abc])$' : 'Lattice parameter, \\1, from Ai and Djk', # N.B. '$' prevents match if any characters follow
1973        u'\u03B1' : u'Lattice parameter, \u03B1, computed from both Ai and Djk',
1974        u'\u03B2' : u'Lattice parameter, \u03B2, computed from both Ai and Djk',
1975        u'\u03B3' : u'Lattice parameter, \u03B3, computed from both Ai and Djk',
1976        # ambiguous, alas:
1977        'Scale' : 'Phase fraction (as p:h:Scale) or Histogram scale factor (as :h:Scale)',
1978        # Phase vars (p::<var>)
1979        'A([0-5])' : ('Reciprocal metric tensor component \\1',1e-5),
1980        '([vV]ol)' : 'Unit cell volume', # probably an error that both upper and lower case are used
1981        # Atom vars (p::<var>:a)
1982        'dA([xyz])$' : ('Refined change to atomic coordinate, \\1',1e-6),
1983        'A([xyz])$' : 'Fractional atomic coordinate, \\1',
1984        'AUiso':('Atomic isotropic displacement parameter',1e-4),
1985        'AU([123][123])':('Atomic anisotropic displacement parameter U\\1',1e-4),
1986        'Afrac': ('Atomic site fraction parameter',1e-5),
1987        'Amul': 'Atomic site multiplicity value',
1988        'AM([xyz])$' : 'Atomic magnetic moment parameter, \\1',
1989        # Hist (:h:<var>) & Phase (HAP) vars (p:h:<var>)
1990        'Back(.*)': 'Background term #\\1',
1991        'BkPkint;(.*)':'Background peak #\\1 intensity',
1992        'BkPkpos;(.*)':'Background peak #\\1 position',
1993        'BkPksig;(.*)':'Background peak #\\1 Gaussian width',
1994        'BkPkgam;(.*)':'Background peak #\\1 Cauchy width',
1995#        'Back File' : 'Background file name',
1996        'BF mult' : 'Background file multiplier',
1997        'Bab([AU])': 'Babinet solvent scattering coef. \\1',
1998        'D([123][123])' : 'Anisotropic strain coef. \\1',
1999        'Extinction' : 'Extinction coef.',
2000        'MD' : 'March-Dollase coef.',
2001        'Mustrain;.*' : 'Microstrain coefficient (delta Q/Q x 10**6)',
2002        'Size;.*' : 'Crystallite size value (in microns)',
2003        'eA$' : 'Cubic mustrain value',
2004        'Ep$' : 'Primary extinction',
2005        'Es$' : 'Secondary type II extinction',
2006        'Eg$' : 'Secondary type I extinction',
2007        'Flack' : 'Flack parameter',
2008        'TwinFr' : 'Twin fraction',
2009        'Layer Disp'  : 'Layer displacement along beam',
2010        #Histogram vars (:h:<var>)
2011        'Absorption' : 'Absorption coef.',
2012        'LayerDisp'  : 'Bragg-Brentano Layer displacement',
2013        'Displace([XY])' : ('Debye-Scherrer sample displacement \\1',0.1),
2014        'Lam' : ('Wavelength',1e-6),
2015        'I\\(L2\\)\\/I\\(L1\\)' : ('Ka2/Ka1 intensity ratio',0.001),
2016        'Polariz.' : ('Polarization correction',1e-3),
2017        'SH/L' : ('FCJ peak asymmetry correction',1e-4),
2018        '([UVW])$' : ('Gaussian instrument broadening \\1',1e-5),
2019        '([XYZ])$' : ('Cauchy instrument broadening \\1',1e-5),
2020        'Zero' : 'Debye-Scherrer zero correction',
2021        'Shift' : 'Bragg-Brentano sample displ.',
2022        'SurfRoughA' : 'Bragg-Brenano surface roughness A',
2023        'SurfRoughB' : 'Bragg-Brenano surface roughness B',
2024        'Transparency' : 'Bragg-Brentano sample tranparency',
2025        'DebyeA' : 'Debye model amplitude',
2026        'DebyeR' : 'Debye model radius',
2027        'DebyeU' : 'Debye model Uiso',
2028        'RBV.*' : 'Vector rigid body parameter',
2029        'RBVO([aijk])' : 'Vector rigid body orientation parameter \\1',
2030        'RBVP([xyz])' : 'Vector rigid body \\1 position parameter',
2031        'RBVf' : 'Vector rigid body site fraction',
2032        'RBV([TLS])([123AB][123AB])' : 'Residue rigid body group disp. param.',
2033        'RBVU' : 'Residue rigid body group Uiso param.',
2034        'RBRO([aijk])' : 'Residue rigid body orientation parameter \\1',
2035        'RBRP([xyz])' : 'Residue rigid body \\1 position parameter',
2036        'RBRTr;.*' : 'Residue rigid body torsion parameter',
2037        'RBRf' : 'Residue rigid body site fraction',
2038        'RBR([TLS])([123AB][123AB])' : 'Residue rigid body group disp. param.',
2039        'RBRU' : 'Residue rigid body group Uiso param.',
2040        'constr([0-9]*)' : 'Generated degree of freedom from constraint',
2041        'nv-(.+)' : 'New variable assignment with name \\1',
2042        # supersymmetry parameters  p::<var>:a:o 'Flen','Fcent'?
2043        'mV([0-2])$' : 'Modulation vector component \\1',
2044        'Fsin'  :   'Sin site fraction modulation',
2045        'Fcos'  :   'Cos site fraction modulation',
2046        'Fzero'  :   'Crenel function offset',      #may go away
2047        'Fwid'   :   'Crenel function width',
2048        'Tmin'   :   'ZigZag/Block min location',
2049        'Tmax'   :   'ZigZag/Block max location',
2050        '([XYZ])max': 'ZigZag/Block max value for \\1',
2051        '([XYZ])sin'  : 'Sin position wave for \\1',
2052        '([XYZ])cos'  : 'Cos position wave for \\1',
2053        'U([123][123])sin$' :  'Sin thermal wave for U\\1',
2054        'U([123][123])cos$' :  'Cos thermal wave for U\\1',
2055        'M([XYZ])sin$' :  'Sin mag. moment wave for \\1',
2056        'M([XYZ])cos$' :  'Cos mag. moment wave for \\1',
2057        # PDF peak parms (l:<var>;l = peak no.)
2058        'PDFpos'  : 'PDF peak position',
2059        'PDFmag'  : 'PDF peak magnitude',
2060        'PDFsig'  : 'PDF peak std. dev.',
2061        # SASD vars (l:<var>;l = component)
2062        'Aspect ratio' : 'Particle aspect ratio',
2063        'Length' : 'Cylinder length',
2064        'Diameter' : 'Cylinder/disk diameter',
2065        'Thickness' : 'Disk thickness',
2066        'Shell thickness' : 'Multiplier to get inner(<1) or outer(>1) sphere radius',
2067        'Dist' : 'Interparticle distance',
2068        'VolFr' : 'Dense scatterer volume fraction',
2069        'epis' : 'Sticky sphere epsilon',
2070        'Sticky' : 'Stickyness',
2071        'Depth' : 'Well depth',
2072        'Width' : 'Well width',
2073        'Volume' : 'Particle volume',
2074        'Radius' : 'Sphere/cylinder/disk radius',
2075        'Mean' : 'Particle mean radius',
2076        'StdDev' : 'Standard deviation in Mean',
2077        'G$': 'Guinier prefactor',
2078        'Rg$': 'Guinier radius of gyration',
2079        'B$': 'Porod prefactor',
2080        'P$': 'Porod power',
2081        'Cutoff': 'Porod cutoff',
2082        'PkInt': 'Bragg peak intensity',
2083        'PkPos': 'Bragg peak position',
2084        'PkSig': 'Bragg peak sigma',
2085        'PkGam': 'Bragg peak gamma',
2086        'e([12][12])' : 'strain tensor e\\1',   # strain vars e11, e22, e12
2087        'Dcalc': 'Calc. d-spacing',
2088        'Back$': 'background parameter',
2089        'pos$': 'peak position',
2090        'int$': 'peak intensity',
2091        'WgtFrac':'phase weight fraction',
2092        'alpha':'TOF profile term',
2093        'alpha-([01])':'Pink profile term',
2094        'beta-([01q])':'TOF/Pink profile term',
2095        'sig-([012q])':'TOF profile term',
2096        'dif([ABC])':'TOF to d-space calibration',
2097        'C\\([0-9]*,[0-9]*\\)' : 'spherical harmonics preferred orientation coef.',
2098        'Pressure': 'Pressure level for measurement in MPa',
2099        'Temperature': 'T value for measurement, K',
2100        'FreePrm([123])': 'User defined measurement parameter \\1',
2101        'Gonio. radius': 'Distance from sample to detector, mm',
2102        }.items():
2103        # Needs documentation: HAP: LeBail, newLeBail
2104        # hist: Azimuth, Chi, Omega, Phi, Bank, nDebye, nPeaks
2105       
2106        if len(value) == 2:
2107            #VarDesc[key] = value[0]
2108            reVarDesc[re.compile(key)] = value[0]
2109            reVarStep[re.compile(key)] = value[1]
2110        else:
2111            #VarDesc[key] = value
2112            reVarDesc[re.compile(key)] = value
2113
2114def removeNonRefined(parmList):
2115    '''Remove items from variable list that are not refined and should not
2116    appear as options for constraints
2117
2118    :param list parmList: a list of strings of form "p:h:VAR:a" where
2119      VAR is the variable name
2120
2121    :returns: a list after removing variables where VAR matches a
2122      entry in local variable NonRefinedList
2123    '''
2124    NonRefinedList = ['Omega','Type','Chi','Phi', 'Azimuth','Gonio. radius',
2125                          'Lam1','Lam2','Back','Temperature','Pressure',
2126                          'FreePrm1','FreePrm2','FreePrm3',
2127                          'Source','nPeaks','LeBail','newLeBail','Bank',
2128                          'nDebye', #'',
2129                    ]
2130    return [prm for prm in parmList if prm.split(':')[2] not in NonRefinedList]
2131       
2132def getDescr(name):
2133    '''Return a short description for a GSAS-II variable
2134
2135    :param str name: The descriptive part of the variable name without colons (:)
2136
2137    :returns: a short description or None if not found
2138    '''
2139
2140    CompileVarDesc() # compile the regular expressions, if needed
2141    for key in reVarDesc:
2142        m = key.match(name)
2143        if m:
2144            reVarDesc[key]
2145            return m.expand(reVarDesc[key])
2146    return None
2147
2148def getVarStep(name,parmDict=None):
2149    '''Return a step size for computing the derivative of a GSAS-II variable
2150
2151    :param str name: A complete variable name (with colons, :)
2152    :param dict parmDict: A dict with parameter values or None (default)
2153
2154    :returns: a float that should be an appropriate step size, either from
2155      the value supplied in :func:`CompileVarDesc` or based on the value for
2156      name in parmDict, if supplied. If not found or the value is zero,
2157      a default value of 1e-5 is used. If parmDict is None (default) and
2158      no value is provided in :func:`CompileVarDesc`, then None is returned.
2159    '''
2160    CompileVarDesc() # compile the regular expressions, if needed
2161    for key in reVarStep:
2162        m = key.match(name)
2163        if m:
2164            return reVarStep[key]
2165    if parmDict is None: return None
2166    val = parmDict.get(key,0.0)
2167    if abs(val) > 0.05:
2168        return abs(val)/1000.
2169    else:
2170        return 1e-5
2171
2172def GenWildCard(varlist):
2173    '''Generate wildcard versions of G2 variables. These introduce '*'
2174    for a phase, histogram or atom number (but only for one of these
2175    fields) but only when there is more than one matching variable in the
2176    input variable list. So if the input is this::
2177
2178      varlist = ['0::AUiso:0', '0::AUiso:1', '1::AUiso:0']
2179
2180    then the output will be this::
2181
2182       wildList = ['*::AUiso:0', '0::AUiso:*']
2183
2184    :param list varlist: an input list of GSAS-II variable names
2185      (such as 0::AUiso:0)
2186
2187    :returns: wildList, the generated list of wild card variable names.
2188    '''
2189    wild = []
2190    for i in (0,1,3):
2191        currentL = varlist[:]
2192        while currentL:
2193            item1 = currentL.pop(0)
2194            i1splt = item1.split(':')
2195            if i >= len(i1splt): continue
2196            if i1splt[i]:
2197                nextL = []
2198                i1splt[i] = '[0-9]+'
2199                rexp = re.compile(':'.join(i1splt))
2200                matchlist = [item1]
2201                for nxtitem in currentL:
2202                    if rexp.match(nxtitem):
2203                        matchlist += [nxtitem]
2204                    else:
2205                        nextL.append(nxtitem)
2206                if len(matchlist) > 1:
2207                    i1splt[i] = '*'
2208                    wild.append(':'.join(i1splt))
2209                currentL = nextL
2210    return wild
2211
2212def LookupWildCard(varname,varlist):
2213    '''returns a list of variable names from list varname
2214    that match wildcard name in varname
2215
2216    :param str varname: a G2 variable name containing a wildcard
2217      (such as \\*::var)
2218    :param list varlist: the list of all variable names used in
2219      the current project
2220    :returns: a list of matching GSAS-II variables (may be empty)
2221    '''
2222    rexp = re.compile(varname.replace('*','[0-9]+'))
2223    return sorted([var for var in varlist if rexp.match(var)])
2224
2225def prmLookup(name,prmDict):
2226    '''Looks for a parameter in a min/max dictionary, optionally
2227    considering a wild card for histogram or atom number (use of
2228    both will never occur at the same time).
2229
2230    :param name: a GSAS-II parameter name (str, see :func:`getVarDescr`
2231      and :func:`CompileVarDesc`) or a :class:`G2VarObj` object.
2232    :param dict prmDict: a min/max dictionary, (parmMinDict
2233      or parmMaxDict in Controls) where keys are :class:`G2VarObj`
2234      objects.
2235    :returns: Two values, (**matchname**, **value**), are returned where:
2236
2237       * **matchname** *(str)* is the :class:`G2VarObj` object
2238         corresponding to the actual matched name,
2239         which could contain a wildcard even if **name** does not; and
2240       * **value** *(float)* which contains the parameter limit.
2241    '''
2242    for key,value in prmDict.items():
2243        if str(key) == str(name): return key,value
2244        if key == name: return key,value
2245    return None,None
2246       
2247
2248def _lookup(dic,key):
2249    '''Lookup a key in a dictionary, where None returns an empty string
2250    but an unmatched key returns a question mark. Used in :class:`G2VarObj`
2251    '''
2252    if key is None:
2253        return ""
2254    elif key == "*":
2255        return "*"
2256    else:
2257        return dic.get(key,'?')
2258
2259def SortVariables(varlist):
2260    '''Sorts variable names in a sensible manner
2261    '''
2262    def cvnnums(var):
2263        v = []
2264        for i in var.split(':'):
2265#            if i == '' or i == '*':
2266#                v.append(-1)
2267#                continue
2268            try:
2269                v.append(int(i))
2270            except:
2271                v.append(-1)
2272        return v
2273    return sorted(varlist,key=cvnnums)
2274
2275class G2VarObj(object):
2276    '''Defines a GSAS-II variable either using the phase/atom/histogram
2277    unique Id numbers or using a character string that specifies
2278    variables by phase/atom/histogram number (which can change).
2279    Note that :func:`GSASIIstrIO.GetUsedHistogramsAndPhases`,
2280    which calls :func:`IndexAllIds` (or
2281    :func:`GSASIIscriptable.G2Project.index_ids`) should be used to
2282    (re)load the current Ids
2283    before creating or later using the G2VarObj object.
2284
2285    This can store rigid body variables, but does not translate the residue # and
2286    body # to/from random Ids
2287
2288    A :class:`G2VarObj` object can be created with a single parameter:
2289
2290    :param str/tuple varname: a single value can be used to create a :class:`G2VarObj`
2291      object. If a string, it must be of form "p:h:var" or "p:h:var:a", where
2292
2293     * p is the phase number (which may be left blank or may be '*' to indicate all phases);
2294     * h is the histogram number (which may be left blank or may be '*' to indicate all histograms);
2295     * a is the atom number (which may be left blank in which case the third colon is omitted).
2296       The atom number can be specified as '*' if a phase number is specified (not as '*').
2297       For rigid body variables, specify a will be a string of form "residue:body#"
2298
2299      Alternately a single tuple of form (Phase,Histogram,VarName,AtomID) can be used, where
2300      Phase, Histogram, and AtomID are None or are ranId values (or one can be '*')
2301      and VarName is a string. Note that if Phase is '*' then the AtomID is an atom number.
2302      For a rigid body variables, AtomID is a string of form "residue:body#".
2303
2304    If four positional arguments are supplied, they are:
2305
2306    :param str/int phasenum: The number for the phase (or None or '*')
2307    :param str/int histnum: The number for the histogram (or None or '*')
2308    :param str varname: a single value can be used to create a :class:`G2VarObj`
2309    :param str/int atomnum: The number for the atom (or None or '*')
2310
2311    '''
2312    IDdict = {}
2313    IDdict['phases'] = {}
2314    IDdict['hists'] = {}
2315    IDdict['atoms'] = {}
2316    def __init__(self,*args):
2317        self.phase = None
2318        self.histogram = None
2319        self.name = ''
2320        self.atom = None
2321        if len(args) == 1 and (type(args[0]) is list or type(args[0]) is tuple) and len(args[0]) == 4:
2322            # single arg with 4 values
2323            self.phase,self.histogram,self.name,self.atom = args[0]
2324        elif len(args) == 1 and ':' in args[0]:
2325            #parse a string
2326            lst = args[0].split(':')
2327            if lst[0] == '*':
2328                self.phase = '*'
2329                if len(lst) > 3:
2330                    self.atom = lst[3]
2331                self.histogram = HistIdLookup.get(lst[1],[None,None])[1]
2332            elif lst[1] == '*':
2333                self.histogram = '*'
2334                self.phase = PhaseIdLookup.get(lst[0],[None,None])[1]
2335            else:
2336                self.histogram = HistIdLookup.get(lst[1],[None,None])[1]
2337                self.phase = PhaseIdLookup.get(lst[0],[None,None])[1]
2338                if len(lst) == 4:
2339                    if lst[3] == '*':
2340                        self.atom = '*'
2341                    else:
2342                        self.atom = AtomIdLookup[lst[0]].get(lst[3],[None,None])[1]
2343                elif len(lst) == 5:
2344                    self.atom = lst[3]+":"+lst[4]
2345                elif len(lst) == 3:
2346                    pass
2347                else:
2348                    raise Exception("Incorrect number of colons in var name "+str(args[0]))
2349            self.name = lst[2]
2350        elif len(args) == 4:
2351            if args[0] == '*':
2352                self.phase = '*'
2353                self.atom = args[3]
2354            else:
2355                self.phase = PhaseIdLookup.get(str(args[0]),[None,None])[1]
2356                if args[3] == '*':
2357                    self.atom = '*'
2358                elif args[0] is not None:
2359                    self.atom = AtomIdLookup[args[0]].get(str(args[3]),[None,None])[1]
2360            if args[1] == '*':
2361                self.histogram = '*'
2362            else:
2363                self.histogram = HistIdLookup.get(str(args[1]),[None,None])[1]
2364            self.name = args[2]
2365        else:
2366            raise Exception("Incorrectly called GSAS-II parameter name")
2367
2368        #print "DEBUG: created ",self.phase,self.histogram,self.name,self.atom
2369
2370    def __str__(self):
2371        return self.varname()
2372
2373    def __hash__(self):
2374        'Allow G2VarObj to be a dict key by implementing hashing'
2375        return hash(self.varname())
2376
2377    def varname(self,hist=None):
2378        '''Formats the GSAS-II variable name as a "traditional" GSAS-II variable
2379        string (p:h:<var>:a) or (p:h:<var>)
2380
2381        :param str/int hist: if specified, overrides the histogram number
2382          with the specified value
2383        :returns: the variable name as a str
2384        '''
2385        a = ""
2386        if self.phase == "*":
2387            ph = "*"
2388            if self.atom:
2389                a = ":" + str(self.atom)
2390        else:
2391            ph = _lookup(PhaseRanIdLookup,self.phase)
2392            if self.atom == '*':
2393                a = ':*'
2394            elif self.atom:
2395                if ":" in str(self.atom):
2396                    a = ":" + str(self.atom)
2397                elif ph in AtomRanIdLookup:
2398                    a = ":" + AtomRanIdLookup[ph].get(self.atom,'?')
2399                else:
2400                    a = ":?"
2401        if hist is not None and self.histogram:
2402            hist = str(hist)
2403        elif self.histogram == "*":
2404            hist = "*"
2405        else:
2406            hist = _lookup(HistRanIdLookup,self.histogram)
2407        s = (ph + ":" + hist + ":" + str(self.name)) + a
2408        return s
2409
2410    def __repr__(self):
2411        '''Return the detailed contents of the object
2412        '''
2413        s = "<"
2414        if self.phase == '*':
2415            s += "Phases: all; "
2416            if self.atom is not None:
2417                if ":" in str(self.atom):
2418                    s += "Rigid body" + str(self.atom) + "; "
2419                else:
2420                    s += "Atom #" + str(self.atom) + "; "
2421        elif self.phase is not None:
2422            ph =  _lookup(PhaseRanIdLookup,self.phase)
2423            s += "Phase: rId=" + str(self.phase) + " (#"+ ph + "); "
2424            if self.atom == '*':
2425                s += "Atoms: all; "
2426            elif ":" in str(self.atom):
2427                s += "Rigid body" + str(self.atom) + "; "
2428            elif self.atom is not None:
2429                s += "Atom rId=" + str(self.atom)
2430                if ph in AtomRanIdLookup:
2431                    s += " (#" + AtomRanIdLookup[ph].get(self.atom,'?') + "); "
2432                else:
2433                    s += " (#? -- not found!); "
2434        if self.histogram == '*':
2435            s += "Histograms: all; "
2436        elif self.histogram is not None:
2437            hist = _lookup(HistRanIdLookup,self.histogram)
2438            s += "Histogram: rId=" + str(self.histogram) + " (#"+ hist + "); "
2439        s += 'Variable name="' + str(self.name) + '">'
2440        return s+" ("+self.varname()+")"
2441
2442    def __eq__(self, other):
2443        '''Allow comparison of G2VarObj to other G2VarObj objects or strings.
2444        If any field is a wildcard ('*') that field matches.
2445        '''
2446        if type(other) is str:
2447            other = G2VarObj(other)
2448        elif type(other) is not G2VarObj:
2449            raise Exception("Invalid type ({}) for G2VarObj comparison with {}"
2450                            .format(type(other),other))
2451        if self.phase != other.phase and self.phase != '*' and other.phase != '*':
2452            return False
2453        if self.histogram != other.histogram and self.histogram != '*' and other.histogram != '*':
2454            return False
2455        if self.atom != other.atom and self.atom != '*' and other.atom != '*':
2456            return False
2457        if self.name != other.name:
2458            return False
2459        return True
2460   
2461    def fmtVarByMode(self, seqmode, note, warnmsg):
2462        '''Format a parameter object for display. Note that these changes
2463        are only temporary and are only shown only when the Constraints
2464        data tree is selected.
2465
2466        * In a non-sequential refinement or where the mode is 'use-all', the
2467          name is converted unchanged to a str
2468        * In a sequential refinement when the mode is 'wildcards-only' the
2469          name is converted unchanged to a str but a warning is added
2470          for non-wildcarded HAP or Histogram parameters
2471        * In a sequential refinement or where the mode is 'auto-wildcard',
2472          a histogram number is converted to a wildcard (*) and then
2473          converted to str
2474
2475        :param str mode: the sequential mode (see above)
2476        :param str note: value displayed on the line of the constraint/equiv.
2477        :param str warnmsg: a message saying the constraint is not used
2478
2479        :returns: varname, explain, note, warnmsg (all str values) where:
2480
2481          * varname is the parameter expressed as a string,
2482          * explain is blank unless there is a warning explanation about
2483            the parameter or blank
2484          * note is the previous value unless overridden
2485          * warnmsg is the previous value unless overridden
2486        '''
2487        explain = ''
2488        s = self.varname()
2489        if seqmode == 'auto-wildcard':
2490            if self.histogram: s = self.varname('*')
2491        elif seqmode == 'wildcards-only' and self.histogram:
2492            if self.histogram != '*':
2493                warnmsg = 'Ignored due to use of a non-wildcarded histogram number'
2494                note = 'Ignored'
2495                explain = '\nIgnoring: '+self.varname()+' does not contain a wildcard.\n'
2496        elif seqmode != 'use-all' and seqmode != 'wildcards-only':
2497            print('Unexpected mode',seqmode,' in fmtVarByMode')
2498        return s,explain,note,warnmsg
2499
2500    def _show(self):
2501        'For testing, shows the current lookup table'
2502        print ('phases'+ self.IDdict['phases'])
2503        print ('hists'+ self.IDdict['hists'])
2504        print ('atomDict'+ self.IDdict['atoms'])
2505
2506#==========================================================================
2507def SetDefaultSample():
2508    'Fills in default items for the Sample dictionary for Debye-Scherrer & SASD'
2509    return {
2510        'InstrName':'',
2511        'ranId':ran.randint(0,sys.maxsize),
2512        'Scale':[1.0,True],'Type':'Debye-Scherrer','Absorption':[0.0,False],
2513        'DisplaceX':[0.0,False],'DisplaceY':[0.0,False],
2514        'Temperature':300.,'Pressure':0.1,'Time':0.0,
2515        'FreePrm1':0.,'FreePrm2':0.,'FreePrm3':0.,
2516        'Gonio. radius':200.0,
2517        'Omega':0.0,'Chi':0.0,'Phi':0.0,'Azimuth':0.0,
2518#SASD items
2519        'Materials':[{'Name':'vacuum','VolFrac':1.0,},{'Name':'vacuum','VolFrac':0.0,}],
2520        'Thick':1.0,'Contrast':[0.0,0.0],       #contrast & anomalous contrast
2521        'Trans':1.0,                            #measured transmission
2522        'SlitLen':0.0,                          #Slit length - in Q(A-1)
2523        }
2524######################################################################
2525class ImportBaseclass(object):
2526    '''Defines a base class for the reading of input files (diffraction
2527    data, coordinates,...). See :ref:`Writing a Import Routine<import_routines>`
2528    for an explanation on how to use a subclass of this class.
2529    '''
2530    class ImportException(Exception):
2531        '''Defines an Exception that is used when an import routine hits an expected error,
2532        usually in .Reader.
2533
2534        Good practice is that the Reader should define a value in self.errors that
2535        tells the user some information about what is wrong with their file.
2536        '''
2537        pass
2538
2539    UseReader = True  # in __init__ set value of self.UseReader to False to skip use of current importer
2540    def __init__(self,formatName,longFormatName=None,
2541                 extensionlist=[],strictExtension=False,):
2542        self.formatName = formatName # short string naming file type
2543        if longFormatName: # longer string naming file type
2544            self.longFormatName = longFormatName
2545        else:
2546            self.longFormatName = formatName
2547        # define extensions that are allowed for the file type
2548        # for windows, remove any extensions that are duplicate, as case is ignored
2549        if sys.platform == 'windows' and extensionlist:
2550            extensionlist = list(set([s.lower() for s in extensionlist]))
2551        self.extensionlist = extensionlist
2552        # If strictExtension is True, the file will not be read, unless
2553        # the extension matches one in the extensionlist
2554        self.strictExtension = strictExtension
2555        self.errors = ''
2556        self.warnings = ''
2557        self.SciPy = False          #image reader needed scipy
2558        # used for readers that will use multiple passes to read
2559        # more than one data block
2560        self.repeat = False
2561        self.selections = []
2562        self.repeatcount = 0
2563        self.readfilename = '?'
2564        self.scriptable = False
2565        #print 'created',self.__class__
2566
2567    def ReInitialize(self):
2568        'Reinitialize the Reader to initial settings'
2569        self.errors = ''
2570        self.warnings = ''
2571        self.SciPy = False          #image reader needed scipy
2572        self.repeat = False
2573        self.repeatcount = 0
2574        self.readfilename = '?'
2575
2576
2577#    def Reader(self, filename, filepointer, ParentFrame=None, **unused):
2578#        '''This method must be supplied in the child class to read the file.
2579#        if the read fails either return False or raise an Exception
2580#        preferably of type ImportException.
2581#        '''
2582#        #start reading
2583#        raise ImportException("Error occurred while...")
2584#        self.errors += "Hint for user on why the error occur
2585#        return False # if an error occurs
2586#        return True # if read OK
2587
2588    def ExtensionValidator(self, filename):
2589        '''This methods checks if the file has the correct extension
2590       
2591        :returns:
2592       
2593          * False if this filename will not be supported by this reader (only
2594            when strictExtension is True)
2595          * True if the extension matches the list supplied by the reader
2596          * None if the reader allows un-registered extensions
2597         
2598        '''
2599        if filename:
2600            ext = os.path.splitext(filename)[1]
2601            if not ext and self.strictExtension: return False
2602            for ext in self.extensionlist:               
2603                if sys.platform == 'windows':
2604                    if filename.lower().endswith(ext): return True
2605                else:
2606                    if filename.endswith(ext): return True
2607        if self.strictExtension:
2608            return False
2609        else:
2610            return None
2611
2612    def ContentsValidator(self, filename):
2613        '''This routine will attempt to determine if the file can be read
2614        with the current format.
2615        This will typically be overridden with a method that
2616        takes a quick scan of [some of]
2617        the file contents to do a "sanity" check if the file
2618        appears to match the selected format.
2619        the file must be opened here with the correct format (binary/text)
2620        '''
2621        #filepointer.seek(0) # rewind the file pointer
2622        return True
2623
2624    def CIFValidator(self, filepointer):
2625        '''A :meth:`ContentsValidator` for use to validate CIF files.
2626        '''
2627        filepointer.seek(0)
2628        for i,l in enumerate(filepointer):
2629            if i >= 1000: return True
2630            '''Encountered only blank lines or comments in first 1000
2631            lines. This is unlikely, but assume it is CIF anyway, since we are
2632            even less likely to find a file with nothing but hashes and
2633            blank lines'''
2634            line = l.strip()
2635            if len(line) == 0: # ignore blank lines
2636                continue
2637            elif line.startswith('#'): # ignore comments
2638                continue
2639            elif line.startswith('data_'): # on the right track, accept this file
2640                return True
2641            else: # found something invalid
2642                self.errors = 'line '+str(i+1)+' contains unexpected data:\n'
2643                if all([ord(c) < 128 and ord(c) != 0 for c in str(l)]): # show only if ASCII
2644                    self.errors += '  '+str(l)
2645                else:
2646                    self.errors += '  (binary)'
2647                self.errors += '\n  Note: a CIF should only have blank lines or comments before'
2648                self.errors += '\n        a data_ statement begins a block.'
2649                return False
2650
2651######################################################################
2652class ImportPhase(ImportBaseclass):
2653    '''Defines a base class for the reading of files with coordinates
2654
2655    Objects constructed that subclass this (in import/G2phase_*.py etc.) will be used
2656    in :meth:`GSASIIdataGUI.GSASII.OnImportPhase` and in
2657    :func:`GSASIIscriptable.import_generic`.
2658    See :ref:`Writing a Import Routine<import_routines>`
2659    for an explanation on how to use this class.
2660
2661    '''
2662    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2663        strictExtension=False,):
2664        # call parent __init__
2665        ImportBaseclass.__init__(self,formatName,longFormatName,
2666            extensionlist,strictExtension)
2667        self.Phase = None # a phase must be created with G2IO.SetNewPhase in the Reader
2668        self.SymOps = {} # specified when symmetry ops are in file (e.g. CIF)
2669        self.Constraints = None
2670
2671######################################################################
2672class ImportStructFactor(ImportBaseclass):
2673    '''Defines a base class for the reading of files with tables
2674    of structure factors.
2675
2676    Structure factors are read with a call to :meth:`GSASIIdataGUI.GSASII.OnImportSfact`
2677    which in turn calls :meth:`GSASIIdataGUI.GSASII.OnImportGeneric`, which calls
2678    methods :meth:`ExtensionValidator`, :meth:`ContentsValidator` and
2679    :meth:`Reader`.
2680
2681    See :ref:`Writing a Import Routine<import_routines>`
2682    for an explanation on how to use import classes in general. The specifics
2683    for reading a structure factor histogram require that
2684    the ``Reader()`` routine in the import
2685    class need to do only a few things: It
2686    should load :attr:`RefDict` item ``'RefList'`` with the reflection list,
2687    and set :attr:`Parameters` with the instrument parameters
2688    (initialized with :meth:`InitParameters` and set with :meth:`UpdateParameters`).
2689    '''
2690    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2691        strictExtension=False,):
2692        ImportBaseclass.__init__(self,formatName,longFormatName,
2693            extensionlist,strictExtension)
2694
2695        # define contents of Structure Factor entry
2696        self.Parameters = []
2697        'self.Parameters is a list with two dicts for data parameter settings'
2698        self.InitParameters()
2699        self.RefDict = {'RefList':[],'FF':{},'Super':0}
2700        self.Banks = []             #for multi bank data (usually TOF)
2701        '''self.RefDict is a dict containing the reflection information, as read from the file.
2702        Item 'RefList' contains the reflection information. See the
2703        :ref:`Single Crystal Reflection Data Structure<XtalRefl_table>`
2704        for the contents of each row. Dict element 'FF'
2705        contains the form factor values for each element type; if this entry
2706        is left as initialized (an empty list) it will be initialized as needed later.
2707        '''
2708    def ReInitialize(self):
2709        'Reinitialize the Reader to initial settings'
2710        ImportBaseclass.ReInitialize(self)
2711        self.InitParameters()
2712        self.Banks = []             #for multi bank data (usually TOF)
2713        self.RefDict = {'RefList':[],'FF':{},'Super':0}
2714
2715    def InitParameters(self):
2716        'initialize the instrument parameters structure'
2717        Lambda = 0.70926
2718        HistType = 'SXC'
2719        self.Parameters = [{'Type':[HistType,HistType], # create the structure
2720                            'Lam':[Lambda,Lambda]
2721                            }, {}]
2722        'Parameters is a list with two dicts for data parameter settings'
2723
2724    def UpdateParameters(self,Type=None,Wave=None):
2725        'Revise the instrument parameters'
2726        if Type is not None:
2727            self.Parameters[0]['Type'] = [Type,Type]
2728        if Wave is not None:
2729            self.Parameters[0]['Lam'] = [Wave,Wave]
2730
2731######################################################################
2732class ImportPowderData(ImportBaseclass):
2733    '''Defines a base class for the reading of files with powder data.
2734
2735    Objects constructed that subclass this (in import/G2pwd_*.py etc.) will be used
2736    in :meth:`GSASIIdataGUI.GSASII.OnImportPowder` and in
2737    :func:`GSASIIscriptable.import_generic`.
2738    See :ref:`Writing a Import Routine<import_routines>`
2739    for an explanation on how to use this class.
2740    '''
2741    def __init__(self,formatName,longFormatName=None,
2742        extensionlist=[],strictExtension=False,):
2743        ImportBaseclass.__init__(self,formatName,longFormatName,
2744            extensionlist,strictExtension)
2745        self.clockWd = None  # used in TOF
2746        self.ReInitialize()
2747
2748    def ReInitialize(self):
2749        'Reinitialize the Reader to initial settings'
2750        ImportBaseclass.ReInitialize(self)
2751        self.powderentry = ['',None,None] #  (filename,Pos,Bank)
2752        self.powderdata = [] # Powder dataset
2753        '''A powder data set is a list with items [x,y,w,yc,yb,yd]:
2754                np.array(x), # x-axis values
2755                np.array(y), # powder pattern intensities
2756                np.array(w), # 1/sig(intensity)^2 values (weights)
2757                np.array(yc), # calc. intensities (zero)
2758                np.array(yb), # calc. background (zero)
2759                np.array(yd), # obs-calc profiles
2760        '''
2761        self.comments = []
2762        self.idstring = ''
2763        self.Sample = SetDefaultSample() # default sample parameters
2764        self.Controls = {}  # items to be placed in top-level Controls
2765        self.GSAS = None     # used in TOF
2766        self.repeat_instparm = True # Should a parm file be
2767        #                             used for multiple histograms?
2768        self.instparm = None # name hint from file of instparm to use
2769        self.instfile = '' # full path name to instrument parameter file
2770        self.instbank = '' # inst parm bank number
2771        self.instmsg = ''  # a label that gets printed to show
2772                           # where instrument parameters are from
2773        self.numbanks = 1
2774        self.instdict = {} # place items here that will be transferred to the instrument parameters
2775        self.pwdparms = {} # place parameters that are transferred directly to the tree
2776                           # here (typically from an existing GPX file)
2777######################################################################
2778class ImportSmallAngleData(ImportBaseclass):
2779    '''Defines a base class for the reading of files with small angle data.
2780    See :ref:`Writing a Import Routine<import_routines>`
2781    for an explanation on how to use this class.
2782    '''
2783    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2784        strictExtension=False,):
2785
2786        ImportBaseclass.__init__(self,formatName,longFormatName,extensionlist,
2787            strictExtension)
2788        self.ReInitialize()
2789
2790    def ReInitialize(self):
2791        'Reinitialize the Reader to initial settings'
2792        ImportBaseclass.ReInitialize(self)
2793        self.smallangleentry = ['',None,None] #  (filename,Pos,Bank)
2794        self.smallangledata = [] # SASD dataset
2795        '''A small angle data set is a list with items [x,y,w,yc,yd]:
2796                np.array(x), # x-axis values
2797                np.array(y), # powder pattern intensities
2798                np.array(w), # 1/sig(intensity)^2 values (weights)
2799                np.array(yc), # calc. intensities (zero)
2800                np.array(yd), # obs-calc profiles
2801                np.array(yb), # preset bkg
2802        '''
2803        self.comments = []
2804        self.idstring = ''
2805        self.Sample = SetDefaultSample()
2806        self.GSAS = None     # used in TOF
2807        self.clockWd = None  # used in TOF
2808        self.numbanks = 1
2809        self.instdict = {} # place items here that will be transferred to the instrument parameters
2810
2811######################################################################
2812class ImportReflectometryData(ImportBaseclass):
2813    '''Defines a base class for the reading of files with reflectometry data.
2814    See :ref:`Writing a Import Routine<import_routines>`
2815    for an explanation on how to use this class.
2816    '''
2817    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2818        strictExtension=False,):
2819
2820        ImportBaseclass.__init__(self,formatName,longFormatName,extensionlist,
2821            strictExtension)
2822        self.ReInitialize()
2823
2824    def ReInitialize(self):
2825        'Reinitialize the Reader to initial settings'
2826        ImportBaseclass.ReInitialize(self)
2827        self.reflectometryentry = ['',None,None] #  (filename,Pos,Bank)
2828        self.reflectometrydata = [] # SASD dataset
2829        '''A small angle data set is a list with items [x,y,w,yc,yd]:
2830                np.array(x), # x-axis values
2831                np.array(y), # powder pattern intensities
2832                np.array(w), # 1/sig(intensity)^2 values (weights)
2833                np.array(yc), # calc. intensities (zero)
2834                np.array(yd), # obs-calc profiles
2835                np.array(yb), # preset bkg
2836        '''
2837        self.comments = []
2838        self.idstring = ''
2839        self.Sample = SetDefaultSample()
2840        self.GSAS = None     # used in TOF
2841        self.clockWd = None  # used in TOF
2842        self.numbanks = 1
2843        self.instdict = {} # place items here that will be transferred to the instrument parameters
2844
2845######################################################################
2846class ImportPDFData(ImportBaseclass):
2847    '''Defines a base class for the reading of files with PDF G(R) data.
2848    See :ref:`Writing a Import Routine<import_routines>`
2849    for an explanation on how to use this class.
2850    '''
2851    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2852        strictExtension=False,):
2853
2854        ImportBaseclass.__init__(self,formatName,longFormatName,extensionlist,
2855            strictExtension)
2856        self.ReInitialize()
2857
2858    def ReInitialize(self):
2859        'Reinitialize the Reader to initial settings'
2860        ImportBaseclass.ReInitialize(self)
2861        self.pdfentry = ['',None,None] #  (filename,Pos,Bank)
2862        self.pdfdata = [] # PDF G(R) dataset
2863        '''A pdf g(r) data set is a list with items [x,y]:
2864                np.array(x), # r-axis values
2865                np.array(y), # pdf g(r)
2866        '''
2867        self.comments = []
2868        self.idstring = ''
2869        self.numbanks = 1
2870
2871######################################################################
2872class ImportImage(ImportBaseclass):
2873    '''Defines a base class for the reading of images
2874
2875    Images are read in only these places:
2876
2877      * Initial reading is typically done from a menu item
2878        with a call to :meth:`GSASIIdataGUI.GSASII.OnImportImage`
2879        which in turn calls :meth:`GSASIIdataGUI.GSASII.OnImportGeneric`. That calls
2880        methods :meth:`ExtensionValidator`, :meth:`ContentsValidator` and
2881        :meth:`Reader`. This returns a list of reader objects for each read image.
2882        Also used in :func:`GSASIIscriptable.import_generic`.
2883
2884      * Images are read alternatively in :func:`GSASIIIO.ReadImages`, which puts image info
2885        directly into the data tree.
2886
2887      * Images are reloaded with :func:`GSASIIIO.GetImageData`.
2888
2889    When reading an image, the ``Reader()`` routine in the ImportImage class
2890    should set:
2891
2892      * :attr:`Comments`: a list of strings (str),
2893      * :attr:`Npix`: the number of pixels in the image (int),
2894      * :attr:`Image`: the actual image as a numpy array (np.array)
2895      * :attr:`Data`: a dict defining image parameters (dict). Within this dict the following
2896        data items are needed:
2897
2898         * 'pixelSize': size of each pixel in microns (such as ``[200.,200.]``.
2899         * 'wavelength': wavelength in :math:`\\AA`.
2900         * 'distance': distance of detector from sample in cm.
2901         * 'center': uncalibrated center of beam on detector (such as ``[204.8,204.8]``.
2902         * 'size': size of image (such as ``[2048,2048]``).
2903         * 'ImageTag': image number or other keyword used to retrieve image from
2904           a multi-image data file (defaults to ``1`` if not specified).
2905         * 'sumfile': holds sum image file name if a sum was produced from a multi image file
2906
2907    optional data items:
2908
2909      * :attr:`repeat`: set to True if there are additional images to
2910        read in the file, False otherwise
2911      * :attr:`repeatcount`: set to the number of the image.
2912
2913    Note that the above is initialized with :meth:`InitParameters`.
2914    (Also see :ref:`Writing a Import Routine<import_routines>`
2915    for an explanation on how to use import classes in general.)
2916    '''
2917    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2918        strictExtension=False,):
2919        ImportBaseclass.__init__(self,formatName,longFormatName,
2920            extensionlist,strictExtension)
2921        self.InitParameters()
2922
2923    def ReInitialize(self):
2924        'Reinitialize the Reader to initial settings -- not used at present'
2925        ImportBaseclass.ReInitialize(self)
2926        self.InitParameters()
2927
2928    def InitParameters(self):
2929        'initialize the instrument parameters structure'
2930        self.Comments = ['No comments']
2931        self.Data = {'samplechangerpos':0.0,'det2theta':0.0,'Gain map':''}
2932        self.Npix = 0
2933        self.Image = None
2934        self.repeat = False
2935        self.repeatcount = 1
2936        self.sumfile = ''
2937
2938    def LoadImage(self,ParentFrame,imagefile,imagetag=None):
2939        '''Optionally, call this after reading in an image to load it into the tree.
2940        This saves time by preventing a reread of the same information.
2941        '''
2942        if ParentFrame:
2943            ParentFrame.ImageZ = self.Image   # store the image for plotting
2944            ParentFrame.oldImagefile = imagefile # save the name of the last image file read
2945            ParentFrame.oldImageTag = imagetag   # save the tag of the last image file read
2946
2947#################################################################################################
2948# shortcut routines
2949exp = np.exp
2950sind = sin = s = lambda x: np.sin(x*np.pi/180.)
2951cosd = cos = c = lambda x: np.cos(x*np.pi/180.)
2952tand = tan = t = lambda x: np.tan(x*np.pi/180.)
2953sqrt = sq = lambda x: np.sqrt(x)
2954pi = lambda: np.pi
2955
2956def FindFunction(f):
2957    '''Find the object corresponding to function f
2958
2959    :param str f: a function name such as 'numpy.exp'
2960    :returns: (pkgdict,pkgobj) where pkgdict contains a dict
2961      that defines the package location(s) and where pkgobj
2962      defines the object associated with the function.
2963      If the function is not found, pkgobj is None.
2964    '''
2965    df = f.split('.')
2966    pkgdict = {}
2967    # no listed module name, try in current namespace
2968    if len(df) == 1:
2969        try:
2970            fxnobj = eval(f)
2971            return pkgdict,fxnobj
2972        except (AttributeError, NameError):
2973            return None,None
2974
2975    # includes a package, see if package is already imported
2976    pkgnam = '.'.join(df[:-1])
2977    try:
2978        fxnobj = eval(f)
2979        pkgdict[pkgnam] = eval(pkgnam)
2980        return pkgdict,fxnobj
2981    except (AttributeError, NameError):
2982        pass
2983    # package not yet imported, so let's try
2984    if '.' not in sys.path: sys.path.append('.')
2985    pkgnam = '.'.join(df[:-1])
2986    #for pkg in f.split('.')[:-1]: # if needed, descend down the tree
2987    #    if pkgname:
2988    #        pkgname += '.' + pkg
2989    #    else:
2990    #        pkgname = pkg
2991    try:
2992        exec('import '+pkgnam)
2993        pkgdict[pkgnam] = eval(pkgnam)
2994        fxnobj = eval(f)
2995    except Exception as msg:
2996        print('load of '+pkgnam+' failed with error='+str(msg))
2997        return {},None
2998    # can we access the function? I am not exactly sure what
2999    #    I intended this to test originally (BHT)
3000    try:
3001        fxnobj = eval(f,globals(),pkgdict)
3002        return pkgdict,fxnobj
3003    except Exception as msg:
3004        print('call to',f,' failed with error=',str(msg))
3005        return None,None # not found
3006               
3007class ExpressionObj(object):
3008    '''Defines an object with a user-defined expression, to be used for
3009    secondary fits or restraints. Object is created null, but is changed
3010    using :meth:`LoadExpression`. This contains only the minimum
3011    information that needs to be stored to save and load the expression
3012    and how it is mapped to GSAS-II variables.
3013    '''
3014    def __init__(self):
3015        self.expression = ''
3016        'The expression as a text string'
3017        self.assgnVars = {}
3018        '''A dict where keys are label names in the expression mapping to a GSAS-II
3019        variable. The value a G2 variable name.
3020        Note that the G2 variable name may contain a wild-card and correspond to
3021        multiple values.
3022        '''
3023        self.freeVars = {}
3024        '''A dict where keys are label names in the expression mapping to a free
3025        parameter. The value is a list with:
3026
3027         * a name assigned to the parameter
3028         * a value for to the parameter and
3029         * a flag to determine if the variable is refined.
3030        '''
3031        self.depVar = None
3032
3033        self.lastError = ('','')
3034        '''Shows last encountered error in processing expression
3035        (list of 1-3 str values)'''
3036
3037        self.distance_dict  = None  # to be used for defining atom phase/symmetry info
3038        self.distance_atoms = None  # to be used for defining atom distances
3039
3040    def LoadExpression(self,expr,exprVarLst,varSelect,varName,varValue,varRefflag):
3041        '''Load the expression and associated settings into the object. Raises
3042        an exception if the expression is not parsed, if not all functions
3043        are defined or if not all needed parameter labels in the expression
3044        are defined.
3045
3046        This will not test if the variable referenced in these definitions
3047        are actually in the parameter dictionary. This is checked when the
3048        computation for the expression is done in :meth:`SetupCalc`.
3049
3050        :param str expr: the expression
3051        :param list exprVarLst: parameter labels found in the expression
3052        :param dict varSelect: this will be 0 for Free parameters
3053          and non-zero for expression labels linked to G2 variables.
3054        :param dict varName: Defines a name (str) associated with each free parameter
3055        :param dict varValue: Defines a value (float) associated with each free parameter
3056        :param dict varRefflag: Defines a refinement flag (bool)
3057          associated with each free parameter
3058        '''
3059        self.expression = expr
3060        self.compiledExpr = None
3061        self.freeVars = {}
3062        self.assgnVars = {}
3063        for v in exprVarLst:
3064            if varSelect[v] == 0:
3065                self.freeVars[v] = [
3066                    varName.get(v),
3067                    varValue.get(v),
3068                    varRefflag.get(v),
3069                    ]
3070            else:
3071                self.assgnVars[v] = varName[v]
3072        self.CheckVars()
3073
3074    def EditExpression(self,exprVarLst,varSelect,varName,varValue,varRefflag):
3075        '''Load the expression and associated settings from the object into
3076        arrays used for editing.
3077
3078        :param list exprVarLst: parameter labels found in the expression
3079        :param dict varSelect: this will be 0 for Free parameters
3080          and non-zero for expression labels linked to G2 variables.
3081        :param dict varName: Defines a name (str) associated with each free parameter
3082        :param dict varValue: Defines a value (float) associated with each free parameter
3083        :param dict varRefflag: Defines a refinement flag (bool)
3084          associated with each free parameter
3085
3086        :returns: the expression as a str
3087        '''
3088        for v in self.freeVars:
3089            varSelect[v] = 0
3090            varName[v] = self.freeVars[v][0]
3091            varValue[v] = self.freeVars[v][1]
3092            varRefflag[v] = self.freeVars[v][2]
3093        for v in self.assgnVars:
3094            varSelect[v] = 1
3095            varName[v] = self.assgnVars[v]
3096        return self.expression
3097
3098    def GetVaried(self):
3099        'Returns the names of the free parameters that will be refined'
3100        return ["::"+self.freeVars[v][0] for v in self.freeVars if self.freeVars[v][2]]
3101
3102    def GetVariedVarVal(self):
3103        'Returns the names and values of the free parameters that will be refined'
3104        return [("::"+self.freeVars[v][0],self.freeVars[v][1]) for v in self.freeVars if self.freeVars[v][2]]
3105
3106    def UpdateVariedVars(self,varyList,values):
3107        'Updates values for the free parameters (after a refinement); only updates refined vars'
3108        for v in self.freeVars:
3109            if not self.freeVars[v][2]: continue
3110            if "::"+self.freeVars[v][0] not in varyList: continue
3111            indx = list(varyList).index("::"+self.freeVars[v][0])
3112            self.freeVars[v][1] = values[indx]
3113
3114    def GetIndependentVars(self):
3115        'Returns the names of the required independent parameters used in expression'
3116        return [self.assgnVars[v] for v in self.assgnVars]
3117
3118    def CheckVars(self):
3119        '''Check that the expression can be parsed, all functions are
3120        defined and that input loaded into the object is internally
3121        consistent. If not an Exception is raised.
3122
3123        :returns: a dict with references to packages needed to
3124          find functions referenced in the expression.
3125        '''
3126        ret = self.ParseExpression(self.expression)
3127        if not ret:
3128            raise Exception("Expression parse error")
3129        exprLblList,fxnpkgdict = ret
3130        # check each var used in expression is defined
3131        defined = list(self.assgnVars.keys()) + list(self.freeVars.keys())
3132        notfound = []
3133        for var in exprLblList:
3134            if var not in defined:
3135                notfound.append(var)
3136        if notfound:
3137            msg = 'Not all variables defined'
3138            msg1 = 'The following variables were not defined: '
3139            msg2 = ''
3140            for var in notfound:
3141                if msg: msg += ', '
3142                msg += var
3143            self.lastError = (msg1,'  '+msg2)
3144            raise Exception(msg)
3145        return fxnpkgdict
3146
3147    def ParseExpression(self,expr):
3148        '''Parse an expression and return a dict of called functions and
3149        the variables used in the expression. Returns None in case an error
3150        is encountered. If packages are referenced in functions, they are loaded
3151        and the functions are looked up into the modules global
3152        workspace.
3153
3154        Note that no changes are made to the object other than
3155        saving an error message, so that this can be used for testing prior
3156        to the save.
3157
3158        :returns: a list of used variables
3159        '''
3160        self.lastError = ('','')
3161        import ast
3162        def ASTtransverse(node,fxn=False):
3163            '''Transverse a AST-parsed expresson, compiling a list of variables
3164            referenced in the expression. This routine is used recursively.
3165
3166            :returns: varlist,fxnlist where
3167              varlist is a list of referenced variable names and
3168              fxnlist is a list of used functions
3169            '''
3170            varlist = []
3171            fxnlist = []
3172            if isinstance(node, list):
3173                for b in node:
3174                    v,f = ASTtransverse(b,fxn)
3175                    varlist += v
3176                    fxnlist += f
3177            elif isinstance(node, ast.AST):
3178                for a, b in ast.iter_fields(node):
3179                    if isinstance(b, ast.AST):
3180                        if a == 'func':
3181                            fxnlist += ['.'.join(ASTtransverse(b,True)[0])]
3182                            continue
3183                        v,f = ASTtransverse(b,fxn)
3184                        varlist += v
3185                        fxnlist += f
3186                    elif isinstance(b, list):
3187                        v,f = ASTtransverse(b,fxn)
3188                        varlist += v
3189                        fxnlist += f
3190                    elif node.__class__.__name__ == "Name":
3191                        varlist += [b]
3192                    elif fxn and node.__class__.__name__ == "Attribute":
3193                        varlist += [b]
3194            return varlist,fxnlist
3195        try:
3196            exprast = ast.parse(expr)
3197        except SyntaxError:
3198            s = ''
3199            import traceback
3200            for i in traceback.format_exc().splitlines()[-3:-1]:
3201                if s: s += "\n"
3202                s += str(i)
3203            self.lastError = ("Error parsing expression:",s)
3204            return
3205        # find the variables & functions
3206        v,f = ASTtransverse(exprast)
3207        varlist = sorted(list(set(v)))
3208        fxnlist = list(set(f))
3209        pkgdict = {}
3210        # check the functions are defined
3211        for fxn in fxnlist:
3212            fxndict,fxnobj = FindFunction(fxn)
3213            if not fxnobj:
3214                self.lastError = ("Error: Invalid function",fxn,
3215                                  "is not defined")
3216                return
3217            if not hasattr(fxnobj,'__call__'):
3218                self.lastError = ("Error: Not a function.",fxn,
3219                                  "cannot be called as a function")
3220                return
3221            pkgdict.update(fxndict)
3222        return varlist,pkgdict
3223
3224    def GetDepVar(self):
3225        'return the dependent variable, or None'
3226        return self.depVar
3227
3228    def SetDepVar(self,var):
3229        'Set the dependent variable, if used'
3230        self.depVar = var
3231#==========================================================================
3232class ExpressionCalcObj(object):
3233    '''An object used to evaluate an expression from a :class:`ExpressionObj`
3234    object.
3235
3236    :param ExpressionObj exprObj: a :class:`~ExpressionObj` expression object with
3237      an expression string and mappings for the parameter labels in that object.
3238    '''
3239    def __init__(self,exprObj):
3240        self.eObj = exprObj
3241        'The expression and mappings; a :class:`ExpressionObj` object'
3242        self.compiledExpr = None
3243        'The expression as compiled byte-code'
3244        self.exprDict = {}
3245        '''dict that defines values for labels used in expression and packages
3246        referenced by functions
3247        '''
3248        self.lblLookup = {}
3249        '''Lookup table that specifies the expression label name that is
3250        tied to a particular GSAS-II parameters in the parmDict.
3251        '''
3252        self.fxnpkgdict = {}
3253        '''a dict with references to packages needed to
3254        find functions referenced in the expression.
3255        '''
3256        self.varLookup = {}
3257        '''Lookup table that specifies the GSAS-II variable(s)
3258        indexed by the expression label name. (Used for only for diagnostics
3259        not evaluation of expression.)
3260        '''
3261        self.su = None
3262        '''Standard error evaluation where supplied by the evaluator
3263        '''
3264        # Patch: for old-style expressions with a (now removed step size)
3265        if '2' in platform.python_version_tuple()[0]: 
3266            basestr = basestring
3267        else:
3268            basestr = str
3269        for v in self.eObj.assgnVars:
3270            if not isinstance(self.eObj.assgnVars[v], basestr):
3271                self.eObj.assgnVars[v] = self.eObj.assgnVars[v][0]
3272        self.parmDict = {}
3273        '''A copy of the parameter dictionary, for distance and angle computation
3274        '''
3275
3276    def SetupCalc(self,parmDict):
3277        '''Do all preparations to use the expression for computation.
3278        Adds the free parameter values to the parameter dict (parmDict).
3279        '''
3280        if self.eObj.expression.startswith('Dist') or self.eObj.expression.startswith('Angle'):
3281            return
3282        self.fxnpkgdict = self.eObj.CheckVars()
3283        # all is OK, compile the expression
3284        self.compiledExpr = compile(self.eObj.expression,'','eval')
3285
3286        # look at first value in parmDict to determine its type
3287        parmsInList = True
3288        if '2' in platform.python_version_tuple()[0]: 
3289            basestr = basestring
3290        else:
3291            basestr = str
3292        for key in parmDict:
3293            val = parmDict[key]
3294            if isinstance(val, basestr):
3295                parmsInList = False
3296                break
3297            try: # check if values are in lists
3298                val = parmDict[key][0]
3299            except (TypeError,IndexError):
3300                parmsInList = False
3301            break
3302
3303        # set up the dicts needed to speed computations
3304        self.exprDict = {}
3305        self.lblLookup = {}
3306        self.varLookup = {}
3307        for v in self.eObj.freeVars:
3308            varname = self.eObj.freeVars[v][0]
3309            varname = "::" + varname.lstrip(':').replace(' ','_').replace(':',';')
3310            self.lblLookup[varname] = v
3311            self.varLookup[v] = varname
3312            if parmsInList:
3313                parmDict[varname] = [self.eObj.freeVars[v][1],self.eObj.freeVars[v][2]]
3314            else:
3315                parmDict[varname] = self.eObj.freeVars[v][1]
3316            self.exprDict[v] = self.eObj.freeVars[v][1]
3317        for v in self.eObj.assgnVars:
3318            varname = self.eObj.assgnVars[v]
3319            if varname in parmDict:
3320                self.lblLookup[varname] = v
3321                self.varLookup[v] = varname
3322                if parmsInList:
3323                    self.exprDict[v] = parmDict[varname][0]
3324                else:
3325                    self.exprDict[v] = parmDict[varname]
3326            elif '*' in varname:
3327                varlist = LookupWildCard(varname,list(parmDict.keys()))
3328                if len(varlist) == 0:
3329                    self.exprDict[v] = None
3330                    self.lblLookup[v] = varname # needed?
3331                    self.exprDict.update(self.fxnpkgdict) # needed?
3332                    return
3333                for var in varlist:
3334                    self.lblLookup[var] = v
3335                if parmsInList:
3336                    self.exprDict[v] = np.array([parmDict[var][0] for var in varlist])
3337                else:
3338                    self.exprDict[v] = np.array([parmDict[var] for var in varlist])
3339                self.varLookup[v] = [var for var in varlist]
3340            else:
3341                self.exprDict[v] = None
3342#                raise Exception,"No value for variable "+str(v)
3343        self.exprDict.update(self.fxnpkgdict)
3344
3345    def UpdateVars(self,varList,valList):
3346        '''Update the dict for the expression with a set of values
3347        :param list varList: a list of variable names
3348        :param list valList: a list of corresponding values
3349        '''
3350        for var,val in zip(varList,valList):
3351            self.exprDict[self.lblLookup.get(var,'undefined: '+var)] = val
3352
3353    def UpdateDict(self,parmDict):
3354        '''Update the dict for the expression with values in a dict
3355        :param dict parmDict: a dict of values, items not in use are ignored
3356        '''
3357        if self.eObj.expression.startswith('Dist') or self.eObj.expression.startswith('Angle'):
3358            self.parmDict = parmDict
3359            return
3360        for var in parmDict:
3361            if var in self.lblLookup:
3362                self.exprDict[self.lblLookup[var]] = parmDict[var]
3363
3364    def EvalExpression(self):
3365        '''Evaluate an expression. Note that the expression
3366        and mapping are taken from the :class:`ExpressionObj` expression object
3367        and the parameter values were specified in :meth:`SetupCalc`.
3368        :returns: a single value for the expression. If parameter
3369        values are arrays (for example, from wild-carded variable names),
3370        the sum of the resulting expression is returned.
3371
3372        For example, if the expression is ``'A*B'``,
3373        where A is 2.0 and B maps to ``'1::Afrac:*'``, which evaluates to::
3374
3375        [0.5, 1, 0.5]
3376
3377        then the result will be ``4.0``.
3378        '''
3379        self.su = None
3380        if self.eObj.expression.startswith('Dist'):
3381#            GSASIIpath.IPyBreak()
3382            dist = G2mth.CalcDist(self.eObj.distance_dict, self.eObj.distance_atoms, self.parmDict)
3383            return dist
3384        elif self.eObj.expression.startswith('Angle'):
3385            angle = G2mth.CalcAngle(self.eObj.angle_dict, self.eObj.angle_atoms, self.parmDict)
3386            return angle
3387        if self.compiledExpr is None:
3388            raise Exception("EvalExpression called before SetupCalc")
3389        try:
3390            val = eval(self.compiledExpr,globals(),self.exprDict)
3391        except TypeError:
3392            val = None
3393        except NameError:
3394            val = None
3395        if not np.isscalar(val):
3396            val = np.sum(val)
3397        return val
3398
3399def makeAngleObj(Phase,Oatom,Tatoms):
3400    General = Phase['General']
3401    cx,ct = General['AtomPtrs'][:2]
3402    pId = Phase['pId']
3403    SGData = General['SGData']
3404    Atoms = Phase['Atoms']
3405    aNames = [atom[ct-1] for atom in Atoms]
3406    tIds = []
3407    symNos = []
3408    cellNos = []
3409    oId = aNames.index(Oatom)
3410    for Tatom in Tatoms.split(';'):
3411        sB = Tatom.find('(')+1
3412        symNo = 0
3413        if sB:
3414            sF = Tatom.find(')')
3415            symNo = int(Tatom[sB:sF])
3416        symNos.append(symNo)
3417        cellNo = [0,0,0]
3418        cB = Tatom.find('[')
3419        if cB>0:
3420            cF = Tatom.find(']')+1
3421            cellNo = eval(Tatom[cB:cF])
3422        cellNos.append(cellNo)
3423        tIds.append(aNames.index(Tatom.split('+')[0]))
3424    # create an expression object
3425    obj = ExpressionObj()
3426    obj.expression = 'Angle(%s,%s,\n%s)'%(Tatoms[0],Oatom,Tatoms[1])
3427    obj.angle_dict = {'pId':pId,'SGData':SGData,'symNo':symNos,'cellNo':cellNos}
3428    obj.angle_atoms = [oId,tIds]
3429    return obj
3430
3431class G2Exception(Exception):
3432    'A generic GSAS-II exception class'
3433    def __init__(self,msg):
3434        self.msg = msg
3435    def __str__(self):
3436        return repr(self.msg)
3437
3438class G2RefineCancel(Exception):
3439    'Raised when Cancel is pressed in a refinement dialog'
3440    def __init__(self,msg):
3441        self.msg = msg
3442    def __str__(self):
3443        return repr(self.msg)
3444   
3445def HowDidIgetHere(wherecalledonly=False):
3446    '''Show a traceback with calls that brought us to the current location.
3447    Used for debugging.
3448    '''
3449    import traceback
3450    if wherecalledonly:
3451        i = traceback.format_list(traceback.extract_stack()[:-1])[-2]
3452        print(i.strip().rstrip())
3453    else:
3454        print (70*'*')
3455        for i in traceback.format_list(traceback.extract_stack()[:-1]): print(i.strip().rstrip())
3456        print (70*'*')
3457
3458# Note that this is GUI code and should be moved at somepoint
3459def CreatePDFitems(G2frame,PWDRtree,ElList,Qlimits,numAtm=1,FltBkg=0,PDFnames=[]):
3460    '''Create and initialize a new set of PDF tree entries
3461
3462    :param Frame G2frame: main GSAS-II tree frame object
3463    :param str PWDRtree: name of PWDR to be used to create PDF item
3464    :param dict ElList: data structure with composition
3465    :param list Qlimits: Q limits to be used for computing the PDF
3466    :param float numAtm: no. atom in chemical formula
3467    :param float FltBkg: flat background value
3468    :param list PDFnames: previously used PDF names
3469
3470    :returns: the Id of the newly created PDF entry
3471    '''
3472    PDFname = 'PDF '+PWDRtree[4:] # this places two spaces after PDF, which is needed is some places
3473    if PDFname in PDFnames:
3474        print('Skipping, entry already exists: '+PDFname)
3475        return None
3476    #PDFname = MakeUniqueLabel(PDFname,PDFnames)
3477    Id = G2frame.GPXtree.AppendItem(parent=G2frame.root,text=PDFname)
3478    Data = {
3479        'Sample':{'Name':PWDRtree,'Mult':1.0},
3480        'Sample Bkg.':{'Name':'','Mult':-1.0,'Refine':False},
3481        'Container':{'Name':'','Mult':-1.0,'Refine':False},
3482        'Container Bkg.':{'Name':'','Mult':-1.0},'ElList':ElList,
3483        'Geometry':'Cylinder','Diam':1.0,'Pack':0.50,'Form Vol':10.0*numAtm,'Flat Bkg':FltBkg,
3484        'DetType':'Area detector','ObliqCoeff':0.3,'Ruland':0.025,'QScaleLim':Qlimits,
3485        'Lorch':False,'BackRatio':0.0,'Rmax':100.,'noRing':False,'IofQmin':1.0,'Rmin':1.0,
3486        'I(Q)':[],'S(Q)':[],'F(Q)':[],'G(R)':[],
3487        #items for sequential PDFfit
3488        'Datarange':[0.,30.],'Fitrange':[0.,30.],'qdamp':[0.03,False],'qbroad':[0,False],'Temp':300}
3489    G2frame.GPXtree.SetItemPyData(G2frame.GPXtree.AppendItem(Id,text='PDF Controls'),Data)
3490    G2frame.GPXtree.SetItemPyData(G2frame.GPXtree.AppendItem(Id,text='PDF Peaks'),
3491        {'Limits':[1.,5.],'Background':[2,[0.,-0.2*np.pi],False],'Peaks':[]})
3492    return Id
3493
3494class ShowTiming(object):
3495    '''An object to use for timing repeated sections of code.
3496
3497    Create the object with::
3498       tim0 = ShowTiming()
3499
3500    Tag sections of code to be timed with::
3501       tim0.start('start')
3502       tim0.start('in section 1')
3503       tim0.start('in section 2')
3504       
3505    etc. (Note that each section should have a unique label.)
3506
3507    After the last section, end timing with::
3508       tim0.end()
3509
3510    Show timing results with::
3511       tim0.show()
3512       
3513    '''
3514    def __init__(self):
3515        self.timeSum =  []
3516        self.timeStart = []
3517        self.label = []
3518        self.prev = None
3519    def start(self,label):
3520        import time
3521        if label in self.label:
3522            i = self.label.index(label)
3523            self.timeStart[i] = time.time()
3524        else:
3525            i = len(self.label)
3526            self.timeSum.append(0.0)
3527            self.timeStart.append(time.time())
3528            self.label.append(label)
3529        if self.prev is not None:
3530            self.timeSum[self.prev] += self.timeStart[i] - self.timeStart[self.prev]
3531        self.prev = i
3532    def end(self):
3533        import time
3534        if self.prev is not None:
3535            self.timeSum[self.prev] += time.time() - self.timeStart[self.prev]
3536        self.prev = None
3537    def show(self):
3538        sumT = sum(self.timeSum)
3539        print('Timing results (total={:.2f} sec)'.format(sumT))
3540        for i,(lbl,val) in enumerate(zip(self.label,self.timeSum)):
3541            print('{} {:20} {:8.2f} ms {:5.2f}%'.format(i,lbl,1000.*val,100*val/sumT))
3542
3543def validateAtomDrawType(typ,generalData={}):
3544    '''Confirm that the selected Atom drawing type is valid for the current
3545    phase. If not, use 'vdW balls'. This is currently used only for setting a
3546    default when atoms are added to the atoms draw list.
3547    '''
3548    if typ in ('lines','vdW balls','sticks','balls & sticks','ellipsoids'):
3549        return typ
3550    # elif generalData.get('Type','') == 'macromolecular':
3551    #     if typ in ('backbone',):
3552    #         return typ
3553    return 'vdW balls'
3554
3555if __name__ == "__main__":
3556    # test variable descriptions
3557    for var in '0::Afrac:*',':1:Scale','1::dAx:0','::undefined':
3558        v = var.split(':')[2]
3559        print(var+':\t', getDescr(v),getVarStep(v))
3560    import sys; sys.exit()
3561    # test equation evaluation
3562    def showEQ(calcobj):
3563        print (50*'=')
3564        print (calcobj.eObj.expression+'='+calcobj.EvalExpression())
3565        for v in sorted(calcobj.varLookup):
3566            print ("  "+v+'='+calcobj.exprDict[v]+'='+calcobj.varLookup[v])
3567        # print '  Derivatives'
3568        # for v in calcobj.derivStep.keys():
3569        #     print '    d(Expr)/d('+v+') =',calcobj.EvalDeriv(v)
3570
3571    obj = ExpressionObj()
3572
3573    obj.expression = "A*np.exp(B)"
3574    obj.assgnVars =  {'B': '0::Afrac:1'}
3575    obj.freeVars =  {'A': [u'A', 0.5, True]}
3576    #obj.CheckVars()
3577    calcobj = ExpressionCalcObj(obj)
3578
3579    obj1 = ExpressionObj()
3580    obj1.expression = "A*np.exp(B)"
3581    obj1.assgnVars =  {'B': '0::Afrac:*'}
3582    obj1.freeVars =  {'A': [u'Free Prm A', 0.5, True]}
3583    #obj.CheckVars()
3584    calcobj1 = ExpressionCalcObj(obj1)
3585
3586    obj2 = ExpressionObj()
3587    obj2.distance_stuff = np.array([[0,1],[1,-1]])
3588    obj2.expression = "Dist(1,2)"
3589    GSASIIpath.InvokeDebugOpts()
3590    parmDict2 = {'0::Afrac:0':[0.0,True], '0::Afrac:1': [1.0,False]}
3591    calcobj2 = ExpressionCalcObj(obj2)
3592    calcobj2.SetupCalc(parmDict2)
3593    showEQ(calcobj2)
3594
3595    parmDict1 = {'0::Afrac:0':1.0, '0::Afrac:1': 1.0}
3596    print ('\nDict = '+parmDict1)
3597    calcobj.SetupCalc(parmDict1)
3598    showEQ(calcobj)
3599    calcobj1.SetupCalc(parmDict1)
3600    showEQ(calcobj1)
3601
3602    parmDict2 = {'0::Afrac:0':[0.0,True], '0::Afrac:1': [1.0,False]}
3603    print ('Dict = '+parmDict2)
3604    calcobj.SetupCalc(parmDict2)
3605    showEQ(calcobj)
3606    calcobj1.SetupCalc(parmDict2)
3607    showEQ(calcobj1)
3608    calcobj2.SetupCalc(parmDict2)
3609    showEQ(calcobj2)
Note: See TracBrowser for help on using the repository browser.