source: trunk/GSASIIobj.py @ 4318

Last change on this file since 4318 was 4318, checked in by toby, 2 years ago

minor table adjustments & finally figure out Angstrom for LaTeX and HTML -- I hope

  • Property svn:eol-style set to native
  • Property svn:keywords set to Date Author Revision URL Id
File size: 141.7 KB
Line 
1# -*- coding: utf-8 -*-
2#GSASIIobj - data objects for GSAS-II
3########### SVN repository information ###################
4# $Date: 2020-02-17 20:03:30 +0000 (Mon, 17 Feb 2020) $
5# $Author: toby $
6# $Revision: 4318 $
7# $URL: trunk/GSASIIobj.py $
8# $Id: GSASIIobj.py 4318 2020-02-17 20:03:30Z toby $
9########### SVN repository information ###################
10
11'''
12*GSASIIobj: Data objects*
13=========================
14
15This module defines and/or documents the data structures used in GSAS-II, as well
16as provides misc. support routines.
17
18.. Next command allows \AA to be used in HTML
19
20.. only:: html
21
22   html :math:`\\require{mediawiki-texvc}`
23
24.. _Constraints_table:
25
26.. index::
27   single: Constraints object description
28   single: Data object descriptions; Constraints
29
30Constraints Tree Item
31----------------------
32
33Constraints are stored in a dict, separated into groups.
34Note that parameter are named in the following pattern,
35p:h:<var>:n, where p is the phase number, h is the histogram number
36<var> is a variable name and n is the parameter number.
37If a parameter does not depend on a histogram or phase or is unnumbered, that
38number is omitted.
39Note that the contents of each dict item is a List where each element in the
40list is a :ref:`constraint definition objects <Constraint_definitions_table>`.
41The constraints in this form are converted in
42:func:`GSASIIstrIO.ProcessConstraints` to the form used in :mod:`GSASIImapvars`
43
44The keys in the Constraints dict are:
45
46.. tabularcolumns:: |l|p{4.5in}|
47
48==========  ====================================================
49  key         explanation
50==========  ====================================================
51Hist        This specifies a list of constraints on
52            histogram-related parameters,
53            which will be of form :h:<var>:n.
54HAP         This specifies a list of constraints on parameters
55            that are defined for every histogram in each phase
56            and are of form p:h:<var>:n.
57Phase       This specifies a list of constraints on phase
58            parameters,
59            which will be of form p::<var>:n.
60Global      This specifies a list of constraints on parameters
61            that are not tied to a histogram or phase and
62            are of form ::<var>:n
63==========  ====================================================
64
65.. _Constraint_definitions_table:
66
67.. index::
68   single: Constraint definition object description
69   single: Data object descriptions; Constraint Definition
70
71Each constraint is defined as an item in a list. Each constraint is of form::
72
73[[<mult1>, <var1>], [<mult2>, <var2>],..., <fixedval>, <varyflag>, <constype>]
74
75Where the variable pair list item containing two values [<mult>, <var>], where:
76
77  * <mult> is a multiplier for the constraint (float)
78  * <var> a :class:`G2VarObj` object (previously a str variable name of form
79      'p:h:name[:at]')
80
81Note that the last three items in the list play a special role:
82
83 * <fixedval> is the fixed value for a `constant equation` (``constype=c``)
84   constraint or is None. For a `New variable` (``constype=f``) constraint,
85   a variable name can be specified as a str (used for externally
86   generated constraints)
87 * <varyflag> is True or False for `New variable` (``constype=f``) constraints
88   or is None. This indicates if this variable should be refined.
89 * <constype> is one of four letters, 'e', 'c', 'h', 'f' that determines the type of constraint:
90
91    * 'e' defines a set of equivalent variables. Only the first variable is refined (if the
92      appropriate refine flag is set) and and all other equivalent variables in the list
93      are generated from that variable, using the appropriate multipliers.
94    * 'c' defines a constraint equation of form,
95      :math:`m_1 \\times var_1 + m_2 \\times var_2 + ... = c`
96    * 'h' defines a variable to hold (not vary). Any variable on this list is not varied,
97      even if its refinement flag is set. Only one [mult,var] pair is allowed in a hold
98      constraint and the mult value is ignored.
99      This is of particular value when needing to hold one or more variables where a
100      single flag controls a set of variables such as, coordinates,
101      the reciprocal metric tensor or anisotropic displacement parameter.
102    * 'f' defines a new variable (function) according to relationship
103      :math:`newvar = m_1 \\times var_1 + m_2 \\times var_2 + ...`
104
105.. _Covariance_table:
106
107.. index::
108   single: Covariance description
109   single: Data object descriptions; Covariance
110
111Covariance Tree Item
112--------------------
113
114The Covariance tree item has results from the last least-squares run. They
115are stored in a dict with these keys:
116
117.. tabularcolumns:: |l|l|p{4in}|
118
119=============  ===============  ====================================================
120  key            sub-key        explanation
121=============  ===============  ====================================================
122newCellDict    \                (dict) ith lattice parameters computed by
123                                :func:`GSASIIstrMath.GetNewCellParms`
124title          \                (str) Name of gpx file(?)
125variables      \                (list) Values for all N refined variables
126                                (list of float values, length N,
127                                ordered to match varyList)
128sig            \                (list) Uncertainty values for all N refined variables
129                                (list of float values, length N,
130                                ordered to match varyList)
131varyList       \                (list of str values, length N) List of directly refined variables
132                               
133newAtomDict    \                (dict) atom position values computed in
134                                :func:`GSASIIstrMath.ApplyXYZshifts`
135Rvals          \                (dict) R-factors, GOF, Marquardt value for last
136                                refinement cycle
137\              Nobs             (int) Number of observed data points
138\              Rwp              (float) overall weighted profile R-factor (%)
139\              chisq            (float) :math:`\sum w*(I_{obs}-I_{calc})^2`                               
140                                for all data.
141                                Note: this is not the reduced :math:`\chi^2`.
142\              lamMax           (float) Marquardt value applied to Hessian diagonal
143\              GOF              (float) The goodness-of-fit, aka square root of
144                                the reduced chi squared.
145covMatrix      \                (np.array) The (NxN) covVariance matrix
146=============  ===============  ====================================================
147
148.. _Phase_table:
149
150.. index::
151   single: Phase object description
152   single: Data object descriptions; Phase
153
154Phase Tree Items
155----------------
156
157Phase information is stored in the GSAS-II data tree as children of the
158Phases item in a dict with keys:
159
160.. tabularcolumns:: |l|l|p{4in}|
161
162==========  ===============     =====================================================================================================
163  key         sub-key           explanation
164==========  ===============     =====================================================================================================
165General         \               (dict) Overall information for the phase
166  \         3Dproj              (list of str) projections for 3D pole distribution plots
167  \         AngleRadii          (list of floats) Default radius for each atom used to compute
168                                interatomic angles
169  \         AtomMass            (list of floats) Masses for atoms
170  \         AtomPtrs            (list of int) four locations (cx,ct,cs & cu) to use to pull info
171                                from the atom records
172  \         AtomTypes           (llist of str) Atom types
173  \         BondRadii           (list of floats) Default radius for each atom used to compute
174                                interatomic distances
175  \         Cell                (list with 8 items) cell refinement flag (bool)
176                                a, b, c, (:math:`\\AA`, float)
177                                alpha, beta & gamma (degrees, float)
178                                volume (:math:`\AA^3`, float)
179  \         Color               (list of (r,b,g) triplets) Colors for atoms
180  \         Compare             (dict) Polygon comparison parameters
181  \         Data plot type      (str) data plot type ('Mustrain', 'Size' or
182                                'Preferred orientation') for powder data
183  \         DisAglCtls          (dDict) with distance/angle search controls,
184                                which has keys 'Name', 'AtomTypes',
185                                'BondRadii', 'AngleRadii' which are as above
186                                except are possibly edited. Also contains
187                                'Factors', which is a 2 element list with
188                                a multiplier for bond and angle search range
189                                [typically (0.85,0.85)].
190  \         F000X               (float) x-ray F(000) intensity
191  \         F000N               (float) neutron F(000) intensity
192  \         Flip                (dict) Charge flip controls
193  \         HydIds              (dict) geometrically generated hydrogen atoms
194  \         Isotope             (dict) Isotopes for each atom type
195  \         Isotopes            (dict) Scattering lengths for each isotope
196                                combination for each element in phase
197  \         MCSA controls       (dict) Monte Carlo-Simulated Annealing controls
198  \         Map                 (dict) Map parameters
199  \         Mass                (float) Mass of unit cell contents in g/mol
200  \         Modulated           (bool) True if phase modulated
201  \         Mydir               (str) Directory of current .gpx file
202  \         Name                (str) Phase name
203  \         NoAtoms             (dict) Number of atoms per unit cell of each type
204  \         POhkl               (list) March-Dollase preferred orientation direction
205  \         Pawley dmin         (float) maximum Q (as d-space) to use for Pawley extraction
206  \         Pawley dmax         (float) minimum Q (as d-space) to use for Pawley extraction
207  \         Pawley neg wt       (float) Restraint value for negative Pawley intensities
208  \         SGData              (object <SGData_table>`) Space group details as a
209                                :ref:`space group (SGData) <SGData_table>`
210                                as defined in :func:`GSASIIspc.SpcGroup`.
211  \         SH Texture          (dict) Spherical harmonic preferred orientation parameters
212  \         Super               (int) dimension of super group (0,1 only)
213  \         Type                (str) phase type (e.g. 'nuclear')
214  \         Z                   (dict) Atomic numbers for each atom type
215  \         doDysnomia          (bool) flag for max ent map modification via Dysnomia
216  \         doPawley            (bool) Flag for Pawley intensity extraction
217  \         vdWRadii            (dict) Van der Waals radii for each atom type
218ranId           \               (int) unique random number Id for phase
219pId             \               (int) Phase Id number for current project.
220Atoms           \               (list of lists) Atoms in phase as a list of lists. The outer list
221                                is for each atom, the inner list contains varying
222                                items depending on the type of phase, see
223                                the :ref:`Atom Records <Atoms_table>` description.
224Drawing         \               (dict) Display parameters
225\           Atoms               (list of lists) with an entry for each atom that is drawn
226\           Plane               (list) Controls for contour density plane display
227\           Quaternion          (4 element np.array) Viewing quaternion
228\           Zclip               (float) clipping distance in :math:`\\AA`
229\           Zstep               (float) Step to de/increase Z-clip
230\           atomPtrs            (list) positions of x, type, site sym, ADP flag in Draw Atoms
231\           backColor           (list) background for plot as and R,G,B triplet
232                                (default = [0, 0, 0], black).
233\           ballScale           (float) Radius of spheres in ball-and-stick display
234\           bondList            (dict) Bonds
235\           bondRadius          (float) Radius of binds in :math:`\\AA`
236\           cameraPos           (float) Viewing position in :math:`\\AA` for plot
237\           contourLevel        (float) map contour level in :math:`e/\\AA^3`
238\           contourMax          (float) map contour maximum
239\           depthFog            (bool) True if use depthFog on plot - set currently as False
240\           ellipseProb         (float) Probability limit for display of thermal
241                                ellipsoids in % .
242\           magMult             (float) multiplier for magnetic moment arrows
243\           mapSize             (float) x & y dimensions of contourmap (fixed internally)
244\           modelView           (4,4 array) from openGL drawing transofmation matrix
245\           oldxy               (list with two floats) previous view point
246\           radiusFactor        (float) Distance ratio for searching for bonds. Bonds
247                                are located that are within r(Ra+Rb) and (Ra+Rb)/r
248                                where Ra and Rb are the atomic radii.
249\           selectedAtoms       (list of int values) List of selected atoms
250\           showABC             (bool) Flag to show view point triplet. True=show.
251\           showHydrogen        (bool) Flag to control plotting of H atoms.
252\           showRigidBodies     (bool) Flag to highlight rigid body placement
253\           showSlice           (bool) flag to show contour map
254\           sizeH               (float) Size ratio for H atoms
255\           unitCellBox         (bool) Flag to control display of the unit cell.
256\           vdwScale            (float) Multiplier of van der Waals radius for display of vdW spheres.
257\           viewDir             (np.array with three floats) cartesian viewing direction
258\           viewPoint           (list of lists) First item in list is [x,y,z]
259                                in fractional coordinates for the center of
260                                the plot. Second item list of previous & current
261                                atom number viewed (may be [0,0])
262RBModels        \               Rigid body assignments (note Rigid body definitions
263                                are stored in their own main top-level tree entry.)
264RMC             \               (dict) RMCProfile & rmcfull controls
265Pawley ref      \               (list) Pawley reflections
266Histograms      \               (dict of dicts) The key for the outer dict is
267                                the histograms tied to this phase. The inner
268                                dict contains the combined phase/histogram
269                                parameters for items such as scale factors,
270                                size and strain parameters. The following are the
271                                keys to the inner dict. (dict)
272\           Babinet             (dict) For protein crystallography. Dictionary with two
273                                entries, 'BabA', 'BabU'
274\           Extinction          (list of float, bool) Extinction parameter
275\           HStrain             (list of two lists) Hydrostatic strain. The first is
276                                a list of the HStrain parameters (1, 2, 3, 4, or 6
277                                depending on unit cell), the second is a list of boolean
278                                refinement parameters (same length)
279\           Histogram           (str) The name of the associated histogram
280\           LeBail              (bool) Flag for LeBail extraction
281\           Mustrain            (list) Microstrain parameters, in order:
282   
283                                0. Type, one of  u'isotropic', u'uniaxial', u'generalized'
284                                1. Isotropic/uniaxial parameters - list of 3 floats
285                                2. Refinement flags - list of 3 bools
286                                3. Microstrain axis - list of 3 ints, [h, k, l]
287                                4. Generalized mustrain parameters - list of 2-6 floats, depending on space group
288                                5. Generalized refinement flags - list of bools, corresponding to the parameters of (4)
289\           Pref.Ori.           (list) Preferred Orientation. List of eight parameters.
290                                Items marked SH are only used for Spherical Harmonics.
291                               
292                                0. (str) Type, 'MD' for March-Dollase or 'SH' for Spherical Harmonics
293                                1. (float) Value
294                                2. (bool) Refinement flag
295                                3. (list) Preferred direction, list of ints, [h, k, l]
296                                4. (int) SH - number of terms
297                                5. (dict) SH - 
298                                6. (list) SH
299                                7. (float) SH
300\           Scale               (list of [float, bool]) Phase fraction & refine flag
301\           Size                List of crystallite size parameters, in order:
302
303                                0. (str) Type, one of  u'isotropic', u'uniaxial', u'ellipsoidal'
304                                1. (list) Isotropic/uniaxial parameters - list of 3 floats
305                                2. (list) Refinement flags - list of 3 bools
306                                3. (list) Size axis - list of 3 ints, [h, k, l]
307                                4. (list) Ellipsoidal size parameters - list of 6 floats
308                                5. (list) Ellipsoidal refinement flags - list of bools, corresponding to the parameters of (4)
309\           Use                 (bool) True if this histogram is to be used in refinement
310\           newLeBail           (bool) Whether to perform a new LeBail extraction
311MCSA            \               (dict) Monte-Carlo simulated annealing parameters
312==========  ===============     =====================================================================================================
313
314.. _RBData_table:
315
316.. index::
317   single: Rigid Body Data description
318   single: Data object descriptions; Rigid Body Data
319
320Rigid Body Objects
321------------------
322
323Rigid body descriptions are available for two types of rigid bodies: 'Vector'
324and 'Residue'. Vector rigid bodies are developed by a sequence of translations each
325with a refinable magnitude and Residue rigid bodies are described as Cartesian coordinates
326with defined refinable torsion angles.
327
328.. tabularcolumns:: |l|l|p{4in}|
329
330==========  ===============     ====================================================
331  key         sub-key           explanation
332==========  ===============     ====================================================
333Vector      RBId                (dict of dict) vector rigid bodies
334\           AtInfo              (dict) Drad, Color: atom drawing radius & color for each atom type
335\           RBname              (str) Name assigned by user to rigid body
336\           VectMag             (list) vector magnitudes in :math:`\\AA`
337\           rbXYZ               (list of 3 float Cartesian coordinates for Vector rigid body )
338\           rbRef               (list of 3 int & 1 bool) 3 assigned reference atom nos. in rigid body for origin
339                                definition, use center of atoms flag
340\           VectRef             (list of bool refinement flags for VectMag values )
341\           rbTypes             (list of str) Atom types for each atom in rigid body
342\           rbVect              (list of lists) Cartesian vectors for each translation used to build rigid body
343\           useCount            (int) Number of times rigid body is used in any structure
344Residue     RBId                (dict of dict) residue rigid bodies
345\           AtInfo              (dict) Drad, Color: atom drawing radius & color for each atom type
346\           RBname              (str) Name assigned by user to rigid body
347\           rbXYZ               (list of 3 float) Cartesian coordinates for Residue rigid body
348\           rbTypes             (list of str) Atom types for each atom in rigid body
349\           atNames             (list of str) Names of each atom in rigid body (e.g. C1,N2...)
350\           rbRef               (list of 3 int & 1 bool) 3 assigned reference atom nos. in rigid body for origin
351                                definition, use center of atoms flag
352\           rbSeq               (list) Orig,Piv,angle,Riding : definition of internal rigid body
353                                torsion; origin atom (int), pivot atom (int), torsion angle (float),
354                                riding atoms (list of int)
355\           SelSeq              (int,int) used by SeqSizer to identify objects
356\           useCount            (int)Number of times rigid body is used in any structure
357RBIds           \               (dict) unique Ids generated upon creation of each rigid body
358\           Vector              (list) Ids for each Vector rigid body
359\           Residue             (list) Ids for each Residue rigid body
360==========  ===============     ====================================================
361
362.. _SGData_table:
363
364.. index::
365   single: Space Group Data description
366   single: Data object descriptions; Space Group Data
367
368Space Group Objects
369-------------------
370
371Space groups are interpreted by :func:`GSASIIspc.SpcGroup`
372and the information is placed in a SGdata object
373which is a dict with these keys. Magnetic ones are marked "mag"
374
375.. tabularcolumns:: |l|p{4.5in}|
376
377==========  ========================================================================================
378  key         explanation
379==========  ========================================================================================
380BNSlattsym  mag - (str) BNS magnetic space group symbol and centering vector
381GenFlg      mag - (list) symmetry generators indices
382GenSym      mag - (list) names for each generator
383MagMom      mag - (list) "time reversals" for each magnetic operator
384MagPtGp     mag - (str) Magnetic point group symbol
385MagSpGrp    mag - (str) Magnetic space group symbol
386OprNames    mag - (list) names for each space group operation
387SGCen       (np.array) Symmetry cell centering vectors. A (n,3) np.array
388            of centers. Will always have at least one row: ``np.array([[0, 0, 0]])``
389SGFixed     (bool) Only True if phase mported from a magnetic cif file
390            then the space group can not be changed by the user because
391            operator set from cif may be nonstandard
392SGGen       (list) generators
393SGGray      (bool) True if space group is a gray group (incommensurate magnetic structures)
394SGInv       (bool) True if centrosymmetric, False if not
395SGLatt      (str)Lattice centering type. Will be one of
396            P, A, B, C, I, F, R
397SGLaue      (str) one of the following 14 Laue classes:
398            -1, 2/m, mmm, 4/m, 4/mmm, 3R,
399            3mR, 3, 3m1, 31m, 6/m, 6/mmm, m3, m3m
400SGOps       (list) symmetry operations as a list of form
401            ``[[M1,T1], [M2,T2],...]``
402            where :math:`M_n` is a 3x3 np.array
403            and :math:`T_n` is a length 3 np.array.
404            Atom coordinates are transformed where the
405            Asymmetric unit coordinates [X is (x,y,z)]
406            are transformed using
407            :math:`X^\prime = M_n*X+T_n`
408SGPolax     (str) Axes for space group polarity. Will be one of
409            '', 'x', 'y', 'x y', 'z', 'x z', 'y z',
410            'xyz'. In the case where axes are arbitrary
411            '111' is used (P 1, and ?).
412SGPtGrp     (str) Point group of the space group
413SGUniq      unique axis if monoclinic. Will be
414            a, b, or c for monoclinic space groups.
415            Will be blank for non-monoclinic.
416SGSpin      mag - (list) of spin flip operatiors (+1 or -1) for the space group operations
417SGSys       (str) symmetry unit cell: type one of
418            'triclinic', 'monoclinic', 'orthorhombic',
419            'tetragonal', 'rhombohedral', 'trigonal',
420            'hexagonal', 'cubic'
421SSGK1       (list) Superspace multipliers
422SpGrp       (str) space group symbol
423SpnFlp      mag - (list) Magnetic spin flips for every magnetic space group operator
424==========  ========================================================================================
425
426.. _SSGData_table:
427
428.. index::
429   single: Superspace Group Data description
430   single: Data object descriptions; Superspace Group Data
431
432Superspace groups [3+1] are interpreted by :func:`GSASIIspc.SSpcGroup`
433and the information is placed in a SSGdata object
434which is a dict with these keys:
435
436.. tabularcolumns:: |l|p{4.5in}|
437
438==========  ====================================================
439  key         explanation
440==========  ====================================================
441SSGCen      (list) 4D cell centering vectors [0,0,0,0] at least
442SSGK1       (list) Superspace multipliers
443SSGOps      (list) 4D symmetry operations as [M,T] so that M*x+T = x'
444SSpGrp      (str) superspace group symbol extension to space group
445            symbol, accidental spaces removed
446modQ        (list) modulation/propagation vector
447modSymb     (list of str) Modulation symbols
448==========  ====================================================
449
450
451.. _Atoms_table:
452
453.. index::
454   single: Atoms record description
455   single: Data object descriptions; Atoms record
456
457Atom Records
458------------
459
460If ``phasedict`` points to the phase information in the data tree, then
461atoms are contained in a list of atom records (list) in
462``phasedict['Atoms']``. Also needed to read atom information
463are four pointers, ``cx,ct,cs,cia = phasedict['General']['AtomPtrs']``,
464which define locations in the atom record, as shown below. Items shown are
465always present; additional ones for macromolecular phases are marked 'mm',
466and those for magnetic structures are marked 'mg'
467
468.. tabularcolumns:: |l|p{4.5in}|
469
470==============      ====================================================
471location            explanation
472==============      ====================================================
473ct-4                mm - (str) residue number
474ct-3                mm - (str) residue name (e.g. ALA)
475ct-2                mm - (str) chain label
476ct-1                (str) atom label
477ct                  (str) atom type
478ct+1                (str) refinement flags; combination of 'F', 'X', 'U', 'M'
479cx,cx+1,cx+2        (3 floats) the x,y and z coordinates
480cx+3                (float) site occupancy
481cx+4,cx+5,cx+6      mg - (list) atom magnetic moment along a,b,c in Bohr magnetons
482cs                  (str) site symmetry
483cs+1                (int) site multiplicity
484cia                 (str) ADP flag: Isotropic ('I') or Anisotropic ('A')
485cia+1               (float) Uiso
486cia+2...cia+7       (6 floats) U11, U22, U33, U12, U13, U23
487atom[cia+8]         (int) unique atom identifier
488
489==============      ====================================================
490
491.. _Drawing_atoms_table:
492
493.. index::
494   single: Drawing atoms record description
495   single: Data object descriptions; Drawing atoms record
496
497Drawing Atom Records
498--------------------
499
500If ``phasedict`` points to the phase information in the data tree, then
501drawing atoms are contained in a list of drawing atom records (list) in
502``phasedict['Drawing']['Atoms']``. Also needed to read atom information
503are four pointers, ``cx,ct,cs,ci = phasedict['Drawing']['AtomPtrs']``,
504which define locations in the atom record, as shown below. Items shown are
505always present; additional ones for macromolecular phases are marked 'mm',
506and those for magnetic structures are marked 'mg'
507
508.. tabularcolumns:: |l|p{4.5in}|
509
510==============   ===================================================================================
511location            explanation
512==============   ===================================================================================
513ct-4                mm - (str) residue number
514ct-3                mm - (str) residue name (e.g. ALA)
515ct-2                mm - (str) chain label
516ct-1                (str) atom label
517ct                  (str) atom type
518cx,cx+1,cx+2        (3 floats) the x,y and z coordinates
519cx+3,cx+4,cx+5      mg - (3 floats) atom magnetic moment along a,b,c in Bohr magnetons
520cs-1                (str) Sym Op symbol; sym. op number + unit cell id (e.g. '1,0,-1')
521cs                  (str) atom drawing style; e.g. 'balls & sticks'
522cs+1                (str) atom label style (e.g. 'name')
523cs+2                (int) atom color (RBG triplet)
524cs+3                (str) ADP flag: Isotropic ('I') or Anisotropic ('A')
525cs+4                (float) Uiso
526cs+5...cs+11        (6 floats) U11, U22, U33, U12, U13, U23
527ci                  (int) unique atom identifier; matches source atom Id in Atom Records
528==============   ===================================================================================
529
530.. _Powder_table:
531
532.. index::
533   single: Powder data object description
534   single: Data object descriptions; Powder Data
535
536Powder Diffraction Tree Items
537-----------------------------
538
539Every powder diffraction histogram is stored in the GSAS-II data tree
540with a top-level entry named beginning with the string "PWDR ". The
541diffraction data for that information are directly associated with
542that tree item and there are a series of children to that item. The
543routines :func:`GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
544and :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` will
545load this information into a dictionary where the child tree name is
546used as a key, and the information in the main entry is assigned
547a key of ``Data``, as outlined below.
548
549.. tabularcolumns:: |p{1in}|p{1in}|p{4in}|
550
551======================     ===============  ===========================================================
552  key                       sub-key          explanation
553======================     ===============  ===========================================================
554Comments                    \               (list of str) Text strings extracted from the original powder
555                                            data header. These cannot be changed by the user;
556                                            it may be empty.
557Limits                      \               (list) two two element lists, as [[Ld,Hd],[L,H]]
558                                            where L and Ld are the current and default lowest
559                                            two-theta value to be used and
560                                            where H and Hd are the current and default highest
561                                            two-theta value to be used.
562Reflection Lists            \               (dict of dicts) with an entry for each phase in the
563                                            histogram. The contents of each dict item
564                                            is a dict containing reflections, as described in
565                                            the :ref:`Powder Reflections <PowderRefl_table>`
566                                            description.
567Instrument Parameters       \               (dict) The instrument parameters uses different dicts
568                                            for the constant wavelength (CW) and time-of-flight (TOF)
569                                            cases. See below for the descriptions of each.
570wtFactor                    \               (float) A weighting factor to increase or decrease
571                                            the leverage of data in the histogram .
572                                            A value of 1.0 weights the data with their
573                                            standard uncertainties and a larger value
574                                            increases the weighting of the data (equivalent
575                                            to decreasing the uncertainties).
576Sample Parameters           \               (dict) Parameters that describe how
577                                            the data were collected, as listed
578                                            below. Refinable parameters are a list containing
579                                            a float and a bool, where the second value
580                                            specifies if the value is refined, otherwise
581                                            the value is a float unless otherwise noted.
582\                           Scale           The histogram scale factor (refinable)
583\                           Absorption      The sample absorption coefficient as
584                                            :math:`\\mu r` where r is the radius
585                                            (refinable). Only valid for Debye-Scherrer geometry.
586\                           SurfaceRoughA   Surface roughness parameter A as defined by
587                                            Surotti, *J. Appl. Cryst*, **5**, 325-331, 1972.
588                                            (refinable - only valid for Bragg-Brentano geometry)
589\                           SurfaceRoughB   Surface roughness parameter B (refinable -
590                                            only valid for Bragg-Brentano geometry)
591\                           DisplaceX,      Sample displacement from goniometer center
592                            DisplaceY       where Y is along the beam direction and
593                                            X is perpendicular. Units are :math:`\\mu m`
594                                            (refinable).
595\                           Phi, Chi,       Goniometer sample setting angles, in degrees.
596                            Omega
597\                           Gonio. radius   Radius of the diffractometer in mm
598\                           InstrName       (str) A name for the instrument, used in preparing
599                                            a CIF .
600\                           Force,          Variables that describe how the measurement
601                            Temperature,    was performed. Not used directly in
602                            Humidity,       any computations.
603                            Pressure,
604                            Voltage
605\                           ranId           (int) The random-number Id for the histogram
606                                            (same value as where top-level key is ranId)
607\                           Type            (str) Type of diffraction data, may be 'Debye-Scherrer'
608                                            or 'Bragg-Brentano' .
609hId                         \               (int) The number assigned to the histogram when
610                                            the project is loaded or edited (can change)
611ranId                       \               (int) A random number id for the histogram
612                                            that does not change
613Background                  \               (list) The background is stored as a list with where
614                                            the first item in the list is list and the second
615                                            item is a dict. The list contains the background
616                                            function and its coefficients; the dict contains
617                                            Debye diffuse terms and background peaks.
618                                            (TODO: this needs to be expanded.)
619Data                        \               (list) The data consist of a list of 6 np.arrays
620                                            containing in order:
621
622                                            0. the x-postions (two-theta in degrees),
623                                            1. the intensity values (Yobs),
624                                            2. the weights for each Yobs value
625                                            3. the computed intensity values (Ycalc)
626                                            4. the background values
627                                            5. Yobs-Ycalc
628======================     ===============  ===========================================================
629
630.. _CWPowder_table:
631
632.. index::
633   single: Powder data CW Instrument Parameters
634
635-----------------------------
636CW Instrument Parameters
637-----------------------------
638
639Instrument Parameters are placed in a list of two dicts,
640where the keys in the first dict are listed below. Note that the dict contents are different for
641constant wavelength (CW) vs. time-of-flight (TOF) histograms.
642The value for each item is a list containing three values: the initial value, the current value
643and a refinement flag which can have a value of True, False or 0 where 0 indicates a value that
644cannot be refined. The first and second values are floats unless otherwise noted.
645Items not refined are noted as [*]
646
647.. tabularcolumns:: |l|p{1in}|p{4in}|
648
649========================    ===============  ===========================================================
650  key                       sub-key           explanation
651========================    ===============  ===========================================================
652Instrument Parameters[0]    Type [*]            (str) Histogram type:
653                                                * 'PXC' for constant wavelength x-ray
654                                                * 'PNC' for constant wavelength neutron
655\                           Bank [*]            (int) Data set number in a multidata file (usually 1)
656\                           Lam                 (float) Specifies a wavelength in :math:`\\AA`
657\                           Lam1 [*]            (float) Specifies the primary wavelength in
658                                                :math:`\\AA`, used in place of Lam
659                                                when an :math:`\\alpha_1, \\alpha_2`
660                                                source is used.
661\                           Lam2 [*]            (float) Specifies the secondary wavelength in
662                                                :math:`\\AA`, used with Lam1
663\                           I(L2)/I(L1)         (float) Ratio of Lam2 to Lam1, used with Lam1
664\                           Zero                (float) Two-theta zero correction in *degrees*
665\                           Azimuth [*]         (float) Azimuthal setting angle for data recorded with differing setting angles
666\                           U, V, W             (float) Cagliotti profile coefficients
667                                                for Gaussian instrumental broadening, where the
668                                                FWHM goes as
669                                                :math:`U \\tan^2\\theta + V \\tan\\theta + W`
670\                           X, Y, Z             (float) Cauchy (Lorentzian) instrumental broadening coefficients
671\                           SH/L                (float) Variant of the Finger-Cox-Jephcoat asymmetric
672                                                peak broadening ratio. Note that this is the
673                                                sum of S/L and H/L where S is
674                                                sample height, H is the slit height and
675                                                L is the goniometer diameter.
676\                           Polariz.            (float) Polarization coefficient.
677Instrument Parameters[1]                        (empty dict)
678========================    ===============  ===========================================================
679
680.. _TOFPowder_table:
681
682.. index::
683   single: Powder data TOF Instrument Parameters
684
685-----------------------------
686TOF Instrument Parameters
687-----------------------------
688
689Instrument Parameters are also placed in a list of two dicts,
690where the keys in each dict listed below, but here for
691time-of-flight (TOF) histograms.
692The value for each item is a list containing three values: the initial value, the current value
693and a refinement flag which can have a value of True, False or 0 where 0 indicates a value that
694cannot be refined. The first and second values are floats unless otherwise noted.
695Items not refined are noted as [*]
696
697.. tabularcolumns:: |l|p{1.5in}|p{4in}|
698
699========================    ===============  ===========================================================
700  key                        sub-key          explanation
701========================    ===============  ===========================================================
702Instrument Parameters[0]    Type [*]            (str) Histogram type:
703                                                * 'PNT' for time of flight neutron
704\                           Bank                (int) Data set number in a multidata file
705\                           2-theta [*]         (float) Nominal scattering angle for the detector
706\                           fltPath [*]         (float) Total flight path source-sample-detector
707\                           Azimuth [*]         (float) Azimuth angle for detector right hand rotation
708                                                from horizontal away from source
709\                           difC,difA,          (float) Diffractometer constants for conversion of d-spacing to TOF
710                            difB                in microseconds
711\                           Zero                (float) Zero point offset (microseconds)
712\                           alpha               (float) Exponential rise profile coefficients
713\                           beta-0              (float) Exponential decay profile coefficients
714                            beta-1
715                            beta-q
716\                           sig-0               (float) Gaussian profile coefficients
717                            sig-1
718                            sig-2
719                            sig-q   
720\                           X,Y,Z               (float) Lorentzian profile coefficients
721Instrument Parameters[1]    Pdabc               (list of 4 float lists) Originally created for use in gsas as optional tables
722                                                of d, alp, bet, d-true; for a reflection alpha & beta are obtained via interpolation
723                                                from the d-spacing and these tables. The d-true column is apparently unused.
724========================    ===============  ===========================================================
725
726
727.. _PowderRefl_table:
728
729.. index::
730   single: Powder reflection object description
731   single: Data object descriptions; Powder Reflections
732
733Powder Reflection Data Structure
734--------------------------------
735
736For every phase in a histogram, the ``Reflection Lists`` value is a dict
737one element of which is `'RefList'`, which is a np.array containing
738reflections. The columns in that array are documented below.
739
740==========  ====================================================
741  index         explanation
742==========  ====================================================
743 0,1,2          h,k,l (float)
744 3              (int) multiplicity
745 4              (float) d-space, :math:`\\AA`
746 5              (float) pos, two-theta
747 6              (float) sig, Gaussian width
748 7              (float) gam, Lorenzian width
749 8              (float) :math:`F_{obs}^2`
750 9              (float) :math:`F_{calc}^2`
751 10             (float) reflection phase, in degrees
752 11             (float) intensity correction for reflection, this times
753                :math:`F_{obs}^2` or :math:`F_{calc}^2` gives Iobs or Icalc
754 12             (float) Preferred orientation correction
755 13             (float) Transmission (absorption correction)
756 14             (float) Extinction correction
757==========  ====================================================
758
759.. _Xtal_table:
760
761.. index::
762   single: Single Crystal data object description
763   single: Data object descriptions; Single crystal data
764
765Single Crystal Tree Items
766-------------------------
767
768Every single crystal diffraction histogram is stored in the GSAS-II data tree
769with a top-level entry named beginning with the string "HKLF ". The
770diffraction data for that information are directly associated with
771that tree item and there are a series of children to that item. The
772routines :func:`GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
773and :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` will
774load this information into a dictionary where the child tree name is
775used as a key, and the information in the main entry is assigned
776a key of ``Data``, as outlined below.
777
778.. tabularcolumns:: |l|l|p{4in}|
779
780======================  ===============     ====================================================
781  key                      sub-key          explanation
782======================  ===============     ====================================================
783Data                        \               (dict) that contains the
784                                            reflection table,
785                                            as described in the
786                                            :ref:`Single Crystal Reflections
787                                            <XtalRefl_table>`
788                                            description.
789
790Instrument Parameters       \               (list) containing two dicts where the possible
791                                            keys in each dict are listed below. The value
792                                            for most items is a list containing two values:
793                                            the initial value, the current value.
794                                            The first and second
795                                            values are floats unless otherwise noted.
796\                           Lam             (two floats) Specifies a wavelength in :math:`\\AA`
797\                           Type            (two str values) Histogram type :
798                                            * 'SXC' for constant wavelength x-ray
799                                            * 'SNC' for constant wavelength neutron
800                                            * 'SNT' for time of flight neutron
801\                           InstrName       (str) A name for the instrument, used in preparing a CIF
802wtFactor                    \               (float) A weighting factor to increase or decrease
803                                            the leverage of data in the histogram.
804                                            A value of 1.0 weights the data with their
805                                            standard uncertainties and a larger value
806                                            increases the weighting of the data (equivalent
807                                            to decreasing the uncertainties).
808
809hId                         \               (int) The number assigned to the histogram when
810                                            the project is loaded or edited (can change)
811ranId                       \               (int) A random number id for the histogram
812                                            that does not change
813======================  ===============     ====================================================
814
815.. _XtalRefl_table:
816
817.. index::
818   single: Single Crystal reflection object description
819   single: Data object descriptions; Single Crystal Reflections
820
821Single Crystal Reflection Data Structure
822----------------------------------------
823
824For every single crystal a histogram, the ``'Data'`` item contains
825the structure factors as an np.array in item `'RefList'`.
826The columns in that array are documented below.
827
828.. tabularcolumns:: |l|p{4in}|
829
830==========  ====================================================
831  index         explanation
832==========  ====================================================
833 0,1,2          (float) h,k,l
834 3              (int) multiplicity
835 4              (float) d-space, :math:`\\AA`
836 5              (float) :math:`F_{obs}^2`
837 6              (float) :math:`\sigma(F_{obs}^2)`
838 7              (float) :math:`F_{calc}^2`
839 8              (float) :math:`F_{obs}^2T`
840 9              (float) :math:`F_{calc}^2T`
841 10             (float) reflection phase, in degrees
842 11             (float) intensity correction for reflection, this times
843                :math:`F_{obs}^2` or :math:`F_{calc}^2`
844                gives Iobs or Icalc
845==========  ====================================================
846
847.. _Image_table:
848
849.. index::
850   image: Image data object description
851   image: Image object descriptions
852
853Image Data Structure
854--------------------
855
856Every 2-dimensional image is stored in the GSAS-II data tree
857with a top-level entry named beginning with the string "IMG ". The
858image data are directly associated with that tree item and there
859are a series of children to that item. The routines :func:`GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
860and :func:`GSASIIstrIO.GetUsedHistogramsAndPhases` will
861load this information into a dictionary where the child tree name is
862used as a key, and the information in the main entry is assigned
863a key of ``Data``, as outlined below.
864
865.. tabularcolumns:: |l|l|p{4in}|
866
867======================  ======================  ====================================================
868  key                      sub-key              explanation
869======================  ======================  ====================================================
870Comments                    \                   (list of str) Text strings extracted from the original image data
871                                                header or a metafile. These cannot be changed by
872                                                the user; it may be empty.
873Image Controls              azmthOff            (float) The offset to be applied to an azimuthal
874                                                value. Accomodates
875                                                detector orientations other than with the detector
876                                                X-axis
877                                                horizontal.
878\                           background image    (list:str,float) The name of a tree item ("IMG ...") that is to be subtracted
879                                                during image integration multiplied by value. It must have the same size/shape as
880                                                the integrated image. NB: value < 0 for subtraction.
881\                           calibrant           (str) The material used for determining the position/orientation
882                                                of the image. The data is obtained from :func:`ImageCalibrants`
883                                                and UserCalibrants.py (supplied by user).
884\                           calibdmin           (float) The minimum d-spacing used during the last calibration run.
885\                           calibskip           (int) The number of expected diffraction lines skipped during the last
886                                                calibration run.
887\                           center              (list:floats) The [X,Y] point in detector coordinates (mm) where the direct beam
888                                                strikes the detector plane as determined by calibration. This point
889                                                does not have to be within the limits of the detector boundaries.
890\                           centerAzm           (bool) If True then the azimuth reported for the integrated slice
891                                                of the image is at the center line otherwise it is at the leading edge.
892\                           color               (str) The name of the colormap used to display the image. Default = 'Paired'.
893\                           cutoff              (float) The minimum value of I/Ib for a point selected in a diffraction ring for
894                                                calibration calculations. See pixLimit for details as how point is found.
895\                           DetDepth            (float) Coefficient for penetration correction to distance; accounts for diffraction
896                                                ring offset at higher angles. Optionally determined by calibration.
897\                           DetDepthRef         (bool) If True then refine DetDepth during calibration/recalibration calculation.
898\                           distance            (float) The distance (mm) from sample to detector plane.
899\                           ellipses            (list:lists) Each object in ellipses is a list [center,phi,radii,color] where
900                                                center (list) is location (mm) of the ellipse center on the detector plane, phi is the
901                                                rotation of the ellipse minor axis from the x-axis, and radii are the minor & major
902                                                radii of the ellipse. If radii[0] is negative then parameters describe a hyperbola. Color
903                                                is the selected drawing color (one of 'b', 'g' ,'r') for the ellipse/hyperbola.
904\                           edgemin             (float) Not used;  parameter in EdgeFinder code.
905\                           fullIntegrate       (bool) If True then integrate over full 360 deg azimuthal range.
906\                           GonioAngles         (list:floats) The 'Omega','Chi','Phi' goniometer angles used for this image.
907                                                Required for texture calculations.
908\                           invert_x            (bool) If True display the image with the x-axis inverted.
909\                           invert_y            (bool) If True display the image with the y-axis inverted.
910\                           IOtth               (list:floats) The minimum and maximum 2-theta values to be used for integration.
911\                           LRazimuth           (list:floats) The minimum and maximum azimuth values to be used for integration.
912\                           Oblique             (list:float,bool) If True apply a detector absorption correction using the value to the
913                                                intensities obtained during integration.
914\                           outAzimuths         (int) The number of azimuth pie slices.
915\                           outChannels         (int) The number of 2-theta steps.
916\                           pixelSize           (list:ints) The X,Y dimensions (microns) of each pixel.
917\                           pixLimit            (int) A box in the image with 2*pixLimit+1 edges is searched to find the maximum.
918                                                This value (I) along with the minimum (Ib) in the box is reported by :func:`GSASIIimage.ImageLocalMax`
919                                                and subject to cutoff in :func:`GSASIIimage.makeRing`.
920                                                Locations are used to construct rings of points for calibration calcualtions.
921\                           PolaVal             (list:float,bool) If type='SASD' and if True, apply polarization correction to intensities from
922                                                integration using value.
923\                           rings               (list:lists) Each entry is [X,Y,dsp] where X & Y are lists of x,y coordinates around a
924                                                diffraction ring with the same d-spacing (dsp)
925\                           ring                (list) The x,y coordinates of the >5 points on an inner ring
926                                                selected by the user,
927\                           Range               (list) The minimum & maximum values of the image
928\                           rotation            (float) The angle between the x-axis and the vector about which the
929                                                detector is tilted. Constrained to -180 to 180 deg.
930\                           SampleShape         (str) Currently only 'Cylinder'. Sample shape for Debye-Scherrer experiments; used for absorption
931                                                calculations.
932\                           SampleAbs           (list: float,bool) Value of absorption coefficient for Debye-Scherrer experimnents, flag if True
933                                                to cause correction to be applied.
934\                           setDefault          (bool) If True the use the image controls values for all new images to be read. (might be removed)
935\                           setRings            (bool) If True then display all the selected x,y ring positions (vida supra rings) used in the calibration.
936\                           showLines           (bool) If True then isplay the integration limits to be used.
937\                           size                (list:int) The number of pixels on the image x & y axes
938\                           type                (str) One of 'PWDR', 'SASD' or 'REFL' for powder, small angle or reflectometry data, respectively.
939\                           tilt                (float) The angle the detector normal makes with the incident beam; range -90 to 90.
940\                           wavelength          (float) The radiation wavelength (:math:`\\AA`) as entered by the user
941                                                (or someday obtained from the image header).
942Masks                       Arcs                (list: lists) Each entry [2-theta,[azimuth[0],azimuth[1]],thickness] describes an arc mask
943                                                to be excluded from integration
944\                           Frames              (list:lists) Each entry describes the x,y points (3 or more - mm) that describe a frame outside
945                                                of which is excluded from recalibration and integration. Only one frame is allowed.
946\                           Points              (list:lists) Each entry [x,y,radius] (mm) describes an excluded spot on the image to be excluded
947                                                from integration.
948\                           Polygons            (list:lists) Each entry is a list of 3+ [x,y] points (mm) that describe a polygon on the image
949                                                to be excluded from integration.
950\                           Rings               (list: lists) Each entry [2-theta,thickness] describes a ring mask
951                                                to be excluded from integration.
952\                           Thresholds          (list:[tuple,list]) [(Imin,Imax),[Imin,Imax]] This gives lower and upper limits for points on the image to be included
953                                                in integrsation. The tuple is the image intensity limits and the list are those set by the user.
954\                           SpotMask            (dict: int & array)
955                                                'esdMul'(int) number of standard deviations above mean ring intensity to mask
956                                                'spotMask' (bool array) the spot mask for every pixel in image         
957
958Stress/Strain               Sample phi          (float) Sample rotation about vertical axis.
959\                           Sample z            (float) Sample translation from the calibration sample position (for Sample phi = 0)
960                                                These will be restricted by space group symmetry; result of strain fit refinement.
961\                           Type                (str) 'True' or 'Conventional': The strain model used for the calculation.
962\                           d-zero              (list:dict) Each item is for a diffraction ring on the image; all items are from the same phase
963                                                and are used to determine the strain tensor.
964                                                The dictionary items are:
965                                                'Dset': (float) True d-spacing for the diffraction ring; entered by the user.
966                                                'Dcalc': (float) Average calculated d-spacing determined from strain coeff.
967                                                'Emat': (list: float) The strain tensor elements e11, e12 & e22 (e21=e12, rest are 0)
968                                                'Esig': (list: float) Esds for Emat from fitting.
969                                                'pixLimit': (int) Search range to find highest point on ring for each data point
970                                                'cutoff': (float) I/Ib cutoff for searching.
971                                                'ImxyObs': (list: lists) [[X],[Y]] observed points to be used for strain calculations.
972                                                'ImtaObs': (list: lists) [[d],[azm]] transformed via detector calibration from ImxyObs.
973                                                'ImtaCalc': (list: lists [[d],[azm]] calculated d-spacing & azimuth from fit.
974
975======================  ======================  ====================================================
976
977.. _parmDict_table:
978
979.. index::
980   single: Parameter dictionary
981
982Parameter Dictionary
983-------------------------
984
985The parameter dictionary contains all of the variable parameters for the refinement.
986The dictionary keys are the name of the parameter (<phase>:<hist>:<name>:<atom>).
987It is prepared in two ways. When loaded from the tree
988(in :meth:`GSASIIdataGUI.GSASII.MakeLSParmDict` and
989:meth:`GSASIIIO.ExportBaseclass.loadParmDict`),
990the values are lists with two elements: ``[value, refine flag]``
991
992When loaded from the GPX file (in
993:func:`GSASIIstrMain.Refine` and :func:`GSASIIstrMain.SeqRefine`), the value in the
994dict is the actual parameter value (usually a float, but sometimes a
995letter or string flag value (such as I or A for iso/anisotropic).
996
997
998*Classes and routines*
999----------------------
1000
1001'''
1002from __future__ import division, print_function
1003import platform
1004import re
1005import imp
1006import random as ran
1007import sys
1008import os.path as ospath
1009if '2' in platform.python_version_tuple()[0]:
1010    import cPickle
1011else:
1012    import pickle as cPickle
1013import GSASIIpath
1014import GSASIImath as G2mth
1015import GSASIIspc as G2spc
1016import numpy as np
1017
1018GSASIIpath.SetVersionNumber("$Revision: 4318 $")
1019
1020DefaultControls = {
1021    'deriv type':'analytic Hessian',
1022    'min dM/M':0.001,'shift factor':1.,'max cyc':3,'F**2':False,'SVDtol':1.e-6,
1023    'UsrReject':{'minF/sig':0.,'MinExt':0.01,'MaxDF/F':100.,'MaxD':500.,'MinD':0.05},
1024    'Copy2Next':False,'Reverse Seq':False,'HatomFix':False,
1025    'Author':'no name',
1026    'FreePrm1':'Sample humidity (%)',
1027    'FreePrm2':'Sample voltage (V)',
1028    'FreePrm3':'Applied load (MN)',
1029    'ShowCell':False,
1030    }
1031'''Values to be used as defaults for the initial contents of the ``Controls``
1032data tree item.
1033'''
1034def StripUnicode(string,subs='.'):
1035    '''Strip non-ASCII characters from strings
1036
1037    :param str string: string to strip Unicode characters from
1038    :param str subs: character(s) to place into string in place of each
1039      Unicode character. Defaults to '.'
1040
1041    :returns: a new string with only ASCII characters
1042    '''
1043    s = ''
1044    for c in string:
1045        if ord(c) < 128:
1046            s += c
1047        else:
1048            s += subs
1049    return s
1050#    return s.encode('ascii','replace')
1051
1052def MakeUniqueLabel(lbl,labellist):
1053    '''Make sure that every a label is unique against a list by adding
1054    digits at the end until it is not found in list.
1055
1056    :param str lbl: the input label
1057    :param list labellist: the labels that have already been encountered
1058    :returns: lbl if not found in labellist or lbl with ``_1-9`` (or
1059      ``_10-99``, etc.) appended at the end
1060    '''
1061    lbl = StripUnicode(lbl.strip(),'_')
1062    if not lbl: # deal with a blank label
1063        lbl = '_1'
1064    if lbl not in labellist:
1065        labellist.append(lbl)
1066        return lbl
1067    i = 1
1068    prefix = lbl
1069    if '_' in lbl:
1070        prefix = lbl[:lbl.rfind('_')]
1071        suffix = lbl[lbl.rfind('_')+1:]
1072        try:
1073            i = int(suffix)+1
1074        except: # suffix could not be parsed
1075            i = 1
1076            prefix = lbl
1077    while prefix+'_'+str(i) in labellist:
1078        i += 1
1079    else:
1080        lbl = prefix+'_'+str(i)
1081        labellist.append(lbl)
1082    return lbl
1083
1084PhaseIdLookup = {}
1085'''dict listing phase name and random Id keyed by sequential phase index as a str;
1086best to access this using :func:`LookupPhaseName`
1087'''
1088PhaseRanIdLookup = {}
1089'''dict listing phase sequential index keyed by phase random Id;
1090best to access this using :func:`LookupPhaseId`
1091'''
1092HistIdLookup = {}
1093'''dict listing histogram name and random Id, keyed by sequential histogram index as a str;
1094best to access this using :func:`LookupHistName`
1095'''
1096HistRanIdLookup = {}
1097'''dict listing histogram sequential index keyed by histogram random Id;
1098best to access this using :func:`LookupHistId`
1099'''
1100AtomIdLookup = {}
1101'''dict listing for each phase index as a str, the atom label and atom random Id,
1102keyed by atom sequential index as a str;
1103best to access this using :func:`LookupAtomLabel`
1104'''
1105AtomRanIdLookup = {}
1106'''dict listing for each phase the atom sequential index keyed by atom random Id;
1107best to access this using :func:`LookupAtomId`
1108'''
1109ShortPhaseNames = {}
1110'''a dict containing a possibly shortened and when non-unique numbered
1111version of the phase name. Keyed by the phase sequential index.
1112'''
1113ShortHistNames = {}
1114'''a dict containing a possibly shortened and when non-unique numbered
1115version of the histogram name. Keyed by the histogram sequential index.
1116'''
1117
1118#VarDesc = {}  # removed 1/30/19 BHT as no longer needed (I think)
1119#''' This dictionary lists descriptions for GSAS-II variables,
1120#as set in :func:`CompileVarDesc`. See that function for a description
1121#for how keys and values are written.
1122#'''
1123
1124reVarDesc = {}
1125''' This dictionary lists descriptions for GSAS-II variables where
1126keys are compiled regular expressions that will match the name portion
1127of a parameter name. Initialized in :func:`CompileVarDesc`.
1128'''
1129
1130reVarStep = {}
1131''' This dictionary lists the preferred step size for numerical
1132derivative computation w/r to a GSAS-II variable. Keys are compiled
1133regular expressions and values are the step size for that parameter.
1134Initialized in :func:`CompileVarDesc`.
1135'''
1136# create a default space group object for P1; N.B. fails when building documentation
1137try:
1138    P1SGData = G2spc.SpcGroup('P 1')[1] # data structure for default space group
1139except:
1140    pass
1141
1142def GetPhaseNames(fl):
1143    ''' Returns a list of phase names found under 'Phases' in GSASII gpx file
1144    NB: there is another one of these in GSASIIstrIO.py that uses the gpx filename
1145
1146    :param file fl: opened .gpx file
1147    :return: list of phase names
1148    '''
1149    PhaseNames = []
1150    while True:
1151        try:
1152            data = cPickle.load(fl)
1153        except EOFError:
1154            break
1155        datum = data[0]
1156        if 'Phases' == datum[0]:
1157            for datus in data[1:]:
1158                PhaseNames.append(datus[0])
1159    fl.seek(0)          #reposition file
1160    return PhaseNames
1161
1162def SetNewPhase(Name='New Phase',SGData=None,cell=None,Super=None):
1163    '''Create a new phase dict with default values for various parameters
1164
1165    :param str Name: Name for new Phase
1166
1167    :param dict SGData: space group data from :func:`GSASIIspc:SpcGroup`;
1168      defaults to data for P 1
1169
1170    :param list cell: unit cell parameter list; defaults to
1171      [1.0,1.0,1.0,90.,90,90.,1.]
1172
1173    '''
1174    if SGData is None: SGData = P1SGData
1175    if cell is None: cell=[1.0,1.0,1.0,90.,90.,90.,1.]
1176    phaseData = {
1177        'ranId':ran.randint(0,sys.maxsize),
1178        'General':{
1179            'Name':Name,
1180            'Type':'nuclear',
1181            'Modulated':False,
1182            'AtomPtrs':[3,1,7,9],
1183            'SGData':SGData,
1184            'Cell':[False,]+cell,
1185            'Pawley dmin':1.0,
1186            'Data plot type':'None',
1187            'SH Texture':{
1188                'Order':0,
1189                'Model':'cylindrical',
1190                'Sample omega':[False,0.0],
1191                'Sample chi':[False,0.0],
1192                'Sample phi':[False,0.0],
1193                'SH Coeff':[False,{}],
1194                'SHShow':False,
1195                'PFhkl':[0,0,1],
1196                'PFxyz':[0,0,1],
1197                'PlotType':'Pole figure',
1198                'Penalty':[['',],0.1,False,1.0]}},
1199        'Atoms':[],
1200        'Drawing':{},
1201        'Histograms':{},
1202        'Pawley ref':[],
1203        'RBModels':{},
1204        }
1205    if Super and Super.get('Use',False):
1206        phaseData['General'].update({'Modulated':True,'Super':True,'SuperSg':Super['ssSymb']})
1207        phaseData['General']['SSGData'] = G2spc.SSpcGroup(SGData,Super['ssSymb'])[1]
1208        phaseData['General']['SuperVec'] = [Super['ModVec'],False,Super['maxH']]
1209
1210    return phaseData
1211
1212def ReadCIF(URLorFile):
1213    '''Open a CIF, which may be specified as a file name or as a URL using PyCifRW
1214    (from James Hester).
1215    The open routine gets confused with DOS names that begin with a letter and colon
1216    "C:\dir\" so this routine will try to open the passed name as a file and if that
1217    fails, try it as a URL
1218
1219    :param str URLorFile: string containing a URL or a file name. Code will try first
1220      to open it as a file and then as a URL.
1221
1222    :returns: a PyCifRW CIF object.
1223    '''
1224    import CifFile as cif # PyCifRW from James Hester
1225
1226    # alternate approach:
1227    #import urllib
1228    #ciffile = 'file:'+urllib.pathname2url(filename)
1229
1230    try:
1231        fp = open(URLorFile,'r')
1232        cf = cif.ReadCif(fp)
1233        fp.close()
1234        return cf
1235    except IOError:
1236        return cif.ReadCif(URLorFile)
1237
1238def IndexAllIds(Histograms,Phases):
1239    '''Scan through the used phases & histograms and create an index
1240    to the random numbers of phases, histograms and atoms. While doing this,
1241    confirm that assigned random numbers are unique -- just in case lightning
1242    strikes twice in the same place.
1243
1244    Note: this code assumes that the atom random Id (ranId) is the last
1245    element each atom record.
1246
1247    This is called in three places (only): :func:`GSASIIstrIO.GetUsedHistogramsAndPhases`
1248    (which loads the histograms and phases from a GPX file),
1249    :meth:`~GSASIIdataGUI.GSASII.GetUsedHistogramsAndPhasesfromTree`
1250    (which loads the histograms and phases from the data tree.) and
1251    :meth:`GSASIIconstrGUI.UpdateConstraints`
1252    (which displays & edits the constraints in a GUI)
1253
1254    TODO: do we need a lookup for rigid body variables?
1255    '''
1256    # process phases and atoms
1257    PhaseIdLookup.clear()
1258    PhaseRanIdLookup.clear()
1259    AtomIdLookup.clear()
1260    AtomRanIdLookup.clear()
1261    ShortPhaseNames.clear()
1262    for ph in Phases:
1263        cx,ct,cs,cia = Phases[ph]['General']['AtomPtrs']
1264        ranId = Phases[ph]['ranId']
1265        while ranId in PhaseRanIdLookup:
1266            # Found duplicate random Id! note and reassign
1267            print ("\n\n*** Phase "+str(ph)+" has repeated ranId. Fixing.\n")
1268            Phases[ph]['ranId'] = ranId = ran.randint(0,sys.maxsize)
1269        pId = str(Phases[ph]['pId'])
1270        PhaseIdLookup[pId] = (ph,ranId)
1271        PhaseRanIdLookup[ranId] = pId
1272        shortname = ph  #[:10]
1273        while shortname in ShortPhaseNames.values():
1274            shortname = ph[:8] + ' ('+ pId + ')'
1275        ShortPhaseNames[pId] = shortname
1276        AtomIdLookup[pId] = {}
1277        AtomRanIdLookup[pId] = {}
1278        for iatm,at in enumerate(Phases[ph]['Atoms']):
1279            ranId = at[cia+8]
1280            while ranId in AtomRanIdLookup[pId]: # check for dups
1281                print ("\n\n*** Phase "+str(ph)+" atom "+str(iatm)+" has repeated ranId. Fixing.\n")
1282                at[cia+8] = ranId = ran.randint(0,sys.maxsize)
1283            AtomRanIdLookup[pId][ranId] = str(iatm)
1284            if Phases[ph]['General']['Type'] == 'macromolecular':
1285                label = '%s_%s_%s_%s'%(at[ct-1],at[ct-3],at[ct-4],at[ct-2])
1286            else:
1287                label = at[ct-1]
1288            AtomIdLookup[pId][str(iatm)] = (label,ranId)
1289    # process histograms
1290    HistIdLookup.clear()
1291    HistRanIdLookup.clear()
1292    ShortHistNames.clear()
1293    for hist in Histograms:
1294        ranId = Histograms[hist]['ranId']
1295        while ranId in HistRanIdLookup:
1296            # Found duplicate random Id! note and reassign
1297            print ("\n\n*** Histogram "+str(hist)+" has repeated ranId. Fixing.\n")
1298            Histograms[hist]['ranId'] = ranId = ran.randint(0,sys.maxsize)
1299        hId = str(Histograms[hist]['hId'])
1300        HistIdLookup[hId] = (hist,ranId)
1301        HistRanIdLookup[ranId] = hId
1302        shortname = hist[:15]
1303        while shortname in ShortHistNames.values():
1304            shortname = hist[:11] + ' ('+ hId + ')'
1305        ShortHistNames[hId] = shortname
1306
1307def LookupAtomId(pId,ranId):
1308    '''Get the atom number from a phase and atom random Id
1309
1310    :param int/str pId: the sequential number of the phase
1311    :param int ranId: the random Id assigned to an atom
1312
1313    :returns: the index number of the atom (str)
1314    '''
1315    if not AtomRanIdLookup:
1316        raise Exception('Error: LookupAtomId called before IndexAllIds was run')
1317    if pId is None or pId == '':
1318        raise KeyError('Error: phase is invalid (None or blank)')
1319    pId = str(pId)
1320    if pId not in AtomRanIdLookup:
1321        raise KeyError('Error: LookupAtomId does not have phase '+pId)
1322    if ranId not in AtomRanIdLookup[pId]:
1323        raise KeyError('Error: LookupAtomId, ranId '+str(ranId)+' not in AtomRanIdLookup['+pId+']')
1324    return AtomRanIdLookup[pId][ranId]
1325
1326def LookupAtomLabel(pId,index):
1327    '''Get the atom label from a phase and atom index number
1328
1329    :param int/str pId: the sequential number of the phase
1330    :param int index: the index of the atom in the list of atoms
1331
1332    :returns: the label for the atom (str) and the random Id of the atom (int)
1333    '''
1334    if not AtomIdLookup:
1335        raise Exception('Error: LookupAtomLabel called before IndexAllIds was run')
1336    if pId is None or pId == '':
1337        raise KeyError('Error: phase is invalid (None or blank)')
1338    pId = str(pId)
1339    if pId not in AtomIdLookup:
1340        raise KeyError('Error: LookupAtomLabel does not have phase '+pId)
1341    if index not in AtomIdLookup[pId]:
1342        raise KeyError('Error: LookupAtomLabel, ranId '+str(index)+' not in AtomRanIdLookup['+pId+']')
1343    return AtomIdLookup[pId][index]
1344
1345def LookupPhaseId(ranId):
1346    '''Get the phase number and name from a phase random Id
1347
1348    :param int ranId: the random Id assigned to a phase
1349    :returns: the sequential Id (pId) number for the phase (str)
1350    '''
1351    if not PhaseRanIdLookup:
1352        raise Exception('Error: LookupPhaseId called before IndexAllIds was run')
1353    if ranId not in PhaseRanIdLookup:
1354        raise KeyError('Error: LookupPhaseId does not have ranId '+str(ranId))
1355    return PhaseRanIdLookup[ranId]
1356
1357def LookupPhaseName(pId):
1358    '''Get the phase number and name from a phase Id
1359
1360    :param int/str pId: the sequential assigned to a phase
1361    :returns:  (phase,ranId) where phase is the name of the phase (str)
1362      and ranId is the random # id for the phase (int)
1363    '''
1364    if not PhaseIdLookup:
1365        raise Exception('Error: LookupPhaseName called before IndexAllIds was run')
1366    if pId is None or pId == '':
1367        raise KeyError('Error: phase is invalid (None or blank)')
1368    pId = str(pId)
1369    if pId not in PhaseIdLookup:
1370        raise KeyError('Error: LookupPhaseName does not have index '+pId)
1371    return PhaseIdLookup[pId]
1372
1373def LookupHistId(ranId):
1374    '''Get the histogram number and name from a histogram random Id
1375
1376    :param int ranId: the random Id assigned to a histogram
1377    :returns: the sequential Id (hId) number for the histogram (str)
1378    '''
1379    if not HistRanIdLookup:
1380        raise Exception('Error: LookupHistId called before IndexAllIds was run')
1381    if ranId not in HistRanIdLookup:
1382        raise KeyError('Error: LookupHistId does not have ranId '+str(ranId))
1383    return HistRanIdLookup[ranId]
1384
1385def LookupHistName(hId):
1386    '''Get the histogram number and name from a histogram Id
1387
1388    :param int/str hId: the sequential assigned to a histogram
1389    :returns:  (hist,ranId) where hist is the name of the histogram (str)
1390      and ranId is the random # id for the histogram (int)
1391    '''
1392    if not HistIdLookup:
1393        raise Exception('Error: LookupHistName called before IndexAllIds was run')
1394    if hId is None or hId == '':
1395        raise KeyError('Error: histogram is invalid (None or blank)')
1396    hId = str(hId)
1397    if hId not in HistIdLookup:
1398        raise KeyError('Error: LookupHistName does not have index '+hId)
1399    return HistIdLookup[hId]
1400
1401def fmtVarDescr(varname):
1402    '''Return a string with a more complete description for a GSAS-II variable
1403
1404    :param str varname: A full G2 variable name with 2 or 3 or 4
1405       colons (<p>:<h>:name[:<a>] or <p>::RBname:<r>:<t>])
1406
1407    :returns: a string with the description
1408    '''
1409    s,l = VarDescr(varname)
1410    return s+": "+l
1411
1412def VarDescr(varname):
1413    '''Return two strings with a more complete description for a GSAS-II variable
1414
1415    :param str name: A full G2 variable name with 2 or 3 or 4
1416       colons (<p>:<h>:name[:<a>] or <p>::RBname:<r>:<t>])
1417
1418    :returns: (loc,meaning) where loc describes what item the variable is mapped
1419      (phase, histogram, etc.) and meaning describes what the variable does.
1420    '''
1421
1422    # special handling for parameter names without a colon
1423    # for now, assume self-defining
1424    if varname.find(':') == -1:
1425        return "Global",varname
1426
1427    l = getVarDescr(varname)
1428    if not l:
1429        return ("invalid variable name ("+str(varname)+")!"),""
1430#        return "invalid variable name!",""
1431
1432    if not l[-1]:
1433        l[-1] = "(variable needs a definition! Set it in CompileVarDesc)"
1434
1435    if len(l) == 3:         #SASD variable name!
1436        s = 'component:'+l[1]
1437        return s,l[-1]
1438    s = ""
1439    if l[0] is not None and l[1] is not None: # HAP: keep short
1440        if l[2] == "Scale": # fix up ambigous name
1441            l[5] = "Phase fraction"
1442        if l[0] == '*':
1443            lbl = 'Seq. ref.'
1444        else:
1445            lbl = ShortPhaseNames.get(l[0],'? #'+str(l[0]))
1446        if l[1] == '*':
1447            hlbl = 'Seq. ref.'
1448        else:
1449            hlbl = ShortHistNames.get(l[1],'? #'+str(l[1]))
1450        if hlbl[:4] == 'HKLF':
1451            hlbl = 'Xtl='+hlbl[5:]
1452        elif hlbl[:4] == 'PWDR':
1453            hlbl = 'Pwd='+hlbl[5:]
1454        else:
1455            hlbl = 'Hist='+hlbl
1456        s = "Ph="+str(lbl)+" * "+str(hlbl)
1457    else:
1458        if l[2] == "Scale": # fix up ambigous name: must be scale factor, since not HAP
1459            l[5] = "Scale factor"
1460        if l[2] == 'Back': # background parameters are "special", alas
1461            s = 'Hist='+ShortHistNames.get(l[1],'? #'+str(l[1]))
1462            l[-1] += ' #'+str(l[3])
1463        elif l[4] is not None: # rigid body parameter or modulation parm
1464            lbl = ShortPhaseNames.get(l[0],'phase?')
1465            if 'RB' in l[2]:    #rigid body parm
1466                s = "Res #"+str(l[3])+" body #"+str(l[4])+" in "+str(lbl)
1467            else: #modulation parm
1468                s = 'Atom %s wave %s in %s'%(LookupAtomLabel(l[0],l[3])[0],l[4],lbl)
1469        elif l[3] is not None: # atom parameter,
1470            lbl = ShortPhaseNames.get(l[0],'phase?')
1471            try:
1472                albl = LookupAtomLabel(l[0],l[3])[0]
1473            except KeyError:
1474                albl = 'Atom?'
1475            s = "Atom "+str(albl)+" in "+str(lbl)
1476        elif l[0] == '*':
1477            s = "All phases "
1478        elif l[0] is not None:
1479            lbl = ShortPhaseNames.get(l[0],'phase?')
1480            s = "Phase "+str(lbl)
1481        elif l[1] == '*':
1482            s = 'All hists'
1483        elif l[1] is not None:
1484            hlbl = ShortHistNames.get(l[1],'? #'+str(l[1]))
1485            if hlbl[:4] == 'HKLF':
1486                hlbl = 'Xtl='+hlbl[5:]
1487            elif hlbl[:4] == 'PWDR':
1488                hlbl = 'Pwd='+hlbl[5:]
1489            else:
1490                hlbl = 'Hist='+hlbl
1491            s = str(hlbl)
1492    if not s:
1493        s = 'Global'
1494    return s,l[-1]
1495
1496def getVarDescr(varname):
1497    '''Return a short description for a GSAS-II variable
1498
1499    :param str name: A full G2 variable name with 2 or 3 or 4
1500       colons (<p>:<h>:name[:<a1>][:<a2>])
1501
1502    :returns: a six element list as [`p`,`h`,`name`,`a1`,`a2`,`description`],
1503      where `p`, `h`, `a1`, `a2` are str values or `None`, for the phase number,
1504      the histogram number and the atom number; `name` will always be
1505      a str; and `description` is str or `None`.
1506      If the variable name is incorrectly formed (for example, wrong
1507      number of colons), `None` is returned instead of a list.
1508    '''
1509    l = varname.split(':')
1510    if len(l) == 2:     #SASD parameter name
1511        return varname,l[0],getDescr(l[1])
1512    if len(l) == 3:
1513        l += [None,None]
1514    elif len(l) == 4:
1515        l += [None]
1516    elif len(l) != 5:
1517        return None
1518    for i in (0,1,3,4):
1519        if l[i] == "":
1520            l[i] = None
1521    l += [getDescr(l[2])]
1522    return l
1523
1524def CompileVarDesc():
1525    '''Set the values in the variable lookup tables
1526    (:attr:`reVarDesc` and :attr:`reVarStep`).
1527    This is called in :func:`getDescr` and :func:`getVarStep` so this
1528    initialization is always done before use.
1529
1530    Note that keys may contain regular expressions, where '[xyz]'
1531    matches 'x' 'y' or 'z' (equivalently '[x-z]' describes this as range
1532    of values). '.*' matches any string. For example::
1533
1534    'AUiso':'Atomic isotropic displacement parameter',
1535
1536    will match variable ``'p::AUiso:a'``.
1537    If parentheses are used in the key, the contents of those parentheses can be
1538    used in the value, such as::
1539
1540    'AU([123][123])':'Atomic anisotropic displacement parameter U\\1',
1541
1542    will match ``AU11``, ``AU23``,.. and `U11`, `U23` etc will be displayed
1543    in the value when used.
1544
1545    '''
1546    if reVarDesc: return # already done
1547    for key,value in {
1548        # derived or other sequential vars
1549        '([abc])$' : 'Lattice parameter, \\1, from Ai and Djk', # N.B. '$' prevents match if any characters follow
1550        u'\u03B1' : u'Lattice parameter, \u03B1, from Ai and Djk',
1551        u'\u03B2' : u'Lattice parameter, \u03B2, from Ai and Djk',
1552        u'\u03B3' : u'Lattice parameter, \u03B3, from Ai and Djk',
1553        # ambiguous, alas:
1554        'Scale' : 'Phase or Histogram scale factor',
1555        # Phase vars (p::<var>)
1556        'A([0-5])' : ('Reciprocal metric tensor component \\1',1e-5),
1557        '[vV]ol' : 'Unit cell volume', # probably an error that both upper and lower case are used
1558        # Atom vars (p::<var>:a)
1559        'dA([xyz])$' : ('change to atomic coordinate, \\1',1e-6),
1560        'A([xyz])$' : '\\1 fractional atomic coordinate',
1561        'AUiso':('Atomic isotropic displacement parameter',1e-4),
1562        'AU([123][123])':('Atomic anisotropic displacement parameter U\\1',1e-4),
1563        'Afrac': ('Atomic site fraction parameter',1e-5),
1564        'Amul': 'Atomic site multiplicity value',
1565        'AM([xyz])$' : 'Atomic magnetic moment parameter, \\1',
1566        # Hist & Phase (HAP) vars (p:h:<var>)
1567        'Back': 'Background term',
1568        'BkPkint;(.*)':'Background peak #\\1 intensity',
1569        'BkPkpos;(.*)':'Background peak #\\1 position',
1570        'BkPksig;(.*)':'Background peak #\\1 Gaussian width',
1571        'BkPkgam;(.*)':'Background peak #\\1 Cauchy width',
1572        'Bab([AU])': 'Babinet solvent scattering coef. \\1',
1573        'D([123][123])' : 'Anisotropic strain coef. \\1',
1574        'Extinction' : 'Extinction coef.',
1575        'MD' : 'March-Dollase coef.',
1576        'Mustrain;.*' : 'Microstrain coef.',
1577        'Size;.*' : 'Crystallite size value',
1578        'eA$' : 'Cubic mustrain value',
1579        'Ep$' : 'Primary extinction',
1580        'Es$' : 'Secondary type II extinction',
1581        'Eg$' : 'Secondary type I extinction',
1582        'Flack' : 'Flack parameter',
1583        'TwinFr' : 'Twin fraction',
1584        #Histogram vars (:h:<var>)
1585        'Absorption' : 'Absorption coef.',
1586        'Displace([XY])' : ('Debye-Scherrer sample displacement \\1',0.1),
1587        'Lam' : ('Wavelength',1e-6),
1588        'I\(L2\)\/I\(L1\)' : ('Ka2/Ka1 intensity ratio',0.001),
1589        'Polariz\.' : ('Polarization correction',1e-3),
1590        'SH/L' : ('FCJ peak asymmetry correction',1e-4),
1591        '([UVW])$' : ('Gaussian instrument broadening \\1',1e-5),
1592        '([XYZ])$' : ('Cauchy instrument broadening \\1',1e-5),
1593        'Zero' : 'Debye-Scherrer zero correction',
1594        'Shift' : 'Bragg-Brentano sample displ.',
1595        'SurfRoughA' : 'Bragg-Brenano surface roughness A',
1596        'SurfRoughB' : 'Bragg-Brenano surface roughness B',
1597        'Transparency' : 'Bragg-Brentano sample tranparency',
1598        'DebyeA' : 'Debye model amplitude',
1599        'DebyeR' : 'Debye model radius',
1600        'DebyeU' : 'Debye model Uiso',
1601        'RBV.*' : 'Vector rigid body parameter',
1602        'RBR.*' : 'Residue rigid body parameter',
1603        'RBRO([aijk])' : 'Residue rigid body orientation parameter',
1604        'RBRP([xyz])' : 'Residue rigid body position parameter',
1605        'RBRTr;.*' : 'Residue rigid body torsion parameter',
1606        'RBR([TLS])([123AB][123AB])' : 'Residue rigid body group disp. param.',
1607        'constr([0-9]*)' : 'Parameter from constraint',
1608        # supersymmetry parameters  p::<var>:a:o 'Flen','Fcent'?
1609        'mV([0-2])$' : 'Modulation vector component \\1',
1610        'Fsin'  :   'Sin site fraction modulation',
1611        'Fcos'  :   'Cos site fraction modulation',
1612        'Fzero'  :   'Crenel function offset',      #may go away
1613        'Fwid'   :   'Crenel function width',
1614        'Tmin'   :   'ZigZag/Block min location',
1615        'Tmax'   :   'ZigZag/Block max location',
1616        '([XYZ])max': 'ZigZag/Block max value for \\1',
1617        '([XYZ])sin'  : 'Sin position wave for \\1',
1618        '([XYZ])cos'  : 'Cos position wave for \\1',
1619        'U([123][123])sin$' :  'Sin thermal wave for U\\1',
1620        'U([123][123])cos$' :  'Cos thermal wave for U\\1',
1621        'M([XYZ])sin$' :  'Sin mag. moment wave for \\1',
1622        'M([XYZ])cos$' :  'Cos mag. moment wave for \\1',
1623        # PDF peak parms (l:<var>;l = peak no.)
1624        'PDFpos'  : 'PDF peak position',
1625        'PDFmag'  : 'PDF peak magnitude',
1626        'PDFsig'  : 'PDF peak std. dev.',
1627        # SASD vars (l:<var>;l = component)
1628        'Aspect ratio' : 'Particle aspect ratio',
1629        'Length' : 'Cylinder length',
1630        'Diameter' : 'Cylinder/disk diameter',
1631        'Thickness' : 'Disk thickness',
1632        'Shell thickness' : 'Multiplier to get inner(<1) or outer(>1) sphere radius',
1633        'Dist' : 'Interparticle distance',
1634        'VolFr' : 'Dense scatterer volume fraction',
1635        'epis' : 'Sticky sphere epsilon',
1636        'Sticky' : 'Stickyness',
1637        'Depth' : 'Well depth',
1638        'Width' : 'Well width',
1639        'Volume' : 'Particle volume',
1640        'Radius' : 'Sphere/cylinder/disk radius',
1641        'Mean' : 'Particle mean radius',
1642        'StdDev' : 'Standard deviation in Mean',
1643        'G$': 'Guinier prefactor',
1644        'Rg$': 'Guinier radius of gyration',
1645        'B$': 'Porod prefactor',
1646        'P$': 'Porod power',
1647        'Cutoff': 'Porod cutoff',
1648        'PkInt': 'Bragg peak intensity',
1649        'PkPos': 'Bragg peak position',
1650        'PkSig': 'Bragg peak sigma',
1651        'PkGam': 'Bragg peak gamma',
1652        'e([12][12])' : 'strain tensor e\1',   # strain vars e11, e22, e12
1653        'Dcalc': 'Calc. d-spacing',
1654        'Back$': 'background parameter',
1655        'pos$': 'peak position',
1656        'int$': 'peak intensity',
1657        'WgtFrac':'phase weight fraction',
1658        'alpha':'TOF profile term',
1659        'beta-[01q]':'TOF profile term',
1660        'sig-[012q]':'TOF profile term',
1661        'dif[ABC]':'TOF to d-space calibration',
1662        'C\([0-9]*,[0-9]*\)' : 'spherical harmonics preferred orientation coef.',
1663        }.items():
1664        if len(value) == 2:
1665            #VarDesc[key] = value[0]
1666            reVarDesc[re.compile(key)] = value[0]
1667            reVarStep[re.compile(key)] = value[1]
1668        else:
1669            #VarDesc[key] = value
1670            reVarDesc[re.compile(key)] = value
1671
1672def removeNonRefined(parmList):
1673    '''Remove items from variable list that are not refined and should not
1674    appear as options for constraints
1675
1676    :param list parmList: a list of strings of form "p:h:VAR:a" where
1677      VAR is the variable name
1678
1679    :returns: a list after removing variables where VAR matches a
1680      entry in local variable NonRefinedList
1681    '''
1682    NonRefinedList = ['Omega','Type','Chi','Phi', 'Azimuth','Gonio. radius',
1683                          'Lam1','Lam2','Back','Temperature','Pressure',
1684                          'FreePrm1','FreePrm2','FreePrm3',
1685                          'Source','nPeaks','LeBail','newLeBail','Bank',
1686                          'nDebye', #'',
1687                    ]
1688    return [prm for prm in parmList if prm.split(':')[2] not in NonRefinedList]
1689       
1690def getDescr(name):
1691    '''Return a short description for a GSAS-II variable
1692
1693    :param str name: The descriptive part of the variable name without colons (:)
1694
1695    :returns: a short description or None if not found
1696    '''
1697
1698    CompileVarDesc() # compile the regular expressions, if needed
1699    for key in reVarDesc:
1700        m = key.match(name)
1701        if m:
1702            reVarDesc[key]
1703            return m.expand(reVarDesc[key])
1704    return None
1705
1706def getVarStep(name,parmDict=None):
1707    '''Return a step size for computing the derivative of a GSAS-II variable
1708
1709    :param str name: A complete variable name (with colons, :)
1710    :param dict parmDict: A dict with parameter values or None (default)
1711
1712    :returns: a float that should be an appropriate step size, either from
1713      the value supplied in :func:`CompileVarDesc` or based on the value for
1714      name in parmDict, if supplied. If not found or the value is zero,
1715      a default value of 1e-5 is used. If parmDict is None (default) and
1716      no value is provided in :func:`CompileVarDesc`, then None is returned.
1717    '''
1718    CompileVarDesc() # compile the regular expressions, if needed
1719    for key in reVarStep:
1720        m = key.match(name)
1721        if m:
1722            return reVarStep[key]
1723    if parmDict is None: return None
1724    val = parmDict.get(key,0.0)
1725    if abs(val) > 0.05:
1726        return abs(val)/1000.
1727    else:
1728        return 1e-5
1729
1730def GenWildCard(varlist):
1731    '''Generate wildcard versions of G2 variables. These introduce '*'
1732    for a phase, histogram or atom number (but only for one of these
1733    fields) but only when there is more than one matching variable in the
1734    input variable list. So if the input is this::
1735
1736      varlist = ['0::AUiso:0', '0::AUiso:1', '1::AUiso:0']
1737
1738    then the output will be this::
1739
1740       wildList = ['*::AUiso:0', '0::AUiso:*']
1741
1742    :param list varlist: an input list of GSAS-II variable names
1743      (such as 0::AUiso:0)
1744
1745    :returns: wildList, the generated list of wild card variable names.
1746    '''
1747    wild = []
1748    for i in (0,1,3):
1749        currentL = varlist[:]
1750        while currentL:
1751            item1 = currentL.pop(0)
1752            i1splt = item1.split(':')
1753            if i >= len(i1splt): continue
1754            if i1splt[i]:
1755                nextL = []
1756                i1splt[i] = '[0-9]+'
1757                rexp = re.compile(':'.join(i1splt))
1758                matchlist = [item1]
1759                for nxtitem in currentL:
1760                    if rexp.match(nxtitem):
1761                        matchlist += [nxtitem]
1762                    else:
1763                        nextL.append(nxtitem)
1764                if len(matchlist) > 1:
1765                    i1splt[i] = '*'
1766                    wild.append(':'.join(i1splt))
1767                currentL = nextL
1768    return wild
1769
1770def LookupWildCard(varname,varlist):
1771    '''returns a list of variable names from list varname
1772    that match wildcard name in varname
1773
1774    :param str varname: a G2 variable name containing a wildcard
1775      (such as \*::var)
1776    :param list varlist: the list of all variable names used in
1777      the current project
1778    :returns: a list of matching GSAS-II variables (may be empty)
1779    '''
1780    rexp = re.compile(varname.replace('*','[0-9]+'))
1781    return sorted([var for var in varlist if rexp.match(var)])
1782
1783
1784def _lookup(dic,key):
1785    '''Lookup a key in a dictionary, where None returns an empty string
1786    but an unmatched key returns a question mark. Used in :class:`G2VarObj`
1787    '''
1788    if key is None:
1789        return ""
1790    elif key == "*":
1791        return "*"
1792    else:
1793        return dic.get(key,'?')
1794
1795def SortVariables(varlist):
1796    '''Sorts variable names in a sensible manner
1797    '''
1798    def cvnnums(var):
1799        v = []
1800        for i in var.split(':'):
1801            if i == '':
1802                v.append(-1)
1803                continue
1804            try:
1805                v.append(int(i))
1806            except:
1807                v.append(i)
1808        return v
1809    return sorted(varlist,key=cvnnums)
1810
1811class G2VarObj(object):
1812    '''Defines a GSAS-II variable either using the phase/atom/histogram
1813    unique Id numbers or using a character string that specifies
1814    variables by phase/atom/histogram number (which can change).
1815    Note that :func:`LoadID` should be used to (re)load the current Ids
1816    before creating or later using the G2VarObj object.
1817
1818    This can store rigid body variables, but does not translate the residue # and
1819    body # to/from random Ids
1820
1821    A :class:`G2VarObj` object can be created with a single parameter:
1822
1823    :param str/tuple varname: a single value can be used to create a :class:`G2VarObj`
1824      object. If a string, it must be of form "p:h:var" or "p:h:var:a", where
1825
1826     * p is the phase number (which may be left blank or may be '*' to indicate all phases);
1827     * h is the histogram number (which may be left blank or may be '*' to indicate all histograms);
1828     * a is the atom number (which may be left blank in which case the third colon is omitted).
1829       The atom number can be specified as '*' if a phase number is specified (not as '*').
1830       For rigid body variables, specify a will be a string of form "residue:body#"
1831
1832      Alternately a single tuple of form (Phase,Histogram,VarName,AtomID) can be used, where
1833      Phase, Histogram, and AtomID are None or are ranId values (or one can be '*')
1834      and VarName is a string. Note that if Phase is '*' then the AtomID is an atom number.
1835      For a rigid body variables, AtomID is a string of form "residue:body#".
1836
1837    If four positional arguments are supplied, they are:
1838
1839    :param str/int phasenum: The number for the phase (or None or '*')
1840    :param str/int histnum: The number for the histogram (or None or '*')
1841    :param str varname: a single value can be used to create a :class:`G2VarObj`
1842    :param str/int atomnum: The number for the atom (or None or '*')
1843
1844    '''
1845    IDdict = {}
1846    IDdict['phases'] = {}
1847    IDdict['hists'] = {}
1848    IDdict['atoms'] = {}
1849    def __init__(self,*args):
1850        self.phase = None
1851        self.histogram = None
1852        self.name = ''
1853        self.atom = None
1854        if len(args) == 1 and (type(args[0]) is list or type(args[0]) is tuple) and len(args[0]) == 4:
1855            # single arg with 4 values
1856            self.phase,self.histogram,self.name,self.atom = args[0]
1857        elif len(args) == 1 and ':' in args[0]:
1858            #parse a string
1859            lst = args[0].split(':')
1860            if lst[0] == '*':
1861                self.phase = '*'
1862                if len(lst) > 3:
1863                    self.atom = lst[3]
1864                self.histogram = HistIdLookup.get(lst[1],[None,None])[1]
1865            elif lst[1] == '*':
1866                self.histogram = '*'
1867                self.phase = PhaseIdLookup.get(lst[0],[None,None])[1]
1868            else:
1869                self.histogram = HistIdLookup.get(lst[1],[None,None])[1]
1870                self.phase = PhaseIdLookup.get(lst[0],[None,None])[1]
1871                if len(lst) == 4:
1872                    if lst[3] == '*':
1873                        self.atom = '*'
1874                    else:
1875                        self.atom = AtomIdLookup[lst[0]].get(lst[3],[None,None])[1]
1876                elif len(lst) == 5:
1877                    self.atom = lst[3]+":"+lst[4]
1878                elif len(lst) == 3:
1879                    pass
1880                else:
1881                    raise Exception("Too many colons in var name "+str(args[0]))
1882            self.name = lst[2]
1883        elif len(args) == 4:
1884            if args[0] == '*':
1885                self.phase = '*'
1886                self.atom = args[3]
1887            else:
1888                self.phase = PhaseIdLookup.get(str(args[0]),[None,None])[1]
1889                if args[3] == '*':
1890                    self.atom = '*'
1891                elif args[0] is not None:
1892                    self.atom = AtomIdLookup[args[0]].get(str(args[3]),[None,None])[1]
1893            if args[1] == '*':
1894                self.histogram = '*'
1895            else:
1896                self.histogram = HistIdLookup.get(str(args[1]),[None,None])[1]
1897            self.name = args[2]
1898        else:
1899            raise Exception("Incorrectly called GSAS-II parameter name")
1900
1901        #print "DEBUG: created ",self.phase,self.histogram,self.name,self.atom
1902
1903    def __str__(self):
1904        return self.varname()
1905
1906    def varname(self):
1907        '''Formats the GSAS-II variable name as a "traditional" GSAS-II variable
1908        string (p:h:<var>:a) or (p:h:<var>)
1909
1910        :returns: the variable name as a str
1911        '''
1912        a = ""
1913        if self.phase == "*":
1914            ph = "*"
1915            if self.atom:
1916                a = ":" + str(self.atom)
1917        else:
1918            ph = _lookup(PhaseRanIdLookup,self.phase)
1919            if self.atom == '*':
1920                a = ':*'
1921            elif self.atom:
1922                if ":" in str(self.atom):
1923                    a = ":" + str(self.atom)
1924                elif ph in AtomRanIdLookup:
1925                    a = ":" + AtomRanIdLookup[ph].get(self.atom,'?')
1926                else:
1927                    a = ":?"
1928        if self.histogram == "*":
1929            hist = "*"
1930        else:
1931            hist = _lookup(HistRanIdLookup,self.histogram)
1932        s = (ph + ":" + hist + ":" + str(self.name)) + a
1933        return s
1934
1935    def __repr__(self):
1936        '''Return the detailed contents of the object
1937        '''
1938        s = "<"
1939        if self.phase == '*':
1940            s += "Phases: all; "
1941            if self.atom is not None:
1942                if ":" in str(self.atom):
1943                    s += "Rigid body" + str(self.atom) + "; "
1944                else:
1945                    s += "Atom #" + str(self.atom) + "; "
1946        elif self.phase is not None:
1947            ph =  _lookup(PhaseRanIdLookup,self.phase)
1948            s += "Phase: rId=" + str(self.phase) + " (#"+ ph + "); "
1949            if self.atom == '*':
1950                s += "Atoms: all; "
1951            elif ":" in str(self.atom):
1952                s += "Rigid body" + str(self.atom) + "; "
1953            elif self.atom is not None:
1954                s += "Atom rId=" + str(self.atom)
1955                if ph in AtomRanIdLookup:
1956                    s += " (#" + AtomRanIdLookup[ph].get(self.atom,'?') + "); "
1957                else:
1958                    s += " (#? -- not found!); "
1959        if self.histogram == '*':
1960            s += "Histograms: all; "
1961        elif self.histogram is not None:
1962            hist = _lookup(HistRanIdLookup,self.histogram)
1963            s += "Histogram: rId=" + str(self.histogram) + " (#"+ hist + "); "
1964        s += 'Variable name="' + str(self.name) + '">'
1965        return s+" ("+self.varname()+")"
1966
1967    def __eq__(self, other):
1968        if type(other) is type(self):
1969            return (self.phase == other.phase and
1970                    self.histogram == other.histogram and
1971                    self.name == other.name and
1972                    self.atom == other.atom)
1973        return False
1974
1975    def _show(self):
1976        'For testing, shows the current lookup table'
1977        print ('phases'+ self.IDdict['phases'])
1978        print ('hists'+ self.IDdict['hists'])
1979        print ('atomDict'+ self.IDdict['atoms'])
1980
1981#==========================================================================
1982def SetDefaultSample():
1983    'Fills in default items for the Sample dictionary for Debye-Scherrer & SASD'
1984    return {
1985        'InstrName':'',
1986        'ranId':ran.randint(0,sys.maxsize),
1987        'Scale':[1.0,True],'Type':'Debye-Scherrer','Absorption':[0.0,False],
1988        'DisplaceX':[0.0,False],'DisplaceY':[0.0,False],
1989        'Temperature':300.,'Pressure':0.1,'Time':0.0,
1990        'FreePrm1':0.,'FreePrm2':0.,'FreePrm3':0.,
1991        'Gonio. radius':200.0,
1992        'Omega':0.0,'Chi':0.0,'Phi':0.0,'Azimuth':0.0,
1993#SASD items
1994        'Materials':[{'Name':'vacuum','VolFrac':1.0,},{'Name':'vacuum','VolFrac':0.0,}],
1995        'Thick':1.0,'Contrast':[0.0,0.0],       #contrast & anomalous contrast
1996        'Trans':1.0,                            #measured transmission
1997        'SlitLen':0.0,                          #Slit length - in Q(A-1)
1998        }
1999######################################################################
2000class ImportBaseclass(object):
2001    '''Defines a base class for the reading of input files (diffraction
2002    data, coordinates,...). See :ref:`Writing a Import Routine<import_routines>`
2003    for an explanation on how to use a subclass of this class.
2004    '''
2005    class ImportException(Exception):
2006        '''Defines an Exception that is used when an import routine hits an expected error,
2007        usually in .Reader.
2008
2009        Good practice is that the Reader should define a value in self.errors that
2010        tells the user some information about what is wrong with their file.
2011        '''
2012        pass
2013
2014    UseReader = True  # in __init__ set value of self.UseReader to False to skip use of current importer
2015    def __init__(self,formatName,longFormatName=None,
2016                 extensionlist=[],strictExtension=False,):
2017        self.formatName = formatName # short string naming file type
2018        if longFormatName: # longer string naming file type
2019            self.longFormatName = longFormatName
2020        else:
2021            self.longFormatName = formatName
2022        # define extensions that are allowed for the file type
2023        # for windows, remove any extensions that are duplicate, as case is ignored
2024        if sys.platform == 'windows' and extensionlist:
2025            extensionlist = list(set([s.lower() for s in extensionlist]))
2026        self.extensionlist = extensionlist
2027        # If strictExtension is True, the file will not be read, unless
2028        # the extension matches one in the extensionlist
2029        self.strictExtension = strictExtension
2030        self.errors = ''
2031        self.warnings = ''
2032        self.SciPy = False          #image reader needed scipy
2033        # used for readers that will use multiple passes to read
2034        # more than one data block
2035        self.repeat = False
2036        self.selections = []
2037        self.repeatcount = 0
2038        self.readfilename = '?'
2039        self.scriptable = False
2040        #print 'created',self.__class__
2041
2042    def ReInitialize(self):
2043        'Reinitialize the Reader to initial settings'
2044        self.errors = ''
2045        self.warnings = ''
2046        self.SciPy = False          #image reader needed scipy
2047        self.repeat = False
2048        self.repeatcount = 0
2049        self.readfilename = '?'
2050
2051
2052#    def Reader(self, filename, filepointer, ParentFrame=None, **unused):
2053#        '''This method must be supplied in the child class to read the file.
2054#        if the read fails either return False or raise an Exception
2055#        preferably of type ImportException.
2056#        '''
2057#        #start reading
2058#        raise ImportException("Error occurred while...")
2059#        self.errors += "Hint for user on why the error occur
2060#        return False # if an error occurs
2061#        return True # if read OK
2062
2063    def ExtensionValidator(self, filename):
2064        '''This methods checks if the file has the correct extension
2065       
2066        :returns:
2067       
2068          * False if this filename will not be supported by this reader (only
2069            when strictExtension is True)
2070          * True if the extension matches the list supplied by the reader
2071          * None if the reader allows un-registered extensions
2072         
2073        '''
2074        if filename:
2075            ext = ospath.splitext(filename)[1]
2076            if not ext and self.strictExtension: return False
2077            for ext in self.extensionlist:               
2078                if sys.platform == 'windows':
2079                    if filename.lower().endswith(ext): return True
2080                else:
2081                    if filename.endswith(ext): return True
2082        if self.strictExtension:
2083            return False
2084        else:
2085            return None
2086
2087    def ContentsValidator(self, filename):
2088        '''This routine will attempt to determine if the file can be read
2089        with the current format.
2090        This will typically be overridden with a method that
2091        takes a quick scan of [some of]
2092        the file contents to do a "sanity" check if the file
2093        appears to match the selected format.
2094        the file must be opened here with the correct format (binary/text)
2095        '''
2096        #filepointer.seek(0) # rewind the file pointer
2097        return True
2098
2099    def CIFValidator(self, filepointer):
2100        '''A :meth:`ContentsValidator` for use to validate CIF files.
2101        '''
2102        filepointer.seek(0)
2103        for i,l in enumerate(filepointer):
2104            if i >= 1000: return True
2105            '''Encountered only blank lines or comments in first 1000
2106            lines. This is unlikely, but assume it is CIF anyway, since we are
2107            even less likely to find a file with nothing but hashes and
2108            blank lines'''
2109            line = l.strip()
2110            if len(line) == 0: # ignore blank lines
2111                continue
2112            elif line.startswith('#'): # ignore comments
2113                continue
2114            elif line.startswith('data_'): # on the right track, accept this file
2115                return True
2116            else: # found something invalid
2117                self.errors = 'line '+str(i+1)+' contains unexpected data:\n'
2118                if all([ord(c) < 128 and ord(c) != 0 for c in str(l)]): # show only if ASCII
2119                    self.errors += '  '+str(l)
2120                else:
2121                    self.errors += '  (binary)'
2122                self.errors += '\n  Note: a CIF should only have blank lines or comments before'
2123                self.errors += '\n        a data_ statement begins a block.'
2124                return False
2125
2126######################################################################
2127class ImportPhase(ImportBaseclass):
2128    '''Defines a base class for the reading of files with coordinates
2129
2130    Objects constructed that subclass this (in import/G2phase_*.py etc.) will be used
2131    in :meth:`GSASIIdataGUI.GSASII.OnImportPhase` and in
2132    :func:`GSASIIscriptable.import_generic`.
2133    See :ref:`Writing a Import Routine<import_routines>`
2134    for an explanation on how to use this class.
2135
2136    '''
2137    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2138        strictExtension=False,):
2139        # call parent __init__
2140        ImportBaseclass.__init__(self,formatName,longFormatName,
2141            extensionlist,strictExtension)
2142        self.Phase = None # a phase must be created with G2IO.SetNewPhase in the Reader
2143        self.Constraints = None
2144
2145######################################################################
2146class ImportStructFactor(ImportBaseclass):
2147    '''Defines a base class for the reading of files with tables
2148    of structure factors.
2149
2150    Structure factors are read with a call to :meth:`GSASIIdataGUI.GSASII.OnImportSfact`
2151    which in turn calls :meth:`GSASIIdataGUI.GSASII.OnImportGeneric`, which calls
2152    methods :meth:`ExtensionValidator`, :meth:`ContentsValidator` and
2153    :meth:`Reader`.
2154
2155    See :ref:`Writing a Import Routine<import_routines>`
2156    for an explanation on how to use import classes in general. The specifics
2157    for reading a structure factor histogram require that
2158    the ``Reader()`` routine in the import
2159    class need to do only a few things: It
2160    should load :attr:`RefDict` item ``'RefList'`` with the reflection list,
2161    and set :attr:`Parameters` with the instrument parameters
2162    (initialized with :meth:`InitParameters` and set with :meth:`UpdateParameters`).
2163    '''
2164    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2165        strictExtension=False,):
2166        ImportBaseclass.__init__(self,formatName,longFormatName,
2167            extensionlist,strictExtension)
2168
2169        # define contents of Structure Factor entry
2170        self.Parameters = []
2171        'self.Parameters is a list with two dicts for data parameter settings'
2172        self.InitParameters()
2173        self.RefDict = {'RefList':[],'FF':{},'Super':0}
2174        self.Banks = []             #for multi bank data (usually TOF)
2175        '''self.RefDict is a dict containing the reflection information, as read from the file.
2176        Item 'RefList' contains the reflection information. See the
2177        :ref:`Single Crystal Reflection Data Structure<XtalRefl_table>`
2178        for the contents of each row. Dict element 'FF'
2179        contains the form factor values for each element type; if this entry
2180        is left as initialized (an empty list) it will be initialized as needed later.
2181        '''
2182    def ReInitialize(self):
2183        'Reinitialize the Reader to initial settings'
2184        ImportBaseclass.ReInitialize(self)
2185        self.InitParameters()
2186        self.Banks = []             #for multi bank data (usually TOF)
2187        self.RefDict = {'RefList':[],'FF':{},'Super':0}
2188
2189    def InitParameters(self):
2190        'initialize the instrument parameters structure'
2191        Lambda = 0.70926
2192        HistType = 'SXC'
2193        self.Parameters = [{'Type':[HistType,HistType], # create the structure
2194                            'Lam':[Lambda,Lambda]
2195                            }, {}]
2196        'Parameters is a list with two dicts for data parameter settings'
2197
2198    def UpdateParameters(self,Type=None,Wave=None):
2199        'Revise the instrument parameters'
2200        if Type is not None:
2201            self.Parameters[0]['Type'] = [Type,Type]
2202        if Wave is not None:
2203            self.Parameters[0]['Lam'] = [Wave,Wave]
2204
2205######################################################################
2206class ImportPowderData(ImportBaseclass):
2207    '''Defines a base class for the reading of files with powder data.
2208
2209    Objects constructed that subclass this (in import/G2pwd_*.py etc.) will be used
2210    in :meth:`GSASIIdataGUI.GSASII.OnImportPowder` and in
2211    :func:`GSASIIscriptable.import_generic`.
2212    See :ref:`Writing a Import Routine<import_routines>`
2213    for an explanation on how to use this class.
2214    '''
2215    def __init__(self,formatName,longFormatName=None,
2216        extensionlist=[],strictExtension=False,):
2217        ImportBaseclass.__init__(self,formatName,longFormatName,
2218            extensionlist,strictExtension)
2219        self.clockWd = None  # used in TOF
2220        self.ReInitialize()
2221
2222    def ReInitialize(self):
2223        'Reinitialize the Reader to initial settings'
2224        ImportBaseclass.ReInitialize(self)
2225        self.powderentry = ['',None,None] #  (filename,Pos,Bank)
2226        self.powderdata = [] # Powder dataset
2227        '''A powder data set is a list with items [x,y,w,yc,yb,yd]:
2228                np.array(x), # x-axis values
2229                np.array(y), # powder pattern intensities
2230                np.array(w), # 1/sig(intensity)^2 values (weights)
2231                np.array(yc), # calc. intensities (zero)
2232                np.array(yb), # calc. background (zero)
2233                np.array(yd), # obs-calc profiles
2234        '''
2235        self.comments = []
2236        self.idstring = ''
2237        self.Sample = SetDefaultSample() # default sample parameters
2238        self.Controls = {}  # items to be placed in top-level Controls
2239        self.GSAS = None     # used in TOF
2240        self.repeat_instparm = True # Should a parm file be
2241        #                             used for multiple histograms?
2242        self.instparm = None # name hint from file of instparm to use
2243        self.instfile = '' # full path name to instrument parameter file
2244        self.instbank = '' # inst parm bank number
2245        self.instmsg = ''  # a label that gets printed to show
2246                           # where instrument parameters are from
2247        self.numbanks = 1
2248        self.instdict = {} # place items here that will be transferred to the instrument parameters
2249        self.pwdparms = {} # place parameters that are transferred directly to the tree
2250                           # here (typically from an existing GPX file)
2251######################################################################
2252class ImportSmallAngleData(ImportBaseclass):
2253    '''Defines a base class for the reading of files with small angle data.
2254    See :ref:`Writing a Import Routine<import_routines>`
2255    for an explanation on how to use this class.
2256    '''
2257    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2258        strictExtension=False,):
2259
2260        ImportBaseclass.__init__(self,formatName,longFormatName,extensionlist,
2261            strictExtension)
2262        self.ReInitialize()
2263
2264    def ReInitialize(self):
2265        'Reinitialize the Reader to initial settings'
2266        ImportBaseclass.ReInitialize(self)
2267        self.smallangleentry = ['',None,None] #  (filename,Pos,Bank)
2268        self.smallangledata = [] # SASD dataset
2269        '''A small angle data set is a list with items [x,y,w,yc,yd]:
2270                np.array(x), # x-axis values
2271                np.array(y), # powder pattern intensities
2272                np.array(w), # 1/sig(intensity)^2 values (weights)
2273                np.array(yc), # calc. intensities (zero)
2274                np.array(yd), # obs-calc profiles
2275                np.array(yb), # preset bkg
2276        '''
2277        self.comments = []
2278        self.idstring = ''
2279        self.Sample = SetDefaultSample()
2280        self.GSAS = None     # used in TOF
2281        self.clockWd = None  # used in TOF
2282        self.numbanks = 1
2283        self.instdict = {} # place items here that will be transferred to the instrument parameters
2284
2285######################################################################
2286class ImportReflectometryData(ImportBaseclass):
2287    '''Defines a base class for the reading of files with reflectometry data.
2288    See :ref:`Writing a Import Routine<import_routines>`
2289    for an explanation on how to use this class.
2290    '''
2291    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2292        strictExtension=False,):
2293
2294        ImportBaseclass.__init__(self,formatName,longFormatName,extensionlist,
2295            strictExtension)
2296        self.ReInitialize()
2297
2298    def ReInitialize(self):
2299        'Reinitialize the Reader to initial settings'
2300        ImportBaseclass.ReInitialize(self)
2301        self.reflectometryentry = ['',None,None] #  (filename,Pos,Bank)
2302        self.reflectometrydata = [] # SASD dataset
2303        '''A small angle data set is a list with items [x,y,w,yc,yd]:
2304                np.array(x), # x-axis values
2305                np.array(y), # powder pattern intensities
2306                np.array(w), # 1/sig(intensity)^2 values (weights)
2307                np.array(yc), # calc. intensities (zero)
2308                np.array(yd), # obs-calc profiles
2309                np.array(yb), # preset bkg
2310        '''
2311        self.comments = []
2312        self.idstring = ''
2313        self.Sample = SetDefaultSample()
2314        self.GSAS = None     # used in TOF
2315        self.clockWd = None  # used in TOF
2316        self.numbanks = 1
2317        self.instdict = {} # place items here that will be transferred to the instrument parameters
2318
2319######################################################################
2320class ImportPDFData(ImportBaseclass):
2321    '''Defines a base class for the reading of files with PDF G(R) data.
2322    See :ref:`Writing a Import Routine<import_routines>`
2323    for an explanation on how to use this class.
2324    '''
2325    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2326        strictExtension=False,):
2327
2328        ImportBaseclass.__init__(self,formatName,longFormatName,extensionlist,
2329            strictExtension)
2330        self.ReInitialize()
2331
2332    def ReInitialize(self):
2333        'Reinitialize the Reader to initial settings'
2334        ImportBaseclass.ReInitialize(self)
2335        self.pdfentry = ['',None,None] #  (filename,Pos,Bank)
2336        self.pdfdata = [] # PDF G(R) dataset
2337        '''A pdf g(r) data set is a list with items [x,y]:
2338                np.array(x), # r-axis values
2339                np.array(y), # pdf g(r)
2340        '''
2341        self.comments = []
2342        self.idstring = ''
2343        self.numbanks = 1
2344
2345######################################################################
2346class ImportImage(ImportBaseclass):
2347    '''Defines a base class for the reading of images
2348
2349    Images are read in only these places:
2350
2351      * Initial reading is typically done from a menu item
2352        with a call to :meth:`GSASIIdataGUI.GSASII.OnImportImage`
2353        which in turn calls :meth:`GSASIIdataGUI.GSASII.OnImportGeneric`. That calls
2354        methods :meth:`ExtensionValidator`, :meth:`ContentsValidator` and
2355        :meth:`Reader`. This returns a list of reader objects for each read image.
2356        Also used in :func:`GSASIIscriptable.import_generic`.
2357
2358      * Images are read alternatively in :func:`GSASIIIO.ReadImages`, which puts image info
2359        directly into the data tree.
2360
2361      * Images are reloaded with :func:`GSASIIIO.GetImageData`.
2362
2363    When reading an image, the ``Reader()`` routine in the ImportImage class
2364    should set:
2365
2366      * :attr:`Comments`: a list of strings (str),
2367      * :attr:`Npix`: the number of pixels in the image (int),
2368      * :attr:`Image`: the actual image as a numpy array (np.array)
2369      * :attr:`Data`: a dict defining image parameters (dict). Within this dict the following
2370        data items are needed:
2371
2372         * 'pixelSize': size of each pixel in microns (such as ``[200.,200.]``.
2373         * 'wavelength': wavelength in :math:`\\AA`.
2374         * 'distance': distance of detector from sample in cm.
2375         * 'center': uncalibrated center of beam on detector (such as ``[204.8,204.8]``.
2376         * 'size': size of image (such as ``[2048,2048]``).
2377         * 'ImageTag': image number or other keyword used to retrieve image from
2378           a multi-image data file (defaults to ``1`` if not specified).
2379         * 'sumfile': holds sum image file name if a sum was produced from a multi image file
2380
2381    optional data items:
2382
2383      * :attr:`repeat`: set to True if there are additional images to
2384        read in the file, False otherwise
2385      * :attr:`repeatcount`: set to the number of the image.
2386
2387    Note that the above is initialized with :meth:`InitParameters`.
2388    (Also see :ref:`Writing a Import Routine<import_routines>`
2389    for an explanation on how to use import classes in general.)
2390    '''
2391    def __init__(self,formatName,longFormatName=None,extensionlist=[],
2392        strictExtension=False,):
2393        ImportBaseclass.__init__(self,formatName,longFormatName,
2394            extensionlist,strictExtension)
2395        self.InitParameters()
2396
2397    def ReInitialize(self):
2398        'Reinitialize the Reader to initial settings -- not used at present'
2399        ImportBaseclass.ReInitialize(self)
2400        self.InitParameters()
2401
2402    def InitParameters(self):
2403        'initialize the instrument parameters structure'
2404        self.Comments = ['No comments']
2405        self.Data = {}
2406        self.Npix = 0
2407        self.Image = None
2408        self.repeat = False
2409        self.repeatcount = 1
2410        self.sumfile = ''
2411
2412    def LoadImage(self,ParentFrame,imagefile,imagetag=None):
2413        '''Optionally, call this after reading in an image to load it into the tree.
2414        This saves time by preventing a reread of the same information.
2415        '''
2416        if ParentFrame:
2417            ParentFrame.ImageZ = self.Image   # store the image for plotting
2418            ParentFrame.oldImagefile = imagefile # save the name of the last image file read
2419            ParentFrame.oldImageTag = imagetag   # save the tag of the last image file read
2420
2421#################################################################################################
2422# shortcut routines
2423exp = np.exp
2424sind = sin = s = lambda x: np.sin(x*np.pi/180.)
2425cosd = cos = c = lambda x: np.cos(x*np.pi/180.)
2426tand = tan = t = lambda x: np.tan(x*np.pi/180.)
2427sqrt = sq = lambda x: np.sqrt(x)
2428pi = lambda: np.pi
2429class ExpressionObj(object):
2430    '''Defines an object with a user-defined expression, to be used for
2431    secondary fits or restraints. Object is created null, but is changed
2432    using :meth:`LoadExpression`. This contains only the minimum
2433    information that needs to be stored to save and load the expression
2434    and how it is mapped to GSAS-II variables.
2435    '''
2436    def __init__(self):
2437        self.expression = ''
2438        'The expression as a text string'
2439        self.assgnVars = {}
2440        '''A dict where keys are label names in the expression mapping to a GSAS-II
2441        variable. The value a G2 variable name.
2442        Note that the G2 variable name may contain a wild-card and correspond to
2443        multiple values.
2444        '''
2445        self.freeVars = {}
2446        '''A dict where keys are label names in the expression mapping to a free
2447        parameter. The value is a list with:
2448
2449         * a name assigned to the parameter
2450         * a value for to the parameter and
2451         * a flag to determine if the variable is refined.
2452        '''
2453        self.depVar = None
2454
2455        self.lastError = ('','')
2456        '''Shows last encountered error in processing expression
2457        (list of 1-3 str values)'''
2458
2459        self.distance_dict  = None  # to be used for defining atom phase/symmetry info
2460        self.distance_atoms = None  # to be used for defining atom distances
2461
2462    def LoadExpression(self,expr,exprVarLst,varSelect,varName,varValue,varRefflag):
2463        '''Load the expression and associated settings into the object. Raises
2464        an exception if the expression is not parsed, if not all functions
2465        are defined or if not all needed parameter labels in the expression
2466        are defined.
2467
2468        This will not test if the variable referenced in these definitions
2469        are actually in the parameter dictionary. This is checked when the
2470        computation for the expression is done in :meth:`SetupCalc`.
2471
2472        :param str expr: the expression
2473        :param list exprVarLst: parameter labels found in the expression
2474        :param dict varSelect: this will be 0 for Free parameters
2475          and non-zero for expression labels linked to G2 variables.
2476        :param dict varName: Defines a name (str) associated with each free parameter
2477        :param dict varValue: Defines a value (float) associated with each free parameter
2478        :param dict varRefflag: Defines a refinement flag (bool)
2479          associated with each free parameter
2480        '''
2481        self.expression = expr
2482        self.compiledExpr = None
2483        self.freeVars = {}
2484        self.assgnVars = {}
2485        for v in exprVarLst:
2486            if varSelect[v] == 0:
2487                self.freeVars[v] = [
2488                    varName.get(v),
2489                    varValue.get(v),
2490                    varRefflag.get(v),
2491                    ]
2492            else:
2493                self.assgnVars[v] = varName[v]
2494        self.CheckVars()
2495
2496    def EditExpression(self,exprVarLst,varSelect,varName,varValue,varRefflag):
2497        '''Load the expression and associated settings from the object into
2498        arrays used for editing.
2499
2500        :param list exprVarLst: parameter labels found in the expression
2501        :param dict varSelect: this will be 0 for Free parameters
2502          and non-zero for expression labels linked to G2 variables.
2503        :param dict varName: Defines a name (str) associated with each free parameter
2504        :param dict varValue: Defines a value (float) associated with each free parameter
2505        :param dict varRefflag: Defines a refinement flag (bool)
2506          associated with each free parameter
2507
2508        :returns: the expression as a str
2509        '''
2510        for v in self.freeVars:
2511            varSelect[v] = 0
2512            varName[v] = self.freeVars[v][0]
2513            varValue[v] = self.freeVars[v][1]
2514            varRefflag[v] = self.freeVars[v][2]
2515        for v in self.assgnVars:
2516            varSelect[v] = 1
2517            varName[v] = self.assgnVars[v]
2518        return self.expression
2519
2520    def GetVaried(self):
2521        'Returns the names of the free parameters that will be refined'
2522        return ["::"+self.freeVars[v][0] for v in self.freeVars if self.freeVars[v][2]]
2523
2524    def GetVariedVarVal(self):
2525        'Returns the names and values of the free parameters that will be refined'
2526        return [("::"+self.freeVars[v][0],self.freeVars[v][1]) for v in self.freeVars if self.freeVars[v][2]]
2527
2528    def UpdateVariedVars(self,varyList,values):
2529        'Updates values for the free parameters (after a refinement); only updates refined vars'
2530        for v in self.freeVars:
2531            if not self.freeVars[v][2]: continue
2532            if "::"+self.freeVars[v][0] not in varyList: continue
2533            indx = list(varyList).index("::"+self.freeVars[v][0])
2534            self.freeVars[v][1] = values[indx]
2535
2536    def GetIndependentVars(self):
2537        'Returns the names of the required independent parameters used in expression'
2538        return [self.assgnVars[v] for v in self.assgnVars]
2539
2540    def CheckVars(self):
2541        '''Check that the expression can be parsed, all functions are
2542        defined and that input loaded into the object is internally
2543        consistent. If not an Exception is raised.
2544
2545        :returns: a dict with references to packages needed to
2546          find functions referenced in the expression.
2547        '''
2548        ret = self.ParseExpression(self.expression)
2549        if not ret:
2550            raise Exception("Expression parse error")
2551        exprLblList,fxnpkgdict = ret
2552        # check each var used in expression is defined
2553        defined = list(self.assgnVars.keys()) + list(self.freeVars.keys())
2554        notfound = []
2555        for var in exprLblList:
2556            if var not in defined:
2557                notfound.append(var)
2558        if notfound:
2559            msg = 'Not all variables defined'
2560            msg1 = 'The following variables were not defined: '
2561            msg2 = ''
2562            for var in notfound:
2563                if msg: msg += ', '
2564                msg += var
2565            self.lastError = (msg1,'  '+msg2)
2566            raise Exception(msg)
2567        return fxnpkgdict
2568
2569    def ParseExpression(self,expr):
2570        '''Parse an expression and return a dict of called functions and
2571        the variables used in the expression. Returns None in case an error
2572        is encountered. If packages are referenced in functions, they are loaded
2573        and the functions are looked up into the modules global
2574        workspace.
2575
2576        Note that no changes are made to the object other than
2577        saving an error message, so that this can be used for testing prior
2578        to the save.
2579
2580        :returns: a list of used variables
2581        '''
2582        self.lastError = ('','')
2583        import ast
2584        def FindFunction(f):
2585            '''Find the object corresponding to function f
2586            :param str f: a function name such as 'numpy.exp'
2587            :returns: (pkgdict,pkgobj) where pkgdict contains a dict
2588              that defines the package location(s) and where pkgobj
2589              defines the object associated with the function.
2590              If the function is not found, pkgobj is None.
2591            '''
2592            df = f.split('.')
2593            pkgdict = {}
2594            # no listed package, try in current namespace
2595            if len(df) == 1:
2596                try:
2597                    fxnobj = eval(f)
2598                    return pkgdict,fxnobj
2599                except (AttributeError, NameError):
2600                    return None,None
2601            else:
2602                try:
2603                    fxnobj = eval(f)
2604                    pkgdict[df[0]] = eval(df[0])
2605                    return pkgdict,fxnobj
2606                except (AttributeError, NameError):
2607                    pass
2608            # includes a package, lets try to load the packages
2609            pkgname = ''
2610            path = sys.path+['./',]
2611            for pkg in f.split('.')[:-1]: # if needed, descend down the tree
2612                if pkgname:
2613                    pkgname += '.' + pkg
2614                else:
2615                    pkgname = pkg
2616                fp = None
2617                try:
2618                    fp, fppath,desc = imp.find_module(pkg,path)
2619                    pkgobj = imp.load_module(pkg,fp,fppath,desc)
2620                    pkgdict[pkgname] = pkgobj
2621                    path = [fppath]
2622                except Exception as msg:
2623                    print('load of '+pkgname+' failed with error='+str(msg))
2624                    return {},None
2625                finally:
2626                    if fp: fp.close()
2627                try:
2628                    #print 'before',pkgdict.keys()
2629                    fxnobj = eval(f,globals(),pkgdict)
2630                    #print 'after 1',pkgdict.keys()
2631                    #fxnobj = eval(f,pkgdict)
2632                    #print 'after 2',pkgdict.keys()
2633                    return pkgdict,fxnobj
2634                except:
2635                    continue
2636            return None # not found
2637        def ASTtransverse(node,fxn=False):
2638            '''Transverse a AST-parsed expresson, compiling a list of variables
2639            referenced in the expression. This routine is used recursively.
2640
2641            :returns: varlist,fxnlist where
2642              varlist is a list of referenced variable names and
2643              fxnlist is a list of used functions
2644            '''
2645            varlist = []
2646            fxnlist = []
2647            if isinstance(node, list):
2648                for b in node:
2649                    v,f = ASTtransverse(b,fxn)
2650                    varlist += v
2651                    fxnlist += f
2652            elif isinstance(node, ast.AST):
2653                for a, b in ast.iter_fields(node):
2654                    if isinstance(b, ast.AST):
2655                        if a == 'func':
2656                            fxnlist += ['.'.join(ASTtransverse(b,True)[0])]
2657                            continue
2658                        v,f = ASTtransverse(b,fxn)
2659                        varlist += v
2660                        fxnlist += f
2661                    elif isinstance(b, list):
2662                        v,f = ASTtransverse(b,fxn)
2663                        varlist += v
2664                        fxnlist += f
2665                    elif node.__class__.__name__ == "Name":
2666                        varlist += [b]
2667                    elif fxn and node.__class__.__name__ == "Attribute":
2668                        varlist += [b]
2669            return varlist,fxnlist
2670        try:
2671            exprast = ast.parse(expr)
2672        except SyntaxError:
2673            s = ''
2674            import traceback
2675            for i in traceback.format_exc().splitlines()[-3:-1]:
2676                if s: s += "\n"
2677                s += str(i)
2678            self.lastError = ("Error parsing expression:",s)
2679            return
2680        # find the variables & functions
2681        v,f = ASTtransverse(exprast)
2682        varlist = sorted(list(set(v)))
2683        fxnlist = list(set(f))
2684        pkgdict = {}
2685        # check the functions are defined
2686        for fxn in fxnlist:
2687            fxndict,fxnobj = FindFunction(fxn)
2688            if not fxnobj:
2689                self.lastError = ("Error: Invalid function",fxn,
2690                                  "is not defined")
2691                return
2692            if not hasattr(fxnobj,'__call__'):
2693                self.lastError = ("Error: Not a function.",fxn,
2694                                  "cannot be called as a function")
2695                return
2696            pkgdict.update(fxndict)
2697        return varlist,pkgdict
2698
2699    def GetDepVar(self):
2700        'return the dependent variable, or None'
2701        return self.depVar
2702
2703    def SetDepVar(self,var):
2704        'Set the dependent variable, if used'
2705        self.depVar = var
2706#==========================================================================
2707class ExpressionCalcObj(object):
2708    '''An object used to evaluate an expression from a :class:`ExpressionObj`
2709    object.
2710
2711    :param ExpressionObj exprObj: a :class:`~ExpressionObj` expression object with
2712      an expression string and mappings for the parameter labels in that object.
2713    '''
2714    def __init__(self,exprObj):
2715        self.eObj = exprObj
2716        'The expression and mappings; a :class:`ExpressionObj` object'
2717        self.compiledExpr = None
2718        'The expression as compiled byte-code'
2719        self.exprDict = {}
2720        '''dict that defines values for labels used in expression and packages
2721        referenced by functions
2722        '''
2723        self.lblLookup = {}
2724        '''Lookup table that specifies the expression label name that is
2725        tied to a particular GSAS-II parameters in the parmDict.
2726        '''
2727        self.fxnpkgdict = {}
2728        '''a dict with references to packages needed to
2729        find functions referenced in the expression.
2730        '''
2731        self.varLookup = {}
2732        '''Lookup table that specifies the GSAS-II variable(s)
2733        indexed by the expression label name. (Used for only for diagnostics
2734        not evaluation of expression.)
2735        '''
2736        self.su = None
2737        '''Standard error evaluation where supplied by the evaluator
2738        '''
2739        # Patch: for old-style expressions with a (now removed step size)
2740        if '2' in platform.python_version_tuple()[0]: 
2741            basestr = basestring
2742        else:
2743            basestr = str
2744        for v in self.eObj.assgnVars:
2745            if not isinstance(self.eObj.assgnVars[v], basestr):
2746                self.eObj.assgnVars[v] = self.eObj.assgnVars[v][0]
2747        self.parmDict = {}
2748        '''A copy of the parameter dictionary, for distance and angle computation
2749        '''
2750
2751    def SetupCalc(self,parmDict):
2752        '''Do all preparations to use the expression for computation.
2753        Adds the free parameter values to the parameter dict (parmDict).
2754        '''
2755        if self.eObj.expression.startswith('Dist') or self.eObj.expression.startswith('Angle'):
2756            return
2757        self.fxnpkgdict = self.eObj.CheckVars()
2758        # all is OK, compile the expression
2759        self.compiledExpr = compile(self.eObj.expression,'','eval')
2760
2761        # look at first value in parmDict to determine its type
2762        parmsInList = True
2763        if '2' in platform.python_version_tuple()[0]: 
2764            basestr = basestring
2765        else:
2766            basestr = str
2767        for key in parmDict:
2768            val = parmDict[key]
2769            if isinstance(val, basestr):
2770                parmsInList = False
2771                break
2772            try: # check if values are in lists
2773                val = parmDict[key][0]
2774            except (TypeError,IndexError):
2775                parmsInList = False
2776            break
2777
2778        # set up the dicts needed to speed computations
2779        self.exprDict = {}
2780        self.lblLookup = {}
2781        self.varLookup = {}
2782        for v in self.eObj.freeVars:
2783            varname = self.eObj.freeVars[v][0]
2784            varname = "::" + varname.lstrip(':').replace(' ','_').replace(':',';')
2785            self.lblLookup[varname] = v
2786            self.varLookup[v] = varname
2787            if parmsInList:
2788                parmDict[varname] = [self.eObj.freeVars[v][1],self.eObj.freeVars[v][2]]
2789            else:
2790                parmDict[varname] = self.eObj.freeVars[v][1]
2791            self.exprDict[v] = self.eObj.freeVars[v][1]
2792        for v in self.eObj.assgnVars:
2793            varname = self.eObj.assgnVars[v]
2794            if varname in parmDict:
2795                self.lblLookup[varname] = v
2796                self.varLookup[v] = varname
2797                if parmsInList:
2798                    self.exprDict[v] = parmDict[varname][0]
2799                else:
2800                    self.exprDict[v] = parmDict[varname]
2801            elif '*' in varname:
2802                varlist = LookupWildCard(varname,list(parmDict.keys()))
2803                if len(varlist) == 0:
2804                    raise Exception("No variables match "+str(v))
2805                for var in varlist:
2806                    self.lblLookup[var] = v
2807                if parmsInList:
2808                    self.exprDict[v] = np.array([parmDict[var][0] for var in varlist])
2809                else:
2810                    self.exprDict[v] = np.array([parmDict[var] for var in varlist])
2811                self.varLookup[v] = [var for var in varlist]
2812            else:
2813                self.exprDict[v] = None
2814#                raise Exception,"No value for variable "+str(v)
2815        self.exprDict.update(self.fxnpkgdict)
2816
2817    def UpdateVars(self,varList,valList):
2818        '''Update the dict for the expression with a set of values
2819        :param list varList: a list of variable names
2820        :param list valList: a list of corresponding values
2821        '''
2822        for var,val in zip(varList,valList):
2823            self.exprDict[self.lblLookup.get(var,'undefined: '+var)] = val
2824
2825    def UpdateDict(self,parmDict):
2826        '''Update the dict for the expression with values in a dict
2827        :param dict parmDict: a dict of values, items not in use are ignored
2828        '''
2829        if self.eObj.expression.startswith('Dist') or self.eObj.expression.startswith('Angle'):
2830            self.parmDict = parmDict
2831            return
2832        for var in parmDict:
2833            if var in self.lblLookup:
2834                self.exprDict[self.lblLookup[var]] = parmDict[var]
2835
2836    def EvalExpression(self):
2837        '''Evaluate an expression. Note that the expression
2838        and mapping are taken from the :class:`ExpressionObj` expression object
2839        and the parameter values were specified in :meth:`SetupCalc`.
2840        :returns: a single value for the expression. If parameter
2841        values are arrays (for example, from wild-carded variable names),
2842        the sum of the resulting expression is returned.
2843
2844        For example, if the expression is ``'A*B'``,
2845        where A is 2.0 and B maps to ``'1::Afrac:*'``, which evaluates to::
2846
2847        [0.5, 1, 0.5]
2848
2849        then the result will be ``4.0``.
2850        '''
2851        self.su = None
2852        if self.eObj.expression.startswith('Dist'):
2853#            GSASIIpath.IPyBreak()
2854            dist = G2mth.CalcDist(self.eObj.distance_dict, self.eObj.distance_atoms, self.parmDict)
2855            return dist
2856        elif self.eObj.expression.startswith('Angle'):
2857            angle = G2mth.CalcAngle(self.eObj.angle_dict, self.eObj.angle_atoms, self.parmDict)
2858            return angle
2859        if self.compiledExpr is None:
2860            raise Exception("EvalExpression called before SetupCalc")
2861        try:
2862            val = eval(self.compiledExpr,globals(),self.exprDict)
2863        except TypeError:
2864            val = None
2865        if not np.isscalar(val):
2866            val = np.sum(val)
2867        return val
2868
2869class G2Exception(Exception):
2870    'A generic GSAS-II exception class'
2871    def __init__(self,msg):
2872        self.msg = msg
2873    def __str__(self):
2874        return repr(self.msg)
2875
2876class G2RefineCancel(Exception):
2877    'Raised when Cancel is pressed in a refinement dialog'
2878    def __init__(self,msg):
2879        self.msg = msg
2880    def __str__(self):
2881        return repr(self.msg)
2882   
2883def HowDidIgetHere(wherecalledonly=False):
2884    '''Show a traceback with calls that brought us to the current location.
2885    Used for debugging.
2886    '''
2887    import traceback
2888    if wherecalledonly:
2889        i = traceback.format_list(traceback.extract_stack()[:-1])[-2]
2890        print(i.strip().rstrip())
2891    else:
2892        print (70*'*')
2893        for i in traceback.format_list(traceback.extract_stack()[:-1]): print(i.strip().rstrip())
2894        print (70*'*')
2895
2896# Note that this is GUI code and should be moved at somepoint
2897def CreatePDFitems(G2frame,PWDRtree,ElList,Qlimits,numAtm=1,FltBkg=0,PDFnames=[]):
2898    '''Create and initialize a new set of PDF tree entries
2899
2900    :param Frame G2frame: main GSAS-II tree frame object
2901    :param str PWDRtree: name of PWDR to be used to create PDF item
2902    :param dict ElList: data structure with composition
2903    :param list Qlimits: Q limits to be used for computing the PDF
2904    :param float numAtm: no. atom in chemical formula
2905    :param float FltBkg: flat background value
2906    :param list PDFnames: previously used PDF names
2907
2908    :returns: the Id of the newly created PDF entry
2909    '''
2910    PDFname = 'PDF '+PWDRtree[4:] # this places two spaces after PDF, which is needed is some places
2911    if PDFname in PDFnames:
2912        print('Skipping, entry already exists: '+PDFname)
2913        return None
2914    #PDFname = MakeUniqueLabel(PDFname,PDFnames)
2915    Id = G2frame.GPXtree.AppendItem(parent=G2frame.root,text=PDFname)
2916    Data = {
2917        'Sample':{'Name':PWDRtree,'Mult':1.0},
2918        'Sample Bkg.':{'Name':'','Mult':-1.0,'Refine':False},
2919        'Container':{'Name':'','Mult':-1.0,'Refine':False},
2920        'Container Bkg.':{'Name':'','Mult':-1.0},'ElList':ElList,
2921        'Geometry':'Cylinder','Diam':1.0,'Pack':0.50,'Form Vol':10.0*numAtm,'Flat Bkg':FltBkg,
2922        'DetType':'Area detector','ObliqCoeff':0.2,'Ruland':0.025,'QScaleLim':Qlimits,
2923        'Lorch':False,'BackRatio':0.0,'Rmax':100.,'noRing':False,'IofQmin':1.0,'Rmin':1.0,
2924        'I(Q)':[],'S(Q)':[],'F(Q)':[],'G(R)':[]}
2925    G2frame.GPXtree.SetItemPyData(G2frame.GPXtree.AppendItem(Id,text='PDF Controls'),Data)
2926    G2frame.GPXtree.SetItemPyData(G2frame.GPXtree.AppendItem(Id,text='PDF Peaks'),
2927        {'Limits':[1.,5.],'Background':[2,[0.,-0.2*np.pi],False],'Peaks':[]})
2928    return Id
2929#%%
2930class ShowTiming(object):
2931    '''An object to use for timing repeated sections of code.
2932
2933    Create the object with::
2934       tim0 = ShowTiming()
2935
2936    Tag sections of code to be timed with::
2937       tim0.start('start')
2938       tim0.start('in section 1')
2939       tim0.start('in section 2')
2940       
2941    etc. (Note that each section should have a unique label.)
2942
2943    After the last section, end timing with::
2944       tim0.end()
2945
2946    Show timing results with::
2947       tim0.show()
2948       
2949    '''
2950    def __init__(self):
2951        self.timeSum =  []
2952        self.timeStart = []
2953        self.label = []
2954        self.prev = None
2955    def start(self,label):
2956        import time
2957        if label in self.label:
2958            i = self.label.index(label)
2959            self.timeStart[i] = time.time()
2960        else:
2961            i = len(self.label)
2962            self.timeSum.append(0.0)
2963            self.timeStart.append(time.time())
2964            self.label.append(label)
2965        if self.prev is not None:
2966            self.timeSum[self.prev] += self.timeStart[i] - self.timeStart[self.prev]
2967        self.prev = i
2968    def end(self):
2969        import time
2970        if self.prev is not None:
2971            self.timeSum[self.prev] += time.time() - self.timeStart[self.prev]
2972        self.prev = None
2973    def show(self):
2974        sumT = sum(self.timeSum)
2975        print('Timing results (total={:.2f} sec)'.format(sumT))
2976        for i,(lbl,val) in enumerate(zip(self.label,self.timeSum)):
2977            print('{} {:20} {:8.2f} ms {:5.2f}%'.format(i,lbl,1000.*val,100*val/sumT))
2978#%%
2979
2980if __name__ == "__main__":
2981    # test variable descriptions
2982    for var in '0::Afrac:*',':1:Scale','1::dAx:0','::undefined':
2983        v = var.split(':')[2]
2984        print(var+':\t', getDescr(v),getVarStep(v))
2985    import sys; sys.exit()
2986    # test equation evaluation
2987    def showEQ(calcobj):
2988        print (50*'=')
2989        print (calcobj.eObj.expression+'='+calcobj.EvalExpression())
2990        for v in sorted(calcobj.varLookup):
2991            print ("  "+v+'='+calcobj.exprDict[v]+'='+calcobj.varLookup[v])
2992        # print '  Derivatives'
2993        # for v in calcobj.derivStep.keys():
2994        #     print '    d(Expr)/d('+v+') =',calcobj.EvalDeriv(v)
2995
2996    obj = ExpressionObj()
2997
2998    obj.expression = "A*np.exp(B)"
2999    obj.assgnVars =  {'B': '0::Afrac:1'}
3000    obj.freeVars =  {'A': [u'A', 0.5, True]}
3001    #obj.CheckVars()
3002    calcobj = ExpressionCalcObj(obj)
3003
3004    obj1 = ExpressionObj()
3005    obj1.expression = "A*np.exp(B)"
3006    obj1.assgnVars =  {'B': '0::Afrac:*'}
3007    obj1.freeVars =  {'A': [u'Free Prm A', 0.5, True]}
3008    #obj.CheckVars()
3009    calcobj1 = ExpressionCalcObj(obj1)
3010
3011    obj2 = ExpressionObj()
3012    obj2.distance_stuff = np.array([[0,1],[1,-1]])
3013    obj2.expression = "Dist(1,2)"
3014    GSASIIpath.InvokeDebugOpts()
3015    parmDict2 = {'0::Afrac:0':[0.0,True], '0::Afrac:1': [1.0,False]}
3016    calcobj2 = ExpressionCalcObj(obj2)
3017    calcobj2.SetupCalc(parmDict2)
3018    showEQ(calcobj2)
3019
3020    parmDict1 = {'0::Afrac:0':1.0, '0::Afrac:1': 1.0}
3021    print ('\nDict = '+parmDict1)
3022    calcobj.SetupCalc(parmDict1)
3023    showEQ(calcobj)
3024    calcobj1.SetupCalc(parmDict1)
3025    showEQ(calcobj1)
3026
3027    parmDict2 = {'0::Afrac:0':[0.0,True], '0::Afrac:1': [1.0,False]}
3028    print ('Dict = '+parmDict2)
3029    calcobj.SetupCalc(parmDict2)
3030    showEQ(calcobj)
3031    calcobj1.SetupCalc(parmDict2)
3032    showEQ(calcobj1)
3033    calcobj2.SetupCalc(parmDict2)
3034    showEQ(calcobj2)
Note: See TracBrowser for help on using the repository browser.