Child pages
  • Meeting with Sector 33 & 34

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


Participants John Hammonds, Christian Schleputz, Nicholas Schwarz, Jon Tischler

Met to discuss status of RSM at sector 33, progress so far for sector 34 energy scans, discuss XSD priorities from Division Management

Discussion of RSM @ 33 somewhat spurred on by comments from Tischler to Schwarz at SAG mtg that it was hard to use.

Jon's comments seem to come from a feeling that "Users need Christian" to work through RSM process.  Some of this is in using the software and some of this is in trying to figure out what data to take.  Some of this will come from improving documentation.  Currently the originals of documentation are in Microsoft Word format.  Chris has suggested moving this to restructured text to allow for this to be more easily placed into version control.  This will allow him to make modifications as he sees changes that may make things easier for users.  Will also allow keeping the documents more up to date.  May need to beef up documentation on processing raw data and also provide some documentation of how to take data.

On the code side of the documentation, need better documentation of inputs and outputs to allow for better scripting.

There are some issues with Spec. - New sample does not work/does not allow entering enough information on what going on.  Seems to be related to some changes by Chris?  Use of TIFF files seems to have issues, too many files, how to find the files, the current method of collection gives long filenames since other info is not stored in spec file.  Need to have understanding of the directory structure.  BCDA should be involved in some of the spec issues.  Make sure that things at least become somewhat universal.

Paraview issues - Need to figure out how to pick points, measure distances to get real numbers,

                           - Need to be able to fit peaks in 3D.

                          - provide measurement tools, etc and document.

                         - Blob, tell morphology, line profiles & slices.

XSD priorities

Fly scan at sector 2.  Chris is also interested in fly scans so he and Tim Mooney should discuss this to see how current work could help sect. 33.

Hammonds will be moving to work on detector for 2ID in October

Sect 34 is getting 2 new Perkin Elmer detectors.  Need this on list for prioritization.

        Who will set up, Nicholas to talk with BCDA.

Christian has been using Matt Newville's PV Archiver.  Needs some work.  Can we add to it?  Are there other options.

Sect 34 Energy scan update

John H demonstrated work so far to integrate 34's energy scans into the program.  Have made progress in calculating qs, reading images and pushing this through the same gridder as sector 33.

  •       Need to look at computing grid size from the data
  • Still need to get HKL axes.  This will require additional input from the sample.  Not clear at what stage this should be added.
  • Need to add line showing direction to origin





Participants: John Hammonds, Christian Schleputz

Christian's focus in the coming weeks is to get people using this method and therefore the application. He has a number of users planned focusing on this in the near term. Things are working well. He will be working on sending people home using the application. I let him know to call on me if any help is needed on this end. We can use the input to smooth the process for future users.
One limit Christian sees is the memory issue on machines with 8GB or less. Christian wants me to look into a method of limiting how much data is processed at one time. Some data will be powder diffraction data with output of 10s of KB. The input can be 5000 images in one scan. This pushes the limits on smaller machines. It should work just as well to process the data in smaller chunks. Perhaps create an application config file which will hold maximum amount of data to process at one time & then chunk the data accordingly.

We still need to figure out more about how xray utilities can be used to properly mask data. Need to look at the code & possibly communicate with developers of xrayutilities.

Chris is using xrayutilities 1D gridder for powder diffraction data. This does not provide feedback on the # of points to use for averaging. This information is needed to provide error values to reitveld refinement.

Should see about getting OpenMP working.

Chris had some problems trying to install with a user last cycle with Windows 8.


Participants: John Hammonds, Christian Schleputz

John went down to talk with Christian about findings on the six circle data. After fixing the code to be more general with the number of angles, the data for six circle seems to be giving strange pole maps. In all other maps the qx, qy is less than one due to the construction of the map to the pole figure.

The test six circle data has qx & qy values > 10000 in some cases and the different scans do not give a contiguous image.

I had found that this data is giving negative qz values. Looking at the code this is causing the transform of qx & qy to get quite large. I made a quick try to fix by taking abs(qz) in the transform and this resulted in a decent looking image. Describing this to Christian he thought about the instrument geometry and determined that qz will indeed be negative in this case. While taking the abs(qz) gave a good picture it does not give a proper image. the x/y axes will be inverted. Need to look at a proper way to introduce a direction vector for the axis of projection. This can then be used to transform coordinates for the projection.


Participants: John Hammonds, Christian Schleputz

Christian tried updated code for Sixc geometry.  Was still gettting message about the wrong number of axes.  Christian needs to get the code from the branch instead of the trunk.



Discussed issues with pseudomotors in psic geometry.  There is no good way to tell the geometry being used from the file. We discussed some options. Look over the files. It is possible that in this situations, real motors (kphi, keta, kappa) may be scanned and the pseudomotors needed in the calculation have only a starting value captured. Need to determine what is scanned and perform a calculation of the pseudomotor positions for each element of the scan. Christian has a method that can be used for this once the values are known.


Participants: John Hammonds, Nicholas Schwarz

Nicholas discussed that during Software Advisory Group meeting this project was discussed.  We should discuss this further with Brian Toby in near future.



Participants: John Hammonds, Christian Schleputz

Let Christian know that monitor and filter scale factors are available in subversion (before leaving for vacation).



Participants: John Hammonds, Christian Schleputz, Nicholas Schwarz, Jon Tischler

Sprint closeout/planning meeting.

Discussed the progress made during 2014-03 sprint and discussed feature requests from Christian.

Feature requests discussed included:

- New transform to give H,K,L (by enabling the UB argument in Ang2Q)
- Corrections to the image data:
    - monitor or exposure time correction (for the 33BMC spec data, the corresponding columns are called 'I0' and 'sec')
     - filter transmission correction (33BMC: 'trans')
    - (eventually, we'll have to include polarization corrections, but not now)
- Export data as tiff image stacks (see attached
- Cancel feature for the file loading and acquisition.

John H added an issue:
- When the output file location is not writable to the user there are errors

Christian mentioned another issue that when a file is loaded that is still collecting data, new data is not shown until the user switches from that dataset and comes back to it.

Some discussion was had concerning the low level threading. We should think about adding a task at the end of the sprint to see if we can get openMP going.

This current sprint is intended to smooth out a few rough edges in the application so that it is more useable. We should evaluate the code for Sector 34 in a near term sprint to keep from getting too far away from looking at energy scans.

Some discussion was had about Mantid (UK) Should possibly take another quick look to see if this has any possibilities for us.


Participants: John Hammonds, Christian Schleputz, outside user

Went down to the beamline to try and help the user with installation on Linux/Windows. He is stuck at the same point that I am on Canopy on Linux. Was able to get him past some hurdles on Windows.



Participants: John Hammonds, Nicholas Schwarz

John  showed results from Christian.  Discussed next set of requirements and starting to close this sprint and plan the next.



Participants: John Hammonds, Christian Schleputz

Christian discussed some of his weekend's work & some experiences using the program.  He demonstrated data with some strange results.  The strangeness appears to be in the data not the program.  Discussed Output to HKL and a stack of images.  Christian did a few hacks on the image stacks and passed along his code.


Participants: John Hammonds, Christian Schleputz

Christian and the users were trying to install on a Linux box at the university. Could not get this to work since there is a problem running Canopy package manager using NX. Worked with Christian to update and get this running on his Mac. There was a problem running on Christian's computer. We suspect that this is a xrayutilities version issue. Looked at setting up PyDev/Eclipse with Christian. For some reason this was not working

Work on installation of RSM application on Christian's computer. Resolve problem with a change of variable in xrayutilities between recent versions. Discuss need for setting the file name for output file and the need for output of projections as an image stack.


Participants: John Hammonds, Christian Schleputz

Demonstration of working code. We were able to look at data that the users had just collected. Spent a bit of time looking at how the users have been looking at their data. Discuss installation of application on user's computer.


Participants: John Hammonds, Christian Schleputz

Update Christian on changes to remove the hardcoded extents for Pole figures.  Discuss plans to get rid of hard-coded ROI and pixels to average and which detector to choose from the config file.  ROIs and Pixels to average will be added to the GUI.  For now the detector choice will remain hard-coded.  Started to discuss getting this running at the beamline.



Participants: John Hammonds, Christian Schleputz

Met with Christian.  Discussed plans for getting rid of hard-coded extents for pole figures.  Helped Christian download and run from SVN.



Participants: John Hammonds, Nicholas Schwarz

Getting close to having something for Sector 33.  Need to work on data flow to remove hardcoded limits for pole figures.


2014-03-08 ( Unsure of actual date)

Separate discussions between John Hammonds with Christian and Jon T.

Due to the issues of Sector 34 not working will with xrayutilities, we will put off working with Sector 34 for now and concentrate on Sector 33.



Participants: John Hammonds, Nicholas Schwarz

Discuss issues with Sector 34.  Planning to get 33 & 34 working assumed being able to use xrayutilities for both.  34 does energy scans in fixed geometry.  This does not fit the mold for xray utilities.  This will require more thinking.  We will probably concentrate on getting 33 working better.


Participants: John Hammonds, Jon Tischler

Talked with Jon Tischler about problems using xrayutilities with his data at 34.  Jon's suggestion is to adapt his Igor code.  This presents a problem based on earlier thinking.


Participants: John Hammonds, Nicholas Schwarz

Discussed documenting parameters needed for processing data with xrayutilities.


Participants: John Hammonds, Christian Schlepuetz, Nicholas Schwarz, Jonathan Tischler

John presented progress made thus far in work for Christian.  Most of this work is described below.  Major addition at this point is adding in pole figures as output.

Also presented the future goals from discussion with Christian on 2/18.

Jon Tischler raised a few points:

  • Since HKL is not necessarily orthogonal, you will need to preserve data in qx, qy, qz
  • Data should be referenced to unrotated sample position

We questioned what is needed to get something out an usable at the beamline.  It was settled that we would focus the next sprint of activity toward:

  • Making the application read all configuration from a file but not necessarily aiming for a unified file between instruments
  • Since Christian does not have a file for storing detector calibration info look at the XML file from Jon T.
  • Make this work for both 33 & 34.
  • Provide a way to present HKL axes on top of qx, qy, qz image
  • Delay any work with user data transforms.


Participants: John Hammonds, Christian Schlepuetz

John demonstrated changes to the python code that allows the user to examine Q-space covered by images in a scan.  Since the last meeting code has been added to

  • Allow display of q coverage of all scans.  The user can select images to display using methods listed below
  • The overall q range of the data can be modified via the "Data Range" form.  Any images falling completely outside this range are not displayed in the overall q coverage and are excluded from the final mapping (see below)
  • It is possible to select/deselect each individual image via the "Scans" form.  Only images selected are displayed in the overall view and processed in the final mapping. 
  • It is now possible to launch q mapping from the application.  The mapping here mostly follows the code originally supplied by Christian with the following exceptions:
    • Christian's code included each image from specified scans.  The modified version does not include images based on the selection criteria mentioned above
    • Christian's code assembled all of the data and made one pass through X-ray utilities gridder routines.  This approach suffered failure as the data size grew.  The data size required to cause failure was not outside normal operating conditions.  This has been modified to process each scan through the gridder building the final results incrementally.  This attempt was a simple minded approach.  It does not take into account anything about data size or available resources.  Further thought may be needed here as we get more experience.

John has run this code on both his Linux box (Single Intel Xeon Quad Core W5580@3.2GHz, 48 GB RAM) and his Mac Laptop (Intel i7 Dual Core @2.66GHz, 8GB RAM).  The demonstration was done using the Mac.

Christian will provide John with pointers to more data sets to try.  This will in part try to stress the code thus far. 

We discussed future direction a bit:

  • One issue is that this code is currently still reading a spec file for some scan information but some important instrument parameters are still hard coded and images are loaded based on an assumed file structure.  The original intent is to migrate towards a data file (NeXus/HDF?) that will hold the instrument metadata and images as well as the scan data.  Christian has been discussed a NeXus solution with Pete Jemian over the last couple of months and Pete seems to have a possible partial solution that reads the Spec file.  John will go down and discuss this with Pete and see if we can start migrating to a solution here.
  • Christian has mentioned wanting to have a method for user defined transforms for the data.  A simple start for this is to add in mapping for a pole figure.
  • Would like to be able to output data in other formats (besides vti files).    It would be nice for instance to output pole figures as a stack of TIFF images.
  • Christian has been working with a set of users doing powder diffraction.  This group of users used to collect data using a special curved detector.  Knowledge of this detector has diminished and Christian is trying to assist the group in moving to collect the data using a series of area detector images.  He has demonstrated a basic proof of principal for this using matlab but would like to keep options open for doing this under this framework
  • In addition to working with a GUI to process data, once a user determines workflow (sets to be used, output options, etc.) it should be possible to script the process and it would be good if the program could provide a script from user inputs to aid the user.
  • Need to look at threading in underlying code.  May be able to speed things up by enabling this.

Should plan to meet with Nicholas and Jon T. in the not too distant future to look more at mid to long term planning.


Participants: John Hammonds, Christian Schlepuetz

 John Demonstrated code that examines the Q-space covered by images in a scan.  The intent here is to give the user a better idea of the range of collected data so that more intelligent choices can be made to select how/what data is analyzed.  At present, the program highlights data that falls outside of a given range of q values.  Based on discussion, John will work to display the range for all data within the selected range in one graphic.  We also need the ability to manually select images so that data that is more difficult to filter by Q can be removed.  The next step will be to work toward submitting the selected range of data for processing/mapping.


Paticipants: John Hammonds, Christian Schlepuetz

Discuss where to go from here.  Christian was able to process some data with pyspec and xrayutilities.  He can run one dataset with this on Mac but not a larger set.  John has been able to duplicate this on Linux, not on Mac.  Size could be an issue for both Christian and John. 

Christial would like to find a way to better examine the data volume and come uo with ways to determine data boundaries and be able to resample quikly.   May need to set up to display an entire dataset with lower resolution ( a thumbnail of sorts) and allow ways to select specific regions with Higher resolution.   Can use the endpoints of the detector somehow to present a reduced dataset which shows the data extent.  Possibly start with the corners but may need some other perimiter points to get a better idea of extent since we are mapping the flat plane of a detector onto a curved space.

The current space currently contains may not a number (NAN) pixels to genereate a regular grid.  Is there a way to get rid of this to save space.


Participants: John Hammonds, Doug Robinson

Met with Doug to discuss his needs in this area.  Doug works primarily with High Energy Diffraction on 6-ID-D.

Doug has a number of users that will present a number of different issues with this.  Different users will use different sources of data.

  • MSD folks would like to measure a fairly complete map of reciprical space.  This causes some problems when bright peaks are encountered.  They have used the Perkin Elmer and GE detector with this.  The problem here is that the strong peak will illuminate the phosphor and this will take some time to dissipate.  This results in persistent streaks in the data caused by prior peaks.  The GE detector has an additional problem that the low level drivers do not allow a complete areaDetector solution.  The low level driver does not allow access to live images.  The only access to images is through a file saved by the driver.  One solution to these problems is to use the Pilatus detector but this has only about 10% efficiency at these energies but the detector has 20 bits of dynamic range.  Another possibility is the ScintX dectector.  This is still a phosphor + CCD like the PE/GE but is much smaller and is therefore less likely for a frame to have legacy data.  This will of course require more datasets to be taken.
  • A group from McMaster university.  These use the Pilatus.
  • Iowa State group sometimes Mar 345 image plate.  This is a much slower process.  They sometimes precess the sample onto the same image.  This will give information on sample symmetries.  Other times they will collect a series of images on an image plate.
  • Doug described his ideal detector (which would extend from an existing detector) as a PixelRad area detector.  PixelRad now has a linear array of these an extension of this would make an area detector possible.

Doug will work up a plan do do some limited scale testing with xrayutilities.

2013-11-15 @ 3 pm

Participants: John Hammonds, NicholasSchwarz, Christian Schlepuetz

Nicholas has been able to make modifications to xrayutilities that allow this to be compiled on OSX.  He has submitted this back to the authors and these changes are being integrated into the main distribution.  Christian and John have both been able to compile with these changes.

Christian has been looking at xrayutilities.  On the surface, it seems to provide most of what is needed.  He was hoping to be a bit further with playing with this but has busy with the beamline.  He needs to figure out reading TIFF files.

Christian has also been talking with Beamline scientists at Diamond and also Sector 13 (Peter Eng).  All seem to agree that putting all of the data into a common HDF/Nexus file would be benificial.  Christian has been trying to gather information for what needs to go into the file here. 

For now will post process existing TIFF/Spec files to push data into correct HDF/Nexus format.

John will meet up with Christian to start helping get some of the code moving along.  Christian & John will meet up Monday afternoon.

2013-10-28 @ 11 am

Participants: John Hammonds, Christian Schlepuetz

John Hammonds stopped by and said he was looking into the project. We looked at some examples of 3DRSM reconstructions performed with MATLAB and also loaded a few volume sets that had been converted to .vti files into ParaView. John was added as an editor to the shared Box folder. Christian uploaded the MATLAB code, python conversion script and some example data into the Box folder.

So far, none of us (including Nicholas) had any luck with installing xrayutilities (on OS X). This needs to be addressed if xrayutilities is to be part of the solution. If it is too cumbersome, alternatives need to be considered.

2013-09-26 @ 3 PM

Participants: Christian M. Schlepuetz, Nicholas Schwarz

We went over the new desired results page on the wiki, which shows various types of renderings that are commonly used.
Mentioned ParaView as a rendering and analysis tool is one possible way to go.

For next time, Christian will prepare some sample datasets and try out x-ray utilities. Nicholas will try out some datasets in ParaView. We should meet again during the week of October 14th.

2013-07-212 from 2 PM - 4 PM

Participants: Pete Jemian, Christian M. Schlepuetz, Nicholas Schwarz, Jon Tischler


Workflow: acquisition -> raw images -> mapping -> real space -> ...
image - angles, i0
detector mount

Pete Jemian mentioned x-ray utilities on Sourceforge and pyspec from BNL

qx, qy, qz + h, k, l
multiple lattices

We had a very long discussion/debate about Python

2013-10-28 @ 11 am

Participants: John Hammonds, Christian Schlepuetz