{"id":108,"date":"2026-10-08T21:48:50","date_gmt":"2026-10-08T21:48:50","guid":{"rendered":"https:\/\/bcsb.als.lbl.gov\/?page_id=108"},"modified":"2026-10-08T21:48:50","modified_gmt":"2026-10-08T21:48:50","slug":"automatic-data-processing","status":"publish","type":"page","link":"https:\/\/bcsb.als.lbl.gov\/?page_id=108","title":{"rendered":"Automatic Data Processing"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">XDS AUTOPROCESSING<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The automated scripts run the programs <strong>XDS<\/strong> (data reduction, including integration and scaling), <strong>pointless<\/strong> (further space group determination), and <strong>aimless<\/strong> (final merging of data to produce the mtz you will need for structure solution).&nbsp;You can see the results directly within the b4 gui.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Pointless runs with default input and aimless uses the ONLYMERGE input. Aimless runs three times. For the first run, it produces the file aimless_all.log and aimless_all.mtz . As part of this run, it identifies the resolution cutoff based on cc(1\/2)&gt;0.3 and IsigI&gt;2.0. It then runs again based on a resolution cutoff of cc(1\/2)&gt;0.3 and produces the files aimless_cc.log and aimless_cc.mtz . It then runs a final time based on IsigI&gt;2.0, and produces the files aimless_IsigI.log and aimless_IsigI.mtz . Look for these files in the XDS folder which appears in your data directories. (Note that pointless and aimless only run when XDS has successfully produced the scaled file XDS_ASCII.HKL)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If processing went smoothly, you should see these files in your directory. You can use any of the <strong>mtz<\/strong> files for further structure solution:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>aimless_all.log<\/strong>\u00a0This is the log resulting from\u00a0autoprocessing using all the data.<\/li>\n\n\n\n<li><strong>aimless_all.mtz<\/strong>\u00a0This is the mtz file resulting from autoprocessing using all the data.<\/li>\n\n\n\n<li><strong>aimless_cc.log<\/strong>\u00a0This is the log file resulting from\u00a0autoprocessing using a resolution cutoff based on a cc1\/2 of 0.3.<\/li>\n\n\n\n<li><strong>aimless_cc.mtz<\/strong>\u00a0This is the mtz file resulting from autoprocessing\u00a0using a resolution cutoff based on a cc1\/2 of 0.3.<\/li>\n\n\n\n<li><strong>aimless_IsigI.log<\/strong>\u00a0This is the log file resulting from\u00a0autoprocessing using a resolution cutoff based on IsigI > 2.0.<\/li>\n\n\n\n<li><strong>aimless_IsigI.mtz<\/strong>\u00a0This is mtz file resulting from\u00a0autoprocessing using a resolution cutoff based on IsigI > 2.0.<\/li>\n\n\n\n<li><strong>XDS.INP<\/strong>\u00a0This is the input file used for processing in XDS. Here is a <a href=\"http:\/\/xds.mpimf-heidelberg.mpg.de\/html_doc\/xds_parameters.html\" target=\"_blank\" rel=\"noopener\">LINK<\/a> which shows the parameters used in this file and what they all mean.<\/li>\n\n\n\n<li><strong>XDS_ASCII.HKL<\/strong>\u00a0This is the scaled reflection file from XDS. Note here that if\u00a0XDS\u00a0does not finish because of an issue with the data, you will not see this\u00a0file in the directory. One thing to try in this case is to edit\u00a0XDS.INP and change the line which reads JOB=XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT\u00a0to: JOB=DEFPIX \u00a0INTEGRATE CORRECT\u00a0and then rerun XDS. (We have seen XDS fail when those first four entries in that line are present, but successfully process when they are not.)<strong>OTHER XDS SPECIFIC FILES OF INTEREST:<\/strong><\/li>\n\n\n\n<li><strong>COLSPOT.LP<\/strong>\u00a0Gives the specifications used (and the results) for picking spots for indexing.<\/li>\n\n\n\n<li><strong>CORRECT.LP<\/strong>\u00a0Gives the specifications (and results) for scaling the data, including R-factor, number of observations, unique reflections. Gives a table towards the end of the file which is similar to the one output by HKL2000.<\/li>\n\n\n\n<li><strong>DEFPIX.LP<\/strong>. Defines the \u201ctrusted\u201d region of the detector.<\/li>\n\n\n\n<li><strong>IDXREF.LP<\/strong>\u00a0Gives results from indexing. Similar to the table given by HKL2000 when indexing.<\/li>\n\n\n\n<li><strong>INIT.LP<\/strong>\u00a0Shows\u00a0the detector gain and dark current corrections used.<\/li>\n\n\n\n<li><strong>INTEGRATE.LP<\/strong>\u00a0Output file from integration.<\/li>\n\n\n\n<li><strong>INTEGRATE.HKL<\/strong>\u00a0The reflection file output.<\/li>\n\n\n\n<li><strong>GXPARM.XDS<\/strong>\u00a0The final refined cell parameters and orientation.<\/li>\n\n\n\n<li><strong>SPOT.XDS<\/strong>\u00a0Gives list of reflections.<\/li>\n\n\n\n<li><strong>XPARM.XDS<\/strong>\u00a0The orientation matrix file.<\/li>\n\n\n\n<li><strong>XPLAN.LP<\/strong>\u00a0Data collection strategy information.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">DIALS AUTOPROCESSING<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We are now running an automatic data processing using xia2 to run DIALS. Similar to xds, it creates a DIALS subdirectory in the data collection directory where the images are. Look at&nbsp;xia2.html file for a nice table showing the processing results. If you open the xia2.html in a browser, you will see the results in a table like this:<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"alignleft is-resized\"><img decoding=\"async\" src=\"https:\/\/bcsb.als.lbl.gov:8443\/wp-content\/uploads\/2015\/05\/xia2html_example.png\" alt=\"\" class=\"wp-image-12236\" style=\"aspect-ratio:0.9090909090909091;width:300px;height:auto\"\/><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">Note that DIALS often takes up to half an hour after the last image was collected to finish, so you won\u2019t see the directory appear right away.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Within the DIALS directory, you will see another directory called DataFiles. This contains the sca and mtz files resulting from processing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ARCHIVED:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Previously, we had automated scripts which processed data on 8.2.1 and 8.2.2 using Xia2, and the results were placed in the user home directory under the directory&nbsp;<strong>~\/rabbit_proc.<\/strong> We are no longer supporting these scripts, but you may still see some old processed data under this directory.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>XDS AUTOPROCESSING The automated scripts run the programs XDS (data reduction, including integration and scaling), pointless (further space group determination), and aimless (final merging of data to produce the mtz you will need for structure solution).&nbsp;You can see the results directly within the b4 gui. Pointless runs with default input and aimless uses the ONLYMERGE&hellip; <a class=\"more-link\" href=\"https:\/\/bcsb.als.lbl.gov\/?page_id=108\">Continue reading <span class=\"screen-reader-text\">Automatic Data Processing<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-108","page","type-page","status-publish","hentry","entry"],"_links":{"self":[{"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=\/wp\/v2\/pages\/108","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=108"}],"version-history":[{"count":1,"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=\/wp\/v2\/pages\/108\/revisions"}],"predecessor-version":[{"id":109,"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=\/wp\/v2\/pages\/108\/revisions\/109"}],"wp:attachment":[{"href":"https:\/\/bcsb.als.lbl.gov\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}