fwrap - a wrapper for EM::Tools::Frealign

SYNOPSIS

  fwrap [-adrv] [-f [frequency]] -c config_file

DESCRIPTION

fwrap is a wrapper application for EM::Tools:Frealign. It allows the user to manage any number of rounds of refinement and/or reconstruction with Frealign. For refinement, the use of multiple processors is supported.

About Frealign

Frealign carries out search and refinement of particle parameters, CTF correction (allowing for astigmatism), and 3D reconstruction using interpolation in Fourier space. The magnification of each dataset and the defocus and astigmatism values of particles grouped by their film number can also be refined. On output, there are also diagnostic data, such as Fourier Shell Correlation, Fourier Shell Phase Residual, Q-factors in resolution zones, Average Particle Phase Residual between particles and 3D reconstruction, variance of the reconstruction, and point spread function indicating resolution in 3 dimensions.

Frealign uses MRC, SPIDER or IMAGIC (eventually) image file formats, and should be linked with MRC library routines for input/output of image files. The MRC mapformat is identical to that used by CCP4 crystallography programs. Current array dimensions provide space for 256x256 pixel images and the program is limited at present to transforms of even dimensions.

Some of the principles of Frealign are explained in N. Grigorieff (1998), J. Mol. Biol. 277, 1033-1046.

Frealign is available from Niko Grigorieff at http://emlab.rose2.brandeis.edu/grigorieff/downloads.html

OPTIONS

-c config_file: Required. Specify the configuration file to use.
-f frequency: Filter the output volume every frequency cycles of refinement. Default: 10.
-n num_proc: Option: specify the number of processors to use for a refinement. If this option is used, then hostfile and rshcommand directives must also be defined in the configuration file.
-p: Option: use dynamic phase residual setting. For each cycle of Frealign refinement, use the previous cycle's average phase residual. Default: Not used. (0)
-r: Option: save final cycle reprojections file. Default: Not used.
-v: Option: show verbose output. This is recommended the first time you use fwrap. Default: Not used.
-d: Option: show debugging output. Default: Not used.

CONFIGURATION

fwrap is configured with a flat text file containing variable settings and directives.

Format

Each line contains either a comment or a variable definition. A comment must start with the '#' symbol. A variable definition consists of a fwrap or Frealign variable and its values, separate by any amount of whitespace (either spaces or tabs). For example:

  # fwrap settings
  refine_variable1          frealign_value1
  refine_variable2          frealign_value2

  # frealign settings
  frealign_variable1        frealign_value1
  frealign_variable2        frealign_value2

Directives

The following fwrap variables may be defined in your configuration file. They are required as indicated.

frealign_exe: Required. Path to Frealign executable.
working_dir: Optional. A directory where fwrap can create temporary files. Any files in it will be cleared out upon normal exit of Frealign. Default: cwd (current working directory).
source_dir: Required. This is the directory that contains your Frealign input volume, parameter file, and raw images.
project_id: Required. A descriptive string used for the generation of intermediate and output files.
first_cycle: Required. The starting Frealign cycle for this refinement.
last_cycle: Required. The ending Frealign cycle for this refinement.
num_proc: Optional. An integer specifying the number of processors to use for refinement. If greater than 1, the hostfile and rshcommand directives must also be defined. Default: 1.
hostfile: Optional. For parallel refinements, the path to a flat text file listing the hosts to be used for the job. Required if num_proc is greater than 1.
rshcommand: Optional. For parallel refinements, the path to a command to be used to obtain a shell on remote hosts. Required if num_proc is greater than 1.
first_par: Optional. Specify an alternate name for your initial Frealign parameter file. Otherwise it is assumed to have a name consistent with the Project ID and cycle number.
last_par: Optional. Specify an alternate name for your final cycle's output Frealign parameter file. Otherwise it will be saved with a name consistent with the Project ID and cycle number.
first_vol: Optional. Specify an alternate name for your initial Frealign volume. Otherwise it is assumed to be named consistent with the Project ID and cycle number.
last_vol: Optional. Specify an alternate name for your final cycle's output Frealign volume. Otherwise it will be saved with a name consistent with the Project ID and cycle number.
use_phase_resid: Optional. If 1 (or true, T, yes), use dynamic phase residual feature. For each cycle of Frealign refinement, use the previous cycle's average phase residual. Default: 0 (not used).

Frealign parameters

The following Frealign parameters may be defined in your configuration file. Required parameters are indicated as such, and default values are given for optional parameters.

fwrap has renamed all Frealign parameters to more clearly indicate their purpose. However, for convenience the original Frealign internal variable are listed at the end of each description, and may be used instead in your configuration file.

When you see [...], this indicates that a variable accepts multiple, comma-delimited values. See Using multiple values below for more information.

file_format: Required. File format of input volume & raw images file. Acceptable values are S (SPIDER format) and M (MRC format). CFORM
mode: Required. Frealign mode key. Acceptable values are as follows. IFLAG [..]; -4 Bootstrap parameter file, then use mode 4.; -3 Bootstrap parameter file, then use mode 3.; 0 Reconstruction only, parameters as read in.; 1 Refinement & reconstruction.; 2 Random search & refinement.; 3 Simple search & refine.; 4 Search, refine, randomise & extend to resolution.
mag_refine: T/F Magnification refinement. Default: F. FMAG [..]
defocus_refine: T/F Defocus refinement. Default: F. FDEF [..]
astig_refine: T/F Astigmatism refinement. Default: F. FASTIG [..]
flip_test: T/F Use handedness "flip" test. Default: F. FFLIP [..]
backup: Not yet supported. Recovery feature. Back up every backup particles. If the program crashes, it can be restarted without much loss of data. 0 = no back up. Default: 0. NBACKUP [..]
reproject: T/F Write out matching projections after the refinement for diagnostic purposes. Default: T. FMATCH [..]
itmax_hist: T/F Write out history of itmax randomization trials. Default: F. FHIST [..]
beautify: T/F Apply extra real space symmetry averaging and masking to beautify final map just prior to output. Default: T. FBEAUT [..]
radius: Required. Particle radius of input images in angstroms. RI [..]
pixel_size: Required. Pixel size of input images. PSIZE [..]
amp_contrast: Required. Amplitude contrast for CTF correction. WGH [..]
phase_frac: Percent of data to be set aside for calculating free phase residual. Recommended values are 0.02 or 0.05. Default: 0.02. PFRAC [..]
num_stdev_mask: Number of standard deviations above mean for masking of input low-pass filtered 3D model. This 3D masking does not use radius. If positive, calculates a mask equivalent to solvent flattening with 5-pixel-cosine-bell smoothed mask boundary. The mask is then used to multiply the input 3D map, which is then used for all parameter refinement and subsequent calculations. If negative, calculates a mask resulting in a sharp binary (0/1) mask boundary which is used for both parameter refinement and reconstruction, and to mask and output the matching projections. Each matching particle image is also always masked with a cosine bell edged function of radius radius. Default: 0.0. XSTD [..]
bfactor_const: Phase residual / pseudo-B-factor conversion constant. Automatic weighting is applied to each particle: a pseudo-temperature (B) factor is applied to each particle according to its relative phase residual against the reference. The weight is calculated as:; with DELTAP = relative phase residual (actual phase residual minus avg_phase_resid), bfactor_const = conversion constant (5.0 in the example), and R^2 the squared resolution in Fourier units (R = 0.0 ... 0.5). A large value for bfactor_const (e.g. 100.0) gives equal weighting to each particle. Default: 5.0. PBC [..]
avg_phase_resid: Average phase residual of particles. Required for mode 0. Default: 60.0. BOFF [..]
angular_step: Angular step size. Modes 3, 4 only. Default: 0.0. DANG [..]
itmax: Number of cycles of search/refine. Modes 2, 4 only. Default: 10. ITMAX [..]
psi
theta
phi
delta_x
delta_y: 0/1 mask to exclude parameters from refinement. There are five numbers to allow psi, theta, phi, deltaX, deltaY to be included or excluded from the search or refinement. The use of 1 1 1 1 1 causes all 5 parameters to be refined for each particle. The use of 0 0 0 1 1 would cause only the position of each particle to be optimised. Default: all 1. MASK
first_part: First particle to use. Default: 1. IFIRST
last_part: Required. Last particle to use. ILAST
symmetry: Symmetry to impose: Cn, Dn, T, O, I, I1, I2 or N. For asymmetric structures, use 0. Default: 0. ASYM [..]
rel_mag: Relative magnification to apply to dataset. If entering data for multiple datasets, use 0 here to indicate that there are no more datasets. Note that fwrap does not currently support multiple datasets. Default: 1.0. RELMAG [..]
step_size: Required. Densitometer step size, in microns. DSTEP [..]
phase_resid_target: Target phase residual. Default: 10.0. TARGET [..]
phase_resid_thresh: Phase residual cut-off. Any particles with a higher overall phase residual will not be included in the reconstruction when mode = 0, 1, 2, 3. This variable is often used with mode 0 in separate runs to calculate maps using various values of phase_resid_thresh to find the optimum value to produce the best map as judged from the statistics. Default: 90.0. THRESH [..]
spherical_aberr: Required. Microscope spherical aberration for this dataset. CS [..]
voltage: Required. Microscope voltage for this dataset. AKV [..]
resolution: Resolution (in angstroms) to which reconstruction is calculated. Default: 10.0. RREC [..]
low_res_cutoff
high_res_cutoff: Low/high resolution cutoff of refinement in angstroms. Together these parameters define resolution of the data included in the search/refinement. These two parameters are very important. The successful alignment of particles depends critically on the signal-to-noise ratio of the cross-correlation or phase residual calculation, and exclusion of weak data at high resolution or spurious, very strong artefacts at low resolution can make a big difference. Success can be judged by whether the X,Y coordinates of the particle centres are reasonable. Defaults: 200.0, 25.0. RMAX1, RMAX2 [..]
bfactor: B-factor to apply to particle image projections before orientation determination or refinement. This allows inclusion of high resolution data but with a reduced (or increased if negative) weight. amp_contrast and bfactor can be manipulated in particle parameter refinement as if they were low pass and high pass filters. amp_contrast and the CTF are used to correct the density in the final map, whereas bfactor is not. Default: 0.0. RBFACT [..]
raw_images: Required. Input particle image stack. If you provide a full path to the stack, then that will be used. Otherwise, the file is assumed to be located in the working directory. FINPAT1

Filter parameters

Optionally, the output volume from a cycle can be filtered using an external program. This is helpful for evaluating the progress of your refinement. Currently the only supported program is Niko Grigorieff's BFACTOR.

The filtered volumes and filter program logfiles are saved in the scratch directory.

To filter output volumes with BFACTOR, define the following variables in your config file.

filter_exe: Required. Path to BFACTOR executable.
filter_bfactor: B-factor in A^2. Default: -3000.0.
filter_type: 1 = gaussian filter, 2 = cosine edge mask filter. Default: 2.
filter_radius: Gaussian or cosine edge filter radius. Default: 20.0.
filter_freq: How often to filter output volumes. For example, a value of 10 means to filter the output volume every 10 cycles. If you are running many cycles, do not choose a very low number. Otherwise you will use a lot of disk space. Default: 10.

Using multiple values

Sometime it is desirable to use a different value each cycle for a Frealign variable. fwrap configfiles may contain multiple values for any numeric Frealign variable. These variables are indicated by the [..] notation above.

Individual values must be comma-separated. To indicate that a particular value should be used more than once, follow it with the character x and an integer indicating the number of cycles to use this value.

DO NOT insert spaces between values in a multivalue entry, ever.

The total number of values specified for any Frealign variable must be equal to or greater than the number of cycles of refinement being run.

The following examples are all valid:

  angular_step              15.0,12.0,12.0,10.0,8.0,8.0
  high_res_cutoff           40,30x2,20x2,15
  phase_resid_thresh        90.0,75.0,60.0x50

Sample configuration file

  # fwrap vars
  debug                 1
  verbose               0
  use_phase_residual    1
  save_reprojs          0
  num_proc              2
  hostfile              hosts
  rshcommand            /usr/bin/ssh

  # frealign factory vars
  location              /usr/local/bin/frealign_v6.exe
  source_dir            /home/slaton/tmp/refine
  working_dir           /home/slaton/tmp/refine/tmp
  project_id            pol
  first_cycle           21
  last_cycle            50
  raw_images            part.spi

  # frealign vars [required]
  file_format           S
  mode                  1
  first_part            1
  last_part             6835
  radius                100.0
  pixel_size            2.5
  amp_contrast          0.12
  step_size             12.7
  spherical_aberr       2.2
  voltage               200.0

  # frealign vars [optional]
  flip_test             T,Fx100
  beautify              T
  bfactor_const         avg_phase_resid
  phase_resid_target    10.0
  phase_resid_thresh    90.0,75.0x50,60.0
  resolution            50,35,25x3,15.0x5,10.0x20
  low_res_cutoff        300,200x20
  high_res_cutoff       40,30,20x5,15x15

  # bfactor vars [optional]
  filter_location       /usr/local/bin/bfactor.exe
  filter_bfactor        -3000.0
  filter_type           2
  filter_radius         25
  filter_freq           3

USAGE

Tips

The recommended initial parameters for more accurate orientation parameter determination are:

a high resolution (e.g. 10-15 Angstroms)

a reasonably but not excessively high value for itmax, e.g. 200.

a high rather than a small value for angular_step, i.e. 200.0 instead of 2.0.

a high value for bfactor to decrease the weight of the noisier data at higher resolution (e.g. use 2000 or 3000) but to keep it in.

a range of values should always be tried since the optimal values may depend on the structure and the quality of the pictures.

Limitations

Currently, only 1 dataset is supported. There is no support for multiple raw image or volume inputs/outputs.

The following Frealign mode keys are supported: 0, 1, 2, 3, 4. Modes -4 and -3 require creating an input parameter file from scratch (bootstrapping), and are not currently supported.

TODO

Clean up code using these three tips:

http://perl.about.com/library/weekly/aa052100c.htm

http://perl.about.com/od/howtoprograminperl/l/aa033100b.htm

http://www.perl.com/pub/a/2000/04/raceinfo.html

Implement bootstrapping for modes -4 and -3.

Create SMP/parallel version.

Add support for Frealign backup recovery feature.

Add support for Card 3 (masking).

Fix benchmarking.

HISTORY

0.1 (05.01.10): First working version.
0.14 (05.01.13): Finished POD documentation.
0.17 (05.01.17): Added multivalue variable support.
0.20 (05.01.20): Improved multivalue support. Added -r option. Fixed intermediate file removal bug.
0.30 (05.03.18): Complete rewrite as a script dependent upon EM-Perl module package.
0.31 (05.09.20): POD modifications/fixes.

AUTHOR

Slaton Lipscomb <slaton@berkeley.edu>

COPYRIGHT

DISCLAIMER

This is alpha-quality software that has undergone only the most superficial testing. Some features are incomplete or missing. It is provided "as is" without a warranty of any kind.

NAME