Slaton's Tips:
Compiling MPI-SPIDER For Linux With Portland Group Compilers

Last updated June 26, 2006
Comments/corrections welcome.

Chao Yang has added support to SPIDER for parallelism using the MPI (Message Passing Interface) standard. The operations AP SH, AP REF, BP RP, BP CG, BP 3F, and BP 32F are supported in releases 12.01 and later.

MPI-SPIDER was developed with support from NIH grant P01064692 (Computational Technology For High-throughput Cryo-EM).

SPIDER was developed at the Resource for the Visualization of Biological Complexity (RVBC) located in the Wadsworth Laboratories, Albany, New York.

0. Introduction And Motivation

The Portland Group's pgf90 is the officially supported Fortran compiler for SPIDER on the Linux platform. A Portland workstation license costs roughly $700, and free 15-day evaluation licenses are available. Additionally, both Chao Yang and ArDean Leith have reported success using the PathScale compiler suite for x86_64 platform.

For the MPI communication, we will use LAM/MPI, a high-quality open source implementation of the Message Passing Interface specification.

SPIDER also supports the use of FFTW ("Fastest Fourier Transform in the West") on the Linux platform. FFTW is "a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of both real and complex data, and of arbitrary input size." It is recommended that you use FFTW, as its routines are faster than SPIDER's. These instructions are for a FFTW-based MPI-SPIDER installation.

It is assumed that the Modules system is used for environment management. While the installation and configuration of Modules is beyond the scope of this document, such information can be obtained from the project's SourceForge page, and from NERSC. Some Linux distributions, such as CAOS Community Assembled OS include the Modules system.

1. Compile And Install LAM-MPI

See this document for complete instructions.

2. Compile And Install FFTW

See this document for complete instructions.

3. Obtain A MPI-SPIDER Makefile For Linux/pgf90

3.1   You will need a Makefile that specifies the required compiler and linker flags for pgf90 and the LAM/MPI libraries. I've packaged a few that support optimization for a couple of CPU types, and you can download them here:

« download MPI-SPIDER Linux/pgf90 Makefiles »

3.2   Select the most appropriate Makefile for your planned installation, and whether or not you are using FFTW, and copy it to your SPIDER source directory.

4. Compile MPI-SPIDER With pgf90

4.1   Make sure your pgf90 environment is loaded. Change to the SPIDER source directory, and enter the following commands:

$ rm -f *.o *.a *.mod
$ make -f Makefile_linux_mpi_fftw_opt64

Use the Makefile name you selected in the previous step, if it is different than Makefile_linux_mpi_fftw_pgi.

The compilation process will take 5 to 15 minutes depending on how fast your machine is. If compilation and linking are successful, then the executables will be built in the spider/bin directory. The executables' names depend on which Makefile you used, as follows:

If you run into any compilation errors, double-check that the FFTW archive libraries are indeed installed in /usr/local/fftw/pgi/lib, and that your pgi environment is properly set up.

5. Create New SPIDER modulefile (optional)

If this is a new SPIDER installation, create a modulefile for setting up the SPIDER environment variables. It is assumed that the Modules system has previously been installed and configured.

5.1   The following SPIDER environment variables must be defined:

setenv  SPMAN_DIR   "/usr/local/spiderweb-14.11/spider/man/"
setenv  SPPROC_DIR  "/usr/local/spiderweb-14.11/spider/proc/"
setenv  SPRGB_DIR   "/usr/local/spiderweb-14.11/web/rgb/"
setenv  WEBMAN_DIR  "/usr/local/spiderweb-14.11/web/docs/"

5.2 The following items are added to the PATH and LD_LIBRARY_PATH environment variables.

append-path  PATH  "/usr/local/spiderweb-14.11/spider/bin"
append-path  PATH  "/usr/local/spiderweb-14.11/web/src"

5.3 For convenience, create a symbolic link in the spider/bin directory.

$ cd /usr/local/spiderweb-14.11/spider/bin
$ ln -s spider_linux_mpi_fftw_opt64 spider_mpi

6. Create The lam-bhost.def File

6.1   Before starting a MPI-SPIDER job, you will need to define which hosts are to be used by LAM/MPI to start jobs. This is more or less the same as the machines.LINUX file used by MPICH.

Create a file /usr/local/lam/7.1.2/pgi/etc/lam-bhost.def. It should like something like the following. Be sure to replace lnx-node01, lnx-node02, etc. with your own hostnames, and to use the correct value for the number of CPUs. Also if you wish LAM to run jobs on your master node as well, you can remove schedule=no from the first line.

# This file is part of the LAM/MPI software package. For license
# information, see the LICENSE file in the top level directory of the
# LAM/MPI source distribution.
#
lnx-master schedule=no
lnx-node01 cpu=2
lnx-node02 cpu=2
lnx-node03 cpu=2
lnx-node04 cpu=2
lnx-node05 cpu=2

6.2   Test your lam-bhost.def configuration by starting lam with the -v (verbose) option, and by running the recon and lamnodes commands.

$ lamboot -v
$ recon -v lam-bhost.def 
$ lamnodes
$ lamhalt

If all goes well, recon should give you a message like the following:

Woo hoo!

recon has completed successfully.  This means that you will most likely
be able to boot LAM successfully with the "lamboot" command (but this
is not a guarantee).  See the lamboot(1) manual page for more
information on the lamboot command.

TODO: Complete this section. Also, need section on generating SSH hostkeys & loading the LAM environment.

7. Test Your New MPI-SPIDER

7.1   Load your SPIDER environment, either by loading the SPIDER module you created above in step 5, or by other means.

7.2   Start a LAM session and run SPIDER. (It is assumed you created a spider_mpi symbolic link in step 5.3, above.)

$ lamboot
$ mpirun -np [number of processors] spider_mpi spi @batchname

7.3   Try running the tests described on this page:

  « http://www.wadsworth.org/spider_doc/spider/docs/testing.html »

7.4   When you are finished, shut down LAM.

$ lamhalt

TODO: Complete this section.