Slaton Lipscomb

Slaton's Tips:
Configuring IMAGIC With MPI for 32-bit Linux Platform

Last updated November 21, 2006
Comments/corrections welcome.

IMAGIC is a product of Image Science Software GmbH in Berlin, Germany.

IMAGIC is not distributed in true "source" form. Rather, it is distributed as a set of precompiled object files, which the user then links to build the executables. Before continuing, make sure you have the right IMAGIC package for your platform.

For the purposes of these instructions, I will install IMAGIC version 040812 and MPICH1 version 1.2.6 onto a machine running Red Hat Linux, with hostname firefox.university.edu. When you see these values referred to, you will need to substitute the versions you are actually installing, and the hostname of the computer you are installing on. The IMAGIC version number is always indicated by an empty file entitled version_xxxxxx in the source directory.

I will use the Intel Fortran Compiler to link and build the IMAGIC programs. Compiler versions prior to 8.0 are called ifc. Releases 8.0 and later are called ifort. In this example I am using ifort release 8.0.34.

The ssh-related configuration here is specific to OpenSSH. If you use another ssh implementation, such as SSH Security Corporation's commercial product, you will need to modify these instructions accordingly.

It is assumed that the Modules system is used for environment management. Initial installation and configuration of Modules is beyond the scope of this document. More information about Modules can be found at these sites:

  « http://modules.sourceforge.net »
  « http://hpcf.nersc.gov/software/os/modules.html »

If you do not yet have ifort installed, please see these instructions.

1   Untar the IMAGIC source to a temporary directory. I prefer to install to /usr/local, with the version and compiler designated by subdirectories. Rename this temporary directory to imagic-040812 where 040812 is the IMAGIC version number.

$ mkdir tmp
$ mv imagic_040812.tar.gz tmp; cd tmp 
$ tar xzvf imagic_040812.tar.gz
$ cd ..
$ mv tmp imagic-040812

2   Now move the imagic-040812 source directory to /usr/local.

1   Download the Unix version of the MPICH1 software from
http://www-unix.mcs.anl.gov/mpi/mpich1/.

2   Unzip and untar the archive. Then make sure the Intel compiler module is loaded, then set the RSHCOMMAND environment variable (required by MPICH1), and the COMPILER environment variable (required by IMAGIC):

$ tar xzvf mpich-1.2.6.tar.gz; cd mpich-1.2.6
$ module load ifort
$ setenv RSHCOMMAND "ssh -q"
$ setenv COMPILER "INTEL"

This builds a version of mpich that uses ssh for starting threads instead of rsh, which is insecure on a public network. For a linux cluster on a private network you may prefer to use rsh

3   Compile and install the mpich package.

$ ./configure --prefix=/usr/local/mpich/1.2.6/intel --with-arch=LINUX \
  --with-fflags="-save"
$ make
$ make install

4   Add a symbolic link so that your IMAGIC environment won't need to be changed in the event of future MPICH updates.

$ ln -s /usr/local/mpich/1.2.6/intel /usr/local/mpich/intel

3   Edit the default machines file and add the machines you want to use. Use the hostname that is returned by the hostname command. Dual processor workstations should be listed twice.

The Linux machine file should be here:

/usr/local/mpich/1.2.6/intel/share/machines.LINUX

1 The following environment variables and aliases must be defined for IMAGIC. This format is appropriate for csh; for other shells, change as needed.

setenv IMAGIC_ROOT  "/usr/local/imagic-040812"
setenv PATH         "${PATH}:."
alias i             "${IMAGIC_ROOT}/imagic.e"
alias disp          "${IMAGIC_ROOT}/display/display.e"
alias plot          "${IMAGIC_ROOT}/plot/plotall.e"
alias em2em         "${IMAGIC_ROOT}/stand/em2em.e"

2 The module should also define the following MPICH environment variables and paths.

setenv MPIHOME      "/usr/local/mpich/intel"
setenv MPIBIN       "${MPIHOME}/bin"
setenv MPIF90       "${MPIBIN}/mpif90"
setenv PATH         "${MPIBIN}:${PATH}"
setenv MANPATH      "${MPIROOT}/man:${MANPATH}"

It's important to prepend the mpi/bin directory to PATH, rather than append it, in case the system has other MPI packages installed.

Unfortunately, dot (.) must be added to PATH, because IMAGIC generates its internal control file for an mpi job in the user's current working directory. It then attempts to execute this file, without specifying the path.

1 Make sure the IMAGIC license file is present in $IMAGIC_ROOT/imagic.drv.

Edit the license file to make sure the # of CPUs listed is correct. This is important for the MULTI-REFERENCE-ALIGNMENT program, which uses OpenMP parallelization for machines with more than one CPU.

2 Make sure the Intel compiler and IMAGIC modules are loaded. Then build the IMAGIC programs:

$ module load ifort imagic/040812
$ cd $IMAGIC_ROOT
$ ./install.b

Check for failures in compilation with:

$ find . -name \*.err

3 Verify that the program paths listed in $IMAGIC_ROOT/lognames.drv are correct. The mpi and MPI paths in particular may need to be fixed if you have decided to use an externally installed mpi package.

4 Now build the IMAGIC MPI programs:

$ cd $IMAGIC_ROOT/align
$ ../fori.b mralign mpi
$ cd ../angrec
$ ../fori.b ar mpi
$ cd ../threed
$ ../fori.b true3d mpi

IMPORTANT fori.b does some tests to determine that the correct MPI version is indicated in the imagic.drv file. These tests may break if you have more than one MPI package installed. For example, some Red Hat Linux distributions install MPICH or LAM-MPI to the system directories. If this occurs, verify that in the IMAGIC module, your mpi/bin directory is being prepended to the user PATH, rather than appended. If this is the case and the script still fails, you may need to edit fori.b to disable the tests that occur between:

# ----- Try to find the MPI (message passing interface) stuff

and

# ----- Check MPI entry in IMAGIC.DRV file

If you edit install.b, make sure the mpidir, mpincl and mpif_h variables are still set correctly.

1 You need to generate an ssh key with a blank passphrase, so that MPI can use ssh instead of rsh for launching worker threads. Then, this key needs to be added to authorized_keys.

$ ssh-keygen -q -C BatchModeKey -t rsa -f ~/.ssh/batchmode -N ""
$ cat ~/.ssh/batchmode.pub >> ~/.ssh/authorized_keys

2 You also need to create (or edit, if it already exists) a user ssh config file, ~/.ssh/config, and specify that this batchmode key is to be used for connections to the local machine (only). This is done with the Host keyword. Add a section like the following to the TOP of the config file. It must be above the Host * wildcard section, if one exists.

Host firefox.university.edu firefox
  IdentityFile ~/.ssh/batchmode

IMAGIC determines the local machine's hostname according to the output of uname -n, so make sure the same name is used here. In this case firefox is just a short convenience alias that is also specified in the system's hosts file.

3 To test the ssh configuration, the following command should give a listing of the user's home directory, without asking for a password or passphrase.

$ ssh firefox ls

If a password is requested, or an error results, the configuration is incorrect. Start over, and go through these steps again.

1 For this example I'll use a machine called firefox, on which you have just installed and configured IMAGIC with MPI.

If you want to use more than one physical machine for parallel computing, you need to NFS mount firefox's /usr/local/imagic directory on the other machines to the same mount point (or set a symbolic link). The user's working directory must also be present on all systems.

ssh must work on all machines as on firefox, using batchmode keys with an empty passphrase.

1 If the backspace key doesn't work in IMAGIC, add this to the user's .cshrc or .bashrc file.

stty erase ^?

Important: This is NOT entered as <caret><question mark>, but rather as <control-V><backspace>. In the shell and most text editors this will appear as ^?.