Slaton Lipscomb

Slaton's Tips:
Configuring IMAGIC With MPI On SGI/IRIX Platform

Last updated September 1, 2002
Comments/corrections welcome.

IMAGIC is a product of Image Science Software GmbH in Berlin, Germany.

IMAGIC is not distributed in true "source" form. Rather, it is distributed as a set of precompiled object files, which the user then links to build the executables. Before continuing, make sure you have the right IMAGIC package for your platform.

For the purposes of these instructions, I will install IMAGIC version 020731 onto a machine running SGI IRIX 6.5, with hostname toro.university.edu. When you see these values referred to, you will need to substitute the IMAGIC version you are actually installing, and the hostname of the computer you are installing on. The IMAGIC version number is always indicated by an empty file entitled version_xxxxxx in the source directory.

Silicon Graphics' MIPSPro compilers will be used.

Starting with the 6.5.19 release, IRIX 6.5 now includes OpenSSH. Therefore, the ssh-related configuration here is specific to OpenSSH. If you use another ssh implementation, such as SSH Security Corporation's commercial product, you will need to modify these instructions accordingly.

It is assumed that the Modules system is used for environment management. Initial installation and configuration of Modules is beyond the scope of this document. More information about Modules can be found at these sites:

   « http://modules.sourceforge.net »
   « http://hpcf.nersc.gov/software/os/modules.html »

1   Unzip and untar the IMAGIC source to a temporary directory. Rename this temporary directory to imagic-020731, where 020731 is the IMAGIC version number.

Move the imagic-020731 source directory to /usr/local.

1   Download the MPI software from:

   « ftp://ftp.mcs.anl.gov/pub/mpi/mpich.tar.gz »

2   Unzip and untar the archive. Move the source to the IMAGIC source directory in /usr/local/ and rename it from mpich-[version] to mpi.

3   Set the RSHCOMMAND environment variable:

$ setenv RSHCOMMAND "/usr/bin/ssh"

This builds a version of mpich that uses ssh for starting threads instead of rsh, which is insecure on a public network. Formerly, the "-rsh=ssh" configure option was used, but this is now deprecated in favor of the environment variable.

4   Compile the mpich package:

$ cd /usr/local/imagic-020731/mpi
$ ./configure --prefix=`pwd` --with-arch=IRIXN32 --with-fflags="-static"
$ make

5   Edit the default machines file and add the machines you want to use. The IRIX machines file should be here:

/usr/local/imagic-020731/mpi/util/machines/machines.IRIXN32

1   Create the modulefile for the IMAGIC environment variables. The following environment variable is required.

$ setenv IMAGIC_ROOT  "/usr/local/imagic-020731"

2   The following IMAGIC alias is defined.

set-alias       I     "/usr/local/imagic-020731/imagic.e"

3   The following mpich paths are added to the PATH environment variable.

prepend-path PATH     "/usr/local/imagic-020731/mpi/bin"
prepend-path MANPATH  "/usr/local/imagic-020731/mpi/man"
append-path PATH      "."

It's important to prepend the mpi/bin directory to PATH, rather than append it, in case the system has other MPI packages installed.

Unfortunately, dot (.) must be added to PATH, because IMAGIC generates its internal control file for an mpi job in the user's current working directory. It then attempts to execute this file, without specifying the path.

1   Make sure the IMAGIC module is loaded, then build the IMAGIC programs:

$ cd $IMAGIC_ROOT
$ ./install.b

Check for failures in compilation with:

$ find . -name \*.err

2   Verify that the program paths listed in $IMAGIC_ROOT/lognames.drv are correct. The mpi and MPI paths in particular may need to be fixed if you have decided to use an externally installed mpi package.

3   Make sure the IMAGIC license file is present in $IMAGIC_ROOT/imagic.drv.

Edit the license file to make sure the # of CPUs listed is correct. This is important for the MULTI-REFERENCE-ALIGNMENT program, which uses OpenMP parallelization for machines with more than one CPU.

4   Now build the IMAGIC MPI programs:

$ cd $IMAGIC_ROOT/align
$ ../fori.b mralign mpi
$ cd ../angrec
$ ../fori.b ar mpi
$ cd ../threed
$ ../fori.b true3d mpi

IMPORTANT fori.b does some tests to determine that the correct MPI version is indicated in the imagic.drv file. These tests may break if you 1) have other MPI packages installed, or 2) are attempting to use a prebuilt MPI package located outside the IMAGIC directory. If this occurs, first verify that in the IMAGIC module, the mpi/bin directory is being prepended to the user PATH, rather than appended. If this is the case and the script still fails, you may need to edit fori.b to disable the tests that occur between:

# ----- Try to find the MPI (message passing interface) stuff

and

# ----- Check MPI entry in IMAGIC.DRV file

If you edit install.b, make sure the mpidir, mpincl and mpif_h variables are still set correctly.

1   You need to generate an ssh key with a blank passphrase so that mpich can use ssh instead of rsh for launching parallel threads. Then this key needs to be added to authorized_keys.

$ ssh-keygen -q -C BatchModeKey -t rsa -f ~/.ssh/batchmode -N ""
$ cat ~/.ssh/batchmode.pub >> ~/.ssh/authorized_keys

2   You also need to edit your user ssh config file, ~/.ssh/config, and specify that this batchmode key is to be used for connections to the local machine (only). This is done with the Host keyword. Add a section like the following to the TOP of the config file. It must be above the Host * wildcard section, if it exists.

Host toro.university.edu toro
  IdentityFile ~/.ssh/batchmode

IMAGIC determines the local machine's hostname according to the output of uname -n, so make sure the same name is used here. In this case toro is just a short convenience alias.

3   To test the ssh configuration, the following command should give a listing of the user's home directory, without asking for a password or passphrase.

$ ssh toro.university.edu ls

If a password is requested, or an error results, the configuration is incorrect. Go through the steps in #9 again.

1   For the point of example I'll use a machine called toro, on which you have just installed and configured IMAGIC with MPI.

If you want to use more than one physical machine for parallel computing, you need to NFS mount toro's /usr/local/imagic directory on the other machines to the same mount point (or set a symbolic link). The user's working directory must also be present on all systems.

ssh must work on all machines as on toro, using batchmode keys with an empty passphrase. An alternative is using hostbased authentication, but that is outside the scope of this document.

1   If the backspace key doesn't work in IMAGIC, add this to the user's .cshrc or .bashrc file.

stty erase ^?

Important: This is NOT entered as <caret><question mark>, but rather as <control-V><backspace>. In the shell and most text editors this will appear as ^?.