Slaton Lipscomb

Slaton's Tips:
Compiling LAM/MPI For Linux With Portland Group Compilers

Last updated June 6, 2006
Comments/corrections welcome.

LAM/MPI is the preferred MPI implementation to use with SPIDER's experimental MPI version.

1.1   The LAM/MPI source can be downloaded from http://www.lam-mpi.org.

At the time of this writing, the current release is 7.1.2. Modify these instructions accordingly if you are compiling a different version.

1.2   Unzip/untar the source, and set environment variables for your desired compilers. LAM/MPI is written in C, C++ and Fortran, so you must define an environment variable for each language.

Here, I am using the bash shell and am compiling with Portland Group compilers pgcc (C), pgCC (C++), and pgf90 (Fortran 90). If you are using another shell, adjust accordingly.

Be sure your compiler /bin directory is in your shell's PATH.

$ export CC=pgcc
$ export CXX=pgCC
$ export F90=pgf90
$ export F77=pgf90
$ export FC=pgf90

1.3   Set FFLAGS, CFLAGS and CXXFLAGS to indicate any desired compiler flags for target architecture, optimization, etc. For example:

$ export CFLAGS="-tp k8-64 -O2"
$ export CXXFLAGS="-tp k8-64 -O2"
$ export FFLAGS="-tp k8-64 -O2"

Some personal experiences with compiler flags and the x86_64 platform:

  • LAM will not compile with the -Mipa=fast,inline flag.
  • LAM binaries segfault when compiled with the -fast flag.
  • Using the -mcmodel=medium flag on x86_64 platform will result in the libpgc.so library being linked to the LAM binaries. This may be a problem on clusters where the library is not available.

1.4   Configure the LAM/MPI source as follows:

$ cd lam-7.1.2
$ ./configure --prefix=/usr/local/lam/7.1.2/pgi --with-rsh="ssh -q" --with-trillium

Some notes:

  • We specifed the use of ssh for starting threads. If you are using a cluster on a private network you might prefer rsh. If so, omit the --with-rsh flag.
  • The -q option to ssh causes banners and diagnostic messages to be suppressed, however this is buggy in OpenSSH versions prior to 3.7.
  • If you do not use the XMPI or TotalView profiling tools, you may omit the --with-trillium flag.

1.5   Compile and install.

$ make
$ make install

If you are using Red Hat Linux 8.0 and get __bswap_32 errors during the compile, this is due to a glibc bug. It is easily fixed by following these instructions.

If this is a new installation, create a modulefile for setting up the LAM environment variables. It is assumed that the Modules system has previously been installed and configured.

2.1   The following environment variable can be defined, if you wish to override the compile-time option specified with --with-rsh in step 1.3 above.

setenv   LAMRSH   ssh -q

2.2 The following item is prepended to the PATH environment variable.

prepend-path   PATH   "/usr/local/lam/7.1.2/pgi/bin"

3.1   Before starting a MPI-SPIDER job, you will need to define which hosts are to be used by LAM/MPI to start jobs. This is more or less the same as the machines.LINUX file used by MPICH.

Create a file /usr/local/lam/7.1.2/pgi/etc/lam-bhost.def. It should like something like the following. Be sure to replace lnx-node01, lnx-node02, etc. with your own hostnames.

# Function:       - bhost.def 
# 
lnx-node01
lnx-node02
lnx-node03
lnx-node04
lnx-node05

3.2   Test your lam-bhost.def configuration by starting lam with the -v (verbose) option, and by running the recon and lamnodes commands.

$ lamboot -v
$ recon -v lam-bhost.def 
$ lamnodes
$ lamhalt

If all goes well, recon should give you a message like the following:

Woo hoo!

recon has completed successfully.  This means that you will most likely
be able to boot LAM successfully with the "lamboot" command (but this
is not a guarantee).  See the lamboot(1) manual page for more
information on the lamboot command.

TODO: Complete this section. Need info on generating SSH hostkeys & loading the LAM environment.

4.1   Load your LAM environment, either by loading the LAM module you created above in step 2, or by other means.

4.2   Start a LAM session and run some of the LAM tests.

$ cd examples/hello
$ mpicc -o hello hello.c
$ lamboot
$ mpirun -np [number of processors] C ./hello

4.3   When you are finished, shut down LAM.

$ lamhalt