Using LAM-MPI

The Message-Passing Interface, or MPI for short, is a library of routines for running jobs across multiple nodes in a Beowulf cluster.

More detailed tutorials are available from LAM-MPI.org.

If you're planning on doing anything serious with LAM-MPI, you should probably read the user manual, which includes information about writing, building, and running programs on a LAM-MPI cluster.

Before you can use the MPI, you must start the service running on the nodes you plan to use.

Starting LAM-MPI

Starting MPI requires you to have SSH keys set up and an SSH agent running so that you don't have to type passwords to connect to different machines. See our SSH tutorial for the details; once you've got SSH working, come back here and try the examples on this page.

Log into the dworkin and start an SSH agent process. Add your key, then run the recon command to check the nodes:

dworkin% recon -v node_list_file

where node_list_file is a file containing the names of the nodes you want to boot and a list of CPUs available on that host. For these systems, you can try setting cpu=2, as they have “hyperthreading” processors. I'm not sure whether hypterthreading gives us any increase in performance over nonhyperthreading with these systems when using MPI, but it's worth trying to see.

A file containing a list of all the possible nodes can be found in /etc/lam/lam-full.bhost on dworkin.

If recon succeeds, try running lamboot to really start the cluster:

dworkin% lamboot -v /etc/lam/lam-full.bhost

If all goes well, you should see something like

LAM 7.1.1/MPI 2 C++/ROMIO–Indiana University

n-1<15868> ssi:boot:base:linear: booting n0 (benedict.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n1 (brand.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n2 (caine.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n3 (corwin.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n4 (deirdre.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n5 (delwin.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n6 (dworkin.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n7 (fiona.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n8 (flora.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n9 (gerard.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n10 (julian.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n11 (martin.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n12 (merlin.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n13 (oberon.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n14 (random.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: booting n15 (sand.math.hmc.edu)
n-1<15868> ssi:boot:base:linear: finished

Listing Active Nodes

The lamnodes command will generate a list of nodes that are part of your LAM cluster. Output should look something like

XOXOX

Monitoring LAM Processes

The mpitask and mpimsg commands allow you to monitor different aspects of the LAM system. mpitask shows information about tasks running on the system, whereas mpimsg shows information about the messages being passed from node to node through the system.

“Cleaning” LAM

In between running processes with mpirun, you should run the lamclean program to “clean up” any leftover processes, memory uses, and so forth.

Ending Your LAM Session

To close down your LAM session on all nodes, run

dworkin% lamhalt -v

If lamhalt fails (because one or more nodes have crashed, for example), you may need to use the wipe tool to attempt to connect to each node from your original lamboot and shut down any LAM processes running on those nodes. The wipe takes a filename containing node names as its argument (just like lamboot).

Some Code Examples

The following code examples are taken from the LAM-MPI user manual.

C

Copy the following code into a file called hello.c:

#include <stdio.h>
#include "mpi.h"

int main(int argc, char *argv[]) {
    int myrank, mysize;
    MPI_Init(&argc,&argv);

    MPI_Comm_size(MPI_COMM_WORLD,&mysize);
    MPI_Comm_rank(MPI_COMM_WORLD,&myrank);

    printf("Hello World! I am %i of %i\n", myrank, mysize);

    MPI_Finalize();
}

Compile and link the program

dworkin% mpicc hello.c -o hello.mpi

Now run the program:

dworkin% mpirun -np 64 hello.mpi

(The -np flag specifies the number of nodes to use; -np n tells mpirun to run n copies of the program on the available nodes; see the manpage for more flags.)

If all goes well, you should see output similar to the following:

Hello World! I am 0 of 64
Hello World! I am 8 of 64
Hello World! I am 24 of 64
Hello World! I am 16 of 64
Hello World! I am 2 of 64
Hello World! I am 4 of 64
Hello World! I am 10 of 64
Hello World! I am 20 of 64
Hello World! I am 18 of 64
Hello World! I am 6 of 64
Hello World! I am 26 of 64
Hello World! I am 22 of 64
Hello World! I am 14 of 64
Hello World! I am 30 of 64
Hello World! I am 12 of 64
Hello World! I am 28 of 64
Hello World! I am 41 of 64
Hello World! I am 57 of 64
Hello World! I am 49 of 64
Hello World! I am 34 of 64
Hello World! I am 53 of 64
Hello World! I am 50 of 64
Hello World! I am 38 of 64
Hello World! I am 42 of 64
Hello World! I am 37 of 64
Hello World! I am 54 of 64
Hello World! I am 1 of 64
Hello World! I am 63 of 64
Hello World! I am 58 of 64
Hello World! I am 46 of 64
Hello World! I am 60 of 64
Hello World! I am 40 of 64
Hello World! I am 56 of 64
Hello World! I am 48 of 64
Hello World! I am 36 of 64
Hello World! I am 39 of 64
Hello World! I am 43 of 64
Hello World! I am 52 of 64
Hello World! I am 55 of 64
Hello World! I am 35 of 64
Hello World! I am 47 of 64
Hello World! I am 51 of 64
Hello World! I am 15 of 64
Hello World! I am 59 of 64
Hello World! I am 62 of 64
Hello World! I am 3 of 64
Hello World! I am 33 of 64
Hello World! I am 11 of 64
Hello World! I am 19 of 64
Hello World! I am 7 of 64
Hello World! I am 23 of 64
Hello World! I am 27 of 64
Hello World! I am 9 of 64
Hello World! I am 61 of 64
Hello World! I am 17 of 64
Hello World! I am 25 of 64
Hello World! I am 31 of 64
Hello World! I am 5 of 64
Hello World! I am 21 of 64
Hello World! I am 29 of 64
Hello World! I am 32 of 64
Hello World! I am 45 of 64
Hello World! I am 44 of 64
Hello World! I am 13 of 64

(The order that the nodes report back in may be different.)

C++

Copy the following code into a file called hello.cc:

#include <iostream>
#include <mpi.h>
using namespace std;

int main(int argc, char *argv[]) {
int rank, size;

MPI::Init(argc, argv);
rank = MPI::COMM_WORLD.Get_rank();
size = MPI::COMM_WORLD.Get_size();

cout << "Hello, world! I am " << rank << " of " << size << endl;

MPI::Finalize();
return 0;
}

Compile and link the program:

dworkinmpiCC hello.cc -o hello++.mpi
or
dworkinmpic++ hello.cc -o hello++.mpi

Run the program with

dworkin% mpirun -np 8 hello++.mpi

If all went well, output should be similar to (order may be different)

Hello World! I am 0 of 8
Hello World! I am 5 of 8
Hello World! I am 1 of 8
Hello World! I am 3 of 8
Hello World! I am 6 of 8
Hello World! I am 4 of 8
Hello World! I am 2 of 8
Hello World! I am 7 of 8

FORTRAN

This tutorial assumes basic knowledge of the UNIX environment. Remember that in fortran, all statements that should be executed must have 6 spaces before it (so 6 spaces then line of code, for every line of code).

Copy the following code to a file called hello.f:

      program main

      include 'mpif.h'

      integer myrank, mysize, rc

      call MPI_INIT( ierr )
      call MPI_COMM_SIZE(MPI_COMM_WORLD,mysize,ierr)
      call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,ierr)

      print *, 'Hello World! I am ', myrank, ' of ', mysize, '.'

      call MPI_FINALIZE(rc)

      stop
      end

Compile and link the program with

dworkin% mpif77 hello.f -o hellof.mpi

Run it with

dworkin% mpirun -np 8 hellof.mpi

You should see results similar to

Hello World! I am 0 of 8
Hello World! I am 5 of 8
Hello World! I am 1 of 8
Hello World! I am 3 of 8
Hello World! I am 6 of 8
Hello World! I am 4 of 8
Hello World! I am 2 of 8
Hello World! I am 7 of 8