Parallel Programming using Message Passing Interface (MPI)

(Part of HPC @ IU Workshop, Indiana University)

  Working on the AVIDD Cluster


  Message Passing Interface (MPI) - Simple Programming Examples


  MPI I/O - Parallel File Access


  Getting Ready for Graphical Applications from Compute Nodes


  Profiling Parallel Code using Intel Trace Analyzer/Collector (ITA/C)


  Debugging Parallel Code using TotalView


  Using MPICH (instead of LAM) to run MPI code (Appendix)


  More Documentation!

MPI - Introduction

Introduction

What will I learn in this tutorial?

You will learn about scientific parallel programming on the AVIDD Linux Cluster at Indiana University.
Previous


HPC @ IU Tutorial - Initial Setup

Initial Setup

Workshop training account on AVIDD

SSH onto AVIDD

Grab Interactive Compute Nodes using qsub -I

  • We will be using interactive nodes through out this workshop to run MPI jobs. Request two compute nodes (2 processors on each node) for 6 hours from PBS.
    Exercise: Do this! 
     [agopu@bh2 agopu]$ qsub -I -l nodes=2:ppn=2 -l walltime=6:00:00
    You can expect to see a message like this: 
     qsub: waiting for job 281988.aviss.avidd.iu.edu to start
 qsub: job 281988.aviss.avidd.iu.edu ready

 [agopu@bc56 agopu]$
    If you want to know what nodes have been allocated to your job, then you can find that information in a file pointed to, by $PBS_NODEFILE. 
     [agopu@bc81 agopu]$ cat $PBS_NODEFILE
 bc81
 bc53
 bc81
 bc53
    Note: Later in the workshop when you run example MPI code, you'll realize that each process' rank is determined by the order of the nodes in $PBS_NODEFILE. We will return to this later in the session.


HPC @ IU - Elementary MPI Programming Example

My First MPI Program

MPI API for SPMD programs

  • MPI stands for "message passing interface." API stands for "application programming interface."

  • MPI lets you write one single program (SP) that all processors execute together.

  • It also equips you with simple interprocessor communication functions for sharing multiple data (MD) among processors.

The "Hello, parallel worlds!" program


HPC @ IU Tutorial - Makefile

Tutorial Makefile

Makefile to compile your example programs

Couple of important points:

  • Make needs a tab, not spaces! If indeed you decided to build your own set of example programs and Makefiles, then after you copy and paste the above Makefile, you must change the spaces into tabs in the twelve or so indented lines. In your text editor backspace each of those lines to the left margin, then hit the tab key once. The make command requires that these command lines begin with exactly one tab; see the above listed tutorial or other documentation for more information.

  • Which compiler to use? If you are going to use the example C source codes from this tutorial, then you do not need to modify this makefile. If you code in C++ or Fortran 77 or Fortran 90, then you must change the CC variable in the following makefile to mpiCC, mpif77 or mpif90 respectively.

  • What is LDFLAGS defined for? In the latter half of this tutorial, you'll be using Intel Trace Analyzer to profile your parallel code. The libraries listed in LDFLAGS (-lVT and its dependencies) in the above Makefile cause profiling files to generated when you run your parallel code.



HPC @ IU - Basic Concepts in Parallel Programming

Basic Concepts in Parallel Programming

In this section, we will briefly look at parallel programming paradigms and the two popular types of architectures used by parallel computers.


Parallel Programming Paradigms

There are different parallel programming paradigms (the names are self-explanatory):

  • Data Parallel Paradigm

  • Message Passing Paradigm

  • Shared Memory Paradigm

  • Mixed Parallel Programming Paradigm

We will use a mixed parallel programming paradigm -- a combination of data parallel and message passing paradigms. We'll refer to it as "Single Program Multiple Data" or SPMD model in this tutorial. The reason it is called so:

  • Single Program (SP)

    • All processors will execute the same program, though they could be executing different sections of the code based on certain criteria. This will be explained in detail in later sections.

  • Multiple Data (MD)

    • Data will be decomposed into chunks, with each processor getting one or more of those chunks.

If you are interested in learning more about the paradigms mentioned above, Gustav Meglicki has some excellent notes explaining parallel programming paradigms in the I590 section of his website.


Hardware Architectures used by Parallel Computers

Distributed memory systems

Shared memory systems


Once again...

Check out "Low-down on AVIDD - Technical Details" subsection in the Primer to working on the AVIDD Linux Cluster guide, if you have not done so yet.




HPC @ IU - RoundRobin: Simple Message Passing Illustrated

RoundRobin - Simple Message Passing Illustrated

The roundRobin program

  • This is our second parallel program and it illustrates basic interprocessor communication.

  • Processor 0 composes a message (in this case, an integer array) and sends it to processor 1; Processor 1 receives the message and sends it on to 2; Processor 2 receives it and sends it to 3, and so forth. Finally the last processor sends it back to 0 who then compares it with the original message to check if the message passing worked correctly.


HPC @ IU - Simple MPI functions explained

The RoundRobin program -- Explanation of MPI function calls

Recall from a previous section...

  • In a distributed-memory machine each processor has access only to its own memory. There is no common or shared memory that processors can access.

  • Therefore, processors can interact only by explicitly transmitting packages of data between each other. In the roundRobin program this interaction was accomplised using MPI function calls MPI_Send () , MPI_Recv () and MPI_Bcast () ; these functions are explained below.

Simple MPI Sends/Receives/Broadcasts

  • Before a processor can send information, it has to pack the relevant data into a "send buffer" located in its own memory.

  • Similary, when a processor receives data from another process, it has to unpack the data sitting in its own "receive buffer".

  • In C the MPI function to send data is: 

     int MPI_Send( void*        sendBuf,
	       int          count,
	       MPI_Datatype datatype,
	       int          destinationRank,
	       int          tag,
	       MPI_Comm     comm )

    • sendBuf is the send buffer or array packed with the data to be sent.

    • count is the length of the sendBuf array.

    • datatype is the datatype of the sendBuf array, typically MPI_INT, MPI_FLOAT , or MPI_DOUBLE.

    • destinationRank is the rank of the intended receiving processor.

    • tag is an identifier that may be used to distinguish among multiple transmissions between the same two processors.

    • comm is the communication domain. In this workshop, we will always specify the default communicator MPI_COMM_WORLD which consists of all processors.

      Note: See Gustav Meglicki's Parallel Diffusion example if you are interested in learning about constructing your own communicator.

  • Be aware that a processor executing an MPI_Send will "block", i.e. it will not proceed executing subsequent code until the send is completed.

  • The MPI receive function in C looks as shown in the roundRobin program is: 

     int MPI_Recv( void*        recvBuf,
	       int          count,
	       MPI_Datatype datatype,
	       int          sourceRank,
	       int          tag,
	       MPI_Comm     comm,
	       MPI_Status   *status )

    • recvBuf is the receive buffer or array into which the MPI API will place the transmitted data.

    • count is the length of the recvBuf array.

    • datatype is the datatype of the recvBuf array, typically MPI_INT, MPI_FLOAT , or MPI_DOUBLE .

    • sourceRank is the rank of the sending processor.

    • tag is an identifier that may be used to distinguish among multiple transmissions between the same two processors.

    • comm is the communication domain.

    • status is a pointer to information about the state of the receive. In this workshop, we will not make use of this status information.

  • Similar to what was explained in the case of MPI_Send () , a processor executing an MPI_Recv will "block", i.e. it will not proceed executing subsequent code, until the receive is completed.

  • And finally, if a process is expected to send some information to all the other processes, then we can use the MPI_Bcast () command. 

     int MPI_Bcast ( void* buffer,
                 int count,
                 MPI_Datatype datatype,
                 int root,
                 MPI_Comm comm )

    The parameters to the above MPI function are similar to the ones in MPI_Send () and MPI_Recv () functions.

  • MPI_Comm_Size () tells the participating processes about the total number of processes in this communicator. This is also something they need to know. For example, if the program is to work on a very long array, the processes will have to know how many of them there are in the pool in order to work out for themselves which portion of the array they should work on.

  • Misc: MPI has a function called MPI_Wtime () that can be used to time your program (or sections of it). It measures wall-clock time down to a microsecond, very accurate indeed. Of course using a tool like Intel Trace Analyzer would give you an even better idea about how your program is performing.



HPC @ IU - Data Decomposition

Data Decomposition

What is Data Decomposition?

  • The roundRobin program was meant only to illustrate basic interprocessor communication, so we did not really focus on data. Processor 0 generated the data. Then each processor received and sent the data back and forth.

  • But in real-world programming, it's never that simple! In most cases, data will have to be shared among the processors so that each cares about only its own portion of the data. "Data decomposition" refers to the sharing data between parallel processes in the best possible (i.e. most efficient) fashion. No process should step on another process' data either.

Parallelism (vs) InterProcessor Communication (IPC) Overhead

It is very important to figure out how suitable an algorithm/piece of code is to parallelization; you should guard against a tendency to parallelize code without enough consideration to the InterProcessor Communication, which is very expensive on distributed memory parallel systems. In general, the following guidelines are recommended:

  • More parallelism is good; maximum amount of CPU time must be spent by each processor on computation (and not message passing)

  • Least interprocessor communication is desirable.

  • Objective: How can we decompose data in our SPMD model in order to maximize parallelization while minimizing interprocessor communication?

Thinking of Parallelization? First write the serial program...

In most cases, when you wish to parallelize any algorithm, then it might be worthwhile to start with the serial version of the code. Then you could follow the steps listed below:




HPC @ IU - Computing PI by Elementary Numerical Integration

A Naturally Parallelizable Computation

Problem statement

  • In this example we would like to compute PI using the following integration:

  • Consider the figure below. To do the integration, we'll divide the domain [0,1] into intervals, compute the integral at the mid point of each interval, then multiply those values by the width of each rectangle and finally add up the areas of the all the rectangles to get an approximate value of PI.

  • In the following sections, we'll show how you could write serial code to perform the above integration and then how you could go about parallelizing the code.


HPC @ IU - Computing PI using Elementary Numerical Integration: Serial version

Computing PI using Serial Code

Program serialPi

  • The serial version of our program to compute PI, written in C, is displayed below: 

    #include <stdio.h>
#include <stdlib.h>
#include <math.h>

const double pi = 3.1415926535897932385;

int main( int argc, char *argv[] ) {
  /*  Arguments required for executing serialPi:     */
  /*  0                         -> serialPi          */
  /*  1                         -> numberOfIntervals */

  int numberOfIntervals, interval;
  double intervalWidth, intervalMidPoint, area = 0.0;

  /*  The number of intervals is a command line argument. */
  numberOfIntervals = atoi( argv[1] );

  /*  Compute the interval width. */
  intervalWidth = 1.0 / numberOfIntervals;

  /*  Now compute the area. */
  for ( interval = 0; interval < numberOfIntervals; interval++ ) {
      intervalMidPoint = (interval + 0.5) * intervalWidth;
      area += 4.0 / ( 1.0 + intervalMidPoint*intervalMidPoint );
  }
  area *= intervalWidth;

  /*  Print the results. */
  printf( "The computed value of the integral is %.15f\n", area );
  printf( "The exact value of the integral is    %.15f\n", pi );

  return 0;
}

    Note: If you did not copy example programs from HPC's account in the initial section of this workshop, then copy the above source code as serialPi.c to your SerialPi directory.

  • Exercise: Do this!

    To compile the serialPi program, at your shell prompt in the SerialPi directory, do: 

     [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/SerialPi/
 [agopu@bh2 SerialPi]$ icc -o serialPi serialPi.c

    Then to run the program on interactive nodes you got through qsub -I (bcXX), do: 

     [agopu@bc81 agopu]$ cd ~/MPI_Tutorial/SerialPi/
 [agopu@bc81 SerialPi]$ ./serialPi 20000
  The computed value of the integral is 3.141592654423135
  The exact value of the integral is    3.141592653589793

    Note: Since this is not a MPI program, we've not used the Makefile. We could have if we'd wanted to, but we would have needed to change the compiler to icc or gcc.

  • Since the command line argument used above is 20000, the domain would have been divided into 20,000 intervals.



HPC @ IU - Computing PI using Elementary Numerical Integration: Parallel version

Computing PI using Parallel Code

Program parallelPi - Actual Parallelization of serialPi


HPC @ IU - Solving Diffusion Equation

The 2-D Diffusion Equation

The equation

  • The next example is more interesting than parallelPi. Our objective is to solve a two-dimensional diffusion PDE:

    where u(x,y,t) is the temperature of the surface being modeled, and D is the so-called diffusion coefficient.

  • Let us choose as the surface a rectangular plate:

Discretized Numerical Solution

  • In this workshop, we will not worry about the exact solution (an infinite sum). Instead we will look at an approximate solution using a discrete grid. We don't bother about the real math because our objective is to learn parallel computing. In any case, the exact solution to the above equation is shown at the end of this page.

  • Going back to the rectangular place shown above, let us superimpose a computational grid upon the plate we are modeling:

    which shows the numbering of the x and y grid indices i and j, respectively. The number of interior grid points in the x-direction is nni, in the y-direction nnj. The boundaries are i = 0, i = nni + 1, j = 0, and j = nnj + 1.

  • Each point in the grid can be computed using the following equation: (based on a finite-difference equation):

    • Let us call the expressions outside of the parentheses mu, c and nu. Using temperature at time step n, the temperature at time step n + 1 can be computed.

  • For the above finite-difference representation of the two-dimensional diffusion equation, we will have to consider a 5-point stencil that must be computed to obtain u_ij at the next time step. The stencil considers the closest neighbors (diagonal points are not considered because they are not the closest).

    Later when considering parallelization we will be especially interested in the stencil. The stencil is the set of grid points needed to compute the single ij grid point at the next time step.

Exact Solution



HPC @ IU - Solving Diffusion - Program serialDiffusion

Serial Numerical Solution of the Diffusion Equation

Program: serialDiffusion

  • As we'd mentioned earlier, if you are beginning a parallel project from scratch, writing the serial code is typically the first step.

  • In C, our serialDiffusion program is shown below: 

    #include <stdio.h>
#include <stdlib.h>
#include <math.h>

const double pi = 3.1415926535897932385;

int main( int argc, char *argv[] ) {
  /*  Arguments required for executing argv[0]:      */
  /*  0                         -> serialDiffusion   */
  /*  1                         -> nni               */
  /*  2                         -> nnj               */
  /*  3                         -> numTimeSteps      */


  double **uo; /* solution at previous time step */
  double **u;  /* solution at current time step  */

  double mu, nu, c;    /* parameters of the FDE                              */
  double ell1,ell2;    /* x,y grid dimensions, 0 <= x <= ell1, etc.          */
  double dx,dy;        /* x,y grid spacing, or spacial step size             */
  int    nni,nnj;      /* x,y number of interior grid points; excl. bdy. pts.*/
  int    numTimeSteps; /* when to end simulation                             */
  double dt;           /* time step size                                     */
  double dcoeff;       /* diffusion coefficient D                            */
  double u_0;          /* uniform initial conditions                         */
  double actual;       /* the true solution                                  */

  int    i,j,m,n,timeStep;

  /*  Get the number of interior grid points and  */
  /*  number of time steps from the command line. */
  nni = atoi( argv[1] );
  nnj = atoi( argv[2] );
  numTimeSteps = atoi( argv[3] );

  /*  Allocate memory for the solution arrays; include boundaries. */
  /*  As declared above, u is a pointer to a pointer to a double.  */
  /*  Allocate memory so u points to the first element of an       */
  /*  (nni+2)-long array of pointers to doubles. Likewise for uo.  */
  u  = (double **) malloc( (size_t) ( (nni+2) * sizeof(double*) ) );
  uo = (double **) malloc( (size_t) ( (nni+2) * sizeof(double*) ) );

  /*  u[0] is a pointer to a double.                               */
  /*  Allocate memory so u[0] points to the first element of an    */
  /*  (nni+2)*(nnj+2)-long array of doubles. Likewise for uo[0].   */
  u[0]  = (double *) malloc( (size_t) ( ((nni+2)*(nnj+2)) * sizeof(double) ) );
  uo[0] = (double *) malloc( (size_t) ( ((nni+2)*(nnj+2)) * sizeof(double) ) );

  /*  Now set pointer u[1] to point to *(u[0] + (nnj+2)).          */
  /*  And set pointer u[2] to point to *(u[0] + 2*(nnj+2))         */
  /*      or *(u[1] + (nnj+2)).                                    */
  /*  And so forth. Likewise for uo[1], uo[2], etc.                */
  for ( i = 1; i < nni+2; i++ )  {
      u[i]  = u[i-1] + (nnj+2);
      uo[i] = uo[i-1] + (nnj+2);
  }

  /*  We have now dynamically allocated memory for the arrays      */
  /*  u[nni+2][nnj+2] and uo[nni+2][nnj+2] which are large         */
  /*  enough to include the boundary points.                       */

  /*  Set the diffusion coefficient, grid dimensions, */
  /*  and the uniform initial conditions.             */
  dcoeff = 1.0;
  ell1 = 7.0;
  ell2 = 1.0;
  u_0 = 1.0;

  /*  Compute dx and dy, and set dt so that the Courant condition */
  /*  is satisfied. */
  dx = ell1 / ( nni + 1 );
  dy = ell2 / ( nnj + 1 );
  dt = 0.49 / ( dcoeff*( 1.0/(dx*dx) + 1.0/(dy*dy) ) );

  /*  Set uniform initial conditions. */
  for ( i = 1; i <= nni; i++ )   {
      for ( j = 1; j <= nnj; j++ )	{
	  u[i][j] = u_0;
      }
  }

  /*  Set boundary conditions. */
  /*  Assume Dirichlet boundary conditions equal to 0. */
  /*  First the y = 0 boundary. */
  for ( i = 0; i <= nni+1; i++ )    {
      u[i][0] = 0.0;
  }
  /*  Second the y = ell2 boundary. */
  for ( i = 0; i <= nni+1; i++ )  {
      u[i][nnj+1] = 0.0;
  }
  /*  Third the x = 0 boundary. */
  for ( j = 1; j <= nnj; j++ )    {
      u[0][j] = 0.0;
  }
  /*  Finally the x = ell1 boundary. */
  for ( j = 1; j <= nnj; j++ )    {
      u[nni+1][j] = 0.0;
  }


  /*  Solve the 2-D diffusion equation by the explicit FTCS method. */

  /*  Pre-compute the FDE coefficients. */
  mu = dcoeff*dt / (dx*dx);
  nu = dcoeff*dt / (dy*dy);
  c  = 1.0 - 2.0*( mu + nu );

  for ( timeStep = 1; timeStep <= numTimeSteps; timeStep++ )    {
      /*  Update previous time step data. */
      for ( i = 0; i <= nni+1; i++ )	{
	  for ( j = 0; j <= nnj+1; j++ )	    {
	      uo[i][j] = u[i][j];
	  }
      }

      /*  Now compute the stencil for each i,j pair. */
      for ( i = 1; i <= nni; i++ )	{
	  for ( j = 1; j <= nnj; j++ )	    {
	      u[i][j] = mu*(uo[i+1][j]+uo[i-1][j]) + c*uo[i][j]
		+ nu*(uo[i][j+1]+uo[i][j-1]);
	  }
      }
      /*  Now proceed to the next time step. */
  }

  /*  Okay, the numerical computation of u is completed. */
  /*  Let's look now at a specific grid point.           */
  i = (int) nni/8;
  j = 10;
  /*  Compute the actual temperature at this point.  */
  actual = 0.0;
  for ( m = 1; m < 1000; m += 2 )    {
      for ( n = 1; n < 1000; n += 2 )	{
	  actual += 1.0/(m*n) * exp( - dcoeff * pi*pi * 
				     ( m*m/(ell1*ell1) + n*n/(ell2*ell2) ) *
				     numTimeSteps*dt )
	    * sin( m*pi*i*dx/ell1 ) * sin( n*pi*j*dy/ell2 );
      }
  }
  actual *= 16.0*u_0 / (pi*pi);
  /*  Compare the actual with the numerical result at this point. */
  printf( "After %d time steps some results are:\n", numTimeSteps );
  printf( "actual   u[%d][%d] = %f\n", i, j, actual );
  printf( "computed u[%d][%d] = %f\n", i, j, u[i][j] );

  /*  And let's look at another specific grid point. */
  i = (int) nni/4;
  /*  Compute the actual temperature at this point.  */
  actual = 0.0;
  for ( m = 1; m < 1000; m += 2 )    {
      for ( n = 1; n < 1000; n += 2 )	{
	  actual += 1.0/(m*n) * exp( - dcoeff * pi*pi * 
				     ( m*m/(ell1*ell1) + n*n/(ell2*ell2) ) *
				     numTimeSteps*dt )
	    * sin( m*pi*i*dx/ell1 ) * sin( n*pi*j*dy/ell2 );
      }
  }
  actual *= 16.0*u_0 / (pi*pi);
  /*  Compare the actual with the numerical result at this point. */
  printf( "After %d time steps some results are:\n", numTimeSteps );
  printf( "actual   u[%d][%d] = %f\n", i, j, actual );
  printf( "computed u[%d][%d] = %f\n", i, j, u[i][j] );

  return 0;
}

    Note: If you didn't copy source files from HPC's tutorial directory, then copy the above source code into a file called serialDiffusion.c inside your ~/MPI_Tutorial/SerialDiffusion/ directory.

  • Exercise: Do this!

    To compile the code, change to the ~/MPI_Tutorial/SerialDiffusion/ directory on the head node (bhX) and then type [since this is not a MPI program, we are not going to use the Makefile we've been using all along]: 

     [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/SerialDiffusion/
 [agopu@bh2 SerialDiffusion]$ icc serialDiffusion.c -o serialDiffusion -lm
    Note: Our serialDiffusion program uses functions like sin () and exp (). That's whay we've used the -lm flag in the above call to icc ...to link with the Math library.

  • To run this program on a interactive node you got through qsub -I (bcXX), do: 

     [agopu@bc81 agopu]$ cd ~/MPI_Tutorial/SerialDiffusion/
 [agopu@bc81 SerialDiffusion]$ ./serialDiffusion 8000 1000 100

The results of serialDiffusion

  • The output of serialDiffusion 8000 1000 100 is: 

      After 100 time steps some results are:
  actual   u[1000][10] = 0.874806
  computed u[1000][10] = 0.873978
  After 100 time steps some results are:
  actual   u[2000][10] = 0.874775
  computed u[2000][10] = 0.873978

    The computed and actual values differ by one part in a thousand, which is not too bad considering we used 8000 and 1000 as parameters (not huge values).



HPC @ IU - Solving Diffusion - Parallelizing Serial Program

Parallelizing the Numerical Solution of the Diffusion Equation

Domain Decomposition

  • In our problem the easiest way to allocate grid points among the processese is to decompose along the x-axis:

  • At each time step, each process computes only it's portion of the grid.

Ghost Points

  • However, when a process computes a grid point that lies adjacent to the edge of its portion, one of the stencil points is actually owned by the neighboring process. This point is referred to as the process' "ghost points".

  • A process never computes the ghost points by itself; these points are computed during the prior time step by neighboring processes.

  • To compute the stencil at any given time step, each processor must receive values (of the grid point) at its ghost points from its neighbors at the conclusion of the previous time step. Thus, processes must share their ghost points between themselves after each time step, i.e. interprocessor communication is required at the conclusion of each time step.

  • A process' ghost points are u[0][j], which are received from its left neighbor's u[mynni][j], and u[mynni + 1][j], which are received from its right neighbor's u[1][j], for all j.

Interprocessor Exchange of Ghost Points

  • We will use the MPI_Send () and MPI_Recv () functions to transfer ghost points between the processors. These are blocking functions; they do not return until the communication is complete (in most installations of MPI). So we will need to make sure that a deadlock situation does not arise.

  • A deadlock arises in a perfectly symmetric situation when, for instance, two processes are sending to each other, and each process' call to MPI_Send () does not return, something which needs to happen for those processes to call their respective MPI_Recv () calls!

    • It's better to program asymmetrically, whereby one process first calls MPI_Send () and its respective receiving process calls MPI_Recv ().

  • To prevent deadlocks we'll choose odd-numbered processes to do MPI_Send () first and MPI_Recv () second, while even-numbered processes will do MPI_Recv () first and MPI_Send () second.



HPC @ IU - Solving Diffusion - Program parallelDiffusion

Parallelized Solution for Diffusion Equation

Program parallelDiffusion.c


HPC @ IU - Simple Illustration of MPI I/O and Minimal Error Handling

Simple MPI I/O and Error Handling

In this section, we will delve into few aspects of MPI we have not looked at before. You'll need to pay careful attention since it introduces multiple (and up to some extent unrelated/tangentially related) concepts.

  • We will see how MPI I/O can be used by multiple processes to read and write to files. All processes will read from/write to the same file using different offsets.

    Note: MPI I/O can only be used with parallel file-systems like GPFS. It is not designed to work with NFS and the like. Refer the MPI distribution's documentation to check what file-systems are supported.

  • Within the program illustrating MPI I/O (shown below), we also introduce very basic error handling mechanisms.

    Important: Do note that, error handling is especially important when you are dealing with files. For example, if you open a file for writing and encounter "write" errors, then it is your responsibility to close the file (handle) and also unlink (delete) it before your program terminates. Refer to the I/O section of the MPI-2 specification.


Program: mpi_io.c


The mpi_io Program Explained

Now that you've seen the program work (hopefully!), let us see how the code worked and what the various MPI functions mean...

  • The MPI_File_open () function opens a file identified by filename on all processes in the MPI_COMM_WORLD communicator. This is a collective function meaning that all processes must do it together and the values of all parameters passed to it must be identical on all processes too.

    The third parameter specifies how the file should be opened, e.g., for writing, reading or both, and whether it should be created if it doesn't exist. In our example, the input file is opened in read-only mode. All file-open modes are listed on the MPI-2 specification for MPI_File_open (). The options can be combined with the boolean ``or'', |, operator.

    The fourth parameter can be used to give the operating system additional hints about how and where the file should be opened - too advanced, so let us not worry about it.

  • The MPI_File_get_size () function gives the file size, which will be used later on to determine the offset for each process.

  • The MPI_File_seek () function points to the position in the file where each process will start reading data. The seek can be performed in one of three ways: MPI_SEEK_SET (the pointer is set exactly to my_offset), MPI_SEEK_CUR (the pointer is advanced by my_offset from its current position), MPI_SEEK_END (the pointer is advanced by my_offset from the end of the file). In general my_offset can be positive and negative, i.e., you can use it to move in both directions within the file.

  • The MPI_File_read () function reads data into the buffer specified as the second parameter. The size to be read is defined by the third parameter.

  • The MPI_File_write_at () function will write data from the buffer, specified by the third parameter, into a specific position on the file specified by the second parameter.

  • The MPI_File_close function closes the file opened by the function MPI_File_open.

  • The MPI_Allreduce () function is similar to MPI_Reduce () but all processes do the reduction and know the result of the operation (rather than just the master process knowing the result)

  • The MPI_Abort () function terminates all the MPI processes associated with the communicator specified. It also allows you to specify a return error code.

    For Kicks

  • Have you spotted a bug in the above shown code?

    Hint: Try running the code on a different input file while using the same output filename. What do you see? Is there a bug? If yes, what is it???

    More hints: Check out the various file open modes in the MPI_File_open () specification while paying special attention to MPI_MODE_EXCL; you'll need to do "ORing" of modes.

  • For more comprehensive information on MPI I/O and (associated) error handling, refer to the MPI  I/O section of Gustav Meglicki's I590 notes.



HPC @ IU Tutorial - Tweaking security settings to allow X-traffic from AVIDD

Tweaking Security Settings

Change security settings to allow X traffic from AVIDD (For TotalView and Intel Trace Analyzer)



HPC @ IU Tutorial - Hoops to jump on Linux (firewall setting, etc.) to open graphical applications from a Unix based server (specifically AVIDD)

Hoops to jump on Linux to open graphical applications from AVIDD

If you are interested in opening GUI applications from AVIDD compute nodes on a Windows workstation then go to the How do I run AVIDD-based graphical applications on my Windows computer? page.

  • Tweak IPTables Rules on your workstation

    • Wait! Skip this part if you are not running a firewall on your Linux box!

    • If you are indeed running a firewall on your Linux box, then you will need to modify your firewall rules on a Linux workstation to allow X-traffic from AVIDD's firewall node bh3.

      Note: The following information is provided for your convenience; there is no guarantee it will enable you to open X applications from AVIDD or any other *n*x based server (ultimately your other rules also have to be taken into consideration)

    • As root , add the rules shown below (check to see if you already have the last two rules): 

       $ /sbin/iptables -A INPUT -i eth0 -p tcp -s 149.165.228.0/255.255.255.0 -d <your_IP_address> \
   --dport 6000 -m state --state NEW -j ACCEPT
 $ /sbin/iptables -A INPUT -i eth0 -p tcp -s 149.165.229.0/255.255.255.0 -d <your_IP_address> \
   --dport 6000 -m state --state NEW -j ACCEPT  	    
 $ /sbin/iptables -A INPUT -i eth0  -m state --state EST,REL -j ACCEPT
 $ /sbin/iptables -A OUTPUT -m state --state EST,REL -j ACCEPT

      All you are doing with the above rules is: allowing NEW packets IN from AVIDD compute nodes (both Bloomington and Indy) into your workstation on TCP ports 6000 as well as established and related packets.

    • If the above rules work, then you can go ahead and save them to your default IP tables ruleset, so they will be applied every time your computer starts!

  • Append Xhost list on your workstation

    • On your workstation you will need to allow X packets from AVIDD compute nodes. To accomplish this, you may want to use the modify-xhost-for-avidd-x-traffic.pl script which is listed below for your convenience.
    • Exercise: Do this!
      Right-click on the above link and save the script to your workstation (or you could copy and paste above text if you wish). Let us assume you call it modify-xhost-for-avidd-x-traffic.pl; Run the script by doing: 
       $ perl modify-xhost-for-avidd-x-traffic.pl b add
      You should see something like: 
        149.165.228.1 being added to access control list
  149.165.228.2 being added to access control list
  . . .
    • If you are using the Indy cluster nodes, then use i instead of b as the first parameter. I.e. do "modify-xhost-for-avidd-x-traffic.pl i add".
    • Once you are done with using graphical applications, use remove as the second parameter to clear out the xhost list back to its original state. I.e. do "modify-xhost-for-avidd-x-traffic.pl b remove".

  • Find your workstation's IP address (assuming you are using the Ethernet interface eth0):

    • On Linux:
       [agopu@peart agopu]$ /sbin/ifconfig eth0 | grep "inet addr" | cut -f2 -d: | cut -f1 -d " " | head -1
    • On Sun Solaris: 
       bash-2.04$ /usr/sbin/ifconfig -a | grep -v "127.0.0.1" | grep "inet" | cut -f2 -d" "
      If you are unable to access ifconfig, then you could use the fully qualified hostname instead. You can get it by doing: 
       bash-2.04$ cat /etc/hosts | grep -v "#" | grep -v "localhost" | cut -f3 | head -1
    You'll be using the result of the above command (whichever was applicable) in the next step.

  • Set the DISPLAY environment variable on the compute node

    • You will need to finally set a DISPLAY environment variable on the compute node that you got using qsub -I; it should point to your workstation's IP address and display unit. Switch to the compute node terminal and do (replace the xx.xx.xx.xx with the real IP address.): 
       [agopu@bc81 agopu]$ export DISPLAY=xxx.xxx.xxx.xxx:0.0


  • For example

    1. On my laptop running Linux: 

       [agopu@laptop agopu]$ perl modify-xhost-for-avidd-x-traffic.pl b add
  149.165.228.1 being added to access control list
  149.165.228.2 being added to access control list
   . . .
  149.165.228.96 being added to access control list

 [agopu@laptop agopu]$ /sbin/ifconfig eth0 | grep "inet addr" | cut -f2 -d: | cut -f1 -d " " | head -1
   156.56.18.241

    2. Then on AVIDD's compute node: 

        [agopu@bc81 agopu]$ export DISPLAY=156.56.18.241:0.0

  • Note: If you are using wireless and VPN (for example, on your laptop) then you'll need to find that interface's IP address. Let us know if you have trouble doing this.

If you are interested in opening GUI applications from AVIDD compute nodes on a Windows workstation then go to the How do I run AVIDD-based graphical applications on my Windows computer? page.
Previous Next


HPC @ IU Tutorial - Using Windows and XLiveCD (especially to open graphical applications from a Unix based server)

Using Windows and XLiveCD

If you are interested in opening GUI applications on a Linux workstation then go to the Tweaking Linux firewall... page.

How do I use XLiveCD on my Windows (XP) computer?

  • Download XLiveCD and burn the ISO image onto a recordable CD.

  • Assuming the CD is inserted into your CD drive, wait to see if the CD auto-loads. If it does not, then go to My Computer or Windows Explorer and double click on the CD rom drive. Then double-click on "X.bat". This should open a command window and run some Cygwin stuff (might take a while).

  • Once the previous step completes, you should see a Xterm window. If you don't see one, then look for a "New Xterm" shortcut on your Desktop and double-click on it.

    Note: At this stage, the Windows XP firewall might prompt you about blocking/unblocking Xwin.exe. You can select "Unblock" if you are comfortable with doing that - see the Tweak Windows XP SP2 firewall section below for more information.

  • To login to AVIDD without default X-forwarding, do: 

     Desktop> ssh -x -l hpctrnXX avidd-b.iu.edu
    (Replace XX with your training account #; Once you finish this workshop replace the whole login account name with your own AVIDD account)

  • Note:

    If you do not expect to use an graphical applications like TotalView or Intel Trace Analyzer, then you can go back to where you came from...most probably the Initial Setup: Login to AVIDD webpage or the Tweaking Security Settings webpage.

How do I run AVIDD-based graphical applications on my Windows computer?

If you are interested in opening GUI applications on a Linux workstation then go to the Tweaking Linux firewall... page.
Previous Next


HPC @ IU - Tracing the Round-Robin SPMD Example

Tracing and Profiling roundRobin using Intel Trace Collector/Analyzer (ITC/A)

Initial ITA timeline and whole trace

  • The basic elements of the initial timeline are:

    • The horizontal axis is time.

    • Each horizontal bar corresponds to a process or a set of processes. In our case, since we were using two nodes and two processors in each of those nodes, ITA merges the timelines on a per node basis. At the left margin they are clearly listed - notice how processes 0 and 2, which ran on node bc43, are represented on the second bar. Soon, we'll show how it's possible to look at each process in finer detail.

    • Red segments show execution of MPI functions. Green segments show execution of the routines of the program. Neither the overhead imposed by the ITC/A tracing application itself nor the messages exchanged between processors is not shown in this initial timeline.

  • Whole Trace :

    Exercise: Do this!

    Next, right-click on the initial timeline frame and select Load -> Whole Trace . This will open up another summary chart - you can see how much time MPI calls took and so forth for the entire application. Shown below is an example of what you could expect to see for the roundRobin program:


Detailed ITA timeline

  • Exercise: Do this!

    Close the initial timeline and the summary chart windows. Then pull down the Global Displays menu and select Timeline . This will open up a more detailed timeline than what you saw initially.

  • Similar to what was explained above in the case of the initial timeline, the basic elements of the detailed timeline are:

    • The horizontal axis is time.

    • Each horizontal bar corresponds to an individual process.

    • Once again, red segments show execution of MPI functions. Green segments show execution of the routines of the program. The overhead imposed by the ITC/A tracing application itself is not shown in this timeline, since it's very negligible. The MPI messages exchanged between processors is shown as solid black lines.

    • Right-click on the detailed timeline window and select Component -> Parallelism Display . An additional section is added to the timeline showing how effectively parallel our program was at various times - it's obvious that till the very end, our program's parallelism was not so good since it was waiting for a blocking call (Send/Receive) to complete.

  • Lets take a closer look at processor 1. The initial green segment is when processor 1 is preparing the message. Then processor 1 calls the MPI_Send function to transmit this message to processor 2. The MPI_Send function is a "blocking" function; this means that processor 1 cannot proceed until the MPI_Send is completed. Once the MPI_Send completes, processor 1 executes a few more lines of the program as can be seen by the thin green line following the MPI_Send. Then processor 1 calls the MPI_Recv function to receive the message back from processor 0. Note that MPI_Recv is also a blocking function. Once processor 1 receives the message, it is free to complete the program (final green stripe) and terminate.

  • The MPI API does make available non-blocking send and receive functions, but we will not consider these in this beginner's tutorial.

Individual Processor activity - ITA Activity Chart

  • Exercise: Do this!

    • To analyze the time spent on various activities by roundRobin in a different format (a bit more detail) pull down the Global Displays menu and select Activity Chart . You now see histograms of the time spent in each activity by each processor.

    • If you prefer pie-charts instead of the histograms, then you can right-click on this activity display, select Mode -> Histogram . The same data is displayed as a bunch of pie-charts this time around.

    • Next, right-click on the activity display, and click on Select -> All Symbols . Now the histograms show finer detail by comparing all states of the program (Some of the states are negligible / nonexistent!). You will learn later how to use Intel Trace Collector/Analyzer's instrumentation to define additional program states in order to increase your resolution of program activity.

  • A sample ITA activity chart associated with the trace file roundRobin.stf is shown below:

  • The histogram view (Activity Chart) provides an overall profile of how each processor spends its time. The bars are in order from left to right as is the function legend from top to bottom. Because in this elementary roundRobin example there is very little computing, the processors spend most of their time executing MPI function calls. Only a small proportion of time is spent executing the application itself (User_Code).



HPC @ IU - Profiling the parallelPi Example

Profiling parallelPi

The Intel Trace Analyzer (ITA) timeline

Processor activity

  • Once again, follow guidelines provided in Individual Processor activity - ITA Activity Chart to view parallelPi's activity.

  • A sample ITA activity chart associated with the trace file parallelPi.stf is shown below:

  • This histogram view provides an overall profile of how each processor spends its time. The bars are in order from left to right as is the function legend from top to bottom. You can immediately see that parallelPi is not an effective parallel program! Far too much of each processor's time is spent on MPI function calls and too little on actually computing and outputting the integral in the application itself (User_Code).



HPC @ IU - More on Tracing the parallelPi Example

More on Tracing parallelPi

Using Intel Trace Collector (ITC) Instrumentation

  • For the preceeding traces of roundRobin and parallelPi, the source code was used unmodified. All that was needed to use ITA was to make sure the linker accessed the libVT.a library at compile and linking time. Although this simple approach is helpful, it does not make use of the powerful tracing instrumentation capabilities of ITC.

  • The basic instrumentation of the ITC API provides the user with several routines that control profiling and record application-specific activities. In addition to profiling of the MPI function calls, this instrumentation permits profiling of loops, user function calls, and other portions of the user's source code.

  • Two simple routines need to be inserted into the user's source code to identify locations that need to be traced. These routines are: 

     int VT_begin (int code)
      
 /* Code to be traced goes in between */
      
 int VT_end (int code)
    Here the integer code is a user-defined identifier for a particular state and activity. The integer return value is negative in case of an error.

  • The user must specify the states and activities, mentioned above, with: 

     int VT_symdef(int code, char *state, char *activity)
    Again the integer return value is negative in case of an error.

  • The state character string names a specific loop or function, or any other clearly identified state of the executing program. The activity character string refers to the activity class of which the specific state is a member of. For instance, a specific call to printf (to print results) may be given the state name print-results and the activity name I/O.

  • The ITC API's predefined activities are:

    • Application
    • MPI
    • VT_API
    • Highlight and
    • Idle

    Some of the predefined states and the activity class, they belong to, are shown below:

    State Activity class
    Various MPI function callsMPI
    User_CodeApplication
    TRACE_ONVT_API

    We will not discuss the Highlight and Idle activity classes in this tutorial.

Adding ITC Instrumentation to the parallelPi source code

  • The modified parallelPi.c, with ITC instrumentation, is as follows. Note the macro USE_VT which determines whether or not the compilation of parallelPiVt.c includes the Intel Trace Collector (a.k.a VampirTrace) instrumentation. 

    
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <mpi.h>

#define USE_VT 1  /* 1 to call Intel Trace Collector API, 0 otherwise */

#if USE_VT
#include "VT.h"
#endif

const double pi = 3.1415926535897932385;

int main( int argc, char *argv[] )
{
  /*  This is a parallelization of serialPi.c.                             */
  /*  The for-loop is partitioned among the processors.                    */
  /*  Function MPI_Reduce sums the processors' results for processor 0.    */
  /*  Processor 0 outputs.                                                 */
  /*                                                                       */
  /*  Remember, it is convenient to reference the typical processor in the */
  /*  first person, using "I" and "me" and "my." In other words,           */
  /*  read through this program as if you were one of the processors.      */

  /*  Arguments required for executing argv[0]:      */
  /*  0                         -> serialPi          */
  /*  1                         -> numberOfIntervals */

  /*  My ID number and the total number of processors: */
  int myrank, numProcs;

  /*  Variables for the computation. */
  int numberOfIntervals, interval;
  double intervalWidth, intervalMidPoint, totalArea, myArea = 0.0;

  /*  For completeness I'll declare an MPI status variable, */
  /*  although I don't plan to use it. */
  MPI_Status status;

  /*  Initialize the MPI API.                                       */
  /*  But don't call any Intel Trace Collector API routines before MPI_Init() */
  /*  or after MPI_Finalize().                                      */
#if USE_VT
  VT_traceoff();		/* Don't want to trace MPI_Init() */
#endif

  MPI_Init(&argc, &argv);

#if USE_VT

/*  Now setup the Intel Trace Collector instrumentation. This involves defining */
/*  source code locations for beginning and ending events.            */
/*  First define state code 200 associated with state "for-loop" and */
/*  activity "Calculation".                                           */
  VT_symdef( 200, "for-loop", "Calculation" );

/*  and define location code 201 associated with state "print" and    */
/*  activity "I/O".                                                   */
  VT_symdef( 201, "print", "I/O" );

/*  See below how we place the routines VT_begin(int code) and        */
/*  VT_end(int code) to see what the states and activities are.       */

  VT_traceon();			/* Switch tracing on */
#endif

  /*  Request my ID number: */
  MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

  /*  I'll also ask how many other processors are out there: */
  MPI_Comm_size(MPI_COMM_WORLD, &numProcs);

  /*  Okay. The preparations have been made. */

  /*  The number of intervals is a command line argument. */
  numberOfIntervals = atoi( argv[1] );

  /*  Compute the interval width. */
  intervalWidth = 1.0 / numberOfIntervals;

  /*  Now I'll compute my area. */
#if USE_VT
  VT_begin(200);   /* Start tracing the for-loop. */
#endif
  for ( interval = myrank; interval < numberOfIntervals; interval += numProcs )
    {
      intervalMidPoint = (interval + 0.5) * intervalWidth;
      myArea += 4.0 / ( 1.0 + intervalMidPoint*intervalMidPoint );
    }
#if USE_VT
  VT_end(200);   /* Stop tracing the for-loop. */
#endif

  /*  Okay. Now I'll submit my results to the MPI API. */
  /*  If I'm processor 0, I'll receive the total area. */
  /*  Otherwise, I'm done. */
  /*  No user-defined tracing is needed here. Intel Trace Collector automatically */
  /*  traces MPI function calls. */
  MPI_Reduce( &myArea,
              &totalArea,
	      1,
	      MPI_DOUBLE,
	      MPI_SUM,
   	      0,
 	      MPI_COMM_WORLD );

  /*  If I'm processor 0, I need to normalize the total area and output. */
  if (myrank == 0)
    {
      totalArea *= intervalWidth;

#if USE_VT
  VT_begin(201);   /* Start tracing printing. */
#endif
      printf( "The computed value of the integral is %.15f\n", totalArea );
      printf( "The exact value of the integral is    %.15f\n", pi );
#if USE_VT
  VT_end(201);   /* Stop tracing printing. */
#endif

    }
  /*  Close the MIP API: */
  MPI_Finalize();

  return 0;
}

  • In the following section, we will see what the ITA timelines and activity charts look like, with the added instrumentation code.



HPC @ IU - View ITC Instrumented Trace of parallelPi

Viewing parallelPi's ITC Instrumented trace

The ITC/A timeline

  • Before we can view the ITC instrumented trace, we'll need to compile and run the new code.

    Exercise: Do this!

    Switch to the ParallelPi_VT directory on a head node (bhX/ihX) and then compile the program. 

     [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/ParallelPi_VT/
 [agopu@bh2 ParallelPi_VT]$ make

    Then run the ITC instrumented code on interactive nodes you got through qsub -I (bcXX/icXX) 

     [agopu@bc81 agopu]$ cd ~/MPI_Tutorial/ParallelPi_VT/
 [agopu@bc81 ParallelPi_VT]$ lamboot $PBS_NODEFILE
 [agopu@bc81 ParallelPi_VT]$ mpirun C parallelPiVt 10000
 [agopu@bc81 ParallelPi_VT]$ lamhalt

    Finally, open the trace file, that's created, by doing: 

     [agopu@bc81 ParallelPi_VT]$ traceanalyzer parallelPiVt.stf

  • In the following figure the ITA timeline of the parallelPiVt program with the additional ITC instrumentation is shown:

  • Do you see the appearance of the states we titled for-loop and print thereby increasing the resolution of our analysis?? Probably not! Because the entire computation took very little time, in the order of milliseconds. The real guts of the computation are in the for-loop when the integral is calculated. Thus, you shouuld try increasing the number of steps from 10000 to, say, 1,000,000 and see how that changes the timeline, i.e. run the program as

    Exercise: Do this! 

     [agopu@bc81 ParallelPi_VT]$ lamboot $PBS_NODEFILE
 [agopu@bc81 ParallelPi_VT]$ mpirun C parallelPiVt 1000000
 [agopu@bc81 ParallelPi_VT]$ lamhalt

     [agopu@bc81 ParallelPi_VT]$ traceanalyzer parallelPiVt.stf

  • A sample detailed timeline if 200,000 is used as the number of steps, is shown below; it's obvious that with the increased number of steps, more time is spent in the computational part of the code (i.e. the for-loop) and our parallelization makes a bit more sense!

Processor activity

  • ITA's activity chart associated with the trace file parallelPiVt.stf, is shown below:

    • This is a detailed histogram view showing all states. The bars are in order from left to right as is the function legend from top to bottom. Here we have the ITC predefined states of the MPI function calls, the User_Code state, and the TRACE_ON state. In addition the histogram breaks out our user-defined states for-loop, main, and print. One must take care with the quantitative relationship among these states in this display. As you can see, the bar heights are to be added together to obtain the total run time on each processor. Although the main program contains all these states, clearly the bar associated with state main must represent the net time after subtracting the times of all the other states from the total.

    • It is again clear that too little of each processor's time is spent on actually computing the integral (for-loop) for a low number of steps in the integration.

    • A sample ITA activity chart if 200,000 is used as the number of steps, is shown below; Once again, it's obvious that with the increased number of steps, more time is spent in the computational part of the code (i.e. the for-loop):



HPC @ IU - parallelDiffusion Results and Profile

Profiling parallelDiffusion Results using Intel Trace Analyzer [ITA]

Recall parallelDiffusion Output

The ITA Profile


HPC @ IU - Introducing TotalView: Debugging RoundRobin Example

Introducing TotalView

Preparing for a TotalView parallel debugging session

  • The roundRobin program appeared to be correctly running. But in general, because of the complexity of most parallel programs, core dumps or simply incorrect results are common. TotalView is the premier parallel dubugging tool available today; it lets you examine your parallel program in detail and identify flaw(s) if any.

    In this section, we will explore the basic functionality of TotalView using the roundRobin program. In the next section, we'll use TV to debug the erroneous Parallel Diffusion program.

  • TotalView is a tool which can be used to interactively debug parallel programs. To use TotalView, the parallel program must be run interactively and not in batch mode, i.e. on nodes gotten through qsub -I.

    Related Info:

    • Other usage modes: There are a few other ways of using TotalView, which are explained in the TV6.6 manual. But we will not go into those details in this workshop.
    • Licensing information on AVIDD: TotalView's license on AVIDD is currently limited to 32 processors, but you will find that, in fact, debugging with 4 or 8 processors (i.e. 2 or 4 nodes respectively, on AVIDD) is sufficient and, more importantly, easier to manage than (say) 32.

      The license also is restricted to 2 concurrent users at any given time. This does not affect workshop attendees, because we have an unlimited user demo license for the workshop.

Starting a TotalView parallel debugging session

  • Let us see how to use TotalView to debug a parallel program while working on interactive nodes provided by PBS. When you invoke mpirun , you will need to append the -tv option to it before specifying your program's name. For example, to debug the roundRobin program, you'll have to type:

    Exercise: Do this! 

     [agopu@bc81 RoundRobin]$ cd ~/MPI_Tutorial/RoundRobin/
 [agopu@bc81 RoundRobin]$ lamboot $PBS_NODEFILE
 [agopu@bc81 RoundRobin]$ mpirun C -tv roundRobin
 [agopu@bc81 RoundRobin]$ lamhalt

    Note to MPICH Users: If you are an MPICH user, please see important note about use of TV with MPICH code on AVIDD in the Compiling and Running Parallel jobs using MPICH on AVIDD section of this workshop.

    As a start, you should see something like this on the console: 

     Linux x86 TotalView 6.6.0-2
 Copyright 1999-2004 by Etnus, LLC. ALL RIGHTS RESERVED.
   . . . 
 Reading symbols for process 1, executing "roundRobin"
 Library /N/u/agopu/MPI_Tutorial/RoundRobin/roundRobin, with 1 asects, was linked at 0x08048000, and initially loaded at 0x90000000
   . . . 
 Skimming 50430 bytes of DWARF '.debug_info' symbols from '/N/u/agopu/MPI_Tutorial/RoundRobin/roundRobin'...done
  . . .
  . . .
    Also two windows should open up:

    1. The TotalView root [main] window where process information and the like are shown.

    2. The TotalView process [debug] window showing a stack trace, program source code, action points, and such. Important features of the Process Window are:

      • Just below the bunch of buttons is an identifier bar. It identifies the current process and its process number and the like. For example, in the sample window shown below, the bar tells us that the information shown currently (again, in the process window) pertain to process 0 and the program name is roundRobin, and so forth. You should be able to cross-reference the process number with the Root Window.

      • Among the buttons, near the upper left-hand corner is a pull-down labeled Group (Control) . You can toggle this to go between Group and Process control. Now what is that supposed to mean?

        If Group (Control) is selected, then when you do any operation (like Next, Step, Halt, etc.)on the process window, it is applicable to all the processes running under mpirun's purview (or mpiexec's purview if you are using MPICH).

        If instead, Process is chosen, then any operation you do is applicable to only the current process i.e. the process indicated by the identifier bar (as explained in the above bullet).

      • To the right are buttons such as Go and Step which move us or step us through the program (Usual debugger functionality)

      • Below the identifier bar, on the left is the Stack Trace which shows all procedures (functions and the like) in the stack. To its right is the Stack Frame displaying local variables and registers for the frame, for the current process. Nothing interesting at this time because the main program has not started for this processor or for any of the others.

      • Below the stack trace and stack frame is the Source Pane displaying the roundRobin.c source code.

    OK, all those details said, the two TotalView windows should look similar to what's shown below

    Note: break points are called action points by TotalView's manuals

  • Exercise: Do this!

  • You should pull down File -> Preferences... -> Action Points; where it says "When breakpoint hit, stop:" select Process. This will prevent one or more processes from stopping prior to a breakpoint.

  • Within the process window, make sure Group (Control) is selected in the drop down menu on the top left corner; left-click on the Go button. You'll trigger parallel processes to be started on your compute nodes and TV will ask if you'd like to stop the parallel code.

  • Click Yes. After a few moments the root window's contents will change: 4 attached processes to which TV has attached itself will be shown with status code T implying stopped state. The process window will also change and TV will stop at its default break point as shown below:



Exploring TotalView

Hopefully you've come this far without any trouble and have managed to checkout various panes on the TV process window. Now let us see how you could put TV to use in a practical situation. Try to follow the steps outlined below to get a feel of how you could use break points (a.k.a. action points in TV manuals) and debug parallel code.

  • Exercise: Do this; and try to follow all the steps below!

    Recall that the parallel code execution stopped at a default break point. Click on main, inside the Stack Trace pane; the roundRobin.c source should be shown in the source window once again. Scroll down to the line shown in red below (line 77 if code is unchanged): 

      . . .
 /*  If I'm processor proc, then I want to send */
 /*  to processor proc + 1:                     */ 
    if (myrank == proc)
     . . .
    ... and left-click right on the line number. A break point indicated by a STOP symbol should show up; this should be reflected in the Action Point pane as well.

  • Now, watch the Root Window carefully and hit the Go button (on the Process window): All four roundRobin processes will momentarily show a status R implying running state, then finally status B for halting at breakpoint we had created. All four processes ran to the breakpoint because we are still toggled to the Group (Control) mode.

  • Now look at the local variables in the Stack Frame: myrank is 0 and numProcs equals 4. A number beginning with 0x is in hexadecimal (base 16); the number to the right, in parenthesis, is the decimal (base 10) equivalent.

  • Now, in the Root Window, dive onto process 2. i.e., right click on process 2 and select Dive. Then look at the Process Window: the identifier bar should display details for process 2, and among the local variables in the Stack Frame myrank should be 1. You may dive into the other two processes and watch the myrank variable (among others) change.

    Another way of dealing with individual process details (instead of diving into them from the Root window) is to use the P+ and P- buttons on the top right corner of the process window. Click on one of those two buttons and see how details change within the process window just as they did if you dived into a particular process.

  • Few more things you should try:

    Stepping through code using Next and also flow of parallel code based on process rank

    • Put in two more break-points at the lines shown in red below (lines 80 and 93 on unchanged code): 

              dest = proc + 1;
        MPI_Send( msg,
          . . .
	  . . .
              src = proc;
        MPI_Recv( msg,

    • Toggle to Process mode using the drop-down menu on the top-left corner.

    • Dive into process 0, and hit Go; Notice where the program stops (should on the MPI_Send( msg, line and also check out variables myrank, src and dest.

    • Then, dive into process 1 (from root window or by hitting P+ till you see process 1 details), hit Next once and then two more times; Notice where the program stops at each point (after 3 nexts, it should be on the MPI_Recv( msg, line; once again check out variables myrank, src and dest.

      You could repeat this exercise for processes 2 and 3. The flow of your parallel code based on process rank should become obvious by now.

    • OK, remove all three breakpoints by clicking on the STOP symbol.

      Set a new breakpoint on the line shown in red below: 

         /*  Close the MPI API: */
   MPI_Finalize();
      Toggle back to Group (Control) mode and hit Go.

    Using Laminate to view variables of multiple processes:

    • You should be on the breakpoint we defined in the last part of the previous bullet. Now dive into the myrank variable (double click on myrank within the Stack Frame pane or within the source code).

    • A new window should be opened up and you should see the value of myrank for the current process. But what if you want to view the value of a variable in all processes?

    • To accomplish that, select View -> Laminate -> Process on the variable window; now you should be able to see each process' rank at the same time.

    • If you are ready to proceed to the next section, then close the Variable window and read on...

  • Finally, make sure you are still toggled to Group (Control) ; Hit Go; On the Root window, all four processes should momentarily go into R (run) status. And finally the four processes shouuld switch to X state. I.e. they have terminated; you should see something similar to this on the console (where you started the TotalView mpirun (mpiexec) run). 

       . . .
   . . .
 Reading symbols for process 4, executing "roundRobin"
 The number of processors in this run is 4.
 Processor 0 has sent the message.
 Processor 1 has received the message.
   . . .
 Processor 0 has received and checked the message. Program terminating normally.
 Processor 3 has sent the message.
 [0] Intel Trace Collector INFO: Writing tracefile roundRobin.stf in /N/u/agopu/MPI_Tutorial/RoundRobin
is acknowledged and notified.

HPC @ IU - Putting TotalView to use: Debugging parallelDiffusion

Use TotalView to Debug parallelDiffusion

TotalView debugging session

Find the bug in parallelDiffusion!

  • First let's run all four processors through half of the variable initializations and check for possible problems.

    • To that effect, left-click on:

      main,

      inside the Stack Trace pane. Then, assuming you have not modified the source code, set a breakpoint at the position indicated in red below (line 124): 

       /*  Set uniform initial conditions on my portion of the grid.  */
       for ( i = 1; i <= mynni; i++ )

      This breakpoint will stop execution after computing dt in line 121.

    • Stay toggled to Group (Control). Hit the Go button. Watch the Root Window to make sure all processes reach this breakpoint.

  • Now let us check some variables.

  • OK, time to move on...

    • Remove the above break point by clicking on the STOP symbol.

      Then place a breakpoint on the line shown in red (line 135 if code is unchanged) 

       /*  First the y = 0 boundary. */
       for ( i = 0; i <= mynni+1; i++ )
       This breakpoint will allow us to examine the temperature array u[][] subsequent to the setting of the initial conditions. Make sure you're toggled to Group (Control), then hit Go. Check to make sure all four processes have reached this second breakpoint (you could use the P+ and P- buttons and check if all processes are at the same line).

    • Among the local variables on the Stack Frame dive on the temperature array u.

      Important: It is usuallynot convenient to laminate in case of a double array; rather it's easier to examine u for process 0 (otherwise there will be way too much data). You might be disappointed to see only:

    • And if you dive into this (by double clicking twice on the variable pointers shown) you will eventually view only the value of u[0][0], which is not very helpful.

      But notice at the top of the window the Type is double** i.e. variable u is of type double** or in simple terms a memory pointer to a memory pointer to a long contiguous chunk of memory. This is illustrated below for the benefit of those unfamiliar with 'C' pointers: 

                        -----------      -----------      -----------                 -----------
      u[0][0] -> |Mem.loc.x_1| -> |Mem.loc.x_2| -> |Mem.loc.x_3| -> ... ... -> |Mem.loc.x_n| 
                  -----------      -----------      -----------                 -----------
      where x_1, x_2, ... x_n are contiguous memory locations and x_n is 2002 x 1002 x size_of_double for each process in our example.

      Note: When displaying normal C or C++ variable, TotalView data types are identical to C/C++ type representations. But for pointers to arrays you need to cast a C/C++ pointer-to-array to a TotalView pointer-to-array if you wish to view its contents. On the other hand, there is no need for such conversion while doing Fortran coding for any F datatype. See the TotalView manual for more information on this.

    • Refer to the parallelDiffusion.c source code to review how we allocated the 2D array u. We allocated one big contiguous chunk of memory for the 2D array. In our test run, we had used 8000 as the first parameter and had used 4 processors (2 nodes) which meant each process worked on 2000 elements. So we can cast u to a TotalView type double[2002][1002]** which is read in reverse as a pointer to a pointer to a 2002 x 1002 array. With this cast TotalView will understand how to display the C array; But how is the casting done?

    • In the Variable Window left-click on the text box showing double**. A cursor should appear; Using your keyboard, modify double** to read double[2002][1002]**; then hit return.

      Still working in the Variable Window, dive twice on the array i.e. double-click or right-click and select Dive on the value shown in the Value pane within the Variable window. Now...you should see processor 0's u in its initialized state as shown below:

    • As you would expect, there are 2,002 x 2,002 = 4,008,004 elements in this array, way too many to make any sense out of! So pick and choose what data to view. Recall our erroneous result was produced along the j = 10 margin. So restrict yourself to only look at data for j = 0 .

      In the Variable Window left-click and release on the Slice field. Use the keyboard to enter slice parameters: 

        [:][10:10]

      and hit return. This will restrict the data in the Variable Window to the j = 10 slice. The resulting variable window should look similar to:

      Note: More information on slicing can be found on the TotalView manual.

    • But isn't 2002 entries still too much to scroll through? Would it not be nice to plot the data?

      . . .

      . . . TotalView lets you do that!

      In the Variable Window pull down the Tools menu and select Visualize. After a moment a 2D plot should appear. In the Plot window pull down on File and select Options. In the resulting dialog box select Points and deselect Lines, then click OK. Your plot should look like:

    • And now you should be able to see the problem! Do you?

      The initialized temperature is zero at u[2001][10], when in fact it should equal 1.

      Along the j = 10 slice the only boundary points that should be set to zero are u[0][10] and u[8001][10]. The point at [2001][10] is one of processor 0's ghost points which should not have been set to zero.

      Some Asides::

      • If this is not obvious to you, then close the visualization windows; edit the slicing text box to [1950:2002][10:10] and then redo the visualization per the steps outlines in the above bullet.

      • Also, as a pure do-it-yourself kind of exercise, if you are interested in playing around with the Visualization tool, try resetting the slice to [:][:], the redo the Visualize step given in the previous bullet; Then try doing: View -> Surface on the smaller Visualization window (it shows "visualize" on the title bar of the window). You could possibly use the surface visualization tool at some point of time?

      • We could repeat the investigation by looking at the other processors, and rest assured, we will find the same sort of error. Apparently when we coded the setting of the initial conditions we coded exactly as we did in the serial version. But doing so proved to be wrong, because we had thereby neglected to set the ghost points that appear only when dealing with the parallel version.

    • The snippet of code we need to investigate lies between our two breakpoints: 

        /*  Set uniform initial conditions on my portion of the grid.           */
  for ( i = 1; i <= mynni; i++ )    {
      for ( j = 1; j <= nnj; j++ )	{
	  u[i][j] = u_0;
      }
  }

      To initialize the ghost points, the for-loop over i must expand to include them: 

        for ( i = 0; i <= mynni+1; i++ )
           ^^^   ^^^

    • However, for processes 0 and 3 this loop will set the i = 0 and i = nni + 1 boundaries to 1, when in fact we want those boundaries to be fixed at 0. Does this present a problem?

      . . .

      . . .

      No! How is it not a problem? Think about it..

    • You probably guessed it right. Because the subsequent code, just beyond the second breakpoint, resets those boundaries to 0 while leaving the ghost points at 1.

      The corrected code snippet with comments is shown below: 

        /*  Set uniform initial conditions on my portion of the grid.           */
  /*  Also set my ghost points. These are grid points not owned by        */
  /*  me but that I need to comput my stencils. If I am processor 0       */
  /*  or processor numProcs-1, then half of these ghost points will       */
  /*  be reset later below during the setting of the boundary conditions. */
  for ( i = 0; i <= mynni+1; i++ ) {
      for ( j = 1; j <= nnj; j++ ) {
	  u[i][j] = u_0;
      }
  }

    • Once you fix the code, recompile your code on a head node (bhX/ihX) by doing: 

       [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/ParallelDiffusion/
 [agopu@bh2 ParallelDiffusion]$ make clean
 [agopu@bh2 ParallelDiffusion]$ make
      and re-run the program on a compute node gotten through qsub -I (bcXX/icXX). You should see correct results this time around! 
       [agopu@bc81 agopu]$ cd ~/MPI_Tutorial/ParallelDiffusion/
 [agopu@bc81 ParallelDiffusion]$ lamboot $PBS_NODEFILE
 [agopu@bc81 ParallelDiffusion]$ mpirun C parallelDiffusion 8000 1000 100
 [agopu@bc81 ParallelDiffusion]$ lamhalt 
  After 100 time steps some results are:
  actual   u[400][10] = 0.874737
  computed u[400][10] = 0.873978
  After 100 time steps some results are:
  actual   u[1000][10] = 0.874806
  computed u[1000][10] = 0.873978

      Hopefully, this example has given you a good idea of how you could use TotalView to debug your parallel code!



HPC @ IU - Conclusion

Concluding Remarks

Hopefully, you have gained enough to write simple MPI parallel code.

  • By now, we hope you have gained sufficient knowledge and experience to begin parallel programming using MPI, to profile your parallel programs for optimization with Intel Trace Collector/Analyzer [ITC/A], and to debug them with Totalview.

Tap into the talent of the High Performance Computing Support team (HPC).

  • HPC is an experienced team of computational scientists; the group has its members located both on the IUB and IUPUI campuses.

  • Our expertise includes:

    • High-performance algorithm development, coding, tuning, and results analysis in distributed memory/shared memory/hybrid computing environments..

    • Use of serial/parallel programming tools, software packages, compilers and computational libraries

  • We have developed high-performance serial/parallel algorithms and code in computational astrophysics, biology, chemistry, medical research and physics in collaboration with researchers in those areas.

  • We are prepared to work with you to augment your research programme if it involves high-performance computing component(s) thereby substantially increasing your ability to achieve results. There is no charge for our contribution as long as you are associated with Indiana University.

  • Please feel free to contact the HPC group at (812) 855 - 2632, or by emailing to hpc [at] iu.edu (remove spaces and replace [at] with @) if you have any questions or think we could be of any help with your computing needs!

Next                            


HPC @ IU Tutorial - Compiling and Running Parallel jobs using MPICH on AVIDD

HPC @ IU Tutorial - Compiling and Running Parallel jobs using MPICH on AVIDD

This section serves, more than anything else, as an appendix to the rest of the notes on this workshop. It addresses the issue of using MPICH instead of LAM to run MPI code on the AVIDD system. You'll find information about:

Setting your environment, using softenv, up to use MPICH (insted of LAM)

As mentioned above, the default AVIDD softenv setup is configured to use MPICH; You do not need to modify any settings if you've left your ~/.soft file unchanged.

But if you modified your ~/.soft file to use per instructions on this workshop (to use LAM), then edit the file and:

  • Comment out/remove the +lam-gm-intel line as well as the @remove mpich-gm-intel line
  • Uncomment/add +mpiexec.
  • Change ita-lam to just ita. 
    ## @remove +mpich-gm-intel
## +lam-gm-intel
## +ita-lam
+mpiexec
+ita
+totalview
## Possibly more lines before the default @avidd line
@avidd
Once you make the changes, you'll need to log out and log back in or do resoft.

There is a file named dot_soft.mpich within the MPI_Tutorial directory (assuming you copied the example programs from hpc's account) which you could use to update/replace your ~/.soft file.


Run examples available in this workshop using MPICH

Assuming you're already logged in to the AVIDD cluster and also have your environment setup to use MPICH, follow the steps given below to run example MPI programs using MPICH.

Re-compile all programs

  • You will need to clean out LAM executables of your MPI programs and re-compile them. As suggested previously, do so on one of the head nodes (bhX/ihX) 
     [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/HelloWorld
 [agopu@bh2 HelloWorld]$ make clean ; make

 [agopu@bh2 HelloWorld]$ cd ~/MPI_Tutorial/RoundRobin; make clean ; make

 [agopu@bh2 RoundRobin]$ cd ~/MPI_Tutorial/ParallelPi; make clean ; make

 [agopu@bh2 ParallelPi]$ cd ~/MPI_Tutorial/ParallelPi_VT; make clean ; make

 [agopu@bh2 ParallelPi_VT]$ cd ~/MPI_Tutorial/ParallelDiffusion; make clean ; make

 [agopu@bh2 ParallelDiffusion]$ cd ~/MPI_Tutorial/MPI_IO; make clean ; make

    Important Note:

    • Even though you used the same Makefile to compile your code, you must note that different Intel Trace Collector includes/libraries as well as MPI includes/libraries would have been (and should have been) used in the Make process. The change is transparent because of the use of softenv, but if you look at environment variables, for example $CPATH, you'll notice that it has different values depending on whether you add MPICH or LAM in your ~/.soft file.

Run programs interactively

  • Once you have your programs compiled, you can use mpiexec to run them. Note that you can use mpirun as you had done in the case of LAM (without the C parameter) but using mpiexec has its advantages.

  • First, grab a couple of interactive nodes from PBS. For example, to get 2 nodes, 2 processors per node, for 30 minutes, you could do: 

     [agopu@bh2 agopu]$ qsub -I -l nodes=2:ppn=2 -l walltime=30:00

  • Get, set, go! 

     [agopu@bc01 agopu]$ cd ~/MPI_Tutorial
 [agopu@bc01 agopu]$ mpiexec HelloWorld/helloWorld
 [agopu@bc01 agopu]$ mpiexec RoundRobin/roundRobin
 [agopu@bc01 agopu]$ mpiexec ParallelPi/parallelPi 2000000
 [agopu@bc01 agopu]$ mpiexec ParallelPi_VT/parallelPiVt 2000000
 [agopu@bc01 agopu]$ mpiexec ParallelDiffusion/parallelDiffusion 8000 1000 100
    Assuming /N/gpfs/${USER}/testfile exists... 
     [agopu@bc01 agopu]$ mpiexec MPI_IO/mpi_io /N/gpfs/${USER}/testfile /N/gpfs/${USER}/testfile2


Run programs in batch mode (non-interactively)


Profiling MPICH code using Intel Trace Analyzer (ITA)


Using Totalview to debug MPICH code

Previous Next 


HPC @ IU Tutorial - Useful Links to Documentation

Useful Links to Documentation

  • The AVIDD system's home page has a comprehensive collection of links regarding the AVIDD cluster.

  • Message Passing Interface MPI Documentation:

  • A good introduction to the TotalView parallel debugger is available at: http://www.etnus.com/Products/TotalView/started/getting_started.html.

    The comprehensive TotalView user's guide is also available online. Here's the HTML version or if you prefer a PDF version.

  • The user's guide for Intel Trace Analyzer is installed on AVIDD. To view it, at your shell prompt type: 

     xpdf /N/soft/linux-rhel3_AS-ia32/intel-traceAC/ita4.0.3.1/doc/Intel_Trace_Analyzer_Users_Guide.pdf &

    You can also copy this pdf file to your workstation using secure scp [Unix flavors] or sftp [MS windows]. For example, if you have a Linux box, go to the directory where you want to put the pdf file, then type: 

     scp your_user_name@avidd-b.iu.edu:/N/soft/linux-rhel3_AS-ia32/intel-traceAC/ita4.0.3.1/doc/Intel_Trace_Analyzer_Users_Guide.pdf .

    Note the "." at the end of the above Unix command, it's required to copy the file into the current directory. If you use MS Windows, you could the Secure File Transfer tool to download and view these files from AVIDD.

  • Similarly Intel Trace Collector user's guide can be viewed from AVIDD. At your shell prompt type: 

     xdf /N/soft/linux-rhel3_AS-ia32/intel-traceAC/itc5.0.1.0/mpich/doc/Intel_Trace_Collector_Users_Guide.pdf &

    The same procedure as above can be used to download the pdf file to your workstation.

  • Data Storage at IU related:

  • Useful books on MPI include the "MPI: The Complete Reference" series published by MIT Press. "Using MPI: Portable Parallel Programming With the Message-Passing Interface" by William Gropp, et al., MIT Press, is an excellent MPI reference book as well.