Shifting data within the Cartesian communicator

Having exchanged niceties with the master, all processes, including the master, initialise their matrices with their Cartesian rank number:

for ( i = 0; i < ROWS; i++ ) {
   for ( j = 0; j < COLUMNS; j++ ) {
      matrix [i][j] = my_cartesian_rank;
   }
}

Next they all send their matrices to the master, which collects them and displays on standard output:

if (! i_am_the_master ) 
   MPI_Send ( matrix, COLUMNS * ROWS, MPI_INT, host_rank, 3003,
              cartesian_communicator );
else
   collect_matrices ( cartesian_communicator, my_cartesian_rank, matrix, 
                      3003 );

The function collect_matrices does the collecting, assembling into a display matrix and printing on standard output. This function is defined at the end of the file.

There is one new MPI function used in collect_matrices which refers to Cartesian grids of processes. It peforms an operation opposite to MPI_Cart_coords. Whereas MPI_Cart_coords yields Cartesian coordinates given a process rank, function MPI_Cart_rank yields a process rank given its Cartesian coordinates.

This time the function collect_matrices really assumes that the master process is also a member of the cartesian_communicator. Thus the program would have to be tweaked a bit more here in order to make it more robust.

Going back to our main program, now really interesting things begin to happen. The two statements:

MPI_Sendrecv ( &matrix[ROWS - 2][0], COLUMNS, MPI_INT, 
               top_neighbour, 4004,
               &matrix[0][0], COLUMNS, MPI_INT, bottom_neighbour,
               4004,
               cartesian_communicator, &status );

MPI_Sendrecv ( &matrix[1][0], COLUMNS, MPI_INT, bottom_neighbour,
               5005,
               &matrix[ROWS - 1][0], COLUMNS, MPI_INT,
               top_neighbour, 5005,
               cartesian_communicator, &status );

Function MPI_Sendrecv, described on page 57, section 3.10, ``MPI: A Message-Passing Interface Standard'', sends the array pointed to by its 1st argument to a process indicated by its 4th argument while at the same time receiving data from a process indicated by its 9th argument into an array pointed to by its 6th argument.

So, the first call above sends the second last row of the worker matrix up to the top_neighbour, while receiving the second last row of the neighbour matrix from the bottom_neighbour into the first row of the matrix at the same time.

The second call sends the second row of the worker matrix down to the bottom_neighbour, while receiving the second row of the neighbour matrix from the top_neighbour into the top row of the matrix at the same time.

Some processes don't have a top neighbour or a bottom neighbour. In their case MPI_Cart_shift would have set top_neighbour or bottom_neighbour to MPI_PROC_NULL. It is perfectly legal to send a message to MPI_PROC_NULL. Such messages are thrown into a black hole, which, as we all know, lurks inside every UNIX computer, and every departmental budget.

If we were to configure our process space into a wrapped grid, the topmost processes would send data to the botommost ones. Running around in circles is what you have to do, if you don't want to end up in a black hole.