MPI_FAQ

A Q&A introduction to MPI.

About MPI

What is MPI?: MPI stands for Message Passing Interface. It is a standard for coordinating actions between processors independently running multiple instances of the same program.; The programs running across processors are called nodes.
How is this coordination achieved?: The different copies of the program are coordinated by sending messages between them as the code executes.; Messages are required because as different instances of the same program executes on different nodes the programmer wants them to execute different calculations (otherwise there would be no benefit to running in parallel). The instances are never synchronized so messages allow them to carry out different tasks and to be in different places in the same code in order to perform a single objective.
What is a message, and how is it transmitted?: A message is some data (such as a vector of numbers) and an integer code for what kind of message it is. This code is called the message tag. MPI supports very sophisticated messaging, but the basic features are often all that is required.; The computer system(s) that are running the program must maintain the MPI environment.; The programmer includes calls to procedures (functions) that access the MPI environment.
What is the MPI library?: The MPI library is the set of procedures (functions) to be called from program in C, FORTRAN or other languages to use the MPI environment.; The programmer only needs to understand and call routines in the MPI library within their program to send and receive messages while the program runs on multiple nodes. The MPI environment handles all the details of information passing for the programmer.
Are messages like email?: Messages sent between nodes are similar to email messages. The environment ensures that messages are delivered to the inbox of the node.; Unlike email, the program must process messages one at a time in the order they arrive.
How are messages sent and received between instances of the program?: The programmer must insert calls to the MPI library of routines in their program to pass messages.
Is MPI the only way to run a program in parallel?: Another common way to execute in parallel is for different instances of program to have shared memory.; Within a shared memory environment messages do not need to be sent between nodes to coordinate action, although this can be done as well.; Ox Professional allows the user to exploit multi-threading on the processor running Ox. Further, Ox version 7x includes features to make multi-threading easier to use.; MPI is designed to coordinate actions when memory is not shared. Ox Console (the free version) does not support multi-threading, but it can run in parallel through MPI as long as the cluster has MPI installed.
How does my program know about the MPI environment.?: If the program is started within the MPI environment then it can and must call MPI_Init() to get information about the environment.; The MPI environment tells the user's program two pieces of information that are required to coordinate actions.; The number of different processes that are running the program and communicating through MPI. These processors are called nodes, and the number of nodes is denoted Nodes. If Nodes=1 then the program is running in serial even though the MPI environment is accessed.; Each node is also told by MPI its rank, or in CFMPI the node's ID. The rank or IDs take on the integer values 0,…,Nodes-1. Usually the user has no control over which physical CPU is assigned to each ID. The MPI environment assigns the IDs arbitrarily.

CFMPI

What is CFMPI?: CFMPI implements an Ox- and niqlow-friendly interface to the basic MPI routines.; The user of CFMPI could learn how to program with MPI and use only Ox routines to send and receive messages.; A user could use CFMPI.ox and CFMPI.c as a template for writing interfaces to other MPI routines or an alternative approach.; Many of the Ox functions in CFMPI have the same form and purpose as routines in the MPI library, but they often have fewer arguments and less flexibility, because the author of CFMPI has found only certain aspects of MPI are needed for this kind of program.
Is CFMPI object oriented?: Yes, CFMPI is object oriented, which makes it easier to exploit parallelization in different ways within the same program.; The OOP aspects of CFMPI also make it possible for the programmer to use more specialized routines in CFMPI. These routines replace several lines of basic MPI code with a single call to a CFMPI routine.
CFMPI uses classes and static members/methods and some C code to interface with MPI.: This allows the calls to MPI in Ox to be simpler than direct calls. It also adds some features to simplify client-server execution.

Parallel Paradigms

There are two main ways to use MPI in parallel execution.

Client-Server Paradigm

Peer-to-Peer Paradigm

Client Server

A Client-Server approach is based on one node (the client) coordinating the actions of the other nodes (the servers). The single program running on the nodes splits into two separate segments of code. The client executes code and calls the server to carry out tasks that can be done simultaneously. The servers execute codes that waits for the client to send a task.

The client is typically the node with ID=0. Servers have ID>0.

When operating in client-server mode a program will typically have a branching statement of the form:

 if (ID==0) Client_Tasks(); else Server_Tasks();

As messages are passed the servers are executing commands in a different part of the program than the client when messages are sent and received.

Client-Server model relies on node-to-node communication: the client sends a message to a client which later sends a reply back. The servers typically do not communicate with each other.

Peer-to-Peer

In Peer-to-Peer mode all the instances of the program are at the same line of code when messages get sent and received.

Peer execution relies on group communication that is coordinated by the MPI environment.

One node (again usually ID=0) will often play a special role before or after messages are sent. That is, they may process the results shared by all the peers and the broadcast the result to every node.

A single program can use both Client-Server and Peer communication.

In more advanced MPI programming the set of nodes can be organized into smaller groups.

For example, fifteen nodes may be split into three worlds, each with five nodes. CFMPI does not support this type of parallelism.

Blocking versus non-blocking

MPI library routines can be either blocking or non-blocking.

A blocking routine means that the program's execution on that node does not proceed until the message passing function completes. A non-blocking routine will let the program continue to the next line even though the communication has not yet completed.

Peer-to-Peer communication is inherently blocking, but node-to-node communication can either.

Most naturally, the send message routine in node-to-node communication is not blocking, but the receive message routine is blocking. That is, programs usually rely on the information received from other nodes to carry out the work. On the other hand, once message is sent to another node it is natural to carry even though the receiving node has not yet processed the message.

Sources

In node-to-node messages are sent to a particular node.

A node waiting (blocking) to receive a message can specify either to wait for a message from a particular node (say the client) or from any node (say the client waiting for any server to respond).

In pure client-server mode the servers only get messages from the client so it does not matter if the source is "any node" or just "the client." However, it is usually more efficient for the client receive messages from any node. How long it takes servers to carry out a task depends on many things and cannot be predicted. So it is better if the client receives whichever messages comes in next rather than waiting for a response from one particular node.