Problems about debugging MPI program

[Robbie]

Hi all,
I have encountered some problems when debugging MPI program with totalview-8.0.1. When run the following command:
mpirun -tv -np 4 MPIDebug
it complains that:

mpiexec_linux: mpd_uncaught_except_tb handling:
exceptions.ImportError: No module named mtv
/usr/local/mpich2-v5-mpd/bin/mpirun 462 mpiexec
import mtv
/usr/local/mpich2-v5-mpd/bin/mpirun 1396 ?
mpiexec()

It seems that the mpirun script (line 462) can't find the required module "mtv", which is needed to run totalview.
Where does the "mtv" module come from? MPICH? Or totalview? Or python?

And how to solve this problem?

Ps:
when invoked without the "-tv" option, the MPI prgram works well.


Regards,
Robbie

[Rob-TotalView]

Hi Robbie,

I think that you are encountering these problems because you are using older code. I suggest that you use the latest release of MPICH2, 1.0.5p4 as well as the current release of TVD, 8.2.0-0.

You should launch TVD with no arguments. You can then use the parallel tab in the "new program dialog" to set up your program. Select MPICH2 as the parallel system, set the number of tasks along with any other necessary arguments.

Rob

Rob Solomon
TotalView Technologies

[Robbie]

Hi Rob,
I have solved this problem. As required by MPICH2 install-guide, to launch MPI job under the control of TV with mpirun command (e.g.: mpirun -tv -np xx App), the MPICH package must be configured with "-with-pm=mpd" and at the same time, the "PATH" environmental variable contains the path to the totalview binary. In this case, MPICH package will compile and link the "mtv" module to be used with totalview.

However, even if I can lauch a debug job with "mpirun -tv -np xx App", I still failed to lauch a MPI debug job in the GUI mode. That is, if totalview is started first, and trying to lauch a new debug process and selecting "MPICH" in the Parallel tab, my totalview complains that:

The starter script is malfunctioning. Possible reasons:
1. The -tv option may be missing from the parallel launch string.
2. This MPI may not support the TOTALVIEW environment variable.
3. Using a parallel configuration (MPICH) for non-parallel program.
4. Parallel configuration "MPICH" is incompatible or missing.
The launch string was:
mpirun -tv -ksq -np 4 -nodes 1 ./app
...

In fact, my MPICH does not support -ksq and -nodes options any more. The command "mpirun -tv -np 4 ./app " should work well.
So this problem must be caused by the incorrect MPICH parallel configuration.
Where can I set the MPICH parallel configuration correcly?
Still in tvdrc?
And how? Will you give me an example?

Ps: My platform is IA64 Linux.

Regards,
Robbie

[Rob-TotalView]

Hello Robbie,

I am glad that you are now able to launch your MPICH2 program with TotalView's command line launch.

I think that you are having problems with the new program dialog because "MPICH2" is not selected as the "Parallel System" on the Parallel panel.

I had previously suggested that you upgrade to TVD 8.2.0-0. It is not clear what version of TVD you are using. I still recommend this upgrade. In fact, MPICH2 was not added to this panel until TVD 8.1.0-0.

There is additional documentation on mpi startup available at:
http://www.totalviewtech.com/Documentation/mpi_startup.php.

Rob