Support #2041

charmrun with mpirun instead of srun?

Added by Geoffrey Lovelace 19 days ago. Updated 14 days ago.

Target version:
Start date:
Due date:
% Done:




I need to use charm++ on a cluster that runs parallel jobs using mpirun. The cluster uses slurm to submit jobs, but for technical reasons slurm can’t be built to be aware of the mpi distribution we need to use, so srun does not correctly run jobs in parallel.

The trouble is charmrun always seems to try to use srun, even if I give it ++mpiexec as an option. But I can run jobs fine by hand by using mpirun directly in a script I pass to sbatch.

Is there a way to be sure charmrun uses mpirun? Either an option when calling charmrun or when compiling charm?


#1 Updated by Sam White 19 days ago

Can you try with this:

./charmrun ++mpiexec ++remote-shell "mpirun <mpirun_args>" ./pgm <application_args>

#2 Updated by Geoffrey Lovelace 18 days ago

I tried this, and it still called srun. Might I need to build charm++ differently? I used

./build charm++ mpi-linux-x86_64 mpicxx smp -j16 --with-production

#3 Updated by Geoffrey Lovelace 18 days ago

I’m using charm++ v6.8.

#4 Updated by Sam White 17 days ago

Is that v6.8.0 or v6.8.2? Can you try with v6.9.0? You shouldn't need to build Charm++ any differently.

#5 Updated by Geoffrey Lovelace 14 days ago

It's v6.8.0. We're working on adding support for v6.9.0 to our code base (, but that will take us a bit. Is it normal for the ++mpiexec option to not work in version 6.8.0?

#6 Updated by Sam White 14 days ago

No, it's not. What MPI library are you using? And could you post the output of what happens when you try the "++mpiexec ++remote-shell" command I posted above?

#7 Updated by Jim Phillips 14 days ago

For the mpi-... builds of Charm++ you can run the binary with mpirun/mpiexec/srun directly rather than via charmrun. In fact charmrun is just a script on these builds. The binary charmrun and ++mpiexec options are only needed for netlrts-, verbs-, and similar builds.

Also available in: Atom PDF