charmrun with mpirun instead of srun?
I need to use charm++ on a cluster that runs parallel jobs using mpirun. The cluster uses slurm to submit jobs, but for technical reasons slurm can’t be built to be aware of the mpi distribution we need to use, so srun does not correctly run jobs in parallel.
The trouble is charmrun always seems to try to use srun, even if I give it ++mpiexec as an option. But I can run jobs fine by hand by using mpirun directly in a script I pass to sbatch.
Is there a way to be sure charmrun uses mpirun? Either an option when calling charmrun or when compiling charm?
#7 Updated by Jim Phillips 5 months ago
For the mpi-... builds of Charm++ you can run the binary with mpirun/mpiexec/srun directly rather than via charmrun. In fact charmrun is just a script on these builds. The binary charmrun and ++mpiexec options are only needed for netlrts-, verbs-, and similar builds.
#8 Updated by Evan Ramos 4 months ago
As Jim said, those
++mpiexec ++remote-shell options are only valid with the netlrts/verbs version of charmrun. For the MPI machine layer,
mpirun was moved ahead of
srun in charmrun's precedence for the 6.9.0 release. You may want to cherry-pick commit
#11 Updated by Geoffrey Lovelace 4 months ago
Thank you for the feedback! Please accept my apologies for the delay, as I have been swamped with travel and with physically installing our new cluster (when I first posted, I was testing the cluster remotely before it was shipped).
Our particular application, spectre (http://github.com/sxs-collaboration/spectre) does not yet support Charm++ version 6.9, although some of our developers are working on adding this support. Once we move to v6.9, I plan to try this again to verify that charmrun works. Since I can run jobs with raw mpirun commands, for now this isn't a dealbreaker. Is there anything I can't do with a raw mpirun call that I can do with a charmrun call?
Meanwhile, I'd like to try a build of v6.8 cherry-picking the commit you suggest. But I probably will need a couple of weeks to be able to carve out the time to do this. Thank you very much for keeping up with this issue and for your patience with the delays in my replies! I appreciate your help.