Feature #190

Build target(s) for Intel MIC

Added by Phil Miller about 6 years ago. Updated about 1 year ago.

Machine Layers
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 3.00 h)


We should incorporate the work done to build on MIC with the main codebase. If it's easily backported, it should go on the stable branch as well.


Feature #238: Intel MPI dual binary method for execution on stampedeClosedEric Bohm

Feature #239: Charmrun wrapper around ibrun.symm for dual binary execution on host+mic with IMPI on stampedeClosedPPL


#1 Updated by Eric Bohm about 6 years ago

  • Assignee changed from Ronak Buch to Eric Bohm

The initial issues here stem from subtle interactions between the Intel cross compiler scheme for MIC (aka Intel Xeon Phi, KNC, Knight's Corner, plus whatever the, arguably overly fecund, brand name developers from Intel come up with next as a name for these things) and Charm++ compiler wrapper settings.

All libraries which a MIC application binary must link to must go through the cross compiler (icc -mmic, or a more gnarly explicit path call to x86_64-k1om-linux-g++, which should probably be avoided as we have no guarantee about name stability for that thing) and its linker.

All binaries which would be executed on the host (usually charmc and various configure tests) should usually not go through the cross compiler. In the past, we have generally used CMK_NATIVE (aka charmc -seq) for these. However, -seq has crept into a bunch of things which must be linked to by application binaries, such as libckqt.a. So we now have some semantic confusion over what -seq means in this context and how it should be configured.

In the general case, on machines like Stampede, the mic0 host is not always directly accessible, so various tools like charmrun, should not be compiled with -mmic. On experimental machines, like NCSA's Xeon Phi, root access to mic0 is available and they might wish to use charmrun natively compiled for the MIC. IMHO, anyone doing that should be expected to solve their own wacky compilation and linking issues. Our main audience is people trying to leverage Phi on Stampede.

On a related note, the MIC environment has its own issues. Notably, there is no MIC ibverbs library to link to on stampede, so you cannot even attempt to use their verbs proxy within the MIC context.

One can build using intel MPI with one binary for host and one for MIC and use ibrun.symm to have PEs running on both host and MIC. The process currently requires some somewhat arduous hackery. That process will be covered in a different issue.

A proper native layer should be able to use the dual binary launch scheme of ibrun.symm, but the necessary underpinning is not yet developed.

#2 Updated by Eric Bohm over 5 years ago

  • Target version changed from 6.6.0 to 6.7.0

#3 Updated by Eric Bohm over 4 years ago

  • Assignee changed from Eric Bohm to Michael Robson

#4 Updated by Michael Robson almost 4 years ago

Just saving some useful changes from knc3 which is being decommissioned today:

#       renamed:    src/arch/mpi-linux-x86_64/conv-mach-mic.h -> src/arch/common/conv-mach-mic.h
#       renamed:    src/arch/mpi-linux-x86_64/ -> src/arch/common/

#5 Updated by Michael Robson almost 4 years ago

  • Target version changed from 6.7.0 to 6.8.0

#6 Updated by Sam White over 2 years ago

is this still relevant or should it be closed?

#7 Updated by Michael Robson over 2 years ago

Sam White wrote:

is this still relevant or should it be closed?

I'm not sure. Kavitha or Seonmyeong can comment further as they've been using new KNL hardware recently.

#8 Updated by Sam White over 2 years ago

  • Category set to Machine Layers
  • Assignee changed from Michael Robson to PPL
  • Target version changed from 6.8.0 to Unscheduled

KNL is self-hosted and doesn't require the complicated dual binary launch scheme. Since this is off the radar for now, but something similar could appear in the future, reassigning to PPL for now.

#9 Updated by Sam White over 1 year ago

  • Target version deleted (Unscheduled)
  • Status changed from New to Closed

Also available in: Atom PDF