[maker-devel] MPI selection

[Carson Holt]

The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.

The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access. As a work around OpenMPI and Intel MPI allow you to disable infiniband libraries via command line flags and use IP over infiniband instead (i.e. they let you drop infiniband features on demand so that your code will run). However MVAPICH2 does not provide the same option. As a result you cannot use MAKER or any MPI program that does system calls to external programs with MVAPICH2 (it results in seg faults). But you can use all other MPI flavors with the appropriate flags detailed below:

#For OpenMPI, use as follows (the example assumes ib0 is your ip over infiniband adapter)
export LD_PRELOAD=/path/to/openmpi/libmpi.so
mpiexec --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 --mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0 maker

#For Intel MPI set these environmental variables before launch
export I_MPI_FABRICS='shm:tcp'
export I_MPI_HYDRA_IFACE='ib0'
mpiexec maker

#For MPICH, nothing is needed as the Infiniband libraries are always disabled, but you can specifically tell it to use the ib0 adapter as the communicator
mpiexec -iface ib0 maker

—Carson


> On Jan 29, 2018, at 4:08 PM, admin at genome.arizona.edu wrote:
> 
> Hi, we have now three versions of MPI installed on our cluster, OpenMPI, MPICH, and MVAPICH2.  Since we have infiniband, the MVAPICH2 is working best with MPI test programs.  MPICH should support infiniband too but currently there are some seg faults with that we are trying to resolve.
> 
> On our cluster we have ~/.mpi-selection file which allows users to pick the MPI installation to use, and sets appropriate PATH and LD_LIBRARY_PATH variables.
> 
> I am looking through the Maker MPI instructions, and it seems that a certain mpicc and mpi.h must be chosen during installation.  So if originally, Maker was installed with MPICH, then would I have to reinstall it if users want to use MVAPICH2?  Or is there config file somewhere I can update so I don't have to reinstall Maker?  Or does nothing need to be done and we can rely on PATH and LD_LIBRARY_PATH variables pointing to correct mpicc and libmpi.so (mpi.h is in include directory)?
> 
> Thanks
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org