[maker-devel] Maker-Error when started with IMPI

[Carson Holt]

Specific things.

1. Do not set LD_PRELOAD. That is only for OpenMPI, but it will cause problems with other MPI's.

2. Make sure you recompiled MAKER for Intel MPI (MPI code always has to be compiled for the flavor you are using, so make sure you have a separate installation of MAKER for Intel MPI). Also validate that the mpicc and libmpi.h listed during the MAKER install belong to Intel MPI. Don’t just assume they do because you loaded the module. Manually verify the paths during MAKER’s setup.

3. The error you got previously should not even be possible with the current version of Intel MPI, which is why I say that when you called mpiexec, something else (that was not Intel MPI) was launched. Easy solution is to give the full path of mpiexec in your job, so are not relying on PATH to be unaltered in your job.

Do not do —>  mpiexec -nc 1 maker
Do this for example —> /opt/bwhpc/common/compiler/intel/compxe.2016.4.258/impi/5.1.3.223/intel64/bin/mpiexec -nc maker

4. Build and run on the same node for your test. If you build on one node and run on another, you may be changing your environment in ways you don’t realize that break things. So if you can build and test on the same node and it works, then it fails when you test it elsewhere, then you have to track down how your environment is changing.

—Carson





> On Feb 24, 2017, at 1:43 AM, Rainer Rutka <rainer.rutka at uni-konstanz.de> wrote:
> 
> Hi Carson.
> 
> First of all THANK YOU FOR YOUR HELP.
> MUCH APPRECIATED. :-)
> 
> Am 23.02.2017 um 21:22 schrieb Carson Holt:
>> It means one of two things.
>> 1. The mpiexec you called is not from Intel MPI (try 'which mpiexec' to verify the location and that it is not some other random mpiexec executable).
> 
> No, I am starting the right version of mpiexec.
> 
> This is a list of all our current available MPI-Versions, corresponding
> to their compiler:
> 
> UC:[kn at uc1n997 ~]$ module avail mpi
> ------------------------------------- /opt/bwhpc/common/modulefiles --------------------------------------
> mpi/impi/4.1.3-gnu-4.4 mpi/openmpi/1.10-intel-15.0
> mpi/impi/4.1.3-gnu-4.7 mpi/openmpi/1.10-intel-16.0
> mpi/impi/4.1.3-intel-14.0 mpi/openmpi/1.8-gnu-4.5
> mpi/impi/5.0.3-gnu-4.4 mpi/openmpi/1.8-gnu-4.7
> mpi/impi/5.0.3-gnu-4.7 mpi/openmpi/1.8-gnu-4.7-m32
> mpi/impi/5.0.3-intel-15.0 mpi/openmpi/1.8-gnu-4.8
> mpi/impi/5.1.3-gnu-4.7 mpi/openmpi/1.8-gnu-4.9
> mpi/impi/5.1.3-gnu-system mpi/openmpi/1.8-intel-14.0(default)
> 
> mpi/impi/5.1.3-intel-16.0(default)
> 
> mpi/openmpi/1.8-intel-15.0
> mpi/openmpi/1.10-gnu-4.5 mpi/openmpi/2.0-gnu-4.7
> mpi/openmpi/1.10-gnu-4.7 mpi/openmpi/2.0-gnu-4.8
> mpi/openmpi/1.10-gnu-4.8 mpi/openmpi/2.0-gnu-5.2
> mpi/openmpi/1.10-gnu-4.9 mpi/openmpi/2.0-intel-15.0
> mpi/openmpi/1.10-gnu-5.2 mpi/openmpi/2.0-intel-16.0
> mpi/openmpi/1.10-intel-14.0
> 
> 
> Here I load the MPI-Module including all it's dependencies:
> 
> UC:[kn at uc1n997 ~]$ module load mpi/impi/5.1.3-intel-16.0
> Loading module dependency 'compiler/intel/16.0'.
> 
> Result:
> UC:[kn at c1n997 ~]$ which mpiexec
> /opt/bwhpc/common/compiler/intel/compxe.2016.4.258/impi/5.1.3.223/intel64/bin/mpiexec
> UC:[kn at uc1n997 ~]$
> 
>> 2. It is an extremely old version of Intel MPI (in which case the instructions I gave you would not apply)
> Extremely old?
> 
> [...]
> Intel(R) MPI Library for Linux* OS, Version 5.1.3 Build 20160601 (build id: 15562)
> 
>> MPD is an old launcher used in MPICH1 and early versions of MPICH2. It’s been abandoned since about 2008 when MPICH2 switched to the hydra launcher which is still used in MPICH3.
> Yes. And we don't even use MPD at our clusters :-)
> 
>> So either you are pointing to an old version of MPICH or you have a very old version of Intel MPI based off of MPICH.
> I do not have any MPICH available on our cluster(s) now. All of the old versions had
> been removed some years ago.
> 
> At a glance
> ---------------
> 
> Maker is available as a so-called module on our cluster system. It's been build
> on a developers' node I can access. But, the MPI-modules are built by other
> fellows (e.g. at the KIT in Karlsruhe/Germany) on other nodes.
> 
> Please check the module-file (included in this mail)
> 
> bio-maker-2.31.8_impi
> 
> to see how Maker was build including the envoronments set by this
> module.
> 
> e.g.:
> ./Build status
> verify dependencies:
> ===================================================
> STATUS MAKER v2.31.8
> ===================================================
> PERL Dependencies: VERIFIED
> External Programs: VERIFIED
> External C Libraries: VERIFIED
> MPI SUPPORT: ENABLED
> MWAS Web Interface: DISABLED
> MAKER PACKAGE: CONFIGURATION OK
> 
> 
> At least, please(!) have a look at our m.moab file (included in this mail).
> This is the way how we submit a Maker job to our cluster. Maybe something
> is wrong here?
> 
> Sorry again for wasting your time. But we imperatively need the Maker
> software running in parallel mode.
> 
> :-)
> 
> -- 
> Rainer Rutka
> University of Konstanz
> Communication, Information, Media Centre (KIM)
> * High-Performance-Computing (HPC)
> * KIM-Support and -Base-Services
> Room: V511
> 78457 Konstanz, Germany
> +49 7531 88-5413
> <bio-maker-2.31.8_impi><m.moab>