[maker-devel] Segmentation fault of MKAER with openmpi on CentOS 7.2

Fri May 13 11:16:49 MDT 2016

It’s possible it was set wrong as there may be more than one libmpi.so on the system. It also has to be set before compiling and every time you run. 

The next issue is that some systems (like ubuntu) will often have extra mpicc, libmpi.so, and mpiexec files that don’t match the OpenMPI you are trying to use. Tracking down those mismatches before compiling and ensuring that they don’t revert with your bashrc/bash_profile can be complicated. In these cases you may also have to additionally specify LD_PRELOAD with the -x parameter of the OpenMPI mpiexec command.

You often have to specify the ‘-mca btl’ parameter explained in the INSTALL file as well.

—Carson

> On May 13, 2016, at 11:08 AM, Ray Cui <platycerus at gmail.com> wrote:
> 
> Hello,
> 
>          I had segfaults even if I set LD_PRELOAD and used gcc for OpenMPI (dealing with Maker 3 beta though). 
>          It works fine with MpiCH so I stopped looking into this.
> 
> Ray
> 
> On Fri, May 13, 2016 at 6:32 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
> With OpenMPI, you must set LD_PRELOAD for libmpi.so and sometimes the ‘-mca btl paramter’. Details can be found in the …/maker/INSTALL file.
> 
> Also we have found a recent issue with maker and intel compiled OpenMPI on CentOS systems. To get around that issue, compile OpenMPI with gcc instead of the intel compiler, or alternatively manually install a separate perl installation without pthread support (i.e. pthreads disabled during the configure step).
> 
> —Carson
> 
> 
> 
>> On May 12, 2016, at 3:49 PM, Ray Cui <platycerus at gmail.com <mailto:platycerus at gmail.com>> wrote:
>> 
>> Dear Yugui
>> 
>>     I had the same problem with openmpi. I think it is not compatible with Maker. I now use mpich, which works.
>> 
>> Ray
>> 
>> On May 12, 2016 11:32 PM, "Yugui Wang" <wangyugui.wei at gmail.com <mailto:wangyugui.wei at gmail.com>> wrote:
>> Hi.
>> 
>> Segmentation fault of MKAER with openmpi on CentOS 7.2.
>> Both MAKER 2.31.8 and 3.00.0 beta have the same error.
>> 
>> $ mpirun -mca btl ^openib -n 4 maker
>> STATUS: Parsing control files...
>> STATUS: Processing and indexing input FASTA files...
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 2 with PID 39507 on node T620 exited
>> on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> $ file core.39505
>> core.39505: ELF 64-bit LSB core file x86-64, version 1 (SYSV),
>> SVR4-style, from '/usr/bin/perl /bio/hpc-bio/maker-3.00.0/bin/make
>> $ gdb /usr/bin/perl core.39505
>> (gdb) where
>> #0  0x00007f0e4a7d2060 in ?? ()
>> #1  <signal handler called>
>> #2  0x00007f0e4a7d2060 in ?? ()
>> #3  <signal handler called>
>> #4  0x00007f0e4bdfba50 in mca_btl_vader_component_progress () from
>> /usr/lib64/openmpi/lib/openmpi/mca_btl_vader.so
>> #5  0x00007f0e63ec8eda in opal_progress () from
>> /usr/lib64/openmpi/lib/libopen-pal.so.13
>> #6  0x00007f0e4a191ac5 in mca_pml_ob1_probe () from
>> /usr/lib64/openmpi/lib/openmpi/mca_pml_ob1.so
>> #7  0x00007f0e65b0dc06 in PMPI_Probe () from /usr/lib64/openmpi/lib/libmpi.so
>> #8  0x00007f0e59007020 in C_MPI_Recv (buf=buf at entry=0x4146b30,
>> source=source at entry=-1, tag=tag at entry=1111) at MPI.xs:56
>> #9  0x00007f0e590071e3 in XS_Parallel__Application__MPI_C_MPI_Recv
>> (my_perl=<optimized out>, cv=<optimized out>) at MPI.c:391
>> #10 0x00007f0e657ce39f in Perl_pp_entersub () from
>> /usr/lib64/perl5/CORE/libperl.so
>> #11 0x00007f0e657c6b16 in Perl_runops_standard () from
>> /usr/lib64/perl5/CORE/libperl.so
>> #12 0x00007f0e65763925 in perl_run () from /usr/lib64/perl5/CORE/libperl.so
>> #13 0x0000000000400d99 in main ()
>> $ echo $LD_PRELOAD
>> /usr/lib64/openmpi/lib/libmpi.so:
>> $ echo $OMPI_MCA_mpi_warn_on_fork
>> 0
>> $ rpm -qa openmpi
>> openmpi-1.10.0-10.el7.x86_64
>> $ uname -a
>> Linux T620 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC
>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>> $ ulimit -a
>> core file size          (blocks, -c) unlimited
>> data seg size           (kbytes, -d) unlimited
>> scheduling priority             (-e) 0
>> file size               (blocks, -f) unlimited
>> pending signals                 (-i) 1029973
>> max locked memory       (kbytes, -l) 64
>> max memory size         (kbytes, -m) unlimited
>> open files                      (-n) 1024
>> pipe size            (512 bytes, -p) 8
>> POSIX message queues     (bytes, -q) 819200
>> real-time priority              (-r) 0
>> stack size              (kbytes, -s) 102400
>> cpu time               (seconds, -t) unlimited
>> max user processes              (-u) 4096
>> virtual memory          (kbytes, -v) unlimited
>> file locks                      (-x) unlimited
>> $ mpiexec --version
>> mpiexec (OpenRTE) 1.10.0
>> 
>> Report bugs to http://www.open-mpi.org/community/help/ <http://www.open-mpi.org/community/help/>
>> $
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20160513/6437f4d8/attachment-0003.html>