[maker-devel] Segmentation fault of MKAER with openmpi on CentOS 7.2

[Yugui Wang]

Hi.

Segmentation fault of MKAER with openmpi on CentOS 7.2.
Both MAKER 2.31.8 and 3.00.0 beta have the same error.

$ mpirun -mca btl ^openib -n 4 maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 39507 on node T620 exited
on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
$ file core.39505
core.39505: ELF 64-bit LSB core file x86-64, version 1 (SYSV),
SVR4-style, from '/usr/bin/perl /bio/hpc-bio/maker-3.00.0/bin/make
$ gdb /usr/bin/perl core.39505
(gdb) where
#0  0x00007f0e4a7d2060 in ?? ()
#1  <signal handler called>
#2  0x00007f0e4a7d2060 in ?? ()
#3  <signal handler called>
#4  0x00007f0e4bdfba50 in mca_btl_vader_component_progress () from
/usr/lib64/openmpi/lib/openmpi/mca_btl_vader.so
#5  0x00007f0e63ec8eda in opal_progress () from
/usr/lib64/openmpi/lib/libopen-pal.so.13
#6  0x00007f0e4a191ac5 in mca_pml_ob1_probe () from
/usr/lib64/openmpi/lib/openmpi/mca_pml_ob1.so
#7  0x00007f0e65b0dc06 in PMPI_Probe () from /usr/lib64/openmpi/lib/libmpi.so
#8  0x00007f0e59007020 in C_MPI_Recv (buf=buf at entry=0x4146b30,
source=source at entry=-1, tag=tag at entry=1111) at MPI.xs:56
#9  0x00007f0e590071e3 in XS_Parallel__Application__MPI_C_MPI_Recv
(my_perl=<optimized out>, cv=<optimized out>) at MPI.c:391
#10 0x00007f0e657ce39f in Perl_pp_entersub () from
/usr/lib64/perl5/CORE/libperl.so
#11 0x00007f0e657c6b16 in Perl_runops_standard () from
/usr/lib64/perl5/CORE/libperl.so
#12 0x00007f0e65763925 in perl_run () from /usr/lib64/perl5/CORE/libperl.so
#13 0x0000000000400d99 in main ()
$ echo $LD_PRELOAD
/usr/lib64/openmpi/lib/libmpi.so:
$ echo $OMPI_MCA_mpi_warn_on_fork
0
$ rpm -qa openmpi
openmpi-1.10.0-10.el7.x86_64
$ uname -a
Linux T620 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1029973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 102400
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
$ mpiexec --version
mpiexec (OpenRTE) 1.10.0

Report bugs to http://www.open-mpi.org/community/help/
$