[maker-devel] ERROR: MPI_Recv(186), dequeue_and_set_error(596)

Wed Jul 25 13:15:59 MDT 2012

Hi everyone,

I ran maker2.25 without a problem, but with maker2.26, I encountered the
following error after running it for ~8 hr with 2 nodes and 24 cpus, do you
have any idea what's going on here? Some contigs did get finished, maybe
this is not a big problem. My mpich2 version 1.4.1p1, job scheduling system
is SGE. Thanks!

running  blast search.
#--------- command -------------#
Widget::blastx:
/home/yunfeiguo/Downloads/maker/bin/../exe/blast/bin/blastx -db
/tmp/6480.1.all.q/maker_PQOTIq/concatPro%2Etxt.mpi.10.4 -query
/tmp/6480.1.all.q/maker_PQOTIq/rank3/scaffold2602.0 -num_alignments 10000
-num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000
-num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/7A/37/scaffold2602//theVoid.scaffold2602/scaffold2602.0.concatPro%2Etxt.blastx.temp_dir/concatPro%2Etxt.mpi.10.4.blastx
#-------------------------------#
deleted:-1 hits
SIGCHLD handler "DEFAULT" not defined.
SIGCHLD handler "DEFAULT" not defined.
running  exonerate search.
#--------- command -------------#
Widget::exonerate::protein2genome:
/home/username/usr/bin/exonerate -q
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/sp%7CQ8N8A2%7CANR44_HUMAN.for.1-3712.8.fasta
-t
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/scaffold2590.1-3712.8.fasta
-Q protein -T dna -m protein2genome --softmasktarget  --percent 20
--showcigar  >
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/scaffold2590.1-3712.sp%7CQ8N8A2%7CANR44_HUMAN.p_exonerate.8
#-------------------------------#
Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(186).............: MPI_Recv(buf=0x7fffa3a2e760, count=2, MPI_INT,
src=MPI_ANY_SOURCE, tag=1111, MPI_COMM_WORLD, status=0x7fffa3a2e740) failed
dequeue_and_set_error(596): Communication error with rank 18
running  blast search.
#--------- command -------------#
Widget::blastx:
/home/yunfeiguo/Downloads/maker/bin/../exe/blast/bin/blastx -db
/tmp/6480.1.all.q/maker_PQOTIq/concatPro%2Etxt.mpi.10.8 -query
/tmp/6480.1.all.q/maker_PQOTIq/rank11/scaffold2575.0 -num_alignments 10000
-num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000
-num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F0/AE/scaffold2575//theVoid.scaffold2575/scaffold2575.0.concatPro%2Etxt.blastx.temp_dir/concatPro%2Etxt.mpi.10.8.blastx
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::tblastx:
/home/yunfeiguo/Downloads/maker/bin/../exe/blast/bin/tblastx -db
/tmp/6480.1.all.q/maker_PQOTIq/AllSebESTs_plus_Rubri%2Efasta.mpi.10.1
-query /tmp/6480.1.all.q/maker_PQOTIq/rank7/scaffold2620.0 -num_alignments
10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp
500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true
-show_gis -out
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/6B/FB/scaffold2620//theVoid.scaffold2620/scaffold2620.0.AllSebESTs_plus_Rubri%2Efasta.tblastx.temp_dir/AllSebESTs_plus_Rubri%2Efasta.mpi.10.1.tblastx
#-------------------------------#
running  exonerate search.
#--------- command -------------#
Widget::exonerate::protein2genome:
/home/username/usr/bin/exonerate -q
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/sp%7CQ8NB46%7CANR52_HUMAN.for.1-3712.8.fasta
-t
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/scaffold2590.1-3712.8.fasta
-Q protein -T dna -m protein2genome --softmasktarget  --percent 20
--showcigar  >
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/scaffold2590.1-3712.sp%7CQ8NB46%7CANR52_HUMAN.p_exonerate.8
#-------------------------------#
cleaning blastx...
in cluster::shadow_cluster...
...finished clustering.
cleaning clusters....
total clusters:1 now processing 0
 ...processing 0 of 2
deleted:0 hits
 ...processing 1 of 2
running  blast search.
#--------- command -------------#
Widget::tblastx:
/home/yunfeiguo/Downloads/maker/bin/../exe/blast/bin/tblastx -db
/tmp/6480.1.all.q/maker_PQOTIq/AllSebESTs_plus_Rubri%2Efasta.mpi.10.6
-query /tmp/6480.1.all.q/maker_PQOTIq/rank9/scaffold2615.0 -num_alignments
10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp
500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true
-show_gis -out
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/E2/6E/scaffold2615//theVoid.scaffold2615/scaffold2615.0.AllSebESTs_plus_Rubri%2Efasta.tblastx.temp_dir/AllSebESTs_plus_Rubri%2Efasta.mpi.10.6.tblastx
#-------------------------------#
deleted:0 hits
running  exonerate search.
#--------- command -------------#
Widget::exonerate::protein2genome:
/home/username/usr/bin/exonerate -q
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/tr%7CE7F7S0%7CE7F7S0_DANRE.for.1-3712.9.fasta
-t
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/scaffold2590.1-3712.9.fasta
-Q protein -T dna -m protein2genome --softmasktarget  --percent 20
--showcigar  >
/home/yunfeiguo/projects/fish/Nigro/run/dir_Nigro-53k00/Nigro-53k_part.maker.output/Nigro-53k_part_datastore/F9/9B/scaffold2590//theVoid.scaffold2590/
scaffold2590.1-3712.tr%7CE7F7S0%7CE7F7S0_DANRE.p_exonerate.9
#-------------------------------#
deleted:0 hits
cleaning blastx...
cleaning clusters....
total clusters:1 now processing 0
cleaning clusters....
total clusters:1 now processing 0
deleted:-1 hits
deleted:-1 hits
deleted:-6 hits
deleted:-3 hits
deleted:-2 hits
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached

Yunfei
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120725/5185d9af/attachment-0002.html>