<div dir="ltr">I use '<span style="font-size:12.800000190734863px">--mca btl ^openib' and it runs on multiple nodes. It works and I see some sequences is done for the test run. </span><div><span style="font-size:12.800000190734863px"><br></span></div><div><span style="font-size:12.800000190734863px">Then I make another run using the large nr database and use local space on the computer cluster, which fails. </span></div><div><span style="font-size:12.800000190734863px">Submit CMD:</span></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">sbatch --gres=lscratch:100 --time=168:00:00 --partition=multinode --constraint=x2680 --mem-per-cpu=64g --ntasks=8 --ntasks-per-core=1 --job-name run05.mpi -o log.mpi.00/run05.mpi.o%A <a href="http://run05.mpi.sh" target="_blank">run05.mpi.sh</a></span></p></div><div><span style="font-size:12.800000190734863px">Error message:</span></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Widget::tblastx:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">/usr/local/apps/blast/ncbi-<wbr>blast-2.5.0+/bin/tblastx -db /lscratch/47455932/maker_<wbr>BLLXNq/rna%2Efasta.mpi.10.21 -query /lscratch/47455932/maker_<wbr>BLLXNq/50/tig00017383_arrow.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp 500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true -show_gis -out /gpfs/gsfs6/users/chenz11/<wbr>goldfish/11549472/sergey_<wbr>canu70x/arrow/maker5/goldfish.<wbr>arrow.renamed.maker.output/<wbr>goldfish.arrow.renamed_<wbr>datastore/5A/65/tig00017383_<wbr>arrow//theVoid.tig00017383_<wbr>arrow/0/tig00017383_arrow.0.<wbr>rna%2Efasta.tblastx.temp_dir/<wbr>rna%2Efasta.mpi.10.21.tblastx</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#-----------------------------<wbr>--#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Thread 1 terminated abnormally: can't open /lscratch/47455932/mpiavG_z: No such file or directory at /home/chenz11/program/maker_<wbr>mpi/bin/maker line 1460 thread 1.</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">--> rank=37, hostname=cn4120</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">FATAL: Thread terminated, causing all processes to fail</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">--> rank=37, hostname=cn4120</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>-------------------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Primary job  terminated normally, but 1 process returned</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">a non-zero exit code.. Per user-direction, the job has been aborted.</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>-------------------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">running  blast search.</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Widget::tblastx:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">/usr/local/apps/blast/ncbi-<wbr>blast-2.5.0+/bin/tblastx -db /lscratch/47455932/maker_Zsg_<wbr>Gg/rna%2Efasta.mpi.10.8 -query /lscratch/47455932/maker_Zsg_<wbr>Gg/62/tig00001111_arrow.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp 500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true -show_gis -out /gpfs/gsfs6/users/chenz11/<wbr>goldfish/11549472/sergey_<wbr>canu70x/arrow/maker5/goldfish.<wbr>arrow.renamed.maker.output/<wbr>goldfish.arrow.renamed_<wbr>datastore/B8/A5/tig00001111_<wbr>arrow//theVoid.tig00001111_<wbr>arrow/0/tig00001111_arrow.0.<wbr>rna%2Efasta.tblastx.temp_dir/<wbr>rna%2Efasta.mpi.10.8.tblastx</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#-----------------------------<wbr>--#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">An MPI communication peer process has unexpectedly disconnected.  This</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">usually indicates a failure in the peer process (e.g., a crash or</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">otherwise exiting without calling MPI_FINALIZE first).</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Although this local MPI process will likely now behave unpredictably</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">(it may even hang or crash), the root cause of this problem is the</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">failure of the peer -- that is what you need to investigate.  For</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">example, there may be a core file that you can examine.  More</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">generally: such peer hangups are frequently caused by application bugs</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">or other external events.</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">  Local host: cn4130</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">  Local PID:  18831</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">  Peer host:  cn3683</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">formating database...</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">


























</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Widget::formater:</span></p><div><span style="font-variant-ligatures:no-common-ligatures"><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">/usr/local/apps/blast/ncbi-<wbr>blast-2.5.0+/bin/makeblastdb -dbtype prot -in /lscratch/47455932/maker_<wbr>rNzO3X/27/blastprep/protein2%<wbr>2Efasta.mpi.10.25</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#-----------------------------<wbr>--#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)">...</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p><div><span style="font-variant-ligatures:no-common-ligatures"><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><br></p><div><span style="font-variant-ligatures:no-common-ligatures">...</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">[cn3683:36010] 59 more processes have sent help message help-mpi-btl-tcp.txt / peer hung up</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">[cn3683:36010] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / </span><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)">error</span><span style="font-variant-ligatures:no-common-ligatures"> messages</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">mpiexec detected that one or more processes exited with non-zero status, thus causing</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">the job to be terminated. The first process to do so was:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">  Process name: [[352,1],37]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">  Exit code:    255</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></p></span></div></span></div></span></div></span></div><div><span style="font-size:12.800000190734863px"> </span></div><div><span style="font-size:12.800000190734863px"><br></span></div><div><span style="font-size:12.800000190734863px">I rebuild the mpi_blast and rerun it again, also get the error:</span></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Widget::formater:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">/usr/local/apps/blast/ncbi-blast-2.5.0+/bin/makeblastdb -dbtype nucl -in /lscratch/47559740/maker_k6a7Hy/32/blastprep/rna%2Efasta.mpi.10.3</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#-------------------------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Thread 1 terminated abnormally: can't open /lscratch/47559740/mpiS84Ju: No such file or directory at /home/chenz11/program/maker_mpi/bin/maker line 1460 thread 1.</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">--> rank=27, hostname=cn4115</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)">FATAL</span><span style="font-variant-ligatures:no-common-ligatures">: Thread terminated, causing all processes to fail</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">--> rank=27, hostname=cn4115</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">deleted:276 hits</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">doing tblastx of alt-ESTs</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">formating database...</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Widget::formater:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">/usr/local/apps/blast/ncbi-blast-2.5.0+/bin/makeblastdb -dbtype nucl -in /lscratch/47559740/maker_nCKTgE/2/blastprep/rna%2Efasta.mpi.10.11</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#-------------------------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">running  blast search.</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Widget::tblastx:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">/usr/local/apps/blast/ncbi-blast-2.5.0+/bin/tblastx -db /lscratch/47559740/maker_0kWZTA/rna%2Efasta.mpi.10.20 -query /lscratch/47559740/maker_0kWZTA/35/tig00027947_arrow.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp 500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true -show_gis -out /gpfs/gsfs6/users/chenz11/goldfish/11549472/sergey_canu70x/arrow/maker5/goldfish.arrow.renamed.maker.output/goldfish.arrow.renamed_datastore/86/7F/tig00027947_arrow//theVoid.tig00027947_arrow/0/tig00027947_arrow.0.rna%2Efasta.tblastx.temp_dir/rna%2Efasta.mpi.10.20.tblastx</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">#-------------------------------#</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">-------------------------------------------------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Primary job  terminated normally, but 1 process returned</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">a non-zero exit code.. Per user-direction, the job has been aborted.</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">-------------------------------------------------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">SIGTERM received</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Perl exited with active threads:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    1 running and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 finished and unjoined</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">    0 running and detached</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">Thanks,</span></div><div><span style="font-variant-ligatures:no-common-ligatures">Zelin</span></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div>--------------------------------------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>]<br></div><div><br></div><div><div>NIH/NHGRI</div><div><span style="color:rgb(0,0,0);font-family:Verdana,Geneva,sans-serif;font-size:12px">Building 50, Room 5531</span><br style="color:rgb(0,0,0);font-family:Verdana,Geneva,sans-serif;font-size:12px"><span style="color:rgb(0,0,0);font-family:Verdana,Geneva,sans-serif;font-size:12px">50 SOUTH DR, MSC 8004 </span><br style="color:rgb(0,0,0);font-family:Verdana,Geneva,sans-serif;font-size:12px"><span style="color:rgb(0,0,0);font-family:Verdana,Geneva,sans-serif;font-size:12px">BETHESDA, MD 20892-8004</span></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Aug 15, 2017 at 5:13 PM, Carson Holt <span dir="ltr"><<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Some notes:<div><br></div><div>First, the mpiexec command still needs the --mca parameters (either  '--mca btl ^openib' or '--mca btl vader,tcp,self --mca btl_tcp_if_include ib0’). Otherwise if you have infiniband on the nodes it will try and use OpenFabrics compatible libraries which will kill code doing system calls (like MAKER does).</div><div><br></div><div>Second, try using a higher count than 2 in your batch. One process is always sacrificed by maker to act only for message management among processes, so with -n 2, you have one process working and one managing data. So only one contig will run at a time. If you set it to a higher number the issue will go away. The message manger process starts to get saturated at ~200 CPUs, so anything above that processor count becomes less beneficial to the job.</div><div><br></div><div>Thanks,</div><div>Carson</div><div><br><div><br></div><div><br></div><div><br><div><blockquote type="cite"><div>On Aug 15, 2017, at 3:05 PM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>> wrote:</div><br class="m_-6541627945015036278Apple-interchange-newline"><div><div dir="ltr">I submit a job:<div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">sbatch --gres=lscratch:100 --time=8:00:00 --mem-per-cpu=8g -N 1-1 --ntasks=2 --ntasks-per-core=1 --job-name run06.mpi -o log/run06.mpi.o%A <a href="http://run06.maker.mpi.sh/" target="_blank">run06.maker.mpi.sh</a></span></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">CMD in <a href="http://run06.maker.mpi.sh/" target="_blank">run06.maker.mpi.sh</a></div><div class="gmail_extra"><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">mpiexec -n $SLURM_NTASKS maker -c 1 -base genome -g genome.fasta</span></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">Another question:</div><div class="gmail_extra">How much temporary space and memory should I use for a ~10Mb sequences and large database like nr and uniref90.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra">zelin</div><div class="gmail_extra"><br clear="all"><div><div class="m_-6541627945015036278gmail_signature"><div dir="ltr"><div dir="ltr"><div>------------------------------<wbr>--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>]<br><br></div></div></div></div>
<br><div class="gmail_quote">On Tue, Aug 15, 2017 at 4:50 PM, Carson Holt <span dir="ltr"><<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>What is your command line? Are you running interactively or as a submitted batch? If it's a batch job what options did you give it?<br><br><div><span style="font-size:13pt">--Carson</span></div><div><br></div><div>Sent from my iPhone</div></div><div><br>On Aug 15, 2017, at 2:47 PM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Hi Carson,  <span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap">Christopher, Daniel,</span><div><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap"><br></span></div><div><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap">Thank you for your kind help.</span></div><div><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap"><br></span></div><div><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap">N</span>ow it works without any other options on one nodes and 4 CPUs. I set the number of task to 2, but there's only one contigs in running. Should it be two contigs running at the same time?<div><br></div><div>Zelin</div></div><div class="gmail_extra"><br clear="all"><div><div class="m_-6541627945015036278gmail-m_-8645049413834822486gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div>------------------------------<wbr>--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>]<br><br></div><div><br></div><div><div>NIH/NHGRI</div><div><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px">Building 50, Room 5531</span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px"><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px">50 SOUTH DR, MSC 8004 </span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px"><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px">BETHESDA, MD 20892-8004</span></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Aug 15, 2017 at 11:47 AM, Carson Holt <span dir="ltr"><<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">Did it die or did you just get a warning?<div><br></div><div>Here is a list of flags to add that suppress warnings and other issues with OpenMPI. You can add them all or one at a time depending on issues you get.</div><div><br></div><div>#add if MPI not using all CPU given</div><div>--oversubscribe --bind-to none</div><div><br></div><div>#workaround for infinaband (use instead of --mca ^openib)</div><div>--mca btl vader,tcp,self --mca btl_tcp_if_include ib0</div><div><br></div><div>#add to stop certain other warnings</div><div>--mca orte_base_help_aggregate 0</div><div><br></div><div>#stop fork warnings</div><div>--mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0</div><div><br></div><div>—Carson</div><div><br></div><div><br></div><div><br><div><blockquote type="cite"><div>On Aug 15, 2017, at 9:34 AM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>> wrote:</div><br class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704Apple-interchange-newline"><div><div dir="ltr"><div class="gmail_extra">Here are some latest message:<div><br></div><div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">[cn3360:57176] 1 more process has sent help message help-opal-runtime.txt / opal_init:warn-fork</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">[cn3360:57176] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><div class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704m_-1862143133597923131gmail-m_5723736450607837839gmail_signature"><div dir="ltr"><div dir="ltr"><div><div>------------------------------<wbr>--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>]<br><br></div><div><br></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Aug 15, 2017 at 10:39 AM, Carson Holt <span dir="ltr"><<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>You may need to delete the .../maker/perl directory before doing the reinstall if not doing a brand new installation. Otherwise you can ignore the subroutine redefined warnings during compile.</div><div><br></div><div>Have you been able to test the alternate flags on the command line for MPI? How about an alternate perl without threads?<br><br><div><div><span style="font-size:13pt">--Carson</span></div><div><br></div><div>Sent from my iPhone</div></div></div><div><br>On Aug 15, 2017, at 8:27 AM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">When I installed by './Build install', I got following some messages:<div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Configuring MAKER with MPI support</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Installing MAKER...</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Configuring MAKER with MPI support</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine dl_load_flags redefined at (eval 125) line 8.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_ANY_SOURCE redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_ANY_TAG redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_SUCCESS redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_Init redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_Finalize redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_Comm_rank redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_Comm_size redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_Send redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::C_<wbr>MPI_Recv redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)"><span style="font-variant-ligatures:no-common-ligatures">Subroutine Parallel::Application::MPI::_c<wbr>omment redefined at (eval 125) line 9.</span></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">I'm not sure whether it's correctly installed.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra"><br clear="all"><div><div class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704m_-1862143133597923131gmail-m_5723736450607837839m_-7744257814608198508gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div>------------------------------<wbr>--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>]<br><br></div><div><div>NIH/NHGRI</div><div><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px">Building 50, Room 5531</span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px"><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px">50 SOUTH DR, MSC 8004 </span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px"><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px">BETHESDA, MD 20892-8004</span></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Aug 14, 2017 at 9:23 PM, Fields, Christopher J <span dir="ltr"><<a href="mailto:cjfields@illinois.edu" target="_blank">cjfields@illinois.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">







<div bgcolor="white" lang="EN-US">
<div class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704m_-1862143133597923131gmail-m_5723736450607837839m_-7744257814608198508m_-5812194475710527082WordSection1"><p class="MsoNormal">Carson,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">It was attached to the initial message (named ‘run05.mpi.o47346077’).  It looks like a Perl issue with threads, though I don’t see why this would crash a cluster.  The fact there is a log file would suggest it just ended the job.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">chris<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0in 0in"><p class="MsoNormal" style="margin-left:0.5in"><b><span style="font-size:12pt">From:
</span></b><span style="font-size:12pt">maker-devel <<a href="mailto:maker-devel-bounces@yandell-lab.org" target="_blank">maker-devel-bounces@yandell-l<wbr>ab.org</a>> on behalf of Carson Holt <<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>><br>
<b>Date: </b>Monday, August 14, 2017 at 2:18 PM<br>
<b>To: </b>zl c <<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>><br>
<b>Cc: </b>"<a href="mailto:maker-devel@yandell-lab.org" target="_blank">maker-devel@yandell-lab.org</a>" <<a href="mailto:maker-devel@yandell-lab.org" target="_blank">maker-devel@yandell-lab.org</a>><br>
<b>Subject: </b>Re: [maker-devel] maker MPI problem<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">This is rather vague —> “crashed the computer cluster”
<u></u><u></u></p>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in">Do you have a specific error?<u></u><u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in">—Carson<u></u><u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div><p class="MsoNormal" style="margin-left:0.5in">On Aug 14, 2017, at 12:59 PM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>> wrote:<u></u><u></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<div>
<div><p class="MsoNormal" style="margin-left:0.5in">
Hello,<u></u><u></u></p>
<div><p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">
I ran maker 3.0 with openmpi 2.0.2 and it crashed the computer cluster. I attached the log file. Could you help me to solve the problem?<u></u><u></u></p>
<div><p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">
CMD:<u></u><u></u></p><p class="MsoNormal" style="margin-left:0.5in">
export LD_PRELOAD=/usr/local/OpenMPI/<wbr>2.0.2/gcc-6.3.0/lib/libmpi.so<u></u><u></u></p><p class="MsoNormal" style="margin-left:0.5in">
export OMPI_MCA_mpi_warn_on_fork=0<u></u><u></u></p><p class="MsoNormal" style="margin-left:0.5in">
mpiexec -mca btl ^openib -n $SLURM_NTASKS maker -c 1 –base genome  -g genome.fasta<u></u><u></u></p>
<div><p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">
Thanks,<u></u><u></u></p><p class="MsoNormal" style="margin-left:0.5in">
Zelin Chen<u></u><u></u></p>
<div><p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div><p class="MsoNormal" style="margin-left:0.5in">------------------------------<wbr>--------------<u></u><u></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank">chzelin@gmail.com</a>]  Ph.D.<u></u><u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal" style="margin-left:0.5in">NIH/NHGRI<u></u><u></u></p>
</div>
<div>
<div><p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:9pt;font-family:Verdana,sans-serif">Building 50, Room 5531<br>
50 SOUTH DR, MSC 8004 <br>
BETHESDA, MD 20892-8004</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div><p class="MsoNormal" style="margin-left:0.5in"><run05.mpi.o47346077>_________<wbr>______________________________<wbr>________<br>
maker-devel mailing list<br>
<a href="mailto:maker-devel@box290.bluehost.com" target="_blank">maker-devel@box290.bluehost.co<wbr>m</a><br>
<a href="http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org" target="_blank">http://box290.bluehost.com/mai<wbr>lman/listinfo/maker-devel_yand<wbr>ell-lab.org</a><u></u><u></u></p>
</div>
</blockquote>
</div><p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
</div>
</div>

</blockquote></div><br></div></div>
</div></blockquote></div></blockquote></div><br></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br></div></div>
</div></blockquote></div></blockquote></div><br></div></div>
</div></blockquote></div><br></div></div></div></blockquote></div><br></div></div>