<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">This is the causal error —> can't open /lscratch/47455932/mpiavG_z<div class=""><br class=""></div><div class="">It kills one process and causes everything else to die in an ugly way.</div><div class=""><br class=""></div><div class="">There are several possible causes:</div><div class=""><br class=""></div><div class="">1. /lscratch/47455932/ is not actually locally mounted. It may be a virtual directory created at run time that exists on the network but not as a true locally mounted disk. If this is the case, there can be a slight IO delay under heavy IO load (common on NFS) that can cause directories and files to appear to not exist. This is one of the reasons TMP= must be sent to a true locally mounted disk. The IO load MAKER can produce can swamp network mounted disks creating strange errors.</div><div class=""><br class=""></div><div class="">2. /lscratch/47455932/ may only exist on the head node and not other nodes for the job. True local temporary storage is not available across nodes. It is only available on the node it is attached to. So if you are creating the location as part of your job, it may only exist on the head node and not the other nodes. Usually this value is set to /tmp because each machine should have it’s own independent /tmp location.</div><div class=""><br class=""></div><div class="">3. /lscratch/47455932/ exists on all nodes, but is full on one of them.</div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Aug 17, 2017, at 7:39 AM, zl c <<a href="mailto:chzelin@gmail.com" class="">chzelin@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I use '<span style="font-size:12.800000190734863px" class="">--mca btl ^openib' and it runs on multiple nodes. It works and I see some sequences is done for the test run. </span><div class=""><span style="font-size:12.800000190734863px" class=""><br class=""></span></div><div class=""><span style="font-size:12.800000190734863px" class="">Then I make another run using the large nr database and use local space on the computer cluster, which fails. </span></div><div class=""><span style="font-size:12.800000190734863px" class="">Submit CMD:</span></div><div class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">sbatch --gres=lscratch:100 --time=168:00:00 --partition=multinode --constraint=x2680 --mem-per-cpu=64g --ntasks=8 --ntasks-per-core=1 --job-name run05.mpi -o log.mpi.00/run05.mpi.o%A <a href="http://run05.mpi.sh/" target="_blank" class="">run05.mpi.sh</a></span></div></div><div class=""><span style="font-size:12.800000190734863px" class="">Error message:</span></div><div class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#--------- command -------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Widget::tblastx:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">/usr/local/apps/blast/ncbi-<wbr class="">blast-2.5.0+/bin/tblastx -db /lscratch/47455932/maker_<wbr class="">BLLXNq/rna%2Efasta.mpi.10.21 -query /lscratch/47455932/maker_<wbr class="">BLLXNq/50/tig00017383_arrow.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp 500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true -show_gis -out /gpfs/gsfs6/users/chenz11/<wbr class="">goldfish/11549472/sergey_<wbr class="">canu70x/arrow/maker5/goldfish.<wbr class="">arrow.renamed.maker.output/<wbr class="">goldfish.arrow.renamed_<wbr class="">datastore/5A/65/tig00017383_<wbr class="">arrow//theVoid.tig00017383_<wbr class="">arrow/0/tig00017383_arrow.0.<wbr class="">rna%2Efasta.tblastx.temp_dir/<wbr class="">rna%2Efasta.mpi.10.21.tblastx</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#-----------------------------<wbr class="">--#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Thread 1 terminated abnormally: can't open /lscratch/47455932/mpiavG_z: No such file or directory at /home/chenz11/program/maker_<wbr class="">mpi/bin/maker line 1460 thread 1.</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">--> rank=37, hostname=cn4120</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">FATAL: Thread terminated, causing all processes to fail</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">--> rank=37, hostname=cn4120</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">-------------------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Primary job terminated normally, but 1 process returned</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">a non-zero exit code.. Per user-direction, the job has been aborted.</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">-------------------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">running blast search.</span></div></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#--------- command -------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Widget::tblastx:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">/usr/local/apps/blast/ncbi-<wbr class="">blast-2.5.0+/bin/tblastx -db /lscratch/47455932/maker_Zsg_<wbr class="">Gg/rna%2Efasta.mpi.10.8 -query /lscratch/47455932/maker_Zsg_<wbr class="">Gg/62/tig00001111_arrow.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp 500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true -show_gis -out /gpfs/gsfs6/users/chenz11/<wbr class="">goldfish/11549472/sergey_<wbr class="">canu70x/arrow/maker5/goldfish.<wbr class="">arrow.renamed.maker.output/<wbr class="">goldfish.arrow.renamed_<wbr class="">datastore/B8/A5/tig00001111_<wbr class="">arrow//theVoid.tig00001111_<wbr class="">arrow/0/tig00001111_arrow.0.<wbr class="">rna%2Efasta.tblastx.temp_dir/<wbr class="">rna%2Efasta.mpi.10.8.tblastx</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#-----------------------------<wbr class="">--#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">------------------------------<wbr class="">--------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">An MPI communication peer process has unexpectedly disconnected. This</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">usually indicates a failure in the peer process (e.g., a crash or</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">otherwise exiting without calling MPI_FINALIZE first).</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150); min-height: 14px;" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Although this local MPI process will likely now behave unpredictably</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">(it may even hang or crash), the root cause of this problem is the</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">failure of the peer -- that is what you need to investigate. For</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">example, there may be a core file that you can examine. More</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">generally: such peer hangups are frequently caused by application bugs</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">or other external events.</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150); min-height: 14px;" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Local host: cn4130</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Local PID: 18831</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Peer host: cn3683</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">------------------------------<wbr class="">--------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">formating database...</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#--------- command -------------#</span></div><p style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">
</span></p><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Widget::formater:</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">/usr/local/apps/blast/ncbi-<wbr class="">blast-2.5.0+/bin/makeblastdb -dbtype prot -in /lscratch/47455932/maker_<wbr class="">rNzO3X/27/blastprep/protein2%<wbr class="">2Efasta.mpi.10.25</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#-----------------------------<wbr class="">--#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class="">...</div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><br class=""></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="">...</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[cn3683:36010] 59 more processes have sent help message help-mpi-btl-tcp.txt / peer hung up</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[cn3683:36010] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / </span><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)" class="">error</span><span style="font-variant-ligatures:no-common-ligatures" class=""> messages</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">------------------------------<wbr class="">--------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">mpiexec detected that one or more processes exited with non-zero status, thus causing</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">the job to be terminated. The first process to do so was:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150); min-height: 14px;" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Process name: [[352,1],37]</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Exit code: 255</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">------------------------------<wbr class="">--------------</span></div></span></div></span></div></span></div></span></div><div class=""><span style="font-size:12.800000190734863px" class=""> </span></div><div class=""><span style="font-size:12.800000190734863px" class=""><br class=""></span></div><div class=""><span style="font-size:12.800000190734863px" class="">I rebuild the mpi_blast and rerun it again, also get the error:</span></div><div class=""><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#--------- command -------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Widget::formater:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">/usr/local/apps/blast/ncbi-blast-2.5.0+/bin/makeblastdb -dbtype nucl -in /lscratch/47559740/maker_k6a7Hy/32/blastprep/rna%2Efasta.mpi.10.3</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#-------------------------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Thread 1 terminated abnormally: can't open /lscratch/47559740/mpiS84Ju: No such file or directory at /home/chenz11/program/maker_mpi/bin/maker line 1460 thread 1.</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">--> rank=27, hostname=cn4115</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)" class="">FATAL</span><span style="font-variant-ligatures:no-common-ligatures" class="">: Thread terminated, causing all processes to fail</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">--> rank=27, hostname=cn4115</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">deleted:276 hits</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">doing tblastx of alt-ESTs</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">formating database...</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#--------- command -------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Widget::formater:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">/usr/local/apps/blast/ncbi-blast-2.5.0+/bin/makeblastdb -dbtype nucl -in /lscratch/47559740/maker_nCKTgE/2/blastprep/rna%2Efasta.mpi.10.11</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#-------------------------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">running blast search.</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#--------- command -------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Widget::tblastx:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">/usr/local/apps/blast/ncbi-blast-2.5.0+/bin/tblastx -db /lscratch/47559740/maker_0kWZTA/rna%2Efasta.mpi.10.20 -query /lscratch/47559740/maker_0kWZTA/35/tig00027947_arrow.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-10 -dbsize 1000 -searchsp 500000000 -num_threads 1 -lcase_masking -seg yes -soft_masking true -show_gis -out /gpfs/gsfs6/users/chenz11/goldfish/11549472/sergey_canu70x/arrow/maker5/goldfish.arrow.renamed.maker.output/goldfish.arrow.renamed_datastore/86/7F/tig00027947_arrow//theVoid.tig00027947_arrow/0/tig00027947_arrow.0.rna%2Efasta.tblastx.temp_dir/rna%2Efasta.mpi.10.20.tblastx</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">#-------------------------------#</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">-------------------------------------------------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Primary job terminated normally, but 1 process returned</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">a non-zero exit code.. Per user-direction, the job has been aborted.</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">-------------------------------------------------------</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">SIGTERM received</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Perl exited with active threads:</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 1 running and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 finished and unjoined</span></div><div style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo; background-color: rgb(255, 222, 150);" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> 0 running and detached</span></div></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Thanks,</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Zelin</span></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">--------------------------------------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>]<br class=""></div><div class=""><br class=""></div><div class=""><div class="">NIH/NHGRI</div><div class=""><span style="font-family: Verdana, Geneva, sans-serif; font-size: 12px;" class="">Building 50, Room 5531</span><br style="font-family: Verdana, Geneva, sans-serif; font-size: 12px;" class=""><span style="font-family: Verdana, Geneva, sans-serif; font-size: 12px;" class="">50 SOUTH DR, MSC 8004 </span><br style="font-family: Verdana, Geneva, sans-serif; font-size: 12px;" class=""><span style="font-family: Verdana, Geneva, sans-serif; font-size: 12px;" class="">BETHESDA, MD 20892-8004</span></div></div></div></div></div></div></div>
<br class=""><div class="gmail_quote">On Tue, Aug 15, 2017 at 5:13 PM, Carson Holt <span dir="ltr" class=""><<a href="mailto:carsonhh@gmail.com" target="_blank" class="">carsonhh@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Some notes:<div class=""><br class=""></div><div class="">First, the mpiexec command still needs the --mca parameters (either '--mca btl ^openib' or '--mca btl vader,tcp,self --mca btl_tcp_if_include ib0’). Otherwise if you have infiniband on the nodes it will try and use OpenFabrics compatible libraries which will kill code doing system calls (like MAKER does).</div><div class=""><br class=""></div><div class="">Second, try using a higher count than 2 in your batch. One process is always sacrificed by maker to act only for message management among processes, so with -n 2, you have one process working and one managing data. So only one contig will run at a time. If you set it to a higher number the issue will go away. The message manger process starts to get saturated at ~200 CPUs, so anything above that processor count becomes less beneficial to the job.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Carson</div><div class=""><br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Aug 15, 2017, at 3:05 PM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>> wrote:</div><br class="m_-6541627945015036278Apple-interchange-newline"><div class=""><div dir="ltr" class="">I submit a job:<div class=""><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">sbatch --gres=lscratch:100 --time=8:00:00 --mem-per-cpu=8g -N 1-1 --ntasks=2 --ntasks-per-core=1 --job-name run06.mpi -o log/run06.mpi.o%A <a href="http://run06.maker.mpi.sh/" target="_blank" class="">run06.maker.mpi.sh</a></span></div></div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">CMD in <a href="http://run06.maker.mpi.sh/" target="_blank" class="">run06.maker.mpi.sh</a></div><div class="gmail_extra"><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">mpiexec -n $SLURM_NTASKS maker -c 1 -base genome -g genome.fasta</span></div></div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Another question:</div><div class="gmail_extra">How much temporary space and memory should I use for a ~10Mb sequences and large database like nr and uniref90.</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra">zelin</div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="m_-6541627945015036278gmail_signature"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">------------------------------<wbr class="">--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>]<br class=""><br class=""></div></div></div></div>
<br class=""><div class="gmail_quote">On Tue, Aug 15, 2017 at 4:50 PM, Carson Holt <span dir="ltr" class=""><<a href="mailto:carsonhh@gmail.com" target="_blank" class="">carsonhh@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="auto" class=""><div class="">What is your command line? Are you running interactively or as a submitted batch? If it's a batch job what options did you give it?<br class=""><br class=""><div class=""><span style="font-size:13pt" class="">--Carson</span></div><div class=""><br class=""></div><div class="">Sent from my iPhone</div></div><div class=""><br class="">On Aug 15, 2017, at 2:47 PM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class="">Hi Carson, <span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap" class="">Christopher, Daniel,</span><div class=""><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap" class=""><br class=""></span></div><div class=""><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap" class="">Thank you for your kind help.</span></div><div class=""><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap" class=""><br class=""></span></div><div class=""><span style="color:rgb(119,119,119);font-size:12.800000190734863px;white-space:nowrap" class="">N</span>ow it works without any other options on one nodes and 4 CPUs. I set the number of task to 2, but there's only one contigs in running. Should it be two contigs running at the same time?<div class=""><br class=""></div><div class="">Zelin</div></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="m_-6541627945015036278gmail-m_-8645049413834822486gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">------------------------------<wbr class="">--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>]<br class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">NIH/NHGRI</div><div class=""><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class="">Building 50, Room 5531</span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class=""><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class="">50 SOUTH DR, MSC 8004 </span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class=""><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class="">BETHESDA, MD 20892-8004</span></div></div></div></div></div></div></div>
<br class=""><div class="gmail_quote">On Tue, Aug 15, 2017 at 11:47 AM, Carson Holt <span dir="ltr" class=""><<a href="mailto:carsonhh@gmail.com" target="_blank" class="">carsonhh@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class="">Did it die or did you just get a warning?<div class=""><br class=""></div><div class="">Here is a list of flags to add that suppress warnings and other issues with OpenMPI. You can add them all or one at a time depending on issues you get.</div><div class=""><br class=""></div><div class="">#add if MPI not using all CPU given</div><div class="">--oversubscribe --bind-to none</div><div class=""><br class=""></div><div class="">#workaround for infinaband (use instead of --mca ^openib)</div><div class="">--mca btl vader,tcp,self --mca btl_tcp_if_include ib0</div><div class=""><br class=""></div><div class="">#add to stop certain other warnings</div><div class="">--mca orte_base_help_aggregate 0</div><div class=""><br class=""></div><div class="">#stop fork warnings</div><div class="">--mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0</div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Aug 15, 2017, at 9:34 AM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>> wrote:</div><br class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_extra">Here are some latest message:<div class=""><br class=""></div><div class=""><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[cn3360:57176] 1 more process has sent help message help-opal-runtime.txt / opal_init:warn-fork</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[cn3360:57176] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages</span></div></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><div class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704m_-1862143133597923131gmail-m_5723736450607837839gmail_signature"><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div class="">------------------------------<wbr class="">--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>]<br class=""><br class=""></div><div class=""><br class=""></div></div></div></div></div>
<br class=""><div class="gmail_quote">On Tue, Aug 15, 2017 at 10:39 AM, Carson Holt <span dir="ltr" class=""><<a href="mailto:carsonhh@gmail.com" target="_blank" class="">carsonhh@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="auto" class=""><div class="">You may need to delete the .../maker/perl directory before doing the reinstall if not doing a brand new installation. Otherwise you can ignore the subroutine redefined warnings during compile.</div><div class=""><br class=""></div><div class="">Have you been able to test the alternate flags on the command line for MPI? How about an alternate perl without threads?<br class=""><br class=""><div class=""><div class=""><span style="font-size:13pt" class="">--Carson</span></div><div class=""><br class=""></div><div class="">Sent from my iPhone</div></div></div><div class=""><br class="">On Aug 15, 2017, at 8:27 AM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class="">When I installed by './Build install', I got following some messages:<div class=""><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Configuring MAKER with MPI support</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Installing MAKER...</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Configuring MAKER with MPI support</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine dl_load_flags redefined at (eval 125) line 8.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_ANY_SOURCE redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_ANY_TAG redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_SUCCESS redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_Init redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_Finalize redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_Comm_rank redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_Comm_size redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_Send redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::C_<wbr class="">MPI_Recv redefined at (eval 125) line 9.</span></div><div style="margin:0px;font-size:12px;line-height:normal;font-family:Menlo;background-color:rgb(255,222,150)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Subroutine Parallel::Application::MPI::_c<wbr class="">omment redefined at (eval 125) line 9.</span></div></div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">I'm not sure whether it's correctly installed.</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704m_-1862143133597923131gmail-m_5723736450607837839m_-7744257814608198508gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">------------------------------<wbr class="">--------------</div>Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>]<br class=""><br class=""></div><div class=""><div class="">NIH/NHGRI</div><div class=""><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class="">Building 50, Room 5531</span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class=""><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class="">50 SOUTH DR, MSC 8004 </span><br style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class=""><span style="font-family:Verdana,Geneva,sans-serif;font-size:12px" class="">BETHESDA, MD 20892-8004</span></div></div></div></div></div></div></div>
<br class=""><div class="gmail_quote">On Mon, Aug 14, 2017 at 9:23 PM, Fields, Christopher J <span dir="ltr" class=""><<a href="mailto:cjfields@illinois.edu" target="_blank" class="">cjfields@illinois.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="white" lang="EN-US" class="">
<div class="m_-6541627945015036278gmail-m_-8645049413834822486m_7520722775390369704m_-1862143133597923131gmail-m_5723736450607837839m_-7744257814608198508m_-5812194475710527082WordSection1"><p class="MsoNormal">Carson,<u class=""></u><u class=""></u></p><p class="MsoNormal"><u class=""></u> <u class=""></u></p><p class="MsoNormal">It was attached to the initial message (named ‘run05.mpi.o47346077’). It looks like a Perl issue with threads, though I don’t see why this would crash a cluster. The fact there is a log file would suggest it just ended the job.<u class=""></u><u class=""></u></p><p class="MsoNormal"><u class=""></u> <u class=""></u></p><p class="MsoNormal">chris<u class=""></u><u class=""></u></p><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0in 0in" class=""><p class="MsoNormal" style="margin-left:0.5in"><b class=""><span style="font-size:12pt" class="">From:
</span></b><span style="font-size:12pt" class="">maker-devel <<a href="mailto:maker-devel-bounces@yandell-lab.org" target="_blank" class="">maker-devel-bounces@yandell-l<wbr class="">ab.org</a>> on behalf of Carson Holt <<a href="mailto:carsonhh@gmail.com" target="_blank" class="">carsonhh@gmail.com</a>><br class="">
<b class="">Date: </b>Monday, August 14, 2017 at 2:18 PM<br class="">
<b class="">To: </b>zl c <<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>><br class="">
<b class="">Cc: </b>"<a href="mailto:maker-devel@yandell-lab.org" target="_blank" class="">maker-devel@yandell-lab.org</a>" <<a href="mailto:maker-devel@yandell-lab.org" target="_blank" class="">maker-devel@yandell-lab.org</a>><br class="">
<b class="">Subject: </b>Re: [maker-devel] maker MPI problem<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">This is rather vague —> “crashed the computer cluster”
<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in">Do you have a specific error?<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in">—Carson<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
<div class="">
<blockquote style="margin-top:5pt;margin-bottom:5pt" class="">
<div class=""><p class="MsoNormal" style="margin-left:0.5in">On Aug 14, 2017, at 12:59 PM, zl c <<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>> wrote:<u class=""></u><u class=""></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
<div class="">
<div class=""><p class="MsoNormal" style="margin-left:0.5in">
Hello,<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"> <u class=""></u><u class=""></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">
I ran maker 3.0 with openmpi 2.0.2 and it crashed the computer cluster. I attached the log file. Could you help me to solve the problem?<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"> <u class=""></u><u class=""></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">
CMD:<u class=""></u><u class=""></u></p><p class="MsoNormal" style="margin-left:0.5in">
export LD_PRELOAD=/usr/local/OpenMPI/<wbr class="">2.0.2/gcc-6.3.0/lib/libmpi.so<u class=""></u><u class=""></u></p><p class="MsoNormal" style="margin-left:0.5in">
export OMPI_MCA_mpi_warn_on_fork=0<u class=""></u><u class=""></u></p><p class="MsoNormal" style="margin-left:0.5in">
mpiexec -mca btl ^openib -n $SLURM_NTASKS maker -c 1 –base genome -g genome.fasta<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"> <u class=""></u><u class=""></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">
Thanks,<u class=""></u><u class=""></u></p><p class="MsoNormal" style="margin-left:0.5in">
Zelin Chen<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"> <u class=""></u><u class=""></u></p>
</div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class=""><p class="MsoNormal" style="margin-left:0.5in">------------------------------<wbr class="">--------------<u class=""></u><u class=""></u></p>
</div><p class="MsoNormal" style="margin-left:0.5in">Zelin Chen [<a href="mailto:chzelin@gmail.com" target="_blank" class="">chzelin@gmail.com</a>] Ph.D.<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal" style="margin-left:0.5in">NIH/NHGRI<u class=""></u><u class=""></u></p>
</div>
<div class="">
<div class=""><p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:9pt;font-family:Verdana,sans-serif" class="">Building 50, Room 5531<br class="">
50 SOUTH DR, MSC 8004 <br class="">
BETHESDA, MD 20892-8004</span><u class=""></u><u class=""></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div><p class="MsoNormal" style="margin-left:0.5in"><run05.mpi.o47346077>_________<wbr class="">______________________________<wbr class="">________<br class="">
maker-devel mailing list<br class="">
<a href="mailto:maker-devel@box290.bluehost.com" target="_blank" class="">maker-devel@box290.bluehost.co<wbr class="">m</a><br class="">
<a href="http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org" target="_blank" class="">http://box290.bluehost.com/mai<wbr class="">lman/listinfo/maker-devel_yand<wbr class="">ell-lab.org</a><u class=""></u><u class=""></u></p>
</div>
</blockquote>
</div><p class="MsoNormal" style="margin-left:0.5in"><u class=""></u> <u class=""></u></p>
</div>
</div>
</div>
</blockquote></div><br class=""></div></div>
</div></blockquote></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div></div>
</div></blockquote></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></body></html>