<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">When running large jobs in MPI semi-random issues can arise as well as tuning issues where hardware configuration, IO performance, buffer sizes etc. all play a role. Using one of the NIH flagship clusters from XSEDE for example, I can run on over 2000 CPUs without issue. But the IT specialists with XSEDE have also spent a lot of time tuning MPI by enabling and disabling certain options for their hardware and network configuration (The IT specialists for the XSEDE project are actually the developers for many of the MPI flavors available, so they actually wrote MPI to work really well on this specific cluster). On other clusters I can’t go over 200 cpus on a single job. Or on another XSEDE cluster I can run on exactly 1424 CPUs. If I increase by a single CPU, the jobs always fails. For these kinds of issues you would have to delve into some of the more obscure parameters of OpenMPI via trial and error (<a href="http://www.open-mpi.org/doc/" class="">http://www.open-mpi.org/doc/</a>). What happens under the hood in OpenMPI is that different buffer sizes and network communication strategies are triggered as the number of nodes increases, so you can often identify a specific CPU count that is stable, and going one over that number causes a failure. You then look in the documentation for a parameter that matches that trigger value and alter it higher or lower. Or if you can identify the stable CPU count, then just submit multiple jobs at exactly that CPU count.<div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 8, 2015, at 8:27 AM, 赵越 <<a href="mailto:jerryzhaosjtu@gmail.com" class="">jerryzhaosjtu@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Carson,<div class=""><br class=""></div><div class="">After using the flag in your example, the warning after runing MAKER was gone, yet after running with MPI in 512 threads for 2 hours, MAKER 'Exited with exit code 1' The stdout info is as followed:<br class=""><div class=""><br class=""></div><div class=""><div style="font-size:13px" class="">[node206][[7968,1],269][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)</div><div style="font-size:13px" class="">[node206][[7968,1],269][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)</div><div style="font-size:13px" class="">SIGTERM received</div><div style="font-size:13px" class="">Perl exited with active threads:</div><div style="font-size:13px" class=""> 1 running and unjoined</div><div style="font-size:13px" class=""> 0 finished and unjoined</div><div style="font-size:13px" class=""> 0 running and detached</div><div class=""><br class=""></div><div class="">Also, my job submission is like:</div><div class=""><div class=""> </div><div class="">#BSUB -J maker_mpi<br class=""></div><div class="">#BSUB -n 512<br class=""></div><div class="">#BSUB -R "span[ptile=16]"</div><div class="">module purge && module load gcc/4.9.1 openmpi/gcc/1.6.5</div><div class="">mpiexec -mca btl ^openib -n 512 perl /lustre/home/clswcc/yzhao/MAKER/maker/bin/maker -fix_nucleotides</div></div><div class=""><br class=""></div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Could you help me find out where is going wrong? The stdout at first is normal as followd :</div><div class="gmail_extra"><div class="gmail_extra">STATUS: Parsing control files...</div><div class="gmail_extra">STATUS: Processing and indexing input FASTA files...</div><div class="gmail_extra">STATUS: Setting up database for any GFF3 input...</div><div class="gmail_extra">A data structure will be created for you at:</div><div class="gmail_extra">/lustre/home/clswcc/SOP_1Krice/gene_prediction/mpi/unaln.maker.output/unaln_datastore</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">To access files for individual sequences use the datastore index:</div><div class="gmail_extra">/lustre/home/clswcc/SOP_1Krice/gene_prediction/mpi/unaln.maker.output/unaln_master_datastore_index.log</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">STATUS: Now running MAKER...</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra"><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Regards,</div><div class="gmail_extra">yue</div></div></div><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-family: Calibri, sans-serif;" class=""><b class=""><i class=""><span lang="EN-US" style="font-size:10pt;color:rgb(38,38,38)" class="">Yue Zhao (Jerry)</span></i></b><br class=""></div><div style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt;" class="">Bachelor Candidate of Plant Biotechnology<u class=""></u><u class=""></u></span></div><div style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt;" class="">Researcher in UCLA-CSST program<u class=""></u><u class=""></u></span></div><div style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" style="font-size: 10pt;" class="">Shanghai Jiao Tong University, Shanghai<u class=""></u><u class=""></u></span></div><div style="margin: 0cm 0cm 0.0001pt; text-align: justify; font-family: Calibri, sans-serif;" class=""><u class=""><span lang="EN-US" style="" class=""><a href="mailto:jerryzhaosjtu@gmail.com" style="color:rgb(17,85,204)" target="_blank" class="">jerryzhaosjtu@gmail.com</a></span></u></div></div></div></div></div>
</div>
</div></blockquote></div><br class=""></div></div></div></body></html>