<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">For Intel MPI, export an environmental variable right before running MAKER —> "export I_MPI_FABRICS=shm:tcp"<div class=""><br class=""></div><div class="">Intel MPI has a similar infiniband segfault issue as OpenMPI when running Perl scripts, but a different workaround.</div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Feb 26, 2020, at 1:15 PM, Devon O'Rourke <<a href="mailto:devon.orourke@gmail.com" class="">devon.orourke@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Much appreciated Carson,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I've submitted a job using the parameters you've suggested and will post the outcome. We definitely have two of three MPI options you've described on our cluster (OpenMPI and MPICH2); I'll check on Intel MPI. Happy to advise my cluster admins to use whichever software you prefer (should there be one).</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Devon<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 26, 2020 at 2:54 PM Carson Holt <<a href="mailto:carsonhh@gmail.com" class="">carsonhh@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Try adding these a few options right after ‘mpiexec’ in your batch script (this will fix infiniband related segfaults as well as some fork related segfaults) —> --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 --mca orte_base_help_aggregate 0 --mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0<div class=""><br class=""></div><div class="">Also remove the -q in the maker command to get full command lines for subprocesses in the STDERR (allows you to run some commands outside of MAKER to test the source of failures if for example BLASt or Exonerate is causing the segfault).</div><div class=""><br class=""></div><div class="">Example —></div><div class="">mpiexec <span style="" class="">--mca btl vader,tcp,self --mca btl_tcp_if_include ib0 </span><font class=""><span class="">--mca orte_base_help_aggregate 0 --mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0 </span></font>-n 28 /packages/maker/3.01.02-beta/bin/maker -base lu -fix_nucleotides </div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">One alternate possibility is that OpenMPI is the problem, I’ve seen a few systems where it has an issue with perl itself, and the only way to get around it is to install your own version of perl without perl threads enabled and install MAKER with that version of Perl (then OpenMPI seems to be ok again). If that’s the case it is often easier to switch to MPICH2 or Intel MPI as the MPI launcher if they are available and then reinstall MAKER with that MPI flavor.</div><div class=""><br class=""></div><div class="">—Carson </div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Feb 26, 2020, at 12:36 PM, Devon O'Rourke <<a href="mailto:devon.orourke@gmail.com" target="_blank" class="">devon.orourke@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Thanks very much for the reply Carson,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I've attached few files file of the most recently failed run: the shell script submitted to Slurm, the _opts.ctl file, and the pair of log files generated from the job. The reason there are a 1a and 1b pair of files is that I had initially set the number of cpus in the _opts.ctl file to "60", but then tried re-running it after setting it to "28". Both seem to have the same result.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I certainly have access to more memory if needed. I'm using a pretty typical (I think?) cluster that controls jobs with Slurm using a Lustre file system - it's the main high performance computing center at our university. I have access to plenty of nodes that contain about 120-150g of RAM each with between 24-28 cpus each, as well a handful of higher memory nodes with about 1.5tb of RAM. As I'm writing this email, I've submitted a similar Maker job (i.e. same fasta/gff inputs) requesting 200g of RAM over 32 cpus; if that fails, I could certainly run again with even more memory.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Appreciate your insights; hope the weather in UT is filled with sun or snow or both.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Devon<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 26, 2020 at 2:10 PM Carson Holt <<a href="mailto:carsonhh@gmail.com" target="_blank" class="">carsonhh@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">If running under MPI, the reason for a failure may be further back in the STDERR (failures tend snowball other failures, so the initial cause is often way back). If you can capture the STDERR and send it, that would be the most informative. If its memory, you can also set all the blast_depth parameters in maker_botpts.ctl to a value like 20.<div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Feb 19, 2020, at 1:54 PM, Devon O'Rourke <<a href="mailto:devon.orourke@gmail.com" target="_blank" class="">devon.orourke@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Hello,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I apologize for not posting directly to the archived forum but it appears that the option to enter new posts is disabled. Perhaps this is by design so emails go directly to this address. I hope this is what you are looking for.<br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br class="">Thank you for your continued support of Maker and your responses to the forum posts. I have been running Maker (V3.01.02-beta) to annotate a mammalian genome that consists of 22 chromosome-length scaffolds (between ~200-20Mb) and about 10,000 smaller fragments from 1Mb to 10kb in length. In my various tests in running Maker, the vast majority of the smaller fragments are annotated successfully, but nearly all the large scaffolds fail with the same error code when I look at the 'run.log.child.0' file:<br class="">```<br class="">DIED RANK 0:6:0:0<br class="">DIED COUNT 2<br class="">```<br class="">(the master 'run.log' file just shows "DIED COUNT 2")<br class=""><br class="">I struggled to find this exact error code anywhere on the forum and was hoping you might be able to help me determine where I should start troubleshooting. I thought perhaps it was an error concerning memory requirements, so I altered the chunk size from the default to a few larger sequence lengths (I've tried 1e6, 1e7, and 999,999,999 - all produce the same outcome). I've tried running the program with parallel support using either openMPI or mpich. I've tried running on a single node using 24 cpus and 120g of RAM. It always stalls at the same step.<br class=""><br class="">Interestingly, one of the 22 large scaffolds always finishes and produces the .maker.proteins.fasta, .maker.transcripts.fasta, and .gff files, but the other 21 of 22 large scaffolds fail. This makes me think perhaps it's not a memory issue?<br class=""><br class="">In the case of both the completed and failed scaffolds, the "theVoid.scaffoldX" subdirectory(ies) containing the .rb.cat.gz, .rb.out, .specific.ori.out, .specific.cat.gz, .specific.out, te_proteins*fasta.repeat runner, the est *fasta.blastn, the altest *fasta.tblastx, and protein *fasta.blastx files are all present (and appear finished from what I can tell).<br class="">However, the particular contents in the parent directory to the "theVoid.scaffold" folder differ. For the failed scaffolds, the contents generally always look something like this (that is, they stall with the same kind of files produced):<br class="">```<br class="">0<br class="">evidence_0.gff<br class="">query.fasta<br class="">query.masked.fasta<br class="">query.masked.fasta.index<br class="">query.masked.gff<br class="">run.log.child.0<br class="">scaffold22.0.final.section<br class="">scaffold22.0.pred.raw.section<br class="">scaffold22.0.raw.section<br class="">scaffold22.gff.ann<br class="">scaffold22.gff.def<br class="">scaffold22.gff.seq<br class="">```<br class=""><br class="">For the completed scaffold, there are many more files created:<br class="">```<br class="">0<br class="">10<br class="">100<br class="">20<br class="">30<br class="">40<br class="">50<br class="">60<br class="">70<br class="">80<br class="">90<br class="">evidence_0.gff<br class="">evidence_10.gff<br class="">evidence_1.gff<br class="">evidence_2.gff<br class="">evidence_3.gff<br class="">evidence_4.gff<br class="">evidence_5.gff<br class="">evidence_6.gff<br class="">evidence_7.gff<br class="">evidence_8.gff<br class="">evidence_9.gff<br class="">query.fasta<br class="">query.masked.fasta<br class="">query.masked.fasta.index<br class="">query.masked.gff<br class="">run.log.child.0<br class="">run.log.child.1<br class="">run.log.child.10<br class="">run.log.child.2<br class="">run.log.child.3<br class="">run.log.child.4<br class="">run.log.child.5<br class="">run.log.child.6<br class="">run.log.child.7<br class="">run.log.child.8<br class="">run.log.child.9<br class="">scaffold4.0-1.raw.section<br class="">scaffold4.0.final.section<br class="">scaffold4.0.pred.raw.section<br class="">scaffold4.0.raw.section<br class="">scaffold4.10.final.section<br class="">scaffold4.10.pred.raw.section<br class="">scaffold4.10.raw.section<br class="">scaffold4.1-2.raw.section<br class="">scaffold4.1.final.section<br class="">scaffold4.1.pred.raw.section<br class="">scaffold4.1.raw.section<br class="">scaffold4.2-3.raw.section<br class="">scaffold4.2.final.section<br class="">scaffold4.2.pred.raw.section<br class="">scaffold4.2.raw.section<br class="">scaffold4.3-4.raw.section<br class="">scaffold4.3.final.section<br class="">scaffold4.3.pred.raw.section<br class="">scaffold4.3.raw.section<br class="">scaffold4.4-5.raw.section<br class="">scaffold4.4.final.section<br class="">scaffold4.4.pred.raw.section<br class="">scaffold4.4.raw.section<br class="">scaffold4.5-6.raw.section<br class="">scaffold4.5.final.section<br class="">scaffold4.5.pred.raw.section<br class="">scaffold4.5.raw.section<br class="">scaffold4.6-7.raw.section<br class="">scaffold4.6.final.section<br class="">scaffold4.6.pred.raw.section<br class="">scaffold4.6.raw.section<br class="">scaffold4.7-8.raw.section<br class="">scaffold4.7.final.section<br class="">scaffold4.7.pred.raw.section<br class="">scaffold4.7.raw.section<br class="">scaffold4.8-9.raw.section<br class="">scaffold4.8.final.section<br class="">scaffold4.8.pred.raw.section<br class="">scaffold4.8.raw.section<br class="">scaffold4.9-10.raw.section<br class="">scaffold4.9.final.section<br class="">scaffold4.9.pred.raw.section<br class="">scaffold4.9.raw.section<br class="">```<br class=""><br class="">Thanks for any troubleshooting tips you can offer. <br class=""><br class="">Cheers,<br class="">Devon</div><br class="">-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><font face="tahoma, sans-serif" class="">Devon O'Rourke</font></div><div class=""><font face="tahoma, sans-serif" class="">Postdoctoral researcher, Northern Arizona University</font></div><font face="tahoma, sans-serif" class=""><span style="font-size:12.8px" class="">Lab of Jeffrey T. Foster - </span></font><a href="https://fozlab.weebly.com/" target="_blank" class="">https://fozlab.weebly.com/</a><br class=""><div class=""><span style="font-size:12.8px" class="">twitter: @thesciencedork<br class=""></span></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br class="">maker-devel mailing list<br class=""><a href="mailto:maker-devel@yandell-lab.org" target="_blank" class="">maker-devel@yandell-lab.org</a><br class=""><a href="http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org" target="_blank" class="">http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org</a><br class=""></div></blockquote></div><br class=""></div></div></blockquote></div><br clear="all" class=""><br class="">-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><font face="tahoma, sans-serif" class="">Devon O'Rourke</font></div><div class=""><font face="tahoma, sans-serif" class="">Postdoctoral researcher, Northern Arizona University</font></div><font face="tahoma, sans-serif" class=""><span style="font-size:12.8px" class="">Lab of Jeffrey T. Foster - </span></font><a href="https://fozlab.weebly.com/" target="_blank" class="">https://fozlab.weebly.com/</a><br class=""><div class=""><span style="font-size:12.8px" class="">twitter: @thesciencedork<br class=""></span></div></div></div></div></div></div></div></div></div></div>
<span id="gmail-m_9061634387225395673cid:f_k73pz0790" class=""><fail-1a.log.gz></span><span id="gmail-m_9061634387225395673cid:f_k73pz07l1" class=""><fail-1b.log.gz></span><span id="gmail-m_9061634387225395673cid:f_k73pz07m2" class=""><run1_maker_opts.ctl></span><span id="gmail-m_9061634387225395673cid:f_k73pz07o3" class=""><run1_slurm.sh></span></div></blockquote></div><br class=""></div></div></blockquote></div><br clear="all" class=""><br class="">-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><font face="tahoma, sans-serif" class="">Devon O'Rourke</font></div><div class=""><font face="tahoma, sans-serif" class="">Postdoctoral researcher, Northern Arizona University</font></div><font face="tahoma, sans-serif" class=""><span style="font-size:12.8px" class="">Lab of Jeffrey T. Foster - </span></font><a href="https://fozlab.weebly.com/" target="_blank" class="">https://fozlab.weebly.com/</a><br class=""><div class=""><span style="font-size:12.8px" class="">twitter: @thesciencedork<br class=""></span></div></div></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></body></html>