<div dir="ltr"><div>Dear Carson:<br><br>(1) Thank you for your explanation. I will try to set max_dna_len as 400kb for our rodent species, which is a little bit higher than the suggested value for large vertebrate genome (in the maker manual it mentioned "300,000 is a good max_dna_len on large vertebrate genomes if memory is not a limiting factor").<br><br></div>(2) By reading some of your replies in the maker google group, and I noticed that it can reduce memory and save time for annotation if I set depth_blast to a certain number. So I changed the following parameters. But I wonder, whether it will decrease the quality of annotation? If it won't affect the quality, can I even use a smaller number (e.g., 20) to save more memory and time?<br><div><br>depth_blastn=30 #Blastn depth cutoff (0 to disable cutoff)<br>depth_blastx=30 #Blastx depth cutoff (0 to disable cutoff)<br>depth_tblastx=30 #tBlastx depth cutoff (0 to disable cutoff)<br>bit_rm_blastx=30 #Blastx bit cutoff for transposable element masking<br></div><div><br></div><div>(3) I also have some concerns about the speed, especially for the long scaffolds (around 100Mb). I wonder which part is the most time consuming for genome annotation (repeat masking, blast, or polishing?). Particularly, I wonder whether the blastx of protein evidence will take majority of time. Now, I have prepared 99k mammalian Swiss protein sequences and 340k rodent TrEMBL protein sequences as protein evidences. I am considering whether I can save much time if I only use the 99k mammalian Swiss protein sequences as evidences.</div><div><br></div><div>(4) For some reasons, I can not run maker though MPI on our cluster. So I
can only start multiple maker. I wonder if it is possible to let
multiple maker to annotate the same long scaffold (i.e., for a single
sequence I start multiple maker, without splitting the long sequence
into shorter ones). </div><div><br></div><div>(5) Still about the speed issue. I read some of your comments about "cpus" parameters in the maker_opts file (<a href="http://gmod.827538.n3.nabble.com/open3-fork-failed-Cannot-allocate-memory-td4025117.html">http://gmod.827538.n3.nabble.com/open3-fork-failed-Cannot-allocate-memory-td4025117.html</a>). And I know it indicate the number of cpus for a single chunk. So if I set "cpus=2" in the maker_opts file, then I can use the following command to submit the job, right? <br></div><div><br></div><div>**************** the bash file used to submit the maker job<br></div><div>#!/bin/bash<br><br>#$ -cwd<br>#$ -S /bin/bash<br>#$ -j y<br>#$ -N makerT2<br>#$ -l h_vmem=8g<br><span style="color:rgb(255,0,0)">#$ -pe smp 2</span><br><br>module load MAKER/2.31.9/perl.5.22.1<br><br>maker --q 2> maker_test.error</div><div><br></div><div><br></div><div><br></div><div>Many thanks</div><div><br></div><div>Best</div><div>Qaunwei<br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2017-09-05 18:08 GMT-04:00 Carson Holt <span dir="ltr"><<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">max_dna_len is the window size for keeping data in RAM. Smaller values do not split genes. But values lower than 100kb can create issues (if a single gene models spans 3 or more windows, it creates a weird failure).<span class="HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">—Carson</font></span><div><div class="h5"><br><div><br></div><div><br></div><div><br><div><blockquote type="cite"><div>On Sep 5, 2017, at 4:04 PM, Quanwei Zhang <<a href="mailto:qwzhang0601@gmail.com" target="_blank">qwzhang0601@gmail.com</a>> wrote:</div><br class="m_-4315328990952190373Apple-interchange-newline"><div><div dir="ltr"><div><div><div>Dear Carson:<br><br></div>Thanks. I wonder whether smaller "max_dna_len" will split longer scaffolds. I set max_dna_len as 1Mb, because there are quite many long scaffolds (e.g., the longest one is about 100Mb). Would you explain whether smaller "max_dna_len" will decrease the quality of annotation (e.g., split some genes in the same scaffold)? <br></div><div><br></div><div><br></div>Best<br></div>Quanwei </div><div class="gmail_extra"><br><div class="gmail_quote">2017-09-05 17:48 GMT-04:00 Carson Holt <span dir="ltr"><<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">You ran out of memory. You probably set max_dna_len too high for the machines you are using. There is a note in the maker_opts.ctl file that tells you that this value affects memory usage.<div><br></div><div>So you can either set it lower, or if running under MPI, use fewer CPUs per node (how you do this is MPI flavor dependent, but some flavors let you do this by setting process count lower combined with the round robin option).</div><span class="m_-4315328990952190373HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="m_-4315328990952190373HOEnZb"><font color="#888888">—Carson</font></span><div><div class="m_-4315328990952190373h5"><br><div><br></div><div><br><div><blockquote type="cite"><div>On Sep 5, 2017, at 2:24 PM, Quanwei Zhang <<a href="mailto:qwzhang0601@gmail.com" target="_blank">qwzhang0601@gmail.com</a>> wrote:</div><br class="m_-4315328990952190373m_-7513552749084071906Apple-interchange-newline"><div><div dir="ltr"><div>Hello:<br></div><div><br></div><div>We are doing genome annotation for a new rodent species. We have finished the training of the ab initio gene predictors successful by setting the following parameters (split_hit=40000, max_dna_len=1000000, and 99k mammalian Swiss protein sequences as evidences. <br></div><div><br></div><div>But when I used the trained model to do the genome annotation, I got the following kinds of errors (shown in red). I used the same parameters as those for training, except for addition of 340k rodent TrEMBL protein sequences for protein evidences (i.e., I use both 99k mammalian Swiss protein sequences and 340k rodent TrEMBL protein sequences). <br></div><div><br></div><div>I am doing the annotation on a cluster and started multiple Maker in the same directory (I had tried to use MPI but met some problems). <br></div><div><br></div><div>Do you have any suggestions? Many thanks<br></div><div>
<span style="font-size:20pt;font-family:Calibri"></span><span style="font-size:20pt;font-family:Calibri"><span></span></span>#some kinds of errors<br></div><div><span style="color:rgb(255,0,0)">open3: fork failed: Cannot allocate memory at /gs/gsfs0/hpc01/apps/MAKER/2.3<wbr>1.9/bin/../lib/Widget/<a href="http://blastx.pm/" target="_blank">blastx.p<wbr>m</a> line 40.<br>--> rank=NA, hostname=n520<br>ERROR: Failed while doing blastx of proteins<br>ERROR: Chunk failed at level:8, tier_type:3<br>FAILED CONTIG:Contig2<br><br><br>setting up GFF3 output and fasta chunks<br>doing repeat masking<br>Can't kill a non-numeric process ID at /gs/gsfs0/hpc01/apps/MAKER/2.3<wbr>1.9/bin/../lib/File/NFSLock.pm line 1050.<br>--> rank=NA, hostname=n513<br>ERROR: Failed while doing repeat masking<br>ERROR: Chunk failed at level:0, tier_type:1<br>FAILED CONTIG:Contig12378</span></div><div><span style="color:rgb(255,0,0)"><br></span></div><div><br></div><div><span>Best</span></div><div><span style="color:rgb(255,0,0)"><span>Quanwei</span><br></span></div></div>
</div></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div></div>