<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Hi, </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I wrote early this day, in reference to a problem of (apparently) space. After I deleted some unnecesary files (despite having plenty of storage left), I killed all the processes, and set 'clean_try=1' as recomended in this <a href="http://gmod.827538.n3.nabble.com/Re-re-run-MAKER-after-out-of-disk-space-td3708059.html">post</a>. Before re-running the processes, we checked that there were no limitation over the size of a directory or something similar.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">After re-running, at first, all seemed correct, but when I re-checked some time after, I found out a lot of contigs with the status FAILED without folder specification in the '_master_datastore_index.log', looking like:</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span style="font-family:monospace,monospace">Contig480</span><span class="gmail-Apple-tab-span" style="font-family:monospace,monospace;white-space:pre">          </span><span style="font-family:monospace,monospace">FAILED</span><br></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">Contig496<span class="gmail-Apple-tab-span" style="white-space:pre">          </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">Contig512<span class="gmail-Apple-tab-span" style="white-space:pre">          </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">Contig528<span class="gmail-Apple-tab-span" style="white-space:pre">          </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">Contig544<span class="gmail-Apple-tab-span" style="white-space:pre">          </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">Contig560<span class="gmail-Apple-tab-span" style="white-space:pre">          </span>FAILED</font><span style="font-family:monospace,monospace">​</span></blockquote><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">But checking the 'nohub.out' of every proccess (16 in total, as the machine has 16 cores), I notice that each run is, from time to time, processing the contig correctly. So, after several (a lot) of FAILED contigs, it process one correctly. As said in the previous email, the ERROR dispolayed in the nohup.out is (including the last part of a processed contig at the beguinning):</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div class="gmail_default" style="font-family:monospace,monospace;display:inline">​</div><span style="font-family:monospace,monospace">#--------- command -------------#</span></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">Widget::blastx:</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">/usr/bin/blastall -p blastx -d /data/ge/tmp/maker_VfDQQU/hsap_ensembl%2Efa.mpi.10.6 -i /data/ge/tmp/maker_VfDQQU/0/Contig20.0 -b 10000 -v 10000 -e 1e-06 -z 300 -Y 500000000 -a 4 -U -F T -I T -o /data/ge/round3/cg.maker.output/cg_datastore/56/AC/Contig20//theVoid.Contig20/0/Contig20.0.hsap_ensembl%2Efa.blastx.temp_dir/hsap_ensembl%2Efa.mpi.10.6.blastx</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">#-------------------------------#</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">deleted:511 hits</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">doing blastx of proteins</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">open3: fork failed: Cannot allocate memory at /home/jmgps/software/maker/bin/../lib/File/NFSLock.pm line 1037.</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Failed while doing blastx of proteins</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Chunk failed at level:8, tier_type:3</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">FAILED CONTIG:Contig20</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace"><br></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Chunk failed at level:4, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">FAILED CONTIG:Contig20</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace"><br></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">FAILED CONTIG:Contig22</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace"><br></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">FAILED CONTIG:Contig24</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace"><br></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">FAILED CONTIG:Contig26</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace"><br></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace">FAILED CONTIG:Contig28</font><span style="font-family:monospace,monospace">​</span></blockquote><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I'm totally lost here, I think it is still processing contigs, but the FAILED attemps slow down the whole process, and we are in a hurry due to the maintenance of the machine. And I can't understand the source of the ERROR.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I will be more than happy to provide more details about the problem, if requested.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Thanks a lot for the help!</div></div>