<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Several things.</div><div class=""><br class=""></div><div class="">1. MAKER does a lot of it’s work in a temporary directory (usually /tmp). This directory must be locally mounted and cannot be a network mounted location. If this location is full you can get issues.</div><div class="">2. MAKER needs at least 1GB of RAM per process (2-3GB is safer), so if you don’t have enough RAM you may need to run fewer processes (with MPI multiply whatever you supplied to the mpiexec -n flag by 1GB).</div><div class="">3. If you are launching MAKER multiple times as opposed to launching once via MPI, you will exacerbate the above limitations as well as open up IO limitations. MAKER can and does saturate IO when run multiple times simultaneously (this is especially true for network mounted locations). If you run via MPI you can greatly reduce IO, so make sure you are using MPI and not just launching MAKER multiple times.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">If you absolutely have to start multiple jobs, you can reduce IO somewhat by splitting the input fasta into pieces (use fasta_tool). Give a separate piece to each job via maker’s -g flag, and set -base so all results from all jobs get written to the same location. Then each job can avoid multiple file locks that would have been encountered by sharing input. Note that you must rebuild the datastore index using 'maker -dsindex’ when all jobs complete.</div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 10, 2017, at 10:21 AM, José Mª G. Perez-Silva <<a href="mailto:ereboperezsilva@gmail.com" class="">ereboperezsilva@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:monospace,monospace">Hi, </div><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><div class="gmail_default" style="font-family:monospace,monospace">I wrote early this day, in reference to a problem of (apparently) space. After I deleted some unnecesary files (despite having plenty of storage left), I killed all the processes, and set 'clean_try=1' as recomended in this <a href="http://gmod.827538.n3.nabble.com/Re-re-run-MAKER-after-out-of-disk-space-td3708059.html" class="">post</a>. Before re-running the processes, we checked that there were no limitation over the size of a directory or something similar.</div><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><div class="gmail_default" style="font-family:monospace,monospace">After re-running, at first, all seemed correct, but when I re-checked some time after, I found out a lot of contigs with the status FAILED without folder specification in the '_master_datastore_index.log', looking like:</div><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span style="font-family:monospace,monospace" class="">Contig480</span><span class="gmail-Apple-tab-span" style="font-family:monospace,monospace;white-space:pre"> </span><span style="font-family:monospace,monospace" class="">FAILED</span><br class=""></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">Contig496<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">Contig512<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">Contig528<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">Contig544<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>FAILED</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">Contig560<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>FAILED</font><span style="font-family:monospace,monospace" class=""></span></blockquote><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><div class="gmail_default" style="font-family:monospace,monospace">But checking the 'nohub.out' of every proccess (16 in total, as the machine has 16 cores), I notice that each run is, from time to time, processing the contig correctly. So, after several (a lot) of FAILED contigs, it process one correctly. As said in the previous email, the ERROR dispolayed in the nohup.out is (including the last part of a processed contig at the beguinning):</div><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div class="gmail_default" style="font-family:monospace,monospace;display:inline"></div><span style="font-family:monospace,monospace" class="">#--------- command -------------#</span></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">Widget::blastx:</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">/usr/bin/blastall -p blastx -d /data/ge/tmp/maker_VfDQQU/hsap_ensembl%2Efa.mpi.10.6 -i /data/ge/tmp/maker_VfDQQU/0/Contig20.0 -b 10000 -v 10000 -e 1e-06 -z 300 -Y 500000000 -a 4 -U -F T -I T -o /data/ge/round3/cg.maker.output/cg_datastore/56/AC/Contig20//theVoid.Contig20/0/Contig20.0.hsap_ensembl%2Efa.blastx.temp_dir/hsap_ensembl%2Efa.mpi.10.6.blastx</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">#-------------------------------#</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">deleted:511 hits</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">doing blastx of proteins</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">open3: fork failed: Cannot allocate memory at /home/jmgps/software/maker/bin/../lib/File/NFSLock.pm line 1037.</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Failed while doing blastx of proteins</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Chunk failed at level:8, tier_type:3</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">FAILED CONTIG:Contig20</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class=""><br class=""></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Chunk failed at level:4, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">FAILED CONTIG:Contig20</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class=""><br class=""></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">FAILED CONTIG:Contig22</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class=""><br class=""></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">FAILED CONTIG:Contig24</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class=""><br class=""></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">FAILED CONTIG:Contig26</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class=""><br class=""></font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: could not make datastore directory</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">--> rank=NA, hostname=Planarian2</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Failed while examining contents of the fasta file and run log</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">ERROR: Chunk failed at level:0, tier_type:0</font></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="monospace, monospace" class="">FAILED CONTIG:Contig28</font><span style="font-family:monospace,monospace" class=""></span></blockquote><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><div class="gmail_default" style="font-family:monospace,monospace">I'm totally lost here, I think it is still processing contigs, but the FAILED attemps slow down the whole process, and we are in a hurry due to the maintenance of the machine. And I can't understand the source of the ERROR.</div><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><div class="gmail_default" style="font-family:monospace,monospace">I will be more than happy to provide more details about the problem, if requested.</div><div class="gmail_default" style="font-family:monospace,monospace"><br class=""></div><div class="gmail_default" style="font-family:monospace,monospace">Thanks a lot for the help!</div></div>
</div></blockquote></div><br class=""></div></body></html>