<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Genome annotation is very data intensive as opposed to CPU intensive. In MAKER, most IO intensive operations will occur in a temporary directory pointed to by the TMP= option in the MAKER control files. If you are setting this value to a location on a network mounted drive then this could be the source of your problem. Also TMP= defaults to the location of the TMPDIR Linux environmental variable, so make sure that TMPDIR is not set to a network mounted location either. The temporary directory needs to be a locally mounted location. There will still need to be a number of global files though; however, we’ve previously ran MAKER on over 8,000 cpus on Lustre file systems with no issues.<div class=""><br class=""></div><div class="">It is possible that it is the metadata server that is having problems as opposed to the object storage server if the genome being annotated has a large number of small contigs. Lots of small contigs in a fragmented genome assembly result in a lot of small result files, but very little reading and writing. Such a situation can be quite stressful for Lustre file systems because they don’t like having large numbers of very small files (it overwhelms the metadata server even though the object storage server will be under more moderate load). Make sure you are setting min_contig= to something like 10000 if that is the case to avoid generating analysis for short un-annotatable contigs (they may number in the hundreds of thousands on lower quality genome assemblies and contain no useful information). You can also set clean_up=1 in the maker control files, to delete files as MAKER advances. This removes restart capability because you won’t have logged results from previous runs, but it will reduce the burden on the Metadata server (which is affected by total file number as opposed to file read/write operations). Also setting clean_up=1 can help you avoid any administrator defined limits on total file number per user (administrators commonly set this limit on Lustre based file systems to avoid taxing the metadata server).</div><div class=""><br class=""></div><div class="">So your issue is likely caused by one of two things:</div><div class=""><br class=""></div><div class="">1. Improperly setting TMP= in the maker_opts.ctl file or the Linux TMPDIR environmental variable to a network mounted location. Fixed by setting these to a locally mounted location (usually /tmp).</div><div class="">2. Too many total files being generated by a fragmented genome assembly. Fixed by either setting min_contig=10000 in order to skip short contigs or by setting clean_up=1 to avoid logging too many files. This happen because it is very difficult to overwhelm Lustre's object storage servers (which perform IO read/write operations), but it’s relatively easy to overwhelming the metadata server (affected by total file count rather than total IO throughput).</div><div class=""><br class=""></div><div class=""><div class=""><div class="">—Carson</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 19, 2015, at 5:55 AM, Stephen Wang <<a href="mailto:wangyichao@sjtu.edu.cn" class="">wangyichao@sjtu.edu.cn</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<div class="">Dear MAKER Team,</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>I am a cluster administrator in the university. The issue is caused by MAKER jobs, which access massive small files and crashed Lustre file system.</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Hardware: 16 cores per node</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Software: OpenMPI 1.6.5 and GCC 4.9.1</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Q1: Does MAKER have to generate a large number of files on the global file system?</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>Q2: Can any parameters help MAKER avoid I/O intensive access? Any experience on Lustre?</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>MAKER is a quite important software for our user. Hope for your help.</div><div class=""><div class=""><br class=""></div><div class="">BR,</div><div class="">Stephen</div><div class=""><span style="font-size: 12px;" class=""><br class=""></span></div><div class="">-- </div><div class=""><span style="font-size: 10pt; color: rgb(51, 51, 51); background-color: rgb(255, 255, 255); line-height: 20px; font-family: 'Lucida Grande', 'Lucida Sans Unicode', sans-serif !important;" class="">Stephen Wang, </span><span style="color: rgb(51, 51, 51); font-size: 10.5pt; line-height: 21px; background-color: rgb(255, 255, 255); font-family: 'Lucida Grande', 'Lucida Sans Unicode', sans-serif !important;" class=""><font style="font-size: 10pt;" class="">GPU Computing Specialist</font></span><span style="color: rgb(51, 51, 51); background-color: rgb(255, 255, 255); font-family: 'Lucida Grande', 'Lucida Sans Unicode', sans-serif; line-height: 20px;" class=""></span><div style="color: rgb(51, 51, 51); background-color: rgb(255, 255, 255); line-height: 20px; font-family: 'Lucida Grande', 'Lucida Sans Unicode', sans-serif !important;" class=""><span style="font-size: 10pt;" class=""></span><span style="font-size: 10pt;" class="">Center for High Performance Computing</span></div><div style="color: rgb(51, 51, 51); background-color: rgb(255, 255, 255); line-height: 20px; font-family: 'Lucida Grande', 'Lucida Sans Unicode', sans-serif !important;" class=""><span style="font-size: 10pt;" class="">Shanghai Jiao Tong University</span></div><div style="color: rgb(51, 51, 51); background-color: rgb(255, 255, 255); line-height: 20px; font-family: 'Lucida Grande', 'Lucida Sans Unicode', sans-serif !important;" class=""><span style="font-size: 10.5pt;" class=""><div style="font-size: 10pt;" class="">Room 205 Network Center, 800 Dongchuan Road, Shanghai 200240 China</div><div class=""><span style="font-size: 9pt;" class="">Mobi:+86-136-6151-1618 Web:http</span><span style="font-size: 9pt;" class="">://<a href="http://hpc.sjtu.edu.cn" class="">hpc.sjtu.edu.cn</a></span></div></span></div></div></div>
</div></blockquote></div><br class=""></div></div></div></body></html>