[maker-devel] maker low cpu utilization

[Carson Holt]

Also the ‘maker’ processes should rarely use very much CPU. All they do is shepherd data between processes like augustus, snap, blast, and exonerate (there are some short intermediate processing steps, but the external tools are the work horses). So each ‘maker’ process is usually just waiting for external tools to complete.  What maker does is divide the input data into reasonable chunks, so that there will always be a blast, snap, or augustus process running somewhere to keep all CPUs busy. If the structure of the actual input data is odd compared to the typical genome project input then there could hypothetically be a situation where not enough reasonable task chunks can be made to keep all CPUs busy. I’d really have to see your data if you think that is the issue.

MAKER has the following points of parallelization.
1. Every contig goes to a separate thread.
2. Large contigs are split into overlapping pieces that go into separate threads (determined using the max_dna_len= paramter with the default being 100,000 bp)
3. BLAST databases for input evidence are split into 10 pieces (so BLAST analysis are split by 10)
4. Ab inito gene prediction on large contigs are split into overlapping sections of 10 megabases each.

So unless you have a small dataset that can’t be split by any of the above parameters it should be able to parallelize.  Also if your assembly contains primarily short contigs and you set min_contig such that the root process spends most of it’s time skipping contigs and less time distributing them for other processes to analyze, then that could create an apparent slowdown. I have had that happen on a couple of assemblies that had > 2 million contigs, but only ~10,000 were usable. By filtering small contigs out of the assembly, you can get around that last issue.

—Carson


> On Mar 10, 2016, at 11:34 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> The ‘ps’ calls should run at startup (they are checking the MPI configuration before MAKER connects to the communication ring and will generate somewhat informative errors for common mis-configurations when users run MAKER with MPI). Because it is one per process (MAKER is not yet connected to MPI at this point) and you have so many CPUs on a single node, it may delay startup by a few seconds, but that’s it. Once MAKER gets into the actual run, you won’t see those processes again.
> 
> If it bothers you there is an alternative to have MAKER query the process table programmatically rather than via ‘ps' (it’s not the default because it works on fewer architectures but should work on AMD).  To do the work around, you will need to install Proc::ProcessTable from CPAN, then replace …/maker/lib/Proc/ProcessTable_simple.pm and …/maker/lib/Proc/Signal.pm with the attached alternate files.
> 
> —Carson
> 
> <ProcessTable_simple.pm_alt><Signal.pm_alt>
> 
> 
>> On Mar 10, 2016, at 7:53 AM, Christian Meesters <meesters at uni-mainz.de> wrote:
>> 
>> Dear maker-developers,
>> 
>> As a computational scientist of our local HPC-Team, I recently installed maker and its tools.
>> 
>> We encountered a most peculiar problem: Distributed over 2 nodes, 64 cores each (AMD OPT6272 "bulldozer"), all started processes take up ~20 % of the possible CPU whilst the node show a full load of processes. Amongst this 20 % there is some system overhead (~4%).
>> 
>> We then wrote a little wrapper / submission script, such that the ctl-Files were altered and all reference input is copied unto ramdisks (each node provides the same path, there are then 2 copies of each reference file, prior to starting maker). Still no change - IO is not a bottleneck, here.
>> 
>> I then wanted to trace individual PIDs, but they are frequently changing. However, I saw > 170 instances ps concurrently running and the same amount of 'sh'.
>> 
>> Only augustus should about 100% CPU usage, all other (except maker itself) showed lower usage.
>> 
>> Have you ever experienced something similar and could perhaps provide a pointer to the cause? Could this perhaps be related to the nature of the input data (can some input data cause frequent switches of processes and therefore OS scheduler overhead)?
>> 
>> Thanks a lot in advance,
>> Best regards,
>> Christian Meesters
>> 
>> -- 
>> ****************************************
>> 
>> Dr. Christian Meesters
>> Johannes Gutenberg-Universität Mainz
>> Zentrum für Datenverarbeitung
>> Anselm-Franz-von-Bentzelweg 12
>> 55099 Mainz
>> 
>> tel. +49 (0)6131 39 26397
>> 
>> ****************************************
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>