[maker-devel] MPI large load

[Carson Holt]

Because MAKER is a pipeline, all processes it calls will generate separate processes (i.e. BLAST etc.). Also it will spawn a couple of helper processes to watch communication and files.  The helper processes use 0% CPU, and the main MAKER process will yield to external system calls and processes until they finish execution. So they will never use a larger % of CPU than is specified.  Also the way MPI works is it spawns a separate process for every CPU specified, so if you specify 40 CPUs you get 40 independent communicating processes rather than 1 process accessing 40 CPUs. So if you take into account the MPI processes, helper processes, and external system calls a 40 CPU specification could result in up to three times that many numbered processes existing simultaneously (even though no more than 40 will be active at a time). However if your system is having an issue letting the required number of processes exist, then it is a ulimit issue. Your administrator has the limit set too low. You can see what limits are set using the command ‘ulimit -a’.  You will need to get your system admin to fix it.

—Carson



> On Oct 20, 2015, at 5:54 PM, Matt Simenc <mcsimenc at gmail.com> wrote:
> 
> Hi,
> 
> 
> 
> I am using OpenMPI to run MAKER on 2 nodes with 40 CPUs/node. The load is distributing across the nodes ok but with a very large number of processes on each node. Sometimes there are several hundred more processes than can be executed at one time by a node. Is this a problem? If so, any suggestions on how to fix?
> 
> 
> 
> Thanks!
> 
> 
> 
> Matt Simenc
> 
> Der Lab
> 
> California State University Fullerton
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20151022/6867d72a/attachment-0003.html>