[maker-devel] MPI vs multiple instance for speed

[Carson Holt]

MAKER can coordinate parallelization under MPI in a way it can’t even with multiple simultaneous runs. Because processes can comunicate among themselves under MPI, MAKER can break larger contigs into chunks or even pull off individual steps and pass them onto another processor, then receive the results back from that processor. So multiple BLAST, RepeatMasker, Exonerate, and prediction processes can all run at the same time for the same contig. Then they all pass their result back to the parent process so it can produce output for that contig. MPI was chosen as the parallelization framework rather than threads because it works both within a single machine as well as across multiple machines, so you can scale up to hundreds of processes if needed.

—Carson


> On Oct 18, 2017, at 5:47 AM, Patrick Tran Van <Patrick.TranVan at unil.ch> wrote:
> 
> Hi Carson, 
> 
> 1) I think I have read one of your post saying that running maker with MPI is faster than multiple instance, can you explain why ?
> 
> 2) I am trying to annotate a 1GB specie but it's superslow.
> I have filtered the transcriptome to speed up the process but do you have other suggestion to increase the speed ? 
> 
> Cheers,
> 
> Patrick Tran Van
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20171018/60854f4e/attachment-0003.html>