[maker-devel] MAKER Commercial User -- Question

[Carson Holt]

Hi Hayley,

Which flavor of MPI are you using? OpenMPI, MVAPICH2, MPICH, or Intel MPI?  There is a known issue with infiniband network cards that causes them to freeze on certain system calls, and I bet that is what you are seeing.  It is not a MAKER issue, but a limitation in order to allow performance benefits elsewhere with MPI.  You need to add extra command line flags to tell MPI to communicate a bit differently to get around it. You can find multiple threads with more info in the maker-devel archives here —> https://groups.google.com/g/maker-devel


For OpenMPI you may need to add these flags to the mpiexec command line —> --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 --mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0

For intel MPI set the environmental variable I_MPI_FABRICS=shm:tcp before running mpiexec

For MPICH, no modifications should be needed (it doesn’t experience the issue).

For MVAPICH2, there is no work around and you just need to use one of the other MPI flavors.


For the second failure, if you can regenerate it, send me the STDERR and I will take a look.

Thanks,
Carson



>  
> From: Hayley Mangelson <hayley at phasegenomics.com <mailto:hayley at phasegenomics.com>> 
> Sent: Thursday, March 4, 2021 12:03 PM
> To: AARON A DUFFY <aaron.duffy at utah.edu <mailto:aaron.duffy at utah.edu>>
> Cc: Mary Wood <mary at phasegenomics.com <mailto:mary at phasegenomics.com>>
> Subject: MAKER Commercial User -- Question
>  
> Hi Aaron, 
>  
> Mary and I are bioinformaticians at Phase Genomics, and we are using the MAKER software (for which Phase purchased a licence) to annotate a couple of genomes. We are using the same workflow that I used for a previous project, but are having the same issues while working on separate projects. I am hoping you could help us out or direct us to someone who can help!
>  
> Here is some background. I am trying to get it to run using mpiexec for parallelization. I let it run for about a week and all processes remained “active”, but no new output seemed to be generated. It was hung up on “prepare section files” for nearly the whole 1-week period. I canceled the run, then tried running again without parallelization (in the same folder as the failed run). I got an error about each contig failing, and something about Augustus. Unfortunately, I didn’t save the error message. I tried to regenerate the error, but it looks like I would probably need to completely rerun… this is the error message now (repeated for each contig), if I run in the same folder again.
> 
> The contig failed 2 times!!
> Maker will not try again!!
> The contig will be stored in a fasta file that you can use for debugging.
> SeqID: PGA_scaffold4__8_contigs__length_31799528
> Length: 31799528
> FASTA: /home/ubuntu/PGScratch/chunk1/chunk1.maker.output/chunk1_datastore/10/91/PGA_scaffold4__8_contigs__length_31799528//PGA_scaffold4__8_contigs__length_31799528.died.fasta
> 
> Any thoughts about why mpiexec seems unable to finish? I tried a run on the test data using parallelization, and it completed in under a minute.
>  
> Thanks!
>  
> Hayley
> -- 
> Hayley Mangelson
> Bioinformatics Scientist
> Phase Genomics, Inc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20210305/3dd471c6/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1376 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20210305/3dd471c6/attachment-0002.p7s>