[maker-devel] maker with mpi support on example still not done after two days

Wed Aug 14 13:05:05 MDT 2019

If no additional output is produced, then it is frozen for some reason. My first suggestion would be to reinstall MPICH and then reinstall MAKER using the MPICH you just installed.  Freezing issues are usually related to MPI communication which is handled by the communicator you have installed (i.e. the mpiexec command that launches MAKER).

Alternatively it can be related to the file system. Some less common network mounted file systems do not support do not correctly support hardlinks which can cause programatic file locks to freeze. You can try running with the -nolock flag if that is the case.

—Carson

> On Aug 8, 2019, at 7:12 AM, Taslim, Cenny <Cenny.Taslim at nationwidechildrens.org> wrote:
> 
> Hi Maker developers,
>  
> Thanks for approving my subscription. 
> I tried running maker with mpi support on the human fasta file provided in example_01_basic. It’s been running for 2 days and 21 hours. I didn’t think the example require a long time to run.
> I’m hoping someone can help me point out the problem.
>  
> I’m running it with 4 processes:
> ~/opt/mpich-3.3.1/bin/mpiexec -n 4 ~/opt/maker.4/maker/bin/maker -f 2> maker.error
>  
> Maker_opts.ctl is the same as opts2.txt
>  
> These are the log:
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> STATUS: Setting up database for any GFF3 input...
> A data structure will be created for you at:
> /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore
>  
> To access files for individual sequences use the datastore index:
> /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_master_datastore_index.log
>  
> STATUS: Now running MAKER...
> examining contents of the fasta file and run log
>  
>  
>  
> --Next Contig--
>  
> #---------------------------------------------------------------------
> Now starting the contig!!
> SeqID: NT_010783.15
> Length: 201444
> #---------------------------------------------------------------------
>  
>  
> setting up GFF3 output and fasta chunks
> doing repeat masking
> doing repeat masking
> running  repeat masker.
> #--------- command -------------#
> Widget::RepeatMasker:
> cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.0.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
> #-------------------------------#
> running  repeat masker.
> #--------- command -------------#
> Widget::RepeatMasker:
> cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
> #-------------------------------#
> doing blastx repeats
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.0
> #-------------------------------#
> doing blastx repeats
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.1
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.0 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.0.repeatrunner
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.1 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.1.repeatrunner
> #-------------------------------#
> deleted:19 hits
> deleted:18 hits
> doing blastx repeats
> doing blastx repeats
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.2
> #-------------------------------#
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.3
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.2 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.2.repeatrunner
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.3 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.3.repeatrunner
> #-------------------------------#
> deleted:9 hits
> deleted:9 hits
> doing blastx repeats
> doing blastx repeats
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.5
> #-------------------------------#
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.4
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.5 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.5.repeatrunner
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.4 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.4.repeatrunner
> #-------------------------------#
> deleted:16 hits
> deleted:16 hits
> doing blastx repeats
> doing blastx repeats
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.6
> #-------------------------------#
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.7
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.6 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.7 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.7.repeatrunner
> #-------------------------------#
> deleted:8 hits
> deleted:12 hits
> doing blastx repeats
> doing blastx repeats
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.8
> #-------------------------------#
> formating database...
> #--------- command -------------#
> Widget::formater:
> /opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.9
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.8 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.8.repeatrunner
> #-------------------------------#
> running  blast search.
> #--------- command -------------#
> Widget::blastx:
> /opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.9 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.9.repeatrunner
> #-------------------------------#
> deleted:5 hits
> deleted:7 hits
> collecting blastx repeatmasking
> processing all repeats
> in cluster::shadow_cluster...
> ...finished clustering.
>  
> Let me know if you need more information. Thank you in advance for your help.
>  
>  
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20190814/827aec6c/attachment-0003.html>