[maker-devel] maker with mpi support on example still not done after two days

Thu Aug 8 07:12:23 MDT 2019

Hi Maker developers,

Thanks for approving my subscription.
I tried running maker with mpi support on the human fasta file provided in example_01_basic. It's been running for 2 days and 21 hours. I didn't think the example require a long time to run.
I'm hoping someone can help me point out the problem.

I'm running it with 4 processes:
~/opt/mpich-3.3.1/bin/mpiexec -n 4 ~/opt/maker.4/maker/bin/maker -f 2> maker.error

Maker_opts.ctl is the same as opts2.txt

These are the log:
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
STATUS: Setting up database for any GFF3 input...
A data structure will be created for you at:
/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore

To access files for individual sequences use the datastore index:
/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_master_datastore_index.log

STATUS: Now running MAKER...
examining contents of the fasta file and run log

--Next Contig--

#---------------------------------------------------------------------
Now starting the contig!!
SeqID: NT_010783.15
Length: 201444
#---------------------------------------------------------------------

setting up GFF3 output and fasta chunks
doing repeat masking
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.0.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
#-------------------------------#
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
#-------------------------------#
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.0
#-------------------------------#
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.1
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.0 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.0.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.1 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.1.repeatrunner
#-------------------------------#
deleted:19 hits
deleted:18 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.2
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.3
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.2 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.2.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.3 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.3.repeatrunner
#-------------------------------#
deleted:9 hits
deleted:9 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.5
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.4
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.5 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.5.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.4 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.4.repeatrunner
#-------------------------------#
deleted:16 hits
deleted:16 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.6
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.7
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.6 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.7 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.7.repeatrunner
#-------------------------------#
deleted:8 hits
deleted:12 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.8
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.9
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.8 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.8.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.9 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.9.repeatrunner
#-------------------------------#
deleted:5 hits
deleted:7 hits
collecting blastx repeatmasking
processing all repeats
in cluster::shadow_cluster...
...finished clustering.

Let me know if you need more information. Thank you in advance for your help.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20190808/b3ec4afb/attachment-0002.html>