<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Your MPI processes may not be seeing each other. So you are getting multiple maker runs all colliding. You need to reinstall MAKER and say ‘yes’ to the compile for MPI question. You may also have to reinstall OpenMPI if just reinstalling MAKER does not work. You can test MAKER for MPI by running the following —> mpiexec -mca btl ^openib -n 40 maker -help<div class=""><br class=""></div><div class="">If you get a single help message then everything is fine. If you get 40 help messages, then MPI is not communicating correctly.</div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 20, 2020, at 10:23 AM, Xu, taosheng <<a href="mailto:taosheng.x@gmail.com" class="">taosheng.x@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thank you very much Carson for your timely response,<div class="">Yes I think so. The Maker MPI should support the multi-processing of an ultra-long single sequence. But I cannot run it successfully for a single sequence.</div><div class="">First I make sure the openMPI with maker has been installed properly. It works well for multiple DNA sequences in a parallel way.</div><div class="">When I submit a maker job for an ultra-long single sequence (mpiexec -mca btl ^openib -n 40 maker -g scaffold1.fasta -fix_nucleotides, The max_dna_len is set to 100000). It always left only one maker thread run. The other maker threads are disappeared and show finished in the output information. See the output information below. Please help me to check it. Thanks for your kind help and your time.</div><div class=""><br class=""></div><div class="">#-----MAKER Behavior Options<br class="">max_dna_len=100000 #length for dividing up contigs into chunks (increases/decreases memory usage)<br class="">min_contig=1 #skip genome contigs below this length (under 10kb are often useless)<br class=""></div><div class=""><br class=""></div><div class="">Best regards,</div><div class="">Taosheng</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">OUTPUT Information<br class="">STATUS: Processing and indexing input FASTA files...<br class="">STATUS: Setting up database for any GFF3 input...<br class="">examining contents of the fasta file and run log<br class="">A data structure will be created for you at:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore<br class=""><br class="">To access files for individual sequences use the datastore index:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_master_datastore_index.log<br class=""><br class="">STATUS: Now running MAKER...<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">Processing run.log file...<br class="">examining contents of the fasta file and run log<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">#---------------------------------------------------------------------<br class="">Another instance of maker is processing this contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class="">examining contents of the fasta file and run log<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">#---------------------------------------------------------------------<br class="">Another instance of maker is processing this contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class=""><br class=""><br class=""><font color="#ff0000" class="">Maker is now finished!!!</font><br class=""><br class=""><br class=""><br class="">Start_time: 1603000030<br class="">End_time: 1603000033<br class="">Elapsed: 3<br class="">A data structure will be created for you at:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore<br class=""><br class="">To access files for individual sequences use the datastore index:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_master_datastore_index.log<br class=""><br class="">STATUS: Now running MAKER...<br class="">MAKER WARNING: The file plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/28/scaffold#1.289.plant_repeatFinal%2Elib%2Empi%2E10%2E2.specific.out<br class="">did not finish on the last run and must be erased<br class="">WARNING: Multiple MAKER processes have been started in the<br class="">same directory.<br class=""><br class="">STATUS: Processing and indexing input FASTA files...<br class="">examining contents of the fasta file and run log<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">#---------------------------------------------------------------------<br class="">Another instance of maker is processing this contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class="">WARNING: Multiple MAKER processes have been started in the<br class="">same directory.<br class=""><br class="">STATUS: Processing and indexing input FASTA files...<br class="">STATUS: Setting up database for any GFF3 input...<br class="">STATUS: Setting up database for any GFF3 input...<br class="">A data structure will be created for you at:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore<br class=""><br class="">To access files for individual sequences use the datastore index:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_master_datastore_index.log<br class=""><br class="">STATUS: Now running MAKER...<br class="">examining contents of the fasta file and run log<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">#---------------------------------------------------------------------<br class="">Another instance of maker is processing this contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class=""><br class=""><br class=""><font color="#cc0000" class=""><b class="">Maker is now finished!!!</b></font><br class=""><br class=""><br class=""><br class="">Start_time: 1603000030<br class="">End_time: 1603000034<br class="">Elapsed: 4<br class="">#---------------------------------------------------------------------<br class="">Now starting the contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class="">examining contents of the fasta file and run log<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">#---------------------------------------------------------------------<br class="">Another instance of maker is processing this contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class="">examining contents of the fasta file and run log<br class=""><br class=""><br class=""><br class="">--Next Contig--<br class=""><br class="">#---------------------------------------------------------------------<br class="">Another instance of maker is processing this contig!!<br class="">SeqID: scaffold#1<br class="">Length: 73580997<br class="">#---------------------------------------------------------------------<br class=""><br class=""><br class=""><br class=""><br class="">Start_time: 1603000030<br class="">End_time: 1603000034<br class="">Elapsed: 4<br class=""><br class=""><br class=""><font color="#cc0000" class=""><b class="">Maker is now finished!!!</b></font><br class=""><br class="">A data structure will be created for you at:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore<br class=""><br class="">To access files for individual sequences use the datastore index:<br class="">/data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_master_datastore_index.log<br class=""><br class="">STATUS: Now running MAKER...<br class="">examining contents of the fasta file and run log</div><div class="">..........</div><div class=""><br class=""></div><div class="">Maker is now finished!!!<br class=""><br class=""><br class=""><br class="">Start_time: 1602920407<br class="">End_time: 1602920580<br class="">Elapsed: 173<br class="">running repeat masker.<br class="">#--------- command -------------#<br class="">Widget::RepeatMasker:<br class="">cd /tmp/maker_VflRYH; ./maker3.01/exe/RepeatMasker/RepeatMasker /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0/scaffold#1.0.plant_repeatFinal%2Elib%2Empi%2E10%2E0.specific -dir /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0 -pa 1 -lib /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/mpi_blastdb/plant_repeatFinal%2Elib.mpi.10/plant_repeatFinal%2Elib.mpi.10.0<br class="">#-------------------------------#<br class="">running repeat masker.<br class="">#--------- command -------------#<br class="">Widget::RepeatMasker:<br class="">cd /tmp/maker_VflRYH; ./maker3.01/exe/RepeatMasker/RepeatMasker /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0/scaffold#1.0.plant_repeatFinal%2Elib%2Empi%2E10%2E1.specific -dir /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0 -pa 1 -lib /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/mpi_blastdb/plant_repeatFinal%2Elib.mpi.10/plant_repeatFinal%2Elib.mpi.10.1<br class="">#-------------------------------#<br class="">running repeat masker.<br class="">#--------- command -------------#<br class="">Widget::RepeatMasker:<br class="">cd /tmp/maker_VflRYH; ./maker3.01/exe/RepeatMasker/RepeatMasker /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0/scaffold#1.0.plant_repeatFinal%2Elib%2Empi%2E10%2E2.specific -dir /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0 -pa 1 -lib /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/mpi_blastdb/plant_repeatFinal%2Elib.mpi.10/plant_repeatFinal%2Elib.mpi.10.2<br class="">#-------------------------------#<br class="">running repeat masker.<br class="">#--------- command -------------#<br class="">Widget::RepeatMasker:<br class="">cd /tmp/maker_VflRYH; ./maker3.01/exe/RepeatMasker/RepeatMasker /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0/scaffold#1.0.plant_repeatFinal%2Elib%2Empi%2E10%2E3.specific -dir /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/plant_datastore/B5/CD/scaffold#1//theVoid.scaffold#1/0 -pa 1 -lib /data/splitgenome/top1000/splitGenome/part1/plant.maker.output/mpi_blastdb/plant_repeatFinal%2Elib.mpi.10/plant_repeatFinal%2Elib.mpi.10.3<br class="">#-------------------------------#<br class="">running repeat masker.<br class="">#--------- command -------------#<br class=""></div><div class=""><br class=""></div><div class="">.....</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 19, 2020 at 10:32 PM Carson Holt <<a href="mailto:carsonhh@gmail.com" class="">carsonhh@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Yes. It will divide contigs into chunks the same size as the max_dna_len parameter.<br class="">
<br class="">
—Carson<br class="">
<br class="">
Sent from my iPhone<br class="">
<br class="">
> On Oct 17, 2020, at 12:48 AM, Xu, taosheng <<a href="mailto:taosheng.x@gmail.com" target="_blank" class="">taosheng.x@gmail.com</a>> wrote:<br class="">
> <br class="">
> <br class="">
> Dear Maker Development Team,<br class="">
> I wonder whether maker supports parallel processing for a single long genome sequence?<br class="">
> <br class="">
> When I submit my maker task using openMPI with multiple cpus (like, mpiexec -n 40 maker) to annotate a single long genome sequence, always only one maker with 4 rmblast run. The other cpus is on idle.<br class="">
> <br class="">
> I want to use the maker parallel processing with openMPI to speed up a single ultra-long genome sequence annotation.<br class="">
> <br class="">
> Best regards,<br class="">
> Taosheng<br class="">
> _______________________________________________<br class="">
> maker-devel mailing list<br class="">
> <a href="mailto:maker-devel@yandell-lab.org" target="_blank" class="">maker-devel@yandell-lab.org</a><br class="">
> <a href="http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org" rel="noreferrer" target="_blank" class="">http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org</a><br class="">
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>