From qlian003 at ucr.edu  Thu Feb  1 12:32:20 2018
From: qlian003 at ucr.edu (Qihua Liang)
Date: Thu, 1 Feb 2018 10:32:20 -0800
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
Message-ID: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>

Hi Carson,

I re-installed Maker and I ran Maker again with the same setting.

This time I still could not get any results. But I have different error messages, saying:
doing blastx repeats
doing blastx repeats
doing blastx repeats
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
doing blastx repeats
doing blastx repeats
gathering ab-init output files
FATAL: Can not identify the version of GeneMark used for the report

--> rank=NA, hostname=H4
ERROR: Failed while gathering ab-init output files
ERROR: Chunk failed at level:1, tier_type:2
FAILED CONTIG:ScsGwly_16;HRSCAF=18

ERROR: Chunk failed at level:4, tier_type:0
FAILED CONTIG:ScsGwly_16;HRSCAF=18

examining contents of the fasta file and run log
#---------------------------------------------------- 


It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  

Could you provide some trouble shooting information regarding this?

Thank you
Qihua

> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
> 
> ?Carson
> 
> 
>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>> 
>> Hi Carson,
>> 
>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>> 
>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>> 
>> Thank you!
>> Qihua
>> 
>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>> 
>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>>> Hi Carson,
>>>> 
>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>> 
>>>> Thanks
>>>> Qihua
>>>> 
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> preparing masked sequence
>>>> preparing ab-inits
>>>> running  snap.
>>>> #--------- command -------------#
>>>> Widget::snap:
>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>> mm.snap
>>>> #-------------------------------#
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> running  augustus.
>>>> #--------- command -------------#
>>>> Widget::augustus:
>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> running  blast search.
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>> #-------------------------------#
>>>> doing blastx repeats
>>>> re reading blast report.
>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>> No such file or directory
>>>> 
>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Failed while builing masking tiers
>>>> --> rank=NA, hostname=H4
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Can not get next level
>>>> running  genemark.
>>>> #--------- command -------------#
>>>> Widget::genemark:
>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>> #-------------------------------#
>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>> 
>>>> examining contents of the fasta file and run log
>>>> 
>>>> 
>>>> 
>>>> --Next Contig--
>>>> 
>>>> #---------------------------------------------------------------------
>>>> Now starting the contig!!
>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>> Length: 1247
>>>> #---------------------------------------------------------------------
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>> 
>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>> 
>>>>> ?Carson
>>>>> 
>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>> 
>>>>>> Hi Ence,
>>>>>> 
>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>> 
>>>>>> Is this what you may need?
>>>>>> 
>>>>>> Qihua
>>>>>> 
>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>> 
>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Daniel 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>> 
>>>>>>>> Dear Maker Develop Team,
>>>>>>>> 
>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>> 
>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>> 
>>>>>>>> Thank you so much
>>>>>>>> Qihua
>>>>>>>> _______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> maker-devel mailing list
>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>> 
>>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180201/536a6cc7/attachment.html>

From carsonhh at gmail.com  Sun Feb  4 10:40:14 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Sun, 4 Feb 2018 17:40:14 +0100
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
	<0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
Message-ID: <A61B15A3-5713-40C0-AC1A-0BCA7699CD98@gmail.com>

It?s saying the GeneMark output file does not have the right number of columns, so MAKER can?t use it. If there is a column mismatch, this likely means you are using a different version of GeneMark that MAKER does not support.

Make sure you are using GeneMark-ES and not GeneMark-S or GeneMark.HMM (yes there are lots of versions of GeneMark)

Found here ?> http://exon.gatech.edu/GeneMark/gmes_instructions.html <http://exon.gatech.edu/GeneMark/gmes_instructions.html>

--Carson


> On Feb 1, 2018, at 7:32 PM, Qihua Liang <qlian003 at ucr.edu> wrote:
> 
> Hi Carson,
> 
> I re-installed Maker and I ran Maker again with the same setting.
> 
> This time I still could not get any results. But I have different error messages, saying:
> doing blastx repeats
> doing blastx repeats
> doing blastx repeats
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> doing blastx repeats
> doing blastx repeats
> gathering ab-init output files
> FATAL: Can not identify the version of GeneMark used for the report
> 
> --> rank=NA, hostname=H4
> ERROR: Failed while gathering ab-init output files
> ERROR: Chunk failed at level:1, tier_type:2
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> ERROR: Chunk failed at level:4, tier_type:0
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> examining contents of the fasta file and run log
> #---------------------------------------------------- 
> 
> 
> It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  
> 
> Could you provide some trouble shooting information regarding this?
> 
> Thank you
> Qihua
> 
>> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>> 
>> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
>> 
>> ?Carson
>> 
>> 
>>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>> Hi Carson,
>>> 
>>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>>> 
>>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>>> 
>>> Thank you!
>>> Qihua
>>> 
>>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>> 
>>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>> 
>>>>> Hi Carson,
>>>>> 
>>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>>> 
>>>>> Thanks
>>>>> Qihua
>>>>> 
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> preparing masked sequence
>>>>> preparing ab-inits
>>>>> running  snap.
>>>>> #--------- command -------------#
>>>>> Widget::snap:
>>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>>> mm.snap
>>>>> #-------------------------------#
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> running  augustus.
>>>>> #--------- command -------------#
>>>>> Widget::augustus:
>>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> running  blast search.
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>>> #-------------------------------#
>>>>> doing blastx repeats
>>>>> re reading blast report.
>>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>>> No such file or directory
>>>>> 
>>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Failed while builing masking tiers
>>>>> --> rank=NA, hostname=H4
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Can not get next level
>>>>> running  genemark.
>>>>> #--------- command -------------#
>>>>> Widget::genemark:
>>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>>> #-------------------------------#
>>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>>> 
>>>>> examining contents of the fasta file and run log
>>>>> 
>>>>> 
>>>>> 
>>>>> --Next Contig--
>>>>> 
>>>>> #---------------------------------------------------------------------
>>>>> Now starting the contig!!
>>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>>> Length: 1247
>>>>> #---------------------------------------------------------------------
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>>> 
>>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>>> 
>>>>>> ?Carson
>>>>>> 
>>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>> 
>>>>>>> Hi Ence,
>>>>>>> 
>>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>>> 
>>>>>>> Is this what you may need?
>>>>>>> 
>>>>>>> Qihua
>>>>>>> 
>>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>>> 
>>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Daniel 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>>> 
>>>>>>>>> Dear Maker Develop Team,
>>>>>>>>> 
>>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>>> 
>>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>>> 
>>>>>>>>> Thank you so much
>>>>>>>>> Qihua
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> maker-devel mailing list
>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>>> 
>>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180204/2e9d58c9/attachment.html>

From admin at genome.arizona.edu  Thu Feb  8 10:53:13 2018
From: admin at genome.arizona.edu (Chandler)
Date: Thu, 8 Feb 2018 09:53:13 -0700
Subject: [maker-devel] MPI selection
In-Reply-To: <F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
Message-ID: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>

Thanks Carson, one other question though.

When you mentioned to keep jobs under 200 CPU cores, did you mean as in 
what is emulated to the OS, or physical hardware?  We are using 
hyperthreading which emulates more CPUs to the OS, and so we have about 
256 emulated CPUs available.  Some of our applications are better 
optimized for this, so I keep it that way.

So suppose we keep hyperthreading enabled, should we specify in the 
machine list file that mpiexec uses to only use 128 of the emulated 
cores?  We have noticed with using all 256 hyperthreaded cores that the 
load can get high, although everything still works great.

Thanks


Chandler / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu


Carson Holt wrote on 01/30/2018 10:37 AM:
> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
> 
> ?Carson
> 
> 
> 
>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>
>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>
>>
>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>
>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> .
> 



From carsonhh at gmail.com  Thu Feb  8 11:10:04 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 8 Feb 2018 18:10:04 +0100
Subject: [maker-devel] MPI selection
In-Reply-To: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
	<b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
Message-ID: <237BC5A6-9008-48EC-86F2-6A55CBEEC214@gmail.com>

Yes, you can oversubscribe and match the hyperthread count (you will need ~1GB of RAM per process though). But you should still keep the overall number of MPI processes given to mpiexec below ~200. This is because, the way I structured the MPI controller in MAKER is that I have one manager process and all the other processes are workers. So if there are too many workers, they can overwhelm the manager with communications, and scaling efficiency drops. So starting another MAKER job, launches a new manager with his own workers. In that way, you get around the communication bottleneck by launching multiple large MAKER jobs and scaling efficiency returns to near linear. You can run with multiple jobs until you start hitting IO bottlenecks (MAKER will perform high IOPS but not necessarily high bandwidth).

?Carson




> On Feb 8, 2018, at 5:53 PM, Chandler <admin at genome.arizona.edu> wrote:
> 
> Thanks Carson, one other question though.
> 
> When you mentioned to keep jobs under 200 CPU cores, did you mean as in what is emulated to the OS, or physical hardware?  We are using hyperthreading which emulates more CPUs to the OS, and so we have about 256 emulated CPUs available.  Some of our applications are better optimized for this, so I keep it that way.
> 
> So suppose we keep hyperthreading enabled, should we specify in the machine list file that mpiexec uses to only use 128 of the emulated cores?  We have noticed with using all 256 hyperthreaded cores that the load can get high, although everything still works great.
> 
> Thanks
> 
> 
> Chandler / Systems Administrator
> Arizona Genomics Institute
> www.genome.arizona.edu
> 
> 
> Carson Holt wrote on 01/30/2018 10:37 AM:
>> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
>> ?Carson
>>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>> 
>>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>> 
>>> 
>>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>> 
>>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> .
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From vsoza at uw.edu  Mon Feb 12 13:41:34 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Mon, 12 Feb 2018 11:41:34 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but no
 errors in screen output
Message-ID: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>

Hi all

I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.

I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.

$ grep FAILED Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED

$ grep RETRY Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY

When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:

#---------------------------------------------------------------------
Now starting the contig!!
SeqID: LG12_ordered_scaffold_101
Length: 89169
#---------------------------------------------------------------------


setting up GFF3 output and fasta chunks
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
#???????????????#

It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 

Thanks.

-Valerie
	
Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From skmccormick19 at gmail.com  Sat Feb 17 11:24:40 2018
From: skmccormick19 at gmail.com (Kevin McCormick)
Date: Sat, 17 Feb 2018 12:24:40 -0500
Subject: [maker-devel] Need Help with maker2jbrowse
Message-ID: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>

Hello,

I have hit a wall attempting to get my MAKER data converted for jbrowse use

I have attached a file showing the problem that is currently leaving me
scratching my head.

Could you please give me some advice, or direct me to someone who may be
able to help.

Thank you

-- 
S. Kevin McCormick

*PhD Candidate*
*Integrative Biology Department - Holekamp Lab*
*Ecology, Evolutionary Biology, and Behavior Program*
*Michigan State University*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf
Type: application/rtf
Size: 9238 bytes
Desc: not available
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment.rtf>

From carsonhh at gmail.com  Tue Feb 20 10:23:05 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:05 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
Message-ID: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>

Hi Valerie,

Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err

Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log

You may get something like this:

unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED

If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.

Thanks,
Carson



> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi all
> 
> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
> 
> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
> 
> $ grep FAILED Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> 
> $ grep RETRY Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> 
> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
> 
> #---------------------------------------------------------------------
> Now starting the contig!!
> SeqID: LG12_ordered_scaffold_101
> Length: 89169
> #---------------------------------------------------------------------
> 
> 
> setting up GFF3 output and fasta chunks
> doing repeat masking
> running  repeat masker.
> #--------- command -------------#
> Widget::RepeatMasker:
> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
> #???????????????#
> 
> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
> 
> Thanks.
> 
> -Valerie
> 	
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Tue Feb 20 10:23:56 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:56 -0700
Subject: [maker-devel] Need Help with maker2jbrowse
In-Reply-To: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
References: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
Message-ID: <ADA97995-86AE-4969-AC1A-B70D1C88F614@gmail.com>

Make sure you ran the setup.sh script that comes with JBrowse. You are missing a perl dependancy, and JBrowse should install it for you when you run the setup script.

?Carson


> On Feb 17, 2018, at 10:24 AM, Kevin McCormick <skmccormick19 at gmail.com> wrote:
> 
> Hello,
> 
> I have hit a wall attempting to get my MAKER data converted for jbrowse use
> 
> I have attached a file showing the problem that is currently leaving me scratching my head.
> 
> Could you please give me some advice, or direct me to someone who may be able to help.
> 
> Thank you
> 
> -- 
> S. Kevin McCormick
> 
> PhD Candidate
> Integrative Biology Department - Holekamp Lab
> Ecology, Evolutionary Biology, and Behavior Program
> Michigan State University
> <MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180220/dbd3a026/attachment.html>

From carsonhh at gmail.com  Tue Feb 20 17:03:11 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 16:03:11 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
	<37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
Message-ID: <E607C645-A8F1-4F27-AE5C-ADE9AFA813E2@gmail.com>

On two, there is a FINISHED entry but it is out of order. There may be a race condition with one process breaking the other?s file lock if you ran multiple jobs at the same time. In which case, one failed and the other finished near the same time. The failure from a broken lock may not say ?ERROR? in the STDERR, but may have another tag. search for ?unclustered_scaffold_3148? in the STDERR, then look at the entries above and below it.

?Carson


> On Feb 20, 2018, at 3:58 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi Carson
> 
> No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:
> 
> $ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED
> 
> $ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED
> 
> $ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> 
> $ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED
> 
> Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.
> 
> -Valerie
> 
> 
>> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>> Hi Valerie,
>> 
>> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
>> 
>> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
>> 
>> You may get something like this:
>> 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
>> 
>> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>>> 
>>> Hi all
>>> 
>>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>>> 
>>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>>> 
>>> $ grep FAILED Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>>> 
>>> $ grep RETRY Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>>> 
>>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>>> 
>>> #---------------------------------------------------------------------
>>> Now starting the contig!!
>>> SeqID: LG12_ordered_scaffold_101
>>> Length: 89169
>>> #---------------------------------------------------------------------
>>> 
>>> 
>>> setting up GFF3 output and fasta chunks
>>> doing repeat masking
>>> running  repeat masker.
>>> #--------- command -------------#
>>> Widget::RepeatMasker:
>>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>>> #???????????????#
>>> 
>>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>>> 
>>> Thanks.
>>> 
>>> -Valerie
>>> 	
>>> Valerie Soza, Ph.D.
>>> c/o Hall Lab
>>> Department of Biology
>>> University of Washington
>>> Johnson Hall 202A
>>> Box 351800
>>> Seattle, WA 98195-1800
>>> 206-543-6740
>>> http://staff.washington.edu/vsoza/
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
> 
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 




From vsoza at uw.edu  Tue Feb 20 16:58:20 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Tue, 20 Feb 2018 14:58:20 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
Message-ID: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>

Hi Carson

No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:

$ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED

$ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED

$ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED

$ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED

Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.

-Valerie


> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> Hi Valerie,
> 
> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
> 
> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
> 
> You may get something like this:
> 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> 
> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
> 
> Thanks,
> Carson
> 
> 
> 
>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>> 
>> Hi all
>> 
>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>> 
>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>> 
>> $ grep FAILED Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>> 
>> $ grep RETRY Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>> 
>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>> 
>> #---------------------------------------------------------------------
>> Now starting the contig!!
>> SeqID: LG12_ordered_scaffold_101
>> Length: 89169
>> #---------------------------------------------------------------------
>> 
>> 
>> setting up GFF3 output and fasta chunks
>> doing repeat masking
>> running  repeat masker.
>> #--------- command -------------#
>> Widget::RepeatMasker:
>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>> #???????????????#
>> 
>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>> 
>> Thanks.
>> 
>> -Valerie
>> 	
>> Valerie Soza, Ph.D.
>> c/o Hall Lab
>> Department of Biology
>> University of Washington
>> Johnson Hall 202A
>> Box 351800
>> Seattle, WA 98195-1800
>> 206-543-6740
>> http://staff.washington.edu/vsoza/
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 

Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From Emily.Giroux at inspection.gc.ca  Fri Feb 23 08:43:49 2018
From: Emily.Giroux at inspection.gc.ca (Giroux, Emily (CFIA/ACIA))
Date: Fri, 23 Feb 2018 14:43:49 +0000
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
Message-ID: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>

Hi,

After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don't waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.

My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I'm not sure if I can just ignore this...

Thanks,

Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180223/de430c44/attachment.html>

From carsonhh at gmail.com  Mon Feb 26 22:30:19 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Mon, 26 Feb 2018 21:30:19 -0700
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
In-Reply-To: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
References: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
Message-ID: <DD5237FE-8F83-4527-AFF4-8F056FCBC425@gmail.com>

Final gene models will have ?maker' in the source column. All others are not gene models, but rather evidence alignments that are there for reference. Make sure you specified a gene predictor during the maker run, or you may only have alignments and no models. Anything with protein_gff:protein2genome in the source column would fall into the category of evidence alignment.

I do not believe GAG supports tRNA?s. Someone may be able to correct me, but I believe they only support the coding annotations.

?Carson




> On Feb 23, 2018, at 7:43 AM, Giroux, Emily (CFIA/ACIA) <Emily.Giroux at inspection.gc.ca> wrote:
> 
> Hi,
>  
> After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don?t waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.
>  
> My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I?m not sure if I can just ignore this?
>  
> Thanks,
>  
> Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20180226/645ffb5f/attachment.html>

From qlian003 at ucr.edu  Thu Feb  1 11:32:20 2018
From: qlian003 at ucr.edu (Qihua Liang)
Date: Thu, 1 Feb 2018 10:32:20 -0800
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
Message-ID: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>

Hi Carson,

I re-installed Maker and I ran Maker again with the same setting.

This time I still could not get any results. But I have different error messages, saying:
doing blastx repeats
doing blastx repeats
doing blastx repeats
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
doing blastx repeats
doing blastx repeats
gathering ab-init output files
FATAL: Can not identify the version of GeneMark used for the report

--> rank=NA, hostname=H4
ERROR: Failed while gathering ab-init output files
ERROR: Chunk failed at level:1, tier_type:2
FAILED CONTIG:ScsGwly_16;HRSCAF=18

ERROR: Chunk failed at level:4, tier_type:0
FAILED CONTIG:ScsGwly_16;HRSCAF=18

examining contents of the fasta file and run log
#---------------------------------------------------- 


It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  

Could you provide some trouble shooting information regarding this?

Thank you
Qihua

> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
> 
> ?Carson
> 
> 
>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>> 
>> Hi Carson,
>> 
>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>> 
>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>> 
>> Thank you!
>> Qihua
>> 
>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>> 
>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>>> Hi Carson,
>>>> 
>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>> 
>>>> Thanks
>>>> Qihua
>>>> 
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> preparing masked sequence
>>>> preparing ab-inits
>>>> running  snap.
>>>> #--------- command -------------#
>>>> Widget::snap:
>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>> mm.snap
>>>> #-------------------------------#
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> running  augustus.
>>>> #--------- command -------------#
>>>> Widget::augustus:
>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> running  blast search.
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>> #-------------------------------#
>>>> doing blastx repeats
>>>> re reading blast report.
>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>> No such file or directory
>>>> 
>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Failed while builing masking tiers
>>>> --> rank=NA, hostname=H4
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Can not get next level
>>>> running  genemark.
>>>> #--------- command -------------#
>>>> Widget::genemark:
>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>> #-------------------------------#
>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>> 
>>>> examining contents of the fasta file and run log
>>>> 
>>>> 
>>>> 
>>>> --Next Contig--
>>>> 
>>>> #---------------------------------------------------------------------
>>>> Now starting the contig!!
>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>> Length: 1247
>>>> #---------------------------------------------------------------------
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>> 
>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>> 
>>>>> ?Carson
>>>>> 
>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>> 
>>>>>> Hi Ence,
>>>>>> 
>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>> 
>>>>>> Is this what you may need?
>>>>>> 
>>>>>> Qihua
>>>>>> 
>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>> 
>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Daniel 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>> 
>>>>>>>> Dear Maker Develop Team,
>>>>>>>> 
>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>> 
>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>> 
>>>>>>>> Thank you so much
>>>>>>>> Qihua
>>>>>>>> _______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> maker-devel mailing list
>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>> 
>>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180201/536a6cc7/attachment-0001.html>

From carsonhh at gmail.com  Sun Feb  4 09:40:14 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Sun, 4 Feb 2018 17:40:14 +0100
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
	<0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
Message-ID: <A61B15A3-5713-40C0-AC1A-0BCA7699CD98@gmail.com>

It?s saying the GeneMark output file does not have the right number of columns, so MAKER can?t use it. If there is a column mismatch, this likely means you are using a different version of GeneMark that MAKER does not support.

Make sure you are using GeneMark-ES and not GeneMark-S or GeneMark.HMM (yes there are lots of versions of GeneMark)

Found here ?> http://exon.gatech.edu/GeneMark/gmes_instructions.html <http://exon.gatech.edu/GeneMark/gmes_instructions.html>

--Carson


> On Feb 1, 2018, at 7:32 PM, Qihua Liang <qlian003 at ucr.edu> wrote:
> 
> Hi Carson,
> 
> I re-installed Maker and I ran Maker again with the same setting.
> 
> This time I still could not get any results. But I have different error messages, saying:
> doing blastx repeats
> doing blastx repeats
> doing blastx repeats
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> doing blastx repeats
> doing blastx repeats
> gathering ab-init output files
> FATAL: Can not identify the version of GeneMark used for the report
> 
> --> rank=NA, hostname=H4
> ERROR: Failed while gathering ab-init output files
> ERROR: Chunk failed at level:1, tier_type:2
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> ERROR: Chunk failed at level:4, tier_type:0
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> examining contents of the fasta file and run log
> #---------------------------------------------------- 
> 
> 
> It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  
> 
> Could you provide some trouble shooting information regarding this?
> 
> Thank you
> Qihua
> 
>> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>> 
>> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
>> 
>> ?Carson
>> 
>> 
>>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>> Hi Carson,
>>> 
>>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>>> 
>>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>>> 
>>> Thank you!
>>> Qihua
>>> 
>>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>> 
>>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>> 
>>>>> Hi Carson,
>>>>> 
>>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>>> 
>>>>> Thanks
>>>>> Qihua
>>>>> 
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> preparing masked sequence
>>>>> preparing ab-inits
>>>>> running  snap.
>>>>> #--------- command -------------#
>>>>> Widget::snap:
>>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>>> mm.snap
>>>>> #-------------------------------#
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> running  augustus.
>>>>> #--------- command -------------#
>>>>> Widget::augustus:
>>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> running  blast search.
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>>> #-------------------------------#
>>>>> doing blastx repeats
>>>>> re reading blast report.
>>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>>> No such file or directory
>>>>> 
>>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Failed while builing masking tiers
>>>>> --> rank=NA, hostname=H4
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Can not get next level
>>>>> running  genemark.
>>>>> #--------- command -------------#
>>>>> Widget::genemark:
>>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>>> #-------------------------------#
>>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>>> 
>>>>> examining contents of the fasta file and run log
>>>>> 
>>>>> 
>>>>> 
>>>>> --Next Contig--
>>>>> 
>>>>> #---------------------------------------------------------------------
>>>>> Now starting the contig!!
>>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>>> Length: 1247
>>>>> #---------------------------------------------------------------------
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>>> 
>>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>>> 
>>>>>> ?Carson
>>>>>> 
>>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>> 
>>>>>>> Hi Ence,
>>>>>>> 
>>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>>> 
>>>>>>> Is this what you may need?
>>>>>>> 
>>>>>>> Qihua
>>>>>>> 
>>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>>> 
>>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Daniel 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>>> 
>>>>>>>>> Dear Maker Develop Team,
>>>>>>>>> 
>>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>>> 
>>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>>> 
>>>>>>>>> Thank you so much
>>>>>>>>> Qihua
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> maker-devel mailing list
>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>>> 
>>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180204/2e9d58c9/attachment-0001.html>

From admin at genome.arizona.edu  Thu Feb  8 09:53:13 2018
From: admin at genome.arizona.edu (Chandler)
Date: Thu, 8 Feb 2018 09:53:13 -0700
Subject: [maker-devel] MPI selection
In-Reply-To: <F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
Message-ID: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>

Thanks Carson, one other question though.

When you mentioned to keep jobs under 200 CPU cores, did you mean as in 
what is emulated to the OS, or physical hardware?  We are using 
hyperthreading which emulates more CPUs to the OS, and so we have about 
256 emulated CPUs available.  Some of our applications are better 
optimized for this, so I keep it that way.

So suppose we keep hyperthreading enabled, should we specify in the 
machine list file that mpiexec uses to only use 128 of the emulated 
cores?  We have noticed with using all 256 hyperthreaded cores that the 
load can get high, although everything still works great.

Thanks


Chandler / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu


Carson Holt wrote on 01/30/2018 10:37 AM:
> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
> 
> ?Carson
> 
> 
> 
>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>
>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>
>>
>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>
>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> .
> 



From carsonhh at gmail.com  Thu Feb  8 10:10:04 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 8 Feb 2018 18:10:04 +0100
Subject: [maker-devel] MPI selection
In-Reply-To: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
	<b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
Message-ID: <237BC5A6-9008-48EC-86F2-6A55CBEEC214@gmail.com>

Yes, you can oversubscribe and match the hyperthread count (you will need ~1GB of RAM per process though). But you should still keep the overall number of MPI processes given to mpiexec below ~200. This is because, the way I structured the MPI controller in MAKER is that I have one manager process and all the other processes are workers. So if there are too many workers, they can overwhelm the manager with communications, and scaling efficiency drops. So starting another MAKER job, launches a new manager with his own workers. In that way, you get around the communication bottleneck by launching multiple large MAKER jobs and scaling efficiency returns to near linear. You can run with multiple jobs until you start hitting IO bottlenecks (MAKER will perform high IOPS but not necessarily high bandwidth).

?Carson




> On Feb 8, 2018, at 5:53 PM, Chandler <admin at genome.arizona.edu> wrote:
> 
> Thanks Carson, one other question though.
> 
> When you mentioned to keep jobs under 200 CPU cores, did you mean as in what is emulated to the OS, or physical hardware?  We are using hyperthreading which emulates more CPUs to the OS, and so we have about 256 emulated CPUs available.  Some of our applications are better optimized for this, so I keep it that way.
> 
> So suppose we keep hyperthreading enabled, should we specify in the machine list file that mpiexec uses to only use 128 of the emulated cores?  We have noticed with using all 256 hyperthreaded cores that the load can get high, although everything still works great.
> 
> Thanks
> 
> 
> Chandler / Systems Administrator
> Arizona Genomics Institute
> www.genome.arizona.edu
> 
> 
> Carson Holt wrote on 01/30/2018 10:37 AM:
>> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
>> ?Carson
>>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>> 
>>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>> 
>>> 
>>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>> 
>>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> .
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From vsoza at uw.edu  Mon Feb 12 12:41:34 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Mon, 12 Feb 2018 11:41:34 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but no
 errors in screen output
Message-ID: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>

Hi all

I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.

I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.

$ grep FAILED Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED

$ grep RETRY Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY

When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:

#---------------------------------------------------------------------
Now starting the contig!!
SeqID: LG12_ordered_scaffold_101
Length: 89169
#---------------------------------------------------------------------


setting up GFF3 output and fasta chunks
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
#???????????????#

It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 

Thanks.

-Valerie
	
Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From skmccormick19 at gmail.com  Sat Feb 17 10:24:40 2018
From: skmccormick19 at gmail.com (Kevin McCormick)
Date: Sat, 17 Feb 2018 12:24:40 -0500
Subject: [maker-devel] Need Help with maker2jbrowse
Message-ID: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>

Hello,

I have hit a wall attempting to get my MAKER data converted for jbrowse use

I have attached a file showing the problem that is currently leaving me
scratching my head.

Could you please give me some advice, or direct me to someone who may be
able to help.

Thank you

-- 
S. Kevin McCormick

*PhD Candidate*
*Integrative Biology Department - Holekamp Lab*
*Ecology, Evolutionary Biology, and Behavior Program*
*Michigan State University*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf
Type: application/rtf
Size: 9238 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment-0001.rtf>

From carsonhh at gmail.com  Tue Feb 20 09:23:05 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:05 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
Message-ID: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>

Hi Valerie,

Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err

Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log

You may get something like this:

unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED

If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.

Thanks,
Carson



> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi all
> 
> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
> 
> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
> 
> $ grep FAILED Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> 
> $ grep RETRY Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> 
> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
> 
> #---------------------------------------------------------------------
> Now starting the contig!!
> SeqID: LG12_ordered_scaffold_101
> Length: 89169
> #---------------------------------------------------------------------
> 
> 
> setting up GFF3 output and fasta chunks
> doing repeat masking
> running  repeat masker.
> #--------- command -------------#
> Widget::RepeatMasker:
> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
> #???????????????#
> 
> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
> 
> Thanks.
> 
> -Valerie
> 	
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Tue Feb 20 09:23:56 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:56 -0700
Subject: [maker-devel] Need Help with maker2jbrowse
In-Reply-To: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
References: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
Message-ID: <ADA97995-86AE-4969-AC1A-B70D1C88F614@gmail.com>

Make sure you ran the setup.sh script that comes with JBrowse. You are missing a perl dependancy, and JBrowse should install it for you when you run the setup script.

?Carson


> On Feb 17, 2018, at 10:24 AM, Kevin McCormick <skmccormick19 at gmail.com> wrote:
> 
> Hello,
> 
> I have hit a wall attempting to get my MAKER data converted for jbrowse use
> 
> I have attached a file showing the problem that is currently leaving me scratching my head.
> 
> Could you please give me some advice, or direct me to someone who may be able to help.
> 
> Thank you
> 
> -- 
> S. Kevin McCormick
> 
> PhD Candidate
> Integrative Biology Department - Holekamp Lab
> Ecology, Evolutionary Biology, and Behavior Program
> Michigan State University
> <MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180220/dbd3a026/attachment-0001.html>

From carsonhh at gmail.com  Tue Feb 20 16:03:11 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 16:03:11 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
	<37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
Message-ID: <E607C645-A8F1-4F27-AE5C-ADE9AFA813E2@gmail.com>

On two, there is a FINISHED entry but it is out of order. There may be a race condition with one process breaking the other?s file lock if you ran multiple jobs at the same time. In which case, one failed and the other finished near the same time. The failure from a broken lock may not say ?ERROR? in the STDERR, but may have another tag. search for ?unclustered_scaffold_3148? in the STDERR, then look at the entries above and below it.

?Carson


> On Feb 20, 2018, at 3:58 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi Carson
> 
> No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:
> 
> $ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED
> 
> $ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED
> 
> $ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> 
> $ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED
> 
> Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.
> 
> -Valerie
> 
> 
>> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>> Hi Valerie,
>> 
>> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
>> 
>> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
>> 
>> You may get something like this:
>> 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
>> 
>> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>>> 
>>> Hi all
>>> 
>>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>>> 
>>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>>> 
>>> $ grep FAILED Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>>> 
>>> $ grep RETRY Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>>> 
>>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>>> 
>>> #---------------------------------------------------------------------
>>> Now starting the contig!!
>>> SeqID: LG12_ordered_scaffold_101
>>> Length: 89169
>>> #---------------------------------------------------------------------
>>> 
>>> 
>>> setting up GFF3 output and fasta chunks
>>> doing repeat masking
>>> running  repeat masker.
>>> #--------- command -------------#
>>> Widget::RepeatMasker:
>>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>>> #???????????????#
>>> 
>>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>>> 
>>> Thanks.
>>> 
>>> -Valerie
>>> 	
>>> Valerie Soza, Ph.D.
>>> c/o Hall Lab
>>> Department of Biology
>>> University of Washington
>>> Johnson Hall 202A
>>> Box 351800
>>> Seattle, WA 98195-1800
>>> 206-543-6740
>>> http://staff.washington.edu/vsoza/
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
> 
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 




From vsoza at uw.edu  Tue Feb 20 15:58:20 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Tue, 20 Feb 2018 14:58:20 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
Message-ID: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>

Hi Carson

No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:

$ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED

$ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED

$ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED

$ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED

Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.

-Valerie


> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> Hi Valerie,
> 
> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
> 
> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
> 
> You may get something like this:
> 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> 
> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
> 
> Thanks,
> Carson
> 
> 
> 
>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>> 
>> Hi all
>> 
>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>> 
>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>> 
>> $ grep FAILED Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>> 
>> $ grep RETRY Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>> 
>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>> 
>> #---------------------------------------------------------------------
>> Now starting the contig!!
>> SeqID: LG12_ordered_scaffold_101
>> Length: 89169
>> #---------------------------------------------------------------------
>> 
>> 
>> setting up GFF3 output and fasta chunks
>> doing repeat masking
>> running  repeat masker.
>> #--------- command -------------#
>> Widget::RepeatMasker:
>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>> #???????????????#
>> 
>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>> 
>> Thanks.
>> 
>> -Valerie
>> 	
>> Valerie Soza, Ph.D.
>> c/o Hall Lab
>> Department of Biology
>> University of Washington
>> Johnson Hall 202A
>> Box 351800
>> Seattle, WA 98195-1800
>> 206-543-6740
>> http://staff.washington.edu/vsoza/
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 

Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From Emily.Giroux at inspection.gc.ca  Fri Feb 23 07:43:49 2018
From: Emily.Giroux at inspection.gc.ca (Giroux, Emily (CFIA/ACIA))
Date: Fri, 23 Feb 2018 14:43:49 +0000
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
Message-ID: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>

Hi,

After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don't waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.

My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I'm not sure if I can just ignore this...

Thanks,

Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180223/de430c44/attachment-0001.html>

From carsonhh at gmail.com  Mon Feb 26 21:30:19 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Mon, 26 Feb 2018 21:30:19 -0700
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
In-Reply-To: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
References: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
Message-ID: <DD5237FE-8F83-4527-AFF4-8F056FCBC425@gmail.com>

Final gene models will have ?maker' in the source column. All others are not gene models, but rather evidence alignments that are there for reference. Make sure you specified a gene predictor during the maker run, or you may only have alignments and no models. Anything with protein_gff:protein2genome in the source column would fall into the category of evidence alignment.

I do not believe GAG supports tRNA?s. Someone may be able to correct me, but I believe they only support the coding annotations.

?Carson




> On Feb 23, 2018, at 7:43 AM, Giroux, Emily (CFIA/ACIA) <Emily.Giroux at inspection.gc.ca> wrote:
> 
> Hi,
>  
> After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don?t waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.
>  
> My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I?m not sure if I can just ignore this?
>  
> Thanks,
>  
> Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180226/645ffb5f/attachment-0001.html>

From qlian003 at ucr.edu  Thu Feb  1 11:32:20 2018
From: qlian003 at ucr.edu (Qihua Liang)
Date: Thu, 1 Feb 2018 10:32:20 -0800
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
Message-ID: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>

Hi Carson,

I re-installed Maker and I ran Maker again with the same setting.

This time I still could not get any results. But I have different error messages, saying:
doing blastx repeats
doing blastx repeats
doing blastx repeats
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
doing blastx repeats
doing blastx repeats
gathering ab-init output files
FATAL: Can not identify the version of GeneMark used for the report

--> rank=NA, hostname=H4
ERROR: Failed while gathering ab-init output files
ERROR: Chunk failed at level:1, tier_type:2
FAILED CONTIG:ScsGwly_16;HRSCAF=18

ERROR: Chunk failed at level:4, tier_type:0
FAILED CONTIG:ScsGwly_16;HRSCAF=18

examining contents of the fasta file and run log
#---------------------------------------------------- 


It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  

Could you provide some trouble shooting information regarding this?

Thank you
Qihua

> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
> 
> ?Carson
> 
> 
>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>> 
>> Hi Carson,
>> 
>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>> 
>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>> 
>> Thank you!
>> Qihua
>> 
>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>> 
>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>>> Hi Carson,
>>>> 
>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>> 
>>>> Thanks
>>>> Qihua
>>>> 
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> preparing masked sequence
>>>> preparing ab-inits
>>>> running  snap.
>>>> #--------- command -------------#
>>>> Widget::snap:
>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>> mm.snap
>>>> #-------------------------------#
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> running  augustus.
>>>> #--------- command -------------#
>>>> Widget::augustus:
>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> running  blast search.
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>> #-------------------------------#
>>>> doing blastx repeats
>>>> re reading blast report.
>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>> No such file or directory
>>>> 
>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Failed while builing masking tiers
>>>> --> rank=NA, hostname=H4
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Can not get next level
>>>> running  genemark.
>>>> #--------- command -------------#
>>>> Widget::genemark:
>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>> #-------------------------------#
>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>> 
>>>> examining contents of the fasta file and run log
>>>> 
>>>> 
>>>> 
>>>> --Next Contig--
>>>> 
>>>> #---------------------------------------------------------------------
>>>> Now starting the contig!!
>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>> Length: 1247
>>>> #---------------------------------------------------------------------
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>> 
>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>> 
>>>>> ?Carson
>>>>> 
>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>> 
>>>>>> Hi Ence,
>>>>>> 
>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>> 
>>>>>> Is this what you may need?
>>>>>> 
>>>>>> Qihua
>>>>>> 
>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>> 
>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Daniel 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>> 
>>>>>>>> Dear Maker Develop Team,
>>>>>>>> 
>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>> 
>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>> 
>>>>>>>> Thank you so much
>>>>>>>> Qihua
>>>>>>>> _______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> maker-devel mailing list
>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>> 
>>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180201/536a6cc7/attachment-0002.html>

From carsonhh at gmail.com  Sun Feb  4 09:40:14 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Sun, 4 Feb 2018 17:40:14 +0100
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
	<0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
Message-ID: <A61B15A3-5713-40C0-AC1A-0BCA7699CD98@gmail.com>

It?s saying the GeneMark output file does not have the right number of columns, so MAKER can?t use it. If there is a column mismatch, this likely means you are using a different version of GeneMark that MAKER does not support.

Make sure you are using GeneMark-ES and not GeneMark-S or GeneMark.HMM (yes there are lots of versions of GeneMark)

Found here ?> http://exon.gatech.edu/GeneMark/gmes_instructions.html <http://exon.gatech.edu/GeneMark/gmes_instructions.html>

--Carson


> On Feb 1, 2018, at 7:32 PM, Qihua Liang <qlian003 at ucr.edu> wrote:
> 
> Hi Carson,
> 
> I re-installed Maker and I ran Maker again with the same setting.
> 
> This time I still could not get any results. But I have different error messages, saying:
> doing blastx repeats
> doing blastx repeats
> doing blastx repeats
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> doing blastx repeats
> doing blastx repeats
> gathering ab-init output files
> FATAL: Can not identify the version of GeneMark used for the report
> 
> --> rank=NA, hostname=H4
> ERROR: Failed while gathering ab-init output files
> ERROR: Chunk failed at level:1, tier_type:2
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> ERROR: Chunk failed at level:4, tier_type:0
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> examining contents of the fasta file and run log
> #---------------------------------------------------- 
> 
> 
> It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  
> 
> Could you provide some trouble shooting information regarding this?
> 
> Thank you
> Qihua
> 
>> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>> 
>> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
>> 
>> ?Carson
>> 
>> 
>>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>> Hi Carson,
>>> 
>>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>>> 
>>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>>> 
>>> Thank you!
>>> Qihua
>>> 
>>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>> 
>>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>> 
>>>>> Hi Carson,
>>>>> 
>>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>>> 
>>>>> Thanks
>>>>> Qihua
>>>>> 
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> preparing masked sequence
>>>>> preparing ab-inits
>>>>> running  snap.
>>>>> #--------- command -------------#
>>>>> Widget::snap:
>>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>>> mm.snap
>>>>> #-------------------------------#
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> running  augustus.
>>>>> #--------- command -------------#
>>>>> Widget::augustus:
>>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> running  blast search.
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>>> #-------------------------------#
>>>>> doing blastx repeats
>>>>> re reading blast report.
>>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>>> No such file or directory
>>>>> 
>>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Failed while builing masking tiers
>>>>> --> rank=NA, hostname=H4
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Can not get next level
>>>>> running  genemark.
>>>>> #--------- command -------------#
>>>>> Widget::genemark:
>>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>>> #-------------------------------#
>>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>>> 
>>>>> examining contents of the fasta file and run log
>>>>> 
>>>>> 
>>>>> 
>>>>> --Next Contig--
>>>>> 
>>>>> #---------------------------------------------------------------------
>>>>> Now starting the contig!!
>>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>>> Length: 1247
>>>>> #---------------------------------------------------------------------
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>>> 
>>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>>> 
>>>>>> ?Carson
>>>>>> 
>>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>> 
>>>>>>> Hi Ence,
>>>>>>> 
>>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>>> 
>>>>>>> Is this what you may need?
>>>>>>> 
>>>>>>> Qihua
>>>>>>> 
>>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>>> 
>>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Daniel 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>>> 
>>>>>>>>> Dear Maker Develop Team,
>>>>>>>>> 
>>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>>> 
>>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>>> 
>>>>>>>>> Thank you so much
>>>>>>>>> Qihua
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> maker-devel mailing list
>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>>> 
>>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180204/2e9d58c9/attachment-0002.html>

From admin at genome.arizona.edu  Thu Feb  8 09:53:13 2018
From: admin at genome.arizona.edu (Chandler)
Date: Thu, 8 Feb 2018 09:53:13 -0700
Subject: [maker-devel] MPI selection
In-Reply-To: <F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
Message-ID: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>

Thanks Carson, one other question though.

When you mentioned to keep jobs under 200 CPU cores, did you mean as in 
what is emulated to the OS, or physical hardware?  We are using 
hyperthreading which emulates more CPUs to the OS, and so we have about 
256 emulated CPUs available.  Some of our applications are better 
optimized for this, so I keep it that way.

So suppose we keep hyperthreading enabled, should we specify in the 
machine list file that mpiexec uses to only use 128 of the emulated 
cores?  We have noticed with using all 256 hyperthreaded cores that the 
load can get high, although everything still works great.

Thanks


Chandler / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu


Carson Holt wrote on 01/30/2018 10:37 AM:
> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
> 
> ?Carson
> 
> 
> 
>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>
>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>
>>
>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>
>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> .
> 



From carsonhh at gmail.com  Thu Feb  8 10:10:04 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 8 Feb 2018 18:10:04 +0100
Subject: [maker-devel] MPI selection
In-Reply-To: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
	<b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
Message-ID: <237BC5A6-9008-48EC-86F2-6A55CBEEC214@gmail.com>

Yes, you can oversubscribe and match the hyperthread count (you will need ~1GB of RAM per process though). But you should still keep the overall number of MPI processes given to mpiexec below ~200. This is because, the way I structured the MPI controller in MAKER is that I have one manager process and all the other processes are workers. So if there are too many workers, they can overwhelm the manager with communications, and scaling efficiency drops. So starting another MAKER job, launches a new manager with his own workers. In that way, you get around the communication bottleneck by launching multiple large MAKER jobs and scaling efficiency returns to near linear. You can run with multiple jobs until you start hitting IO bottlenecks (MAKER will perform high IOPS but not necessarily high bandwidth).

?Carson




> On Feb 8, 2018, at 5:53 PM, Chandler <admin at genome.arizona.edu> wrote:
> 
> Thanks Carson, one other question though.
> 
> When you mentioned to keep jobs under 200 CPU cores, did you mean as in what is emulated to the OS, or physical hardware?  We are using hyperthreading which emulates more CPUs to the OS, and so we have about 256 emulated CPUs available.  Some of our applications are better optimized for this, so I keep it that way.
> 
> So suppose we keep hyperthreading enabled, should we specify in the machine list file that mpiexec uses to only use 128 of the emulated cores?  We have noticed with using all 256 hyperthreaded cores that the load can get high, although everything still works great.
> 
> Thanks
> 
> 
> Chandler / Systems Administrator
> Arizona Genomics Institute
> www.genome.arizona.edu
> 
> 
> Carson Holt wrote on 01/30/2018 10:37 AM:
>> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
>> ?Carson
>>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>> 
>>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>> 
>>> 
>>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>> 
>>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> .
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From vsoza at uw.edu  Mon Feb 12 12:41:34 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Mon, 12 Feb 2018 11:41:34 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but no
 errors in screen output
Message-ID: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>

Hi all

I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.

I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.

$ grep FAILED Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED

$ grep RETRY Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY

When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:

#---------------------------------------------------------------------
Now starting the contig!!
SeqID: LG12_ordered_scaffold_101
Length: 89169
#---------------------------------------------------------------------


setting up GFF3 output and fasta chunks
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
#???????????????#

It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 

Thanks.

-Valerie
	
Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From skmccormick19 at gmail.com  Sat Feb 17 10:24:40 2018
From: skmccormick19 at gmail.com (Kevin McCormick)
Date: Sat, 17 Feb 2018 12:24:40 -0500
Subject: [maker-devel] Need Help with maker2jbrowse
Message-ID: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>

Hello,

I have hit a wall attempting to get my MAKER data converted for jbrowse use

I have attached a file showing the problem that is currently leaving me
scratching my head.

Could you please give me some advice, or direct me to someone who may be
able to help.

Thank you

-- 
S. Kevin McCormick

*PhD Candidate*
*Integrative Biology Department - Holekamp Lab*
*Ecology, Evolutionary Biology, and Behavior Program*
*Michigan State University*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf
Type: application/rtf
Size: 9238 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment-0002.rtf>

From carsonhh at gmail.com  Tue Feb 20 09:23:05 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:05 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
Message-ID: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>

Hi Valerie,

Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err

Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log

You may get something like this:

unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED

If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.

Thanks,
Carson



> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi all
> 
> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
> 
> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
> 
> $ grep FAILED Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> 
> $ grep RETRY Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> 
> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
> 
> #---------------------------------------------------------------------
> Now starting the contig!!
> SeqID: LG12_ordered_scaffold_101
> Length: 89169
> #---------------------------------------------------------------------
> 
> 
> setting up GFF3 output and fasta chunks
> doing repeat masking
> running  repeat masker.
> #--------- command -------------#
> Widget::RepeatMasker:
> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
> #???????????????#
> 
> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
> 
> Thanks.
> 
> -Valerie
> 	
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Tue Feb 20 09:23:56 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:56 -0700
Subject: [maker-devel] Need Help with maker2jbrowse
In-Reply-To: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
References: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
Message-ID: <ADA97995-86AE-4969-AC1A-B70D1C88F614@gmail.com>

Make sure you ran the setup.sh script that comes with JBrowse. You are missing a perl dependancy, and JBrowse should install it for you when you run the setup script.

?Carson


> On Feb 17, 2018, at 10:24 AM, Kevin McCormick <skmccormick19 at gmail.com> wrote:
> 
> Hello,
> 
> I have hit a wall attempting to get my MAKER data converted for jbrowse use
> 
> I have attached a file showing the problem that is currently leaving me scratching my head.
> 
> Could you please give me some advice, or direct me to someone who may be able to help.
> 
> Thank you
> 
> -- 
> S. Kevin McCormick
> 
> PhD Candidate
> Integrative Biology Department - Holekamp Lab
> Ecology, Evolutionary Biology, and Behavior Program
> Michigan State University
> <MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180220/dbd3a026/attachment-0002.html>

From carsonhh at gmail.com  Tue Feb 20 16:03:11 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 16:03:11 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
	<37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
Message-ID: <E607C645-A8F1-4F27-AE5C-ADE9AFA813E2@gmail.com>

On two, there is a FINISHED entry but it is out of order. There may be a race condition with one process breaking the other?s file lock if you ran multiple jobs at the same time. In which case, one failed and the other finished near the same time. The failure from a broken lock may not say ?ERROR? in the STDERR, but may have another tag. search for ?unclustered_scaffold_3148? in the STDERR, then look at the entries above and below it.

?Carson


> On Feb 20, 2018, at 3:58 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi Carson
> 
> No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:
> 
> $ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED
> 
> $ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED
> 
> $ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> 
> $ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED
> 
> Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.
> 
> -Valerie
> 
> 
>> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>> Hi Valerie,
>> 
>> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
>> 
>> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
>> 
>> You may get something like this:
>> 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
>> 
>> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>>> 
>>> Hi all
>>> 
>>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>>> 
>>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>>> 
>>> $ grep FAILED Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>>> 
>>> $ grep RETRY Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>>> 
>>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>>> 
>>> #---------------------------------------------------------------------
>>> Now starting the contig!!
>>> SeqID: LG12_ordered_scaffold_101
>>> Length: 89169
>>> #---------------------------------------------------------------------
>>> 
>>> 
>>> setting up GFF3 output and fasta chunks
>>> doing repeat masking
>>> running  repeat masker.
>>> #--------- command -------------#
>>> Widget::RepeatMasker:
>>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>>> #???????????????#
>>> 
>>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>>> 
>>> Thanks.
>>> 
>>> -Valerie
>>> 	
>>> Valerie Soza, Ph.D.
>>> c/o Hall Lab
>>> Department of Biology
>>> University of Washington
>>> Johnson Hall 202A
>>> Box 351800
>>> Seattle, WA 98195-1800
>>> 206-543-6740
>>> http://staff.washington.edu/vsoza/
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
> 
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 




From vsoza at uw.edu  Tue Feb 20 15:58:20 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Tue, 20 Feb 2018 14:58:20 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
Message-ID: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>

Hi Carson

No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:

$ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED

$ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED

$ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED

$ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED

Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.

-Valerie


> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> Hi Valerie,
> 
> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
> 
> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
> 
> You may get something like this:
> 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> 
> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
> 
> Thanks,
> Carson
> 
> 
> 
>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>> 
>> Hi all
>> 
>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>> 
>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>> 
>> $ grep FAILED Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>> 
>> $ grep RETRY Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>> 
>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>> 
>> #---------------------------------------------------------------------
>> Now starting the contig!!
>> SeqID: LG12_ordered_scaffold_101
>> Length: 89169
>> #---------------------------------------------------------------------
>> 
>> 
>> setting up GFF3 output and fasta chunks
>> doing repeat masking
>> running  repeat masker.
>> #--------- command -------------#
>> Widget::RepeatMasker:
>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>> #???????????????#
>> 
>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>> 
>> Thanks.
>> 
>> -Valerie
>> 	
>> Valerie Soza, Ph.D.
>> c/o Hall Lab
>> Department of Biology
>> University of Washington
>> Johnson Hall 202A
>> Box 351800
>> Seattle, WA 98195-1800
>> 206-543-6740
>> http://staff.washington.edu/vsoza/
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 

Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From Emily.Giroux at inspection.gc.ca  Fri Feb 23 07:43:49 2018
From: Emily.Giroux at inspection.gc.ca (Giroux, Emily (CFIA/ACIA))
Date: Fri, 23 Feb 2018 14:43:49 +0000
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
Message-ID: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>

Hi,

After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don't waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.

My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I'm not sure if I can just ignore this...

Thanks,

Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180223/de430c44/attachment-0002.html>

From carsonhh at gmail.com  Mon Feb 26 21:30:19 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Mon, 26 Feb 2018 21:30:19 -0700
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
In-Reply-To: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
References: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
Message-ID: <DD5237FE-8F83-4527-AFF4-8F056FCBC425@gmail.com>

Final gene models will have ?maker' in the source column. All others are not gene models, but rather evidence alignments that are there for reference. Make sure you specified a gene predictor during the maker run, or you may only have alignments and no models. Anything with protein_gff:protein2genome in the source column would fall into the category of evidence alignment.

I do not believe GAG supports tRNA?s. Someone may be able to correct me, but I believe they only support the coding annotations.

?Carson




> On Feb 23, 2018, at 7:43 AM, Giroux, Emily (CFIA/ACIA) <Emily.Giroux at inspection.gc.ca> wrote:
> 
> Hi,
>  
> After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don?t waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.
>  
> My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I?m not sure if I can just ignore this?
>  
> Thanks,
>  
> Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180226/645ffb5f/attachment-0002.html>

From qlian003 at ucr.edu  Thu Feb  1 11:32:20 2018
From: qlian003 at ucr.edu (Qihua Liang)
Date: Thu, 1 Feb 2018 10:32:20 -0800
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
Message-ID: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>

Hi Carson,

I re-installed Maker and I ran Maker again with the same setting.

This time I still could not get any results. But I have different error messages, saying:
doing blastx repeats
doing blastx repeats
doing blastx repeats
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
collecting blastx repeatmasking
processing all repeats
preparing masked sequence
preparing ab-inits
doing blastx repeats
doing blastx repeats
gathering ab-init output files
FATAL: Can not identify the version of GeneMark used for the report

--> rank=NA, hostname=H4
ERROR: Failed while gathering ab-init output files
ERROR: Chunk failed at level:1, tier_type:2
FAILED CONTIG:ScsGwly_16;HRSCAF=18

ERROR: Chunk failed at level:4, tier_type:0
FAILED CONTIG:ScsGwly_16;HRSCAF=18

examining contents of the fasta file and run log
#---------------------------------------------------- 


It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  

Could you provide some trouble shooting information regarding this?

Thank you
Qihua

> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
> 
> ?Carson
> 
> 
>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>> 
>> Hi Carson,
>> 
>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>> 
>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>> 
>> Thank you!
>> Qihua
>> 
>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>> 
>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>>> Hi Carson,
>>>> 
>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>> 
>>>> Thanks
>>>> Qihua
>>>> 
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> preparing masked sequence
>>>> preparing ab-inits
>>>> running  snap.
>>>> #--------- command -------------#
>>>> Widget::snap:
>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>> mm.snap
>>>> #-------------------------------#
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>> running  augustus.
>>>> #--------- command -------------#
>>>> Widget::augustus:
>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>> #-------------------------------#
>>>> deleted:0 hits
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> running  blast search.
>>>> #--------- command -------------#
>>>> Widget::blastx:
>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>> #-------------------------------#
>>>> doing blastx repeats
>>>> re reading blast report.
>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>> deleted:0 hits
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> doing blastx repeats
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> in cluster::shadow_cluster...
>>>> ...finished clustering.
>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>> No such file or directory
>>>> 
>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Failed while builing masking tiers
>>>> --> rank=NA, hostname=H4
>>>> --> rank=NA, hostname=H4
>>>> ERROR: Can not get next level
>>>> running  genemark.
>>>> #--------- command -------------#
>>>> Widget::genemark:
>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>> #-------------------------------#
>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>> 
>>>> examining contents of the fasta file and run log
>>>> 
>>>> 
>>>> 
>>>> --Next Contig--
>>>> 
>>>> #---------------------------------------------------------------------
>>>> Now starting the contig!!
>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>> Length: 1247
>>>> #---------------------------------------------------------------------
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>> 
>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>> 
>>>>> ?Carson
>>>>> 
>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>> 
>>>>>> Hi Ence,
>>>>>> 
>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>> 
>>>>>> Is this what you may need?
>>>>>> 
>>>>>> Qihua
>>>>>> 
>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>> 
>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Daniel 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>> 
>>>>>>>> Dear Maker Develop Team,
>>>>>>>> 
>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>> 
>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>> 
>>>>>>>> Thank you so much
>>>>>>>> Qihua
>>>>>>>> _______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> maker-devel mailing list
>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>> 
>>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180201/536a6cc7/attachment-0003.html>

From carsonhh at gmail.com  Sun Feb  4 09:40:14 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Sun, 4 Feb 2018 17:40:14 +0100
Subject: [maker-devel] questions on master_datastore_index.log file
In-Reply-To: <0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
References: <EEF26992-985D-4E3A-BE9D-1C83EB66A538@ucr.edu>
	<C495B996-34AF-4D0D-A55F-D4296FE19ED2@mail.ufl.edu>
	<BA535508-3C42-44C8-976C-814F11E4187A@ucr.edu>
	<C428C432-2313-4A44-8EB6-3B7BFCF38860@gmail.com>
	<0BECB285-BB11-4F46-B6D7-072640F311B2@ucr.edu>
	<0E5E8721-E814-4BA5-891B-B1C312BC0D4A@gmail.com>
	<87A06F3B-82C1-4B21-906E-69DC1308DEC6@ucr.edu>
	<36B45AA1-3D02-4E83-9EF8-85D56C4D3020@gmail.com>
	<0FBF82B1-222E-4E93-8E1F-92E1D2032DCE@ucr.edu>
Message-ID: <A61B15A3-5713-40C0-AC1A-0BCA7699CD98@gmail.com>

It?s saying the GeneMark output file does not have the right number of columns, so MAKER can?t use it. If there is a column mismatch, this likely means you are using a different version of GeneMark that MAKER does not support.

Make sure you are using GeneMark-ES and not GeneMark-S or GeneMark.HMM (yes there are lots of versions of GeneMark)

Found here ?> http://exon.gatech.edu/GeneMark/gmes_instructions.html <http://exon.gatech.edu/GeneMark/gmes_instructions.html>

--Carson


> On Feb 1, 2018, at 7:32 PM, Qihua Liang <qlian003 at ucr.edu> wrote:
> 
> Hi Carson,
> 
> I re-installed Maker and I ran Maker again with the same setting.
> 
> This time I still could not get any results. But I have different error messages, saying:
> doing blastx repeats
> doing blastx repeats
> doing blastx repeats
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> collecting blastx repeatmasking
> processing all repeats
> preparing masked sequence
> preparing ab-inits
> doing blastx repeats
> doing blastx repeats
> gathering ab-init output files
> FATAL: Can not identify the version of GeneMark used for the report
> 
> --> rank=NA, hostname=H4
> ERROR: Failed while gathering ab-init output files
> ERROR: Chunk failed at level:1, tier_type:2
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> ERROR: Chunk failed at level:4, tier_type:0
> FAILED CONTIG:ScsGwly_16;HRSCAF=18
> 
> examining contents of the fasta file and run log
> #---------------------------------------------------- 
> 
> 
> It said something related to ?GeneMark?, but I was able to run GeneMark by the path in maker_exe.ctl  
> 
> Could you provide some trouble shooting information regarding this?
> 
> Thank you
> Qihua
> 
>> On Jan 10, 2018, at 11:05 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>> 
>> The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn?t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
>> 
>> ?Carson
>> 
>> 
>>> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>> 
>>> Hi Carson,
>>> 
>>> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>>> 
>>> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>>> 
>>> Thank you!
>>> Qihua
>>> 
>>>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>> 
>>>> Your contig names may create issues. Specifically the ?;? character, but you should also remove the ?=? character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>> 
>>>>> Hi Carson,
>>>>> 
>>>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq?, the seq name of ?ScsGwly? is ">ScsGwly_6124;HRSCAF=6247?, is it because of the seq naming that makes the temp file name weird?
>>>>> 
>>>>> Thanks
>>>>> Qihua
>>>>> 
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes 
>>>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> preparing masked sequence
>>>>> preparing ab-inits
>>>>> running  snap.
>>>>> #--------- command -------------#
>>>>> Widget::snap:
>>>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm 
>>>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>>>> mm.snap
>>>>> #-------------------------------#
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>>>> running  augustus.
>>>>> #--------- command -------------#
>>>>> Widget::augustus:
>>>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>>>> #-------------------------------#
>>>>> deleted:0 hits
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> running  blast search.
>>>>> #--------- command -------------#
>>>>> Widget::blastx:
>>>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>>>> #-------------------------------#
>>>>> doing blastx repeats
>>>>> re reading blast report.
>>>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>>>> deleted:0 hits
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> doing blastx repeats
>>>>> collecting blastx repeatmasking
>>>>> processing all repeats
>>>>> in cluster::shadow_cluster...
>>>>> ...finished clustering.
>>>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>>>> No such file or directory
>>>>> 
>>>>>  at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>>>>         Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>>>>         Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>         Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>>>>         Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Failed while builing masking tiers
>>>>> --> rank=NA, hostname=H4
>>>>> --> rank=NA, hostname=H4
>>>>> ERROR: Can not get next level
>>>>> running  genemark.
>>>>> #--------- command -------------#
>>>>> Widget::genemark:
>>>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>>>> #-------------------------------#
>>>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>>> 
>>>>> examining contents of the fasta file and run log
>>>>> 
>>>>> 
>>>>> 
>>>>> --Next Contig--
>>>>> 
>>>>> #---------------------------------------------------------------------
>>>>> Now starting the contig!!
>>>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>>>> Length: 1247
>>>>> #---------------------------------------------------------------------
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>>> 
>>>>>> That?s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>>> 
>>>>>> ?Carson
>>>>>> 
>>>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>> 
>>>>>>> Hi Ence,
>>>>>>> 
>>>>>>> When I searched for ?E/error? in the output file, here is what first showed up:
>>>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>>>>         Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>>>>         Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>>>>         Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>>>>         eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>>>>         Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>>>>         Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>>> 
>>>>>>> Is this what you may need?
>>>>>>> 
>>>>>>> Qihua
>>>>>>> 
>>>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>>> 
>>>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the ?maker is finished? message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list. 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Daniel 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>>> 
>>>>>>>>> Dear Maker Develop Team,
>>>>>>>>> 
>>>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>>> 
>>>>>>>>> I saw the message of "Maker is now finished!!!? but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of ?failed?s and ?retry?s and ?failed? again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>>> 
>>>>>>>>> Thank you so much
>>>>>>>>> Qihua
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=> 
>>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> maker-devel mailing list
>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>>> 
>>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180204/2e9d58c9/attachment-0003.html>

From admin at genome.arizona.edu  Thu Feb  8 09:53:13 2018
From: admin at genome.arizona.edu (Chandler)
Date: Thu, 8 Feb 2018 09:53:13 -0700
Subject: [maker-devel] MPI selection
In-Reply-To: <F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
Message-ID: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>

Thanks Carson, one other question though.

When you mentioned to keep jobs under 200 CPU cores, did you mean as in 
what is emulated to the OS, or physical hardware?  We are using 
hyperthreading which emulates more CPUs to the OS, and so we have about 
256 emulated CPUs available.  Some of our applications are better 
optimized for this, so I keep it that way.

So suppose we keep hyperthreading enabled, should we specify in the 
machine list file that mpiexec uses to only use 128 of the emulated 
cores?  We have noticed with using all 256 hyperthreaded cores that the 
load can get high, although everything still works great.

Thanks


Chandler / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu


Carson Holt wrote on 01/30/2018 10:37 AM:
> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
> 
> ?Carson
> 
> 
> 
>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>
>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>
>>
>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>
>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> .
> 



From carsonhh at gmail.com  Thu Feb  8 10:10:04 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 8 Feb 2018 18:10:04 +0100
Subject: [maker-devel] MPI selection
In-Reply-To: <b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
References: <77cfb864-4de1-a9af-aeea-9d3e7cf45ce5@genome.arizona.edu>
	<34C36A98-A87F-4B28-8E05-FCD412CFEBEA@gmail.com>
	<4825e452-aab6-aa13-ebc7-3d3d1832cc60@genome.arizona.edu>
	<F6EB5790-01B4-4C83-B395-1F8AD6C632FA@gmail.com>
	<b4cc2d5e-fe0d-0e34-6940-4614acdf373c@genome.arizona.edu>
Message-ID: <237BC5A6-9008-48EC-86F2-6A55CBEEC214@gmail.com>

Yes, you can oversubscribe and match the hyperthread count (you will need ~1GB of RAM per process though). But you should still keep the overall number of MPI processes given to mpiexec below ~200. This is because, the way I structured the MPI controller in MAKER is that I have one manager process and all the other processes are workers. So if there are too many workers, they can overwhelm the manager with communications, and scaling efficiency drops. So starting another MAKER job, launches a new manager with his own workers. In that way, you get around the communication bottleneck by launching multiple large MAKER jobs and scaling efficiency returns to near linear. You can run with multiple jobs until you start hitting IO bottlenecks (MAKER will perform high IOPS but not necessarily high bandwidth).

?Carson




> On Feb 8, 2018, at 5:53 PM, Chandler <admin at genome.arizona.edu> wrote:
> 
> Thanks Carson, one other question though.
> 
> When you mentioned to keep jobs under 200 CPU cores, did you mean as in what is emulated to the OS, or physical hardware?  We are using hyperthreading which emulates more CPUs to the OS, and so we have about 256 emulated CPUs available.  Some of our applications are better optimized for this, so I keep it that way.
> 
> So suppose we keep hyperthreading enabled, should we specify in the machine list file that mpiexec uses to only use 128 of the emulated cores?  We have noticed with using all 256 hyperthreaded cores that the load can get high, although everything still works great.
> 
> Thanks
> 
> 
> Chandler / Systems Administrator
> Arizona Genomics Institute
> www.genome.arizona.edu
> 
> 
> Carson Holt wrote on 01/30/2018 10:37 AM:
>> MAKER does not really move a lot of data with MPI, it?s just moving around command lines and small variables. So not getting full infiniband performance will not hurt you. I doubt you see any issues using ib0. For MPI flavor, I get the best performance with Intel MPI followed by OpenMPI. Overall you will find that MAKER is IO bound as opposed to CPU or communications bound. So pointing it at your best performing network based storage will be the greatest performance factor (if you have Lustre storage, point it there for example). Pull back on job size and count if other users have issues accessing the disk (too many jobs can bring NFS to it?s knees). The one suggestion I have as far as job size, it to keep jobs sizes under 200 CPU cores. Over that, you will get better performance by splitting up datasets and submitting multiple job. Also MAKER keeps a log of it?s progress, so you can kill jobs or restart failed jobs, and they pick up right where they left off.
>> ?Carson
>>> On Jan 30, 2018, at 10:24 AM, admin at genome.arizona.edu wrote:
>>> 
>>> Carson Holt wrote on 01/30/2018 09:47 AM:
>>>> The libraries used by MVAPICH2, Intel MPI, and OpenMPI to access infiniband have a known bug. For performance reasons, infiniband libraries use registered memory in a way that makes it impossible to do system calls to external programs under MPI (doing so results in seg faults). MAKER has to call out to external programs like BLAST, exonerate, etc., so it triggers this bug.
>>>> The infiniband bug is well known, and unfortunately will not be fixed because fixing it causes infiniband to lose some advertised features like direct memory access.
>>> 
>>> 
>>> Well that stinks!  Maybe that's why we got such a good deal on new-old-stock infiniband equipment!  Still it has allowed us to use full speed of our NFS RAIDs, which has been nice.  I will try with using ib0, the speed is still about 10Gb, but I was under the impression using IPoIB would cause packet loss or other problems...
>>> 
>>> Thanks for clearing that up.  So is there a fabric/protocol you would recommend for clusters running maker?
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> .
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From vsoza at uw.edu  Mon Feb 12 12:41:34 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Mon, 12 Feb 2018 11:41:34 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but no
 errors in screen output
Message-ID: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>

Hi all

I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.

I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.

$ grep FAILED Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED

$ grep RETRY Rwill4_master_datastore_index.log 
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY

When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:

#---------------------------------------------------------------------
Now starting the contig!!
SeqID: LG12_ordered_scaffold_101
Length: 89169
#---------------------------------------------------------------------


setting up GFF3 output and fasta chunks
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
#???????????????#

It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 

Thanks.

-Valerie
	
Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From skmccormick19 at gmail.com  Sat Feb 17 10:24:40 2018
From: skmccormick19 at gmail.com (Kevin McCormick)
Date: Sat, 17 Feb 2018 12:24:40 -0500
Subject: [maker-devel] Need Help with maker2jbrowse
Message-ID: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>

Hello,

I have hit a wall attempting to get my MAKER data converted for jbrowse use

I have attached a file showing the problem that is currently leaving me
scratching my head.

Could you please give me some advice, or direct me to someone who may be
able to help.

Thank you

-- 
S. Kevin McCormick

*PhD Candidate*
*Integrative Biology Department - Holekamp Lab*
*Ecology, Evolutionary Biology, and Behavior Program*
*Michigan State University*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf
Type: application/rtf
Size: 9238 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180217/c5824aa0/attachment-0003.rtf>

From carsonhh at gmail.com  Tue Feb 20 09:23:05 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:05 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
Message-ID: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>

Hi Valerie,

Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err

Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log

You may get something like this:

unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED

If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.

Thanks,
Carson



> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi all
> 
> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
> 
> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
> 
> $ grep FAILED Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> 
> $ grep RETRY Rwill4_master_datastore_index.log 
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> 
> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
> 
> #---------------------------------------------------------------------
> Now starting the contig!!
> SeqID: LG12_ordered_scaffold_101
> Length: 89169
> #---------------------------------------------------------------------
> 
> 
> setting up GFF3 output and fasta chunks
> doing repeat masking
> running  repeat masker.
> #--------- command -------------#
> Widget::RepeatMasker:
> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
> #???????????????#
> 
> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
> 
> Thanks.
> 
> -Valerie
> 	
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Tue Feb 20 09:23:56 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 09:23:56 -0700
Subject: [maker-devel] Need Help with maker2jbrowse
In-Reply-To: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
References: <CAAd6-xerrMtb4_B0Q9iPKO_E9GsNTLsvR8wcEPk01mRLGieHfg@mail.gmail.com>
Message-ID: <ADA97995-86AE-4969-AC1A-B70D1C88F614@gmail.com>

Make sure you ran the setup.sh script that comes with JBrowse. You are missing a perl dependancy, and JBrowse should install it for you when you run the setup script.

?Carson


> On Feb 17, 2018, at 10:24 AM, Kevin McCormick <skmccormick19 at gmail.com> wrote:
> 
> Hello,
> 
> I have hit a wall attempting to get my MAKER data converted for jbrowse use
> 
> I have attached a file showing the problem that is currently leaving me scratching my head.
> 
> Could you please give me some advice, or direct me to someone who may be able to help.
> 
> Thank you
> 
> -- 
> S. Kevin McCormick
> 
> PhD Candidate
> Integrative Biology Department - Holekamp Lab
> Ecology, Evolutionary Biology, and Behavior Program
> Michigan State University
> <MobaXterm_mccor187 at dev-intel14jbrowseJBrowse-1.12.3_20180217_115511.rtf>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180220/dbd3a026/attachment-0003.html>

From carsonhh at gmail.com  Tue Feb 20 16:03:11 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Tue, 20 Feb 2018 16:03:11 -0700
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
	<37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>
Message-ID: <E607C645-A8F1-4F27-AE5C-ADE9AFA813E2@gmail.com>

On two, there is a FINISHED entry but it is out of order. There may be a race condition with one process breaking the other?s file lock if you ran multiple jobs at the same time. In which case, one failed and the other finished near the same time. The failure from a broken lock may not say ?ERROR? in the STDERR, but may have another tag. search for ?unclustered_scaffold_3148? in the STDERR, then look at the entries above and below it.

?Carson


> On Feb 20, 2018, at 3:58 PM, Valerie Soza <vsoza at uw.edu> wrote:
> 
> Hi Carson
> 
> No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:
> 
> $ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED
> 
> $ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED
> 
> $ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> 
> $ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED
> 
> Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.
> 
> -Valerie
> 
> 
>> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>> Hi Valerie,
>> 
>> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
>> 
>> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
>> 
>> You may get something like this:
>> 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
>> 
>> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>>> 
>>> Hi all
>>> 
>>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>>> 
>>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>>> 
>>> $ grep FAILED Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>>> 
>>> $ grep RETRY Rwill4_master_datastore_index.log 
>>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>>> 
>>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>>> 
>>> #---------------------------------------------------------------------
>>> Now starting the contig!!
>>> SeqID: LG12_ordered_scaffold_101
>>> Length: 89169
>>> #---------------------------------------------------------------------
>>> 
>>> 
>>> setting up GFF3 output and fasta chunks
>>> doing repeat masking
>>> running  repeat masker.
>>> #--------- command -------------#
>>> Widget::RepeatMasker:
>>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>>> #???????????????#
>>> 
>>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>>> 
>>> Thanks.
>>> 
>>> -Valerie
>>> 	
>>> Valerie Soza, Ph.D.
>>> c/o Hall Lab
>>> Department of Biology
>>> University of Washington
>>> Johnson Hall 202A
>>> Box 351800
>>> Seattle, WA 98195-1800
>>> 206-543-6740
>>> http://staff.washington.edu/vsoza/
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
> 
> Valerie Soza, Ph.D.
> c/o Hall Lab
> Department of Biology
> University of Washington
> Johnson Hall 202A
> Box 351800
> Seattle, WA 98195-1800
> 206-543-6740
> http://staff.washington.edu/vsoza/
> 




From vsoza at uw.edu  Tue Feb 20 15:58:20 2018
From: vsoza at uw.edu (Valerie Soza)
Date: Tue, 20 Feb 2018 14:58:20 -0800
Subject: [maker-devel] failed contigs in master_datastore_index.log but
 no errors in screen output
In-Reply-To: <0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
References: <B457512E-26FB-4E27-B273-8BF5FC29869B@uw.edu>
	<0F96DD87-5E20-40C6-90C6-F7714C3C8BF3@gmail.com>
Message-ID: <37417ED7-31B8-47E2-AE2C-ABC8DCA50B36@uw.edu>

Hi Carson

No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:

$ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED

$ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED

$ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED

$ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED

Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.

-Valerie


> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> Hi Valerie,
> 
> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
> 
> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep ?unclustered_scaffold_3490? Rwill4_master_datastore_index.log
> 
> You may get something like this:
> 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> 
> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
> 
> Thanks,
> Carson
> 
> 
> 
>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>> 
>> Hi all
>> 
>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>> 
>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>> 
>> $ grep FAILED Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>> 
>> $ grep RETRY Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>> 
>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!? 
>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>> 
>> #---------------------------------------------------------------------
>> Now starting the contig!!
>> SeqID: LG12_ordered_scaffold_101
>> Length: 89169
>> #---------------------------------------------------------------------
>> 
>> 
>> setting up GFF3 output and fasta chunks
>> doing repeat masking
>> running  repeat masker.
>> #--------- command -------------#
>> Widget::RepeatMasker:
>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>> #???????????????#
>> 
>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>> 
>> Thanks.
>> 
>> -Valerie
>> 	
>> Valerie Soza, Ph.D.
>> c/o Hall Lab
>> Department of Biology
>> University of Washington
>> Johnson Hall 202A
>> Box 351800
>> Seattle, WA 98195-1800
>> 206-543-6740
>> http://staff.washington.edu/vsoza/
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 

Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/




From Emily.Giroux at inspection.gc.ca  Fri Feb 23 07:43:49 2018
From: Emily.Giroux at inspection.gc.ca (Giroux, Emily (CFIA/ACIA))
Date: Fri, 23 Feb 2018 14:43:49 +0000
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
Message-ID: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>

Hi,

After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don't waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.

My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I'm not sure if I can just ignore this...

Thanks,

Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180223/de430c44/attachment-0003.html>

From carsonhh at gmail.com  Mon Feb 26 21:30:19 2018
From: carsonhh at gmail.com (Carson Holt)
Date: Mon, 26 Feb 2018 21:30:19 -0700
Subject: [maker-devel] gff3 valiadation with GAG and loss of Name
 information, start-stop codons
In-Reply-To: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
References: <397E3572255740488AA9993F4D41A3B036B5C2EF@ONOTTAXES2.AGR.GC.CA>
Message-ID: <DD5237FE-8F83-4527-AFF4-8F056FCBC425@gmail.com>

Final gene models will have ?maker' in the source column. All others are not gene models, but rather evidence alignments that are there for reference. Make sure you specified a gene predictor during the maker run, or you may only have alignments and no models. Anything with protein_gff:protein2genome in the source column would fall into the category of evidence alignment.

I do not believe GAG supports tRNA?s. Someone may be able to correct me, but I believe they only support the coding annotations.

?Carson




> On Feb 23, 2018, at 7:43 AM, Giroux, Emily (CFIA/ACIA) <Emily.Giroux at inspection.gc.ca> wrote:
> 
> Hi,
>  
> After running my rounds with Maker, I went on to using the Genome Annotation Generator (GAG) and tbl2asn to find errors and fix them. My plan is to run BLAST and InterProScan after so that I don?t waste time annotating invalid features. However, when running GAG, all the protein_gff:protein2genome annotations get removed, along with the Name matches that I got to related species. Somehow I feel it would be better to find the errors, and go back to the last Maker generated gff3 instead of continuing on successively with each GAG-cleaned-up gff. However, another thing is that the start and stop codons are not included in the Maker gff, and there is no way that I can fix this all in the original manually.
>  
> My question is, is it a real loss to lose the protein_gff:protein2genome information, and all the Name info, or does this get added back once I run BLAST and InterProScan? Does Maker have a function to add the Start and Stop codons? Should I really be running BLAST and InterProScan before going on to GAG? Lastly, MAKER adds tRNA annotations, but for each of these, tbl2asn results in errors: SEQ_FEAT.MissingTrnaAA, and I?m not sure if I can just ignore this?
>  
> Thanks,
>  
> Emily

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180226/645ffb5f/attachment-0003.html>