[maker-devel] failed contigs in master_datastore_index.log but no errors in screen output

Tue Feb 20 15:58:20 MST 2018

Hi Carson

No worries. Thanks for your response. I am using qsub on an SGE cluster and get logs of the jobs so I have the entire screen output when I searched for errors and the 4 scaffolds that failed in STDERR. I also did a search for the 4 scaffolds that failed in the master_datastore_index.log and this is what I got:

$ grep LG12_ordered_scaffold_101 Rwill4_master_datastore_index.log
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	STARTED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY
LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FINISHED

$ grep unclustered_scaffold_3148 Rwill4_master_datastore_index.log
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	STARTED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FINISHED
unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED

$ grep unclustered_scaffold_3490 Rwill4_master_datastore_index.log
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED

$ grep unclustered_scaffold_7506 Rwill4_master_datastore_index.log
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	STARTED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FINISHED

Based on this, it seems like 2 of the scaffolds were retried and finished successfully, while the other 2 were retried but failed for some reason. I am now rerunning this with 4 retrys instead of the default of 2, but it is weird that I did not get any errors in the STDERR though.

-Valerie

> On Feb 20, 2018, at 8:23 AM, Carson Holt <carsonhh at gmail.com> wrote:
> 
> Hi Valerie,
> 
> Sorry for the slow reply. If you are running in a screen session, instead try redirecting STDERR to a file so you can capture all errors. Example: maker &> log.err
> 
> Also the datastore_index.log is a cumulative file. Rather than just just grepping for FAILED. grep for the contig of interest. Example: grep “unclustered_scaffold_3490” Rwill4_master_datastore_index.log
> 
> You may get something like this:
> 
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	STARTED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	RETRY
> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FINISHED
> 
> If rather than FINISHED, it shows DIED_SKIPPED_PERMANENT, then increase the maker retry count on the next run (in maker_opts file or command line flag). Then you can see why it fails on the next run by capturing all STDERR to a file.
> 
> Thanks,
> Carson
> 
> 
> 
>> On Feb 12, 2018, at 12:41 PM, Valerie Soza <vsoza at uw.edu> wrote:
>> 
>> Hi all
>> 
>> I ran 3 instances of Maker 2.31.9 on a genome assembly using a SNAP training parameters file and my training parameters from running BUSCO on our genome. The job completed on our departmental computing cluster but when I looked at the master_datastore_index.log, 4 scaffolds had FAILED and 2 were indicated as RETRY.
>> 
>> I previously ran Maker on this same genome just using the SNAP training parameters file and it worked fine so I am perplexed.
>> 
>> $ grep FAILED Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	FAILED 
>> unclustered_scaffold_3148	Rwill4_datastore/62/82/unclustered_scaffold_3148/	FAILED 
>> unclustered_scaffold_3490	Rwill4_datastore/1D/F9/unclustered_scaffold_3490/	FAILED 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	FAILED
>> 
>> $ grep RETRY Rwill4_master_datastore_index.log 
>> LG12_ordered_scaffold_101	Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101/	RETRY 
>> unclustered_scaffold_7506	Rwill4_datastore/69/B8/unclustered_scaffold_7506/	RETRY
>> 
>> When I looked at the screen output from all 3 instances, there are no errors when I grep for Error, error, or ERROR.
>> None of the 3 screen outputs indicated that Maker had finished, so I restarted the job and then immediately got that "Maker is now finished!!!” 
>> However when I grep for the 4 failed scaffolds above in the 3 screen outputs, I only get something for 1 of the scaffolds, which also happens to be the last lines in one of the screens' output's:
>> 
>> #---------------------------------------------------------------------
>> Now starting the contig!!
>> SeqID: LG12_ordered_scaffold_101
>> Length: 89169
>> #---------------------------------------------------------------------
>> 
>> 
>> setting up GFF3 output and fasta chunks
>> doing repeat masking
>> running  repeat masker.
>> #--------- command -------------#
>> Widget::RepeatMasker:
>> cd /tmp/935482.1.ravana.q/maker_ul9sWE; /net/gs/vol3/software/modules-sw/RepeatMasker/4.0.7/Linux/RHEL6/x86_64/RepeatMasker /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0/LG12_ordered_scaffold_101.0.all.rb -species all -dir /net/shendure/vol8/projects/R.williamsianum.annotation/Maker_analyses/SNAP_training/Rwill4.maker.output/Rwill4_datastore/B7/F2/LG12_ordered_scaffold_101//theVoid.LG12_ordered_scaffold_101/0 -pa 10
>> #———————————————#
>> 
>> It seems like the run did not finish properly. Am I interpreting this correctly? Does anyone have suggestions on what I should do or how to troubleshoot? 
>> 
>> Thanks.
>> 
>> -Valerie
>> 	
>> Valerie Soza, Ph.D.
>> c/o Hall Lab
>> Department of Biology
>> University of Washington
>> Johnson Hall 202A
>> Box 351800
>> Seattle, WA 98195-1800
>> 206-543-6740
>> http://staff.washington.edu/vsoza/
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 

Valerie Soza, Ph.D.
c/o Hall Lab
Department of Biology
University of Washington
Johnson Hall 202A
Box 351800
Seattle, WA 98195-1800
206-543-6740
http://staff.washington.edu/vsoza/