From Sean.Li at csiro.au  Wed Aug  1 20:21:08 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 11:21:08 +1000
Subject: [maker-devel] The est_gff option
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120802/50cfa71d/attachment.html>

From carsonhh at gmail.com  Wed Aug  1 20:40:09 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 21:40:09 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F4EB7.10283%carsonhh@gmail.com>

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

Thanks,
Carson



From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 9:21 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] The est_gff option

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120801/ba79d284/attachment.html>

From Sean.Li at csiro.au  Wed Aug  1 21:45:10 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 12:45:10 +1000
Subject: [maker-devel] The est_gff option
In-Reply-To: <CC3F4EB7.10283%carsonhh@gmail.com>
References: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
	<CC3F4EB7.10283%carsonhh@gmail.com>
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>

Dear Carson,

Thank you for your explanation! Now I understand how the RNA-Seq data's been interpreted in MAKER.  I check the tag est_gff and only found it in a binary db file: the scaffold_56.db, which should be a SQLite file. I guess this is right? Will this file be only generated when est data's been provided or I turned the alt_splice to 1? As the db file didn't exist for my first run of Maker when no EST flags have been switched on.

Best regards,
Sean

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished because they are already aligned (so no run.log entry is generated), they are just popped into an alignment object in memory and integrated directly.  The run.log file is used primarily to track finished calls to external programs (I.e. to avoid partial results files on failure). So since there is nothing to run, there is no run.log entry.  You will just get a message saying reading GFF3 in STDERR. The GFF3 results are combined with any other results produced within maker and are then used to generate hints for gene predictors you run as to the location of introns/exons.  This is during the step where you see SNAP or augustus running again and again.  They also become part of the AED score for selecting from the pool of gene models, and can be used to add UTR to resulting gene models.  If you specify alt_splice=1 they can even be used to infer alternate splice forms.  For reference purposes, they will end up in the maker results with the tag est_gff:Cufflinks.

Thanks,
Carson



From: <Sean.Li at csiro.au<mailto:Sean.Li at csiro.au>>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: [maker-devel] The est_gff option

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



_______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com<mailto:maker-devel at box290.bluehost.com> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120802/d8018277/attachment.html>

From carsonhh at gmail.com  Wed Aug  1 21:50:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 22:50:14 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F629F.102AC%carsonhh@gmail.com>

The file is generated any time you provide GFF3 input.  It is just a quick
way to pull out the features for a region via SQLite.  It's generated at the
start of a MAKER run.

Thanks,
Carson

From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 10:45 PM
To:  Carson Holt <carsonhh at gmail.com>, <maker-devel at yandell-lab.org>
Subject:  RE: [maker-devel] The est_gff option

Dear Carson,
 
Thank you for your explanation! Now I understand how the RNA-Seq data?s been
interpreted in MAKER.  I check the tag est_gff and only found it in a binary
db file: the scaffold_56.db, which should be a SQLite file. I guess this is
right? Will this file be only generated when est data?s been provided or I
turned the alt_splice to 1? As the db file didn?t exist for my first run of
Maker when no EST flags have been switched on.
 
Best regards,
Sean   
 

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option
 

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

 

Thanks,

Carson

 

 

 

From: <Sean.Li at csiro.au>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org>
Subject: [maker-devel] The est_gff option

 

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120801/e466dcaa/attachment.html>

From alangfan at gmail.com  Thu Aug  2 00:36:24 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Thu, 2 Aug 2012 13:36:24 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
Message-ID: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
[alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
#location of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
#location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
#location of probuild executable (required for genemark)

============================================

-- 
      Alang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120802/03155f36/attachment.html>

From carsonhh at gmail.com  Thu Aug  2 07:23:21 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 08:23:21 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
Message-ID: <CC3FE8DD.102BC%carsonhh@gmail.com>

The timing of the fault suggests either a problem with your AnyDBM_File
module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
--debug and send me the output as well as your version of maker?  Also try
updating those 3 modules on your system.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 1:36 AM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
 [alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMas
ker #location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
of probuild executable (required for genemark)

============================================

-- 
      Alang

_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120802/32122c14/attachment.html>

From mike.thon at gmail.com  Thu Aug  2 07:48:31 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Thu, 2 Aug 2012 14:48:31 +0200
Subject: [maker-devel] segmentation fault
Message-ID: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>

I have the same problem as Fan Wen-Lang.  I get a segmentation fault when processing the fasta file.  The strange thing is, when I run maker with the -debug parameter, I don't get the segmentation fault.  I'm using maker 2.26 on ubuntu linux 64 bit.
Mike




From carsonhh at gmail.com  Thu Aug  2 08:49:47 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 09:49:47 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>
Message-ID: <CC3FFDA2.102D5%carsonhh@gmail.com>

Odd.  Have you tried updating the 3 modules I mentioned.

Thanks,
Carson



On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>processing the fasta file.  The strange thing is, when I run maker with
>the -debug parameter, I don't get the segmentation fault.  I'm using
>maker 2.26 on ubuntu linux 64 bit.
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From felix.bemm at uni-wuerzburg.de  Thu Aug  2 10:56:55 2012
From: felix.bemm at uni-wuerzburg.de (Felix Bemm)
Date: Thu, 02 Aug 2012 17:56:55 +0200
Subject: [maker-devel] segmentation fault
In-Reply-To: <CC3FFDA2.102D5%carsonhh@gmail.com>
References: <CC3FFDA2.102D5%carsonhh@gmail.com>
Message-ID: <501AA347.2080304@uni-wuerzburg.de>

This snippet will fix your problem temporarily:

cd maker/lib
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)

But do it with caution, I still do not know what problems are caused by 
this fix.

Best regards
Felix

On 02.08.2012 15:49, Carson Holt wrote:
> Odd.  Have you tried updating the 3 modules I mentioned.
>
> Thanks,
> Carson
>
>
>
> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>> processing the fasta file.  The strange thing is, when I run maker with
>> the -debug parameter, I don't get the segmentation fault.  I'm using
>> maker 2.26 on ubuntu linux 64 bit.
>> Mike
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



From carsonhh at gmail.com  Thu Aug  2 11:31:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 12:31:37 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <501AA347.2080304@uni-wuerzburg.de>
Message-ID: <CC40233B.102F6%carsonhh@gmail.com>

Thanks Felix.

You may also have to run the fix below on wherever Bioperl is installed as
well, because it contains the same (DB_File GDBM_File NDBM_File SDBM_File)
statement in the Bio::DB::Fasta module in one of the BEGIN statements.

Thanks,
Carson



On 12-08-02 11:56 AM, "Felix Bemm" <felix.bemm at uni-wuerzburg.de> wrote:

>This snippet will fix your problem temporarily:
>
>cd maker/lib
>sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>
>But do it with caution, I still do not know what problems are caused by
>this fix.
>
>Best regards
>Felix
>
>On 02.08.2012 15:49, Carson Holt wrote:
>> Odd.  Have you tried updating the 3 modules I mentioned.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault
>>>when
>>> processing the fasta file.  The strange thing is, when I run maker with
>>> the -debug parameter, I don't get the segmentation fault.  I'm using
>>> maker 2.26 on ubuntu linux 64 bit.
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From kippjohnson at uchicago.edu  Thu Aug  2 14:12:15 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Thu, 2 Aug 2012 14:12:15 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
Message-ID: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The
first time I ran maker, I ignored these and the maker run seemingly
proceeded to the process the contigs correctly. Is this something I should
be worried about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for
each restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120802/70c4c837/attachment.html>

From alangfan at gmail.com  Thu Aug  2 19:45:17 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Fri, 3 Aug 2012 08:45:17 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CC3FE8DD.102BC%carsonhh@gmail.com>
References: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
	<CC3FE8DD.102BC%carsonhh@gmail.com>
Message-ID: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>

Thanks Carson.

I'm using a single server with centOS6 64 bit.
==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep
-l 'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>

> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
>
> Thanks,
> Carson
>
>
>
> From: "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date: Thursday, 2 August, 2012 1:36 AM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
>
> Dear All,
>
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
>
> Best,
>
> -------command and error message-------------------------------------
> [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
>
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
>
> #-----Protein Homology Evidence
> #protein=protein.fasta
>
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
>
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
> of NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
> of NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
> #location of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
> #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
>
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
> of eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
>
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
> #location of probuild executable (required for genemark)
>
> ============================================
>
> --
>       Alang
>
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120803/36149b1d/attachment.html>

From carsonhh at gmail.com  Thu Aug  2 21:07:53 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 22:07:53 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>
Message-ID: <CC40AA6F.10327%carsonhh@gmail.com>

Ru with the ?f flag before rerunning in the same directory as your previous
error.  Also collect the STDERR both with and without the -debug flag and
send it to me.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 8:45 PM
To:  Carson Holt <carsonhh at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] Question: maker Segmentation fault (core dumped)

Thanks Carson.

I'm using a single server with centOS6 64 bit.
============================================================================
==
STATUS MAKER 2.26
============================================================================
==
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------
------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep -l
'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>
> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
> 
> Thanks,
> Carson
> 
> 
> 
> From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date:  Thursday, 2 August, 2012 1:36 AM
> To:  <maker-devel at yandell-lab.org>
> Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)
> 
> Dear All,
> 
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
> 
> Best,
> 
> -------command and error message-------------------------------------
>  [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
> 
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
> 
> #-----Protein Homology Evidence
> #protein=protein.fasta
> 
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
> 
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location of
> NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location of
> NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
> of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMaske
> r #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
> 
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location of
> eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
> 
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
> of probuild executable (required for genemark)
> 
> ============================================
> 
> -- 
>       Alang
> 
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak
> er-devel_yandell-lab.org



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------
------------------------------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120802/9a29692b/attachment.html>

From mike.thon at gmail.com  Thu Aug  2 23:30:15 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 06:30:15 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been started
	in the same directory.
Message-ID: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>

I get this warning when running the mpi version of maker.  Should I be concerned?  the command I use is this:

 mpiexec -n 8 maker

-Mike




From carsonhh at gmail.com  Fri Aug  3 00:11:39 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 01:11:39 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>
Message-ID: <CC40CC6D.10338%carsonhh@gmail.com>

I'm sorry I'm confused.  Which warning? the segmentation fault you
mentioned before?  For now I would say it's best to first try maker once
outside of MPI to verify that the segmentation fault no longer happens
before moving onto the MPI environment.

Make sure to run with the -f flag when retrying after the fix suggested by
Felix, so maker completely restarts from scratch.  That way you can see
that the fasta database is being generated from start to finish (and is
not just being reread from the successful run under the --debug flag).

Make sure that the fix Felix sent is also applied to the Bioperl
Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
fault will not go away otherwise (if it is infact one of perl's DB modules
causing the issue, which it likely is).  Your error sounds similar to an
error Felix and Thomas encountered some time ago
(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
h/000941.html).  Basically one of perl's native DB modules or the database
the module relies on are broken, so the command line snippet is cutting
any calls to those other databases out of maker and Bioperl (which should
be ok if you have at minimum a working Berkley DB and DB_File module on
your system).

Use this command line to get the correct directory for applying the
changes to Bioperl as well -->
perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
s/[^\/]+$//; print "$dir\n"'

Then copy the directory location-->

cd <to_that_directory>
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)



Finally run maker with the -f flag.  If it works let me know, and then you
can give MPI a shot.  Make said yes to the "compile for MPI question"
asked during the maker install process or mpiexec will fail when running
maker.  If it doesn?t work without MPI or if it works and then MPI fails,
make sure to collect the STDERR in a text file and send it to me.  I will
probably have you run it with the --debug flag as well following any
initial failure (just to get the extra status messages about your system
configuration).

Thanks,
Carson



On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I get this warning when running the mpi version of maker.  Should I be
>concerned?  the command I use is this:
>
> mpiexec -n 8 maker
>
>-Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From mike.thon at gmail.com  Fri Aug  3 02:56:34 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 09:56:34 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>

I meant the warning that is in the subject line of my first email:

WARNING: Multiple MAKER processes have been started in the same directory.

I get that warning when I run mpi maker.  For now, I'm running it without mpi.  When the run finishes I can try it again with mpi and make sure I get the same output.

For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.

best
Mike

On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From carsonhh at gmail.com  Fri Aug  3 07:37:24 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 08:37:24 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC413D27.1038C%carsonhh@gmail.com>

That's good to know the seg fault is gone.  Sorry for the confusion, but I
don't seem to have received your other e-mail about the "multiple process
warning".

Could you run '/maker/src/Build status' and send me the output?

Thanks,
Carson

On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 08:11:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:11:03 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC414316.103A1%carsonhh@gmail.com>

One more thing.

There are three locations where this exact line of code are called
--> @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
run_log
ds_utility
Bio::Fasta::DB

Because of the way perl works only the last one to run needs to be fixed
to stop the error (because it overrides the other two calls), but that
means if the line is ever removed from the last loaded module or load
order changes, then the error will come back because it is the last one to
run that fixes the issue.  So to be safe you should fix it in all
instances.

Currently ds_utility loads last, but it really doesn't need AnyDBM_File
any more to perform it's function and the call will likely be removed in
future releases, so changing the line in Bio::Fasta::DB should avoid
potential future errors on your system.

Thanks,
Carson



On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 08:27:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:27:37 -0400
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
Message-ID: <CC414867.103D1%carsonhh@gmail.com>

It means that your GFF3 results will not be integrated.  Does it happen if
you start in a different directory?

The lock error is coming from SQLite, and may be the result of your working
directory being mounted on an NFS filesystem which is a known issue with
SQLite.  If you give maker a location for TMP that is not NFS mounted (I.e.
A locally mounted filesystem), then maker will try and move the SQLite
database off of the NFS mount.

You can check the mount status of the current working directory and the
directory you choose fro TMP by using the Linux df command.

df <directory>

Any NFS mount will follow the format of host:/location.

Example:
152.234.123.234:/data/research

Or 

storage.utah.edu:/data/research

If you are not running on an NFS device let me know if running in a
different directory results in the same error.

Thanks,
Carson

From:  Kipp Johnson <kippjohnson at uchicago.edu>
Date:  Thursday, 2 August, 2012 3:12 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The first
time I ran maker, I ignored these and the maker run seemingly proceeded to
the process the contigs correctly. Is this something I should be worried
about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for each
restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120803/5517b922/attachment.html>

From cjfields at illinois.edu  Fri Aug  3 08:51:04 2012
From: cjfields at illinois.edu (Fields, Christopher J)
Date: Fri, 3 Aug 2012 13:51:04 +0000
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>

I just want to add in, if there are any fixes that need to be made to Bioperl let me know.  We're trying to move towards an AnyDBM_File approach, but I think this effort has stalled.

chris

On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
 wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug  3 09:28:44 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 10:28:44 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>
Message-ID: <CC414FC9.1040E%carsonhh@gmail.com>

I think I've finally figured it out, as well as a fix to quash this error
for good.  It not strictly a Bioperl issue.  It's an an issue with
AnyDBM_File as well as it's documentation (as it recommends setting
@AnyDBM_File::ISA in code that uses it but never mentions potential
caveats of multiple modules using it).

Basically if you set @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File
SDBM_File) and then call 'use AnyDBM_File', the AnyDBM_File module will
actually preprocess and trim @AnyDBM_File::ISA using this code -->

my $mod;
for $mod (@ISA) {
	if (eval "require $mod") {
		@ISA = ($mod);  # if we leave @ISA alone, warnings abound
		return 1;
	}
}


However if you load any other module that also sets @AnyDBM_File::ISA that
code doesn?t get called on the next 'use AnyDBM_File'.  So you will loose
the preprocessing and basically mess up @AnyDBM_File::ISA if you ever have
two modules loaded at the same time that use AnyDBM_File.

So loading Bio::DB::Fasta before or after a module that also uses
AnyDBM_File has the potential to result in errors.

Example:
use Bio::DB::Fasta;
print join(',', @AnyDBM_File::ISA );

Result --> DB_File


use Bio::DB::Qual;

use Bio::DB::Fasta;

print join(',', @AnyDBM_File::ISA );


Result --> DB_File,GDBM_File,NDBM_File,SDBM_File


The second module messes up @AnyDBM_File::ISA.  It also happens if you
load them in reverse.
Perhaps changing the BEGIN block in Bio::DB::Fasta, Bio::DB::Qual, and any
other any other code using AnyDBM_File to include one of these alternate
BEGIN blocks.

BEGIN {
	@AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
if(!$INC{'AnyDBM_File.pm'});
}


Or 

BEGIN {
	for my $mod (qw(DB_File GDBM_File NDBM_File SDBM_File)) {
		if (eval "require $mod") {
			@AnyDBM_File::ISA = ($mod);
			last;
		}
	}

}


As far as maker goes, I've actually removed all use of AnyDBM_File from
it's modules now, so it won't be an issue there any more, so Bioperl is
the only one left to fix.

Thanks,
Carson



On 12-08-03 9:51 AM, "Fields, Christopher J" <cjfields at illinois.edu> wrote:

>I just want to add in, if there are any fixes that need to be made to
>Bioperl let me know.  We're trying to move towards an AnyDBM_File
>approach, but I think this effort has stalled.
>
>chris
>
>On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>





From kippjohnson at uchicago.edu  Fri Aug  3 11:16:33 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Fri, 3 Aug 2012 11:16:33 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CC414867.103D1%carsonhh@gmail.com>
References: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
	<CC414867.103D1%carsonhh@gmail.com>
Message-ID: <CAGJuV2XTG0GWGi9Dbuk7xdT=OOJodWhbUORsj9XcZJwdz3PYYw@mail.gmail.com>

It does happen if I start in a different directory. My current working
directory is NFS. I changed the tmp directory in maker_opts from the NFS to
a local partition (tmpfs), and I'm still getting this error. Does the
working directory also have to be locally mounted?

On Fri, Aug 3, 2012 at 8:27 AM, Carson Holt <carsonhh at gmail.com> wrote:

> It means that your GFF3 results will not be integrated.  Does it happen if
> you start in a different directory?
>
> The lock error is coming from SQLite, and may be the result of your
> working directory being mounted on an NFS filesystem which is a known issue
> with SQLite.  If you give maker a location for TMP that is not NFS mounted
> (I.e. A locally mounted filesystem), then maker will try and move the
> SQLite database off of the NFS mount.
>
> You can check the mount status of the current working directory and the
> directory you choose fro TMP by using the Linux df command.
>
> df <directory>
>
> Any NFS mount will follow the format of host:/location.
>
> Example:
> 152.234.123.234:/data/research
>
> Or
>
> storage.utah.edu:/data/research
>
> If you are not running on an NFS device let me know if running in a
> different directory results in the same error.
>
> Thanks,
> Carson
>
> From: Kipp Johnson <kippjohnson at uchicago.edu>
> Date: Thursday, 2 August, 2012 3:12 PM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
>
> Hi,
>
>      I am running Maker version 2.26. I have two questions:
>
> 1)
>
>     When I run maker, I get errors like the following printed to STDOUT:
>
> DBD::SQLite::db selectcol_arrayref failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.
>
>      I reinstalled the indicated module via cpanm and restarted Maker from
> the beginning in a new directory, and I am getting the same error. The
> first time I ran maker, I ignored these and the maker run seemingly
> proceeded to the process the contigs correctly. Is this something I should
> be worried about?
>
>
> 2)
>
>     I am running Maker on a single server with 64 processors. Would it be
> faster to use MPICH2, or to simply restart Maker in the same directory ~16
> times, with the # of CPUs option that gets passed to Blast set to 4 for
> each restart?
>
>     I believe that I have MPICH2 correctly compiled and installed; if it
> would be faster to use MPICH2, could someone provide me with a sample
> command line entry to use for Maker? I am confused about how to run MPICH2
> on a single node (without a hostfile?), and searching the maker-dev-list
> archive seems to give me results for older versions of MAKER and MPICH2.
>
> Thank you,
>
>
> Best,
>
>      Kipp Johnson
>      kippjohnson at uchicago.edu
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120803/b5f2b0cd/attachment.html>

From mike.thon at gmail.com  Sat Aug  4 00:48:52 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 07:48:52 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC413D27.1038C%carsonhh@gmail.com>
References: <CC413D27.1038C%carsonhh@gmail.com>
Message-ID: <E44B4BF9-7D27-4B11-B33F-42A975222F99@gmail.com>

./Build status


==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:	VERIFIED
External Programs:	VERIFIED
External C Libraries:	VERIFIED
MPI SUPPORT:		ENABLED
MWAS Web Interface:	DISABLED
MAKER PACKAGE:		CONFIGURATION OK

...

I was impatient and I ran maker with MPI.  In the master log file I see a lot of failed contigs.  Now I'm running it without MPI to see if those contigs fail again. 

On Aug 3, 2012, at 2:37 PM, Carson Holt <carsonhh at gmail.com> wrote:

> That's good to know the seg fault is gone.  Sorry for the confusion, but I
> don't seem to have received your other e-mail about the "multiple process
> warning".
> 
> Could you run '/maker/src/Build status' and send me the output?
> 
> Thanks,
> Carson
> 
> On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I meant the warning that is in the subject line of my first email:
>> 
>> WARNING: Multiple MAKER processes have been started in the same directory.
>> 
>> I get that warning when I run mpi maker.  For now, I'm running it without
>> mpi.  When the run finishes I can try it again with mpi and make sure I
>> get the same output.
>> 
>> For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>> to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>> 
>> best
>> Mike
>> 
>> On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>>> mentioned before?  For now I would say it's best to first try maker once
>>> outside of MPI to verify that the segmentation fault no longer happens
>>> before moving onto the MPI environment.
>>> 
>>> Make sure to run with the -f flag when retrying after the fix suggested
>>> by
>>> Felix, so maker completely restarts from scratch.  That way you can see
>>> that the fasta database is being generated from start to finish (and is
>>> not just being reread from the successful run under the --debug flag).
>>> 
>>> Make sure that the fix Felix sent is also applied to the Bioperl
>>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>>> fault will not go away otherwise (if it is infact one of perl's DB
>>> modules
>>> causing the issue, which it likely is).  Your error sounds similar to an
>>> error Felix and Thomas encountered some time ago
>>> 
>>> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>> rc
>>> h/000941.html).  Basically one of perl's native DB modules or the
>>> database
>>> the module relies on are broken, so the command line snippet is cutting
>>> any calls to those other databases out of maker and Bioperl (which
>>> should
>>> be ok if you have at minimum a working Berkley DB and DB_File module on
>>> your system).
>>> 
>>> Use this command line to get the correct directory for applying the
>>> changes to Bioperl as well -->
>>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>>> s/[^\/]+$//; print "$dir\n"'
>>> 
>>> Then copy the directory location-->
>>> 
>>> cd <to_that_directory>
>>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>> $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>>> 
>>> 
>>> 
>>> Finally run maker with the -f flag.  If it works let me know, and then
>>> you
>>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>>> asked during the maker install process or mpiexec will fail when running
>>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>> fails,
>>> make sure to collect the STDERR in a text file and send it to me.  I
>>> will
>>> probably have you run it with the --debug flag as well following any
>>> initial failure (just to get the extra status messages about your system
>>> configuration).
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> 
>>> 
>>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>> 
>>>> I get this warning when running the mpi version of maker.  Should I be
>>>> concerned?  the command I use is this:
>>>> 
>>>> mpiexec -n 8 maker
>>>> 
>>>> -Mike
>>>> 
>>>> 
>>>> _______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From mike.thon at gmail.com  Sat Aug  4 06:51:04 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 13:51:04 +0200
Subject: [maker-devel] failed contigs
Message-ID: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>

running maker 2.26 some contigs seem to consistently fail to run, according to XXX_master_datastore_index.log

Here are the last few lines of run.log for one of the failed contigs:

CTL_OPTIONS     run     
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
DIED    RANK    0
DIED    COUNT   1


the contents of run.log.child.1 for the same contig:
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
FINISHED        cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.te_proteins%2Efasta.repeatrunner  
DIED    RANK    0:2:0:1
DIED    COUNT   1

So perhaps its a problem with the repeat masking step.  I already have annotated repeats using RepeatMasker and I have the gff2 file produced by RepeatMasker.  Do I need to convert this to gff3 before adding to to my maker_opts file?  Better yet, does anyone know why these contigs are failing and how to fix it?
Mike




From carsonhh at gmail.com  Sat Aug  4 08:44:25 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Sat, 04 Aug 2012 09:44:25 -0400
Subject: [maker-devel] failed contigs
In-Reply-To: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>
Message-ID: <CC429E5D.104F1%carsonhh@gmail.com>

Do you have STDERR log stored somewhere?  Could you send it to me?
Sometimes the error causing the failure is upstream in the log and then
cascades into other errors.  Also if it is the repeatrunner step failing,
the log will have the command to run the BLAST step outside of MAKER (is
helpful for debugging).  If you don't have the captured STDERR, could you
run again and send it to me.

Thanks,
Carson

On 12-08-04 7:51 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>running maker 2.26 some contigs seem to consistently fail to run,
>according to XXX_master_datastore_index.log
>
>Here are the last few lines of run.log for one of the failed contigs:
>
>CTL_OPTIONS     run
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>DIED    RANK    0
>DIED    COUNT   1
>
>
>the contents of run.log.child.1 for the same contig:
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>FINISHED        
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.te_proteins%2Efasta.repeatrunner
>DIED    RANK    0:2:0:1
>DIED    COUNT   1
>
>So perhaps its a problem with the repeat masking step.  I already have
>annotated repeats using RepeatMasker and I have the gff2 file produced by
>RepeatMasker.  Do I need to convert this to gff3 before adding to to my
>maker_opts file?  Better yet, does anyone know why these contigs are
>failing and how to fix it?
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From barry.moore at genetics.utah.edu  Fri Aug 17 20:05:00 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 17 Aug 2012 19:05:00 -0600
Subject: [maker-devel] Exonerate web access is back
References: <201208160913.q7G9DTbB005179@rabbit.ebi.ac.uk>
Message-ID: <C52C60DC-1052-4FD7-9CAC-2F4830E78827@genetics.utah.edu>



Begin forwarded message:

> From: <datasubs at ebi.ac.uk>
> Subject: SUP/EMBL/ 24308 moved from EBIHELP by apc (rasko) (SUB#818751)
> Date: August 16, 2012 3:13:29 AM MDT
> To: <barry.moore at genetics.utah.edu>
> 
> Dear Barry,
> 
> Access to http://www.ebi.ac.uk/~guy/exonerate/ has been restored.
> 
> Kind Regards,
> Rasko Leinonen 
> European Nucleotide Archive (ENA)
> 

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120817/0a68e03e/attachment.html>

From kippjohnson at uchicago.edu  Tue Aug 21 16:25:49 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Tue, 21 Aug 2012 16:25:49 -0500
Subject: [maker-devel] Maker Re-tries
In-Reply-To: <6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
References: <CC414867.103D1%carsonhh@gmail.com>
	<6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
Message-ID: <1E417039-68CF-4EFE-90BF-964DC86B1864@uchicago.edu>

Hi Carson,

	Simple question: If I start maker and then kill the job before a thread finishes processing a contig, does this failure to complete the contig count in the number of tries maker will try to complete the contig before skipping to another?	

	Basically, does the "tries=2 #number of times to try a contig if there is a failure for some reason" parameter take into account failures resulting from me killing the job?	


	Best,

		Kipp Johnson
		kippjohnson at uchicago.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120821/74551a9f/attachment.html>

From jeremy.semeiks at utsw.edu  Tue Aug 21 11:26:32 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Tue, 21 Aug 2012 11:26:32 -0500
Subject: [maker-devel] maker dying on a contig
Message-ID: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>

Hi,

I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
~36,000 contigs, and per the master log maker FINISHED all of them
except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
run. It didn't report ERROR for that contig in the master log, but it
did output two DIED lines in the contig's run.log and then just hung
for a week until I killed it.

I reran maker on just that contig by deleting the START line from the
master log. maker again died on that contig.

I attach the relevant part of the stderr stream; the rest was just
"--Next Contig--"-type lines. Any ideas?

A related question. Is deleting lines in the master log, as I did
here, the best way to get maker to rerun on just a few contigs? I
often find myself needing to do this, sometimes because of ERROR lines
or other assumed quirks in maker and sometimes just because I needed
to pause a big run for a while. One disadvantage is that when I
restart, maker needs to manually scan over many completed contigs,
which takes a while.

(I'd think the best solution would be for maker to automatically
recognize whether it's really currently running on contigs marked
STARTED but not FINISHED. It doesn't seem to do this now, but I
realize that might be hard to implement.)

Thanks,
Jeremy
Grad student, Grishin lab
UT Southwestern, Dallas TX
510.384.8959
-------------- next part --------------
A non-text attachment was scrubbed...
Name: maker.stderr
Type: application/octet-stream
Size: 11421 bytes
Desc: not available
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120821/7f29efb2/attachment.obj>

From mike.thon at gmail.com  Fri Aug 24 02:44:13 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 24 Aug 2012 09:44:13 +0200
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
Message-ID: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>

I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.


On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:

> Hi,
> 
> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
> ~36,000 contigs, and per the master log maker FINISHED all of them
> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
> run. It didn't report ERROR for that contig in the master log, but it
> did output two DIED lines in the contig's run.log and then just hung
> for a week until I killed it.
> 
> I reran maker on just that contig by deleting the START line from the
> master log. maker again died on that contig.
> 
> I attach the relevant part of the stderr stream; the rest was just
> "--Next Contig--"-type lines. Any ideas?
> 
> A related question. Is deleting lines in the master log, as I did
> here, the best way to get maker to rerun on just a few contigs? I
> often find myself needing to do this, sometimes because of ERROR lines
> or other assumed quirks in maker and sometimes just because I needed
> to pause a big run for a while. One disadvantage is that when I
> restart, maker needs to manually scan over many completed contigs,
> which takes a while.
> 
> (I'd think the best solution would be for maker to automatically
> recognize whether it's really currently running on contigs marked
> STARTED but not FINISHED. It doesn't seem to do this now, but I
> realize that might be hard to implement.)
> 
> Thanks,
> Jeremy
> Grad student, Grishin lab
> UT Southwestern, Dallas TX
> 510.384.8959
> <maker.stderr>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From kmegy at ebi.ac.uk  Fri Aug 24 03:38:32 2012
From: kmegy at ebi.ac.uk (Karyn Megy)
Date: Fri, 24 Aug 2012 09:38:32 +0100
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
	<BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <463EA6A1-58CC-4358-9812-1B659DED9F13@ebi.ac.uk>

Hi Jeremy, 

Could it be a contig with lots of repeats, or that would hit lots of proteins?

One of my colleague had this issue and he ended splitting the few problematic contigs in sections... but then you have to reconciliate the results because the gene/BLAST coordinates are based in the bits of sub-contigs rather than the initial contig.  

You could try running MAKER on half of this contig and see what happen.

Cheers,
Karyn.

On 24 Aug 2012, at 08:44, Michael Thon wrote:

> I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.
> 
> 
> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:
> 
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug 24 07:24:56 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 08:24:56 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <CC5CEA43.11379%carsonhh@gmail.com>

Could you send the blast version informations for reference purposes? I
can see this being caused by BLAST.

Thanks,
Carson



On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I'm not sure if this is relevant to your problem but I had problems with
>maker failing on some contigs but not others.  I reinstalled maker and
>used the Build script to have it install all of its own dependencies.
>Now maker is running fine, although the culprit seems to have been the
>blast version I was using before.
>
>
>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>wrote:
>
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From jeremy.semeiks at utsw.edu  Fri Aug 24 14:13:39 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 14:13:39 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC5CEA43.11379%carsonhh@gmail.com>
References: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
	<CC5CEA43.11379%carsonhh@gmail.com>
Message-ID: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>

Carson,

$ blastx -version
blastx: 2.2.26+
Package: blast 2.2.26, build Feb 22 2012 16:04:28

- J

On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
> Could you send the blast version informations for reference purposes? I
> can see this being caused by BLAST.
>
> Thanks,
> Carson
>
>
>
> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>>I'm not sure if this is relevant to your problem but I had problems with
>>maker failing on some contigs but not others.  I reinstalled maker and
>>used the Build script to have it install all of its own dependencies.
>>Now maker is running fine, although the culprit seems to have been the
>>blast version I was using before.
>>
>>
>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>> run. It didn't report ERROR for that contig in the master log, but it
>>> did output two DIED lines in the contig's run.log and then just hung
>>> for a week until I killed it.
>>>
>>> I reran maker on just that contig by deleting the START line from the
>>> master log. maker again died on that contig.
>>>
>>> I attach the relevant part of the stderr stream; the rest was just
>>> "--Next Contig--"-type lines. Any ideas?
>>>
>>> A related question. Is deleting lines in the master log, as I did
>>> here, the best way to get maker to rerun on just a few contigs? I
>>> often find myself needing to do this, sometimes because of ERROR lines
>>> or other assumed quirks in maker and sometimes just because I needed
>>> to pause a big run for a while. One disadvantage is that when I
>>> restart, maker needs to manually scan over many completed contigs,
>>> which takes a while.
>>>
>>> (I'd think the best solution would be for maker to automatically
>>> recognize whether it's really currently running on contigs marked
>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>> realize that might be hard to implement.)
>>>
>>> Thanks,
>>> Jeremy
>>> Grad student, Grishin lab
>>> UT Southwestern, Dallas TX
>>> 510.384.8959
>>> <maker.stderr>_______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>_______________________________________________
>>maker-devel mailing list
>>maker-devel at box290.bluehost.com
>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>



From carsonhh at gmail.com  Fri Aug 24 14:17:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 15:17:03 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
Message-ID: <CC5D4ADD.11459%carsonhh@gmail.com>

Let maker install it's own blast (a version I know should work).

cd ./maker/src
./Build blast

Then to finish up your job (this command will just update executable
locations in the maker_exe.ctl file) -->
maker -EXE

Then run maker again with a higher retry count.  Let me know if the contig
still fails.

Thanks,
Carson




On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Carson,
>
>$ blastx -version
>blastx: 2.2.26+
>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>
>- J
>
>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send the blast version informations for reference purposes? I
>> can see this being caused by BLAST.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>>I'm not sure if this is relevant to your problem but I had problems with
>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>used the Build script to have it install all of its own dependencies.
>>>Now maker is running fine, although the culprit seems to have been the
>>>blast version I was using before.
>>>
>>>
>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>> did output two DIED lines in the contig's run.log and then just hung
>>>> for a week until I killed it.
>>>>
>>>> I reran maker on just that contig by deleting the START line from the
>>>> master log. maker again died on that contig.
>>>>
>>>> I attach the relevant part of the stderr stream; the rest was just
>>>> "--Next Contig--"-type lines. Any ideas?
>>>>
>>>> A related question. Is deleting lines in the master log, as I did
>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>> or other assumed quirks in maker and sometimes just because I needed
>>>> to pause a big run for a while. One disadvantage is that when I
>>>> restart, maker needs to manually scan over many completed contigs,
>>>> which takes a while.
>>>>
>>>> (I'd think the best solution would be for maker to automatically
>>>> recognize whether it's really currently running on contigs marked
>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>> realize that might be hard to implement.)
>>>>
>>>> Thanks,
>>>> Jeremy
>>>> Grad student, Grishin lab
>>>> UT Southwestern, Dallas TX
>>>> 510.384.8959
>>>> <maker.stderr>_______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> 
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>>>_______________________________________________
>>>maker-devel mailing list
>>>maker-devel at box290.bluehost.com
>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>





From jeremy.semeiks at utsw.edu  Fri Aug 24 16:11:10 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 16:11:10 -0500
Subject: [maker-devel]  maker dying on a contig
In-Reply-To: <CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
References: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
	<CC5D4ADD.11459%carsonhh@gmail.com>
	<CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
Message-ID: <CABPMbe3nSDGzf_VqDj1Nucd0DPuPUB_tKcqQj64sV-2SyuUAQQ@mail.gmail.com>

After doing this, I still get the same errors as I attached before.
There was also nothing in the log supporting that maker tried this
more than once. (Initially, I had tries=2, but I changed it to
tries=10 for this run. I'm guessing that's what you mean by "retry
count".) It also doesn't matter if I keep the same old contig
directory and master log before rerunning, or if I delete them both.

FWIW, maker's own version of blast is 2.2.26+, which is the same as my
native version.

Thanks,
Jeremy

On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Let maker install it's own blast (a version I know should work).
>
> cd ./maker/src
> ./Build blast
>
> Then to finish up your job (this command will just update executable
> locations in the maker_exe.ctl file) -->
> maker -EXE
>
> Then run maker again with a higher retry count.  Let me know if the contig
> still fails.
>
> Thanks,
> Carson
>
>
>
>
> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Carson,
>>
>>$ blastx -version
>>blastx: 2.2.26+
>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>
>>- J
>>
>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send the blast version informations for reference purposes? I
>>> can see this being caused by BLAST.
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>>
>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>
>>>>I'm not sure if this is relevant to your problem but I had problems with
>>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>>used the Build script to have it install all of its own dependencies.
>>>>Now maker is running fine, although the culprit seems to have been the
>>>>blast version I was using before.
>>>>
>>>>
>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>>> did output two DIED lines in the contig's run.log and then just hung
>>>>> for a week until I killed it.
>>>>>
>>>>> I reran maker on just that contig by deleting the START line from the
>>>>> master log. maker again died on that contig.
>>>>>
>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>
>>>>> A related question. Is deleting lines in the master log, as I did
>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>>> or other assumed quirks in maker and sometimes just because I needed
>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>> which takes a while.
>>>>>
>>>>> (I'd think the best solution would be for maker to automatically
>>>>> recognize whether it's really currently running on contigs marked
>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>> realize that might be hard to implement.)
>>>>>
>>>>> Thanks,
>>>>> Jeremy
>>>>> Grad student, Grishin lab
>>>>> UT Southwestern, Dallas TX
>>>>> 510.384.8959
>>>>> <maker.stderr>_______________________________________________
>>>>> maker-devel mailing list
>>>>> maker-devel at box290.bluehost.com
>>>>>
>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>>
>>>>
>>>>_______________________________________________
>>>>maker-devel mailing list
>>>>maker-devel at box290.bluehost.com
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>
>



From jokelley at stanford.edu  Thu Aug 30 19:18:29 2012
From: jokelley at stanford.edu (Joanna Kelley)
Date: Thu, 30 Aug 2012 17:18:29 -0700
Subject: [maker-devel] convert gff to ncbi table format?
Message-ID: <CA+77YqHuS8CUK+YmDCQX8KMt1FWiMkJ00eXsgwdzHTM1PxsZ1g@mail.gmail.com>

Hello,

I am trying to submit the maker annotations to NCBI and
I was wondering if there are any tools to convert gff to
the NCBI table format for submission?

Thank you,
Joanna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120830/9a5ce895/attachment.html>

From chrisi.hahni at gmail.com  Fri Aug 31 07:43:21 2012
From: chrisi.hahni at gmail.com (Christoph Hahn)
Date: Fri, 31 Aug 2012 14:43:21 +0200
Subject: [maker-devel] keep_preds option?
Message-ID: <5040B169.1030909@gmail.com>

Hello maker users and developers,

I am new to gene prediction and I am trying to use maker 2.25 on a newly 
assembled non-model organisms draft genome. Within maker I use genemark, 
SNAP and Augustus. I have a few questions:

1. I was wondering what the keep_preds option means exactly.

I found two slightly different explanations on the option
#Add unsupported gene prediction to final annotation set (maker2.25)
#Add non-overlapping ab-inito gene prediction to final annotation set 
(found on the net - probably older maker version)

As far as I understood keep_preds=0 only retains gene models for which 
the ab initio predictions agree. But how many, all three? two of three?
keep_preds=1 instead keeps all gene models regardless if the different 
programs agree, right?

In my case I get substantial differences in the number of gene models 
found between the two settings, while with =1 I get a number that is 
close to what we would expect. How would you interpret that? What would 
you recommend me to do? Obiously =0 is the saver option.

2. I tried to use EST data of an alternative organism in altest= 
(#EST/cDNA sequence file in fasta format from an alternate organism). 
The organism is quite distantly related, but its the closest I have so I 
thought I d give it a shot. I ran maker twice with identical settigs 
expect in altest and est2genome=0/1. The number of genes predicted is 
identical with both approaches, so I am not sure whether or not the EST 
data was actually used or its just to distantly related. Any easy way to 
assess this?

3. I am running maker in several passes and after each pass I am 
training SNAP using the result of the previous pass. Then for every pass 
I run maker from scratch. Would you recommend to supply the gff of the 
previous pass in "#-----Re-annotation Using MAKER Derived GFF3
maker_gff= #re-annotate genome based on this gff3 file", instead?

Thanks in advance for any thoughts/advice on these things! I appreciate 
your help!

much obliged,
Christoph



From barry.moore at genetics.utah.edu  Fri Aug 31 11:52:10 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 31 Aug 2012 10:52:10 -0600
Subject: [maker-devel] keep_preds option?
In-Reply-To: <5040B169.1030909@gmail.com>
References: <5040B169.1030909@gmail.com>
Message-ID: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly assembled non-model organisms draft genome. Within maker I use genemark, SNAP and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical evidence.  There are two pieces of information that are required for every MAKER annotation (by default).  MAKER runs the ab initio gene predictors and aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to the genome.  For each locus where one or more gene predictions exist MAKER checks to see if there is any physical evidence for gene expression at that locus (RNA/protein sequence alignments) and if there is (splice EST or protein alignments) evidence overlapping the predictions, MAKER decides which prediction best matches the evidence and promotes that prediction to an annotation.  If there is no evidence overlapping any of the predictions then those predictions are not included in the output annotation file (although they are saved).  If you set keep_preds=1 then for each locus where prediction(s) exist maker keeps one and promotes it to an annotation even though there is no physical evidence.  The description of 'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as evidence and thus in the absence of aligned physical evidence MAKER will not output an annotation even if all three ab initio tools predict a gene at that locus.

> In my case I get substantial differences in the number of gene models found between the two settings, while with =1 I get a number that is close to what we would expect. How would you interpret that? What would you recommend me to do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is too few, you could run MAKER with keep_preds=1.  After the run is finished, use a tool like IPRScan to search all MAKER predictions for protein domain content and push that IPRScan output back into the MAKER GFF file with the ipr_update_gff script.  Then as a final step you can run over the GFF file and remove any gene model that doesn't have either physical evidence (AED < 1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA sequence file in fasta format from an alternate organism). The organism is quite distantly related, but its the closest I have so I thought I d give it a shot. I ran maker twice with identical settigs expect in altest and est2genome=0/1. The number of genes predicted is identical with both approaches, so I am not sure whether or not the EST data was actually used or its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in the way of additional support (compared to just using protein evidence from say Swiss-prot) and this would be especially true if your alt_est is distantly related.  I'm not sure I really understand you alt_est/est2genome combo's to comment in more detail.  I could see four possible combinations there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP using the result of the previous pass. Then for every pass I run maker from scratch. Would you recommend to supply the gff of the previous pass in "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a genome when you want certain parts of the previous run to be passed through unchanged, but with retraining SNAP you want MAKER to re-evaluate each annotation in light of the new predictions made by the retrained SNAP.  MAKER should run really fast in all of the runs after the first one because as long as you haven't changed the evidence files it won't have to redo any of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120831/8b0671bd/attachment.html>

From carsonhh at gmail.com  Fri Aug 31 14:03:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 15:03:14 -0400
Subject: [maker-devel] keep_preds option?
In-Reply-To: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>
Message-ID: <CC667EFB.117DD%carsonhh@gmail.com>

I concur with everything Barry said.  Also let me add that alt-ESTs do not
get polished around splice sites (exonerate won't handle them).  However
ESTs and proteins do.  This means that the utility of alt-ESTs in adding
UTR, and splice information is zero.  They just function as an anchor to
maintain gene models that might have otherwise been rejected.  This also
means alt_est=some.fasta  together with est2genome=1 will produce virtually
no additional results because est2genome requires that the splice site makes
sense.  You are better off using proteins with protein2genome=1 if you don?t
have ESTs and want partial models for training.  Once you have a trained ab
initio gene predictor, turn the est2genome and protein2genome options off.
Otherwise they will give weird partial models that decrease the quality of
your final annotations. (partial models are ok for training but that's about
it).

If you are getting too low a gene count with keep_preds=0, then you probably
need to add more evidence.  Try adding all proteins from a couple of related
species (the protein= option accepts comma separated lists of files). If
your genome is a fungi, oomycete, or a prokaryote, then setting keep_preds=1
is usually safe.  Theses are genomes with high gene density and simple gene
structure, so ab initio predictors do really well and don't need as much
help from the evidence.  For other organisms, leave it set to 0 or you will
get a lot of false positives (false positives on some genomes with complex
gene structure can outnumber the genes by 10 to 1 if you turn this on).

Thanks,
Carson




From:  Barry Moore <barry.moore at genetics.utah.edu>
Date:  Friday, 31 August, 2012 12:52 PM
To:  Christoph Hahn <chrisi.hahni at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] keep_preds option?

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly
> assembled non-model organisms draft genome. Within maker I use genemark, SNAP
> and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found
> on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical
evidence.  There are two pieces of information that are required for every
MAKER annotation (by default).  MAKER runs the ab initio gene predictors and
aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to
the genome.  For each locus where one or more gene predictions exist MAKER
checks to see if there is any physical evidence for gene expression at that
locus (RNA/protein sequence alignments) and if there is (splice EST or
protein alignments) evidence overlapping the predictions, MAKER decides
which prediction best matches the evidence and promotes that prediction to
an annotation.  If there is no evidence overlapping any of the predictions
then those predictions are not included in the output annotation file
(although they are saved).  If you set keep_preds=1 then for each locus
where prediction(s) exist maker keeps one and promotes it to an annotation
even though there is no physical evidence.  The description of
'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio
predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab
> initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different
> programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as
evidence and thus in the absence of aligned physical evidence MAKER will not
output an annotation even if all three ab initio tools predict a gene at
that locus.

> In my case I get substantial differences in the number of gene models found
> between the two settings, while with =1 I get a number that is close to what
> we would expect. How would you interpret that? What would you recommend me to
> do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is
too few, you could run MAKER with keep_preds=1.  After the run is finished,
use a tool like IPRScan to search all MAKER predictions for protein domain
content and push that IPRScan output back into the MAKER GFF file with the
ipr_update_gff script.  Then as a final step you can run over the GFF file
and remove any gene model that doesn't have either physical evidence (AED <
1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a
script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA
> sequence file in fasta format from an alternate organism). The organism is
> quite distantly related, but its the closest I have so I thought I d give it a
> shot. I ran maker twice with identical settigs expect in altest and
> est2genome=0/1. The number of genes predicted is identical with both
> approaches, so I am not sure whether or not the EST data was actually used or
> its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in
the way of additional support (compared to just using protein evidence from
say Swiss-prot) and this would be especially true if your alt_est is
distantly related.  I'm not sure I really understand you alt_est/est2genome
combo's to comment in more detail.  I could see four possible combinations
there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP
> using the result of the previous pass. Then for every pass I run maker from
> scratch. Would you recommend to supply the gff of the previous pass in
> "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a
genome when you want certain parts of the previous run to be passed through
unchanged, but with retraining SNAP you want MAKER to re-evaluate each
annotation in light of the new predictions made by the retrained SNAP.
MAKER should run really fast in all of the runs after the first one because
as long as you haven't changed the evidence files it won't have to redo any
of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your
> help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120831/ed670166/attachment.html>

From carsonhh at gmail.com  Fri Aug 31 15:47:33 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 16:47:33 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
Message-ID: <CC66941F.118C8%carsonhh@gmail.com>

Looks like there is a slight format variation in one line of the
RepeatMasker outputs.  Not sure what could have caused it -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          0    109    (0)

The position/compliment information in columns 12-14 give a start position
of 0 for part of the alignment.  A 0 is invalid because repeat masker
doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
asks for the starting position of the alignment in another part of the
code.

This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
wu-blast and then reconfigure RepeatMasker I get this line -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          1    109    (0)

Notice the 0 becomes a 1 and all logic is restored to the world.

>From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
output, then I just turn it into a 1.
I've attached a file that you should use to replace the
./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
I'll also make a note to the RepeatMasker developers.

Thanks,
Carson



On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Attached.
>
>Thanks,
>J
>
>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send me the entire directory for just the failed contig?
>>
>> Thanks,
>> Carson
>>
>>
>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>
>>>After doing this, I still get the same errors as I attached before.
>>>There was also nothing in the log supporting that maker tried this
>>>more than once. (Initially, I had tries=2, but I changed it to
>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>count".) It also doesn't matter if I keep the same old contig
>>>directory and master log before rerunning, or if I delete them both.
>>>
>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>native version.
>>>
>>>Thanks,
>>>Jeremy
>>>
>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Let maker install it's own blast (a version I know should work).
>>>>
>>>> cd ./maker/src
>>>> ./Build blast
>>>>
>>>> Then to finish up your job (this command will just update executable
>>>> locations in the maker_exe.ctl file) -->
>>>> maker -EXE
>>>>
>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>contig
>>>> still fails.
>>>>
>>>> Thanks,
>>>> Carson
>>>>
>>>>
>>>>
>>>>
>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>
>>>>>Carson,
>>>>>
>>>>>$ blastx -version
>>>>>blastx: 2.2.26+
>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>
>>>>>- J
>>>>>
>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>wrote:
>>>>>> Could you send the blast version informations for reference
>>>>>>purposes?
>>>>>>I
>>>>>> can see this being caused by BLAST.
>>>>>>
>>>>>> Thanks,
>>>>>> Carson
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>
>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>with
>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>and
>>>>>>>used the Build script to have it install all of its own
>>>>>>>dependencies.
>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>the
>>>>>>>blast version I was using before.
>>>>>>>
>>>>>>>
>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>have
>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>first
>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>it
>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>hung
>>>>>>>> for a week until I killed it.
>>>>>>>>
>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>the
>>>>>>>> master log. maker again died on that contig.
>>>>>>>>
>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>
>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>lines
>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>needed
>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>> which takes a while.
>>>>>>>>
>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>> realize that might be hard to implement.)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jeremy
>>>>>>>> Grad student, Grishin lab
>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>> 510.384.8959
>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>.o
>>>>>>>>rg
>>>>>>>
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>maker-devel mailing list
>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>or
>>>>>>>g
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: RepeatMasker.pm
Type: text/x-perl-script
Size: 9225 bytes
Desc: not available
URL: <http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/attachments/20120831/c248081c/attachment.bin>

From jeremy.semeiks at utsw.edu  Fri Aug 31 17:48:37 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 31 Aug 2012 17:48:37 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC66941F.118C8%carsonhh@gmail.com>
References: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
	<CC66941F.118C8%carsonhh@gmail.com>
Message-ID: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>

Carson, maker completed on the scaffold without error when I used your
patch. Thanks!

- J

On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Looks like there is a slight format variation in one line of the
> RepeatMasker outputs.  Not sure what could have caused it -->
> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          0    109    (0)
>
> The position/compliment information in columns 12-14 give a start position
> of 0 for part of the alignment.  A 0 is invalid because repeat masker
> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
> asks for the starting position of the alignment in another part of the
> code.
>
> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
> wu-blast and then reconfigure RepeatMasker I get this line -->
> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          1    109    (0)
>
> Notice the 0 becomes a 1 and all logic is restored to the world.
>
> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
> output, then I just turn it into a 1.
> I've attached a file that you should use to replace the
> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
> I'll also make a note to the RepeatMasker developers.
>
> Thanks,
> Carson
>
>
>
> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Attached.
>>
>>Thanks,
>>J
>>
>>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send me the entire directory for just the failed contig?
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>
>>>>After doing this, I still get the same errors as I attached before.
>>>>There was also nothing in the log supporting that maker tried this
>>>>more than once. (Initially, I had tries=2, but I changed it to
>>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>count".) It also doesn't matter if I keep the same old contig
>>>>directory and master log before rerunning, or if I delete them both.
>>>>
>>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>native version.
>>>>
>>>>Thanks,
>>>>Jeremy
>>>>
>>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>> Let maker install it's own blast (a version I know should work).
>>>>>
>>>>> cd ./maker/src
>>>>> ./Build blast
>>>>>
>>>>> Then to finish up your job (this command will just update executable
>>>>> locations in the maker_exe.ctl file) -->
>>>>> maker -EXE
>>>>>
>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>contig
>>>>> still fails.
>>>>>
>>>>> Thanks,
>>>>> Carson
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>
>>>>>>Carson,
>>>>>>
>>>>>>$ blastx -version
>>>>>>blastx: 2.2.26+
>>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>
>>>>>>- J
>>>>>>
>>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>wrote:
>>>>>>> Could you send the blast version informations for reference
>>>>>>>purposes?
>>>>>>>I
>>>>>>> can see this being caused by BLAST.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Carson
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>
>>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>with
>>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>and
>>>>>>>>used the Build script to have it install all of its own
>>>>>>>>dependencies.
>>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>>the
>>>>>>>>blast version I was using before.
>>>>>>>>
>>>>>>>>
>>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>have
>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>first
>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>it
>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>hung
>>>>>>>>> for a week until I killed it.
>>>>>>>>>
>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>the
>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>
>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>
>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>lines
>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>needed
>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>> which takes a while.
>>>>>>>>>
>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jeremy
>>>>>>>>> Grad student, Grishin lab
>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>> 510.384.8959
>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>
>>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>.o
>>>>>>>>>rg
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>maker-devel mailing list
>>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>or
>>>>>>>>g
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>



From carsonhh at gmail.com  Fri Aug 31 18:00:27 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 19:00:27 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>
Message-ID: <CC66BA4B.820D%carsonhh@gmail.com>

Your welcome.  I also submitted a bug report to RepeatMasker.  They said the
issue will go away with the next release since simple repeat detection is
going to be handled by TRF rather than rmblast.

Thanks,
Carson


On 8/31/12 6:48 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

> Carson, maker completed on the scaffold without error when I used your
> patch. Thanks!
> 
> - J
> 
> On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Looks like there is a slight format variation in one line of the
>> RepeatMasker outputs.  Not sure what could have caused it -->
>> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          0    109    (0)
>> 
>> The position/compliment information in columns 12-14 give a start position
>> of 0 for part of the alignment.  A 0 is invalid because repeat masker
>> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
>> asks for the starting position of the alignment in another part of the
>> code.
>> 
>> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
>> wu-blast and then reconfigure RepeatMasker I get this line -->
>> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          1    109    (0)
>> 
>> Notice the 0 becomes a 1 and all logic is restored to the world.
>> 
>> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
>> output, then I just turn it into a 1.
>> I've attached a file that you should use to replace the
>> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
>> I'll also make a note to the RepeatMasker developers.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>> 
>>> Attached.
>>> 
>>> Thanks,
>>> J
>>> 
>>> On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Could you send me the entire directory for just the failed contig?
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> 
>>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>> 
>>>>> After doing this, I still get the same errors as I attached before.
>>>>> There was also nothing in the log supporting that maker tried this
>>>>> more than once. (Initially, I had tries=2, but I changed it to
>>>>> tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>> count".) It also doesn't matter if I keep the same old contig
>>>>> directory and master log before rerunning, or if I delete them both.
>>>>> 
>>>>> FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>> native version.
>>>>> 
>>>>> Thanks,
>>>>> Jeremy
>>>>> 
>>>>> On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>>> Let maker install it's own blast (a version I know should work).
>>>>>> 
>>>>>> cd ./maker/src
>>>>>> ./Build blast
>>>>>> 
>>>>>> Then to finish up your job (this command will just update executable
>>>>>> locations in the maker_exe.ctl file) -->
>>>>>> maker -EXE
>>>>>> 
>>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>> contig
>>>>>> still fails.
>>>>>> 
>>>>>> Thanks,
>>>>>> Carson
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>> 
>>>>>>> Carson,
>>>>>>> 
>>>>>>> $ blastx -version
>>>>>>> blastx: 2.2.26+
>>>>>>> Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>> 
>>>>>>> - J
>>>>>>> 
>>>>>>> On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>> wrote:
>>>>>>>> Could you send the blast version informations for reference
>>>>>>>> purposes?
>>>>>>>> I
>>>>>>>> can see this being caused by BLAST.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Carson
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>> with
>>>>>>>>> maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>> and
>>>>>>>>> used the Build script to have it install all of its own
>>>>>>>>> dependencies.
>>>>>>>>> Now maker is running fine, although the culprit seems to have been
>>>>>>>>> the
>>>>>>>>> blast version I was using before.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>>> <jeremy.semeiks at utsw.edu>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>> have
>>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>> first
>>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>> it
>>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>> hung
>>>>>>>>>> for a week until I killed it.
>>>>>>>>>> 
>>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>> the
>>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>> 
>>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>> 
>>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>> lines
>>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>> needed
>>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>>> which takes a while.
>>>>>>>>>> 
>>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Jeremy
>>>>>>>>>> Grad student, Grishin lab
>>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>>> 510.384.8959
>>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>>> maker-devel mailing list
>>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>> 
>>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>> .o
>>>>>>>>>> rg
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>> or
>>>>>>>>> g
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 





From Sean.Li at csiro.au  Wed Aug  1 19:21:08 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 11:21:08 +1000
Subject: [maker-devel] The est_gff option
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/50cfa71d/attachment-0001.html>

From carsonhh at gmail.com  Wed Aug  1 19:40:09 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 21:40:09 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F4EB7.10283%carsonhh@gmail.com>

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

Thanks,
Carson



From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 9:21 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] The est_gff option

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120801/ba79d284/attachment-0001.html>

From Sean.Li at csiro.au  Wed Aug  1 20:45:10 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 12:45:10 +1000
Subject: [maker-devel] The est_gff option
In-Reply-To: <CC3F4EB7.10283%carsonhh@gmail.com>
References: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
	<CC3F4EB7.10283%carsonhh@gmail.com>
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>

Dear Carson,

Thank you for your explanation! Now I understand how the RNA-Seq data's been interpreted in MAKER.  I check the tag est_gff and only found it in a binary db file: the scaffold_56.db, which should be a SQLite file. I guess this is right? Will this file be only generated when est data's been provided or I turned the alt_splice to 1? As the db file didn't exist for my first run of Maker when no EST flags have been switched on.

Best regards,
Sean

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished because they are already aligned (so no run.log entry is generated), they are just popped into an alignment object in memory and integrated directly.  The run.log file is used primarily to track finished calls to external programs (I.e. to avoid partial results files on failure). So since there is nothing to run, there is no run.log entry.  You will just get a message saying reading GFF3 in STDERR. The GFF3 results are combined with any other results produced within maker and are then used to generate hints for gene predictors you run as to the location of introns/exons.  This is during the step where you see SNAP or augustus running again and again.  They also become part of the AED score for selecting from the pool of gene models, and can be used to add UTR to resulting gene models.  If you specify alt_splice=1 they can even be used to infer alternate splice forms.  For reference purposes, they will end up in the maker results with the tag est_gff:Cufflinks.

Thanks,
Carson



From: <Sean.Li at csiro.au<mailto:Sean.Li at csiro.au>>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: [maker-devel] The est_gff option

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



_______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com<mailto:maker-devel at box290.bluehost.com> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/d8018277/attachment-0001.html>

From carsonhh at gmail.com  Wed Aug  1 20:50:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 22:50:14 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F629F.102AC%carsonhh@gmail.com>

The file is generated any time you provide GFF3 input.  It is just a quick
way to pull out the features for a region via SQLite.  It's generated at the
start of a MAKER run.

Thanks,
Carson

From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 10:45 PM
To:  Carson Holt <carsonhh at gmail.com>, <maker-devel at yandell-lab.org>
Subject:  RE: [maker-devel] The est_gff option

Dear Carson,
 
Thank you for your explanation! Now I understand how the RNA-Seq data?s been
interpreted in MAKER.  I check the tag est_gff and only found it in a binary
db file: the scaffold_56.db, which should be a SQLite file. I guess this is
right? Will this file be only generated when est data?s been provided or I
turned the alt_splice to 1? As the db file didn?t exist for my first run of
Maker when no EST flags have been switched on.
 
Best regards,
Sean   
 

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option
 

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

 

Thanks,

Carson

 

 

 

From: <Sean.Li at csiro.au>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org>
Subject: [maker-devel] The est_gff option

 

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120801/e466dcaa/attachment-0001.html>

From alangfan at gmail.com  Wed Aug  1 23:36:24 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Thu, 2 Aug 2012 13:36:24 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
Message-ID: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
[alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
#location of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
#location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
#location of probuild executable (required for genemark)

============================================

-- 
      Alang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/03155f36/attachment-0001.html>

From carsonhh at gmail.com  Thu Aug  2 06:23:21 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 08:23:21 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
Message-ID: <CC3FE8DD.102BC%carsonhh@gmail.com>

The timing of the fault suggests either a problem with your AnyDBM_File
module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
--debug and send me the output as well as your version of maker?  Also try
updating those 3 modules on your system.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 1:36 AM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
 [alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMas
ker #location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
of probuild executable (required for genemark)

============================================

-- 
      Alang

_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/32122c14/attachment-0001.html>

From mike.thon at gmail.com  Thu Aug  2 06:48:31 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Thu, 2 Aug 2012 14:48:31 +0200
Subject: [maker-devel] segmentation fault
Message-ID: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>

I have the same problem as Fan Wen-Lang.  I get a segmentation fault when processing the fasta file.  The strange thing is, when I run maker with the -debug parameter, I don't get the segmentation fault.  I'm using maker 2.26 on ubuntu linux 64 bit.
Mike




From carsonhh at gmail.com  Thu Aug  2 07:49:47 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 09:49:47 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>
Message-ID: <CC3FFDA2.102D5%carsonhh@gmail.com>

Odd.  Have you tried updating the 3 modules I mentioned.

Thanks,
Carson



On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>processing the fasta file.  The strange thing is, when I run maker with
>the -debug parameter, I don't get the segmentation fault.  I'm using
>maker 2.26 on ubuntu linux 64 bit.
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From felix.bemm at uni-wuerzburg.de  Thu Aug  2 09:56:55 2012
From: felix.bemm at uni-wuerzburg.de (Felix Bemm)
Date: Thu, 02 Aug 2012 17:56:55 +0200
Subject: [maker-devel] segmentation fault
In-Reply-To: <CC3FFDA2.102D5%carsonhh@gmail.com>
References: <CC3FFDA2.102D5%carsonhh@gmail.com>
Message-ID: <501AA347.2080304@uni-wuerzburg.de>

This snippet will fix your problem temporarily:

cd maker/lib
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)

But do it with caution, I still do not know what problems are caused by 
this fix.

Best regards
Felix

On 02.08.2012 15:49, Carson Holt wrote:
> Odd.  Have you tried updating the 3 modules I mentioned.
>
> Thanks,
> Carson
>
>
>
> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>> processing the fasta file.  The strange thing is, when I run maker with
>> the -debug parameter, I don't get the segmentation fault.  I'm using
>> maker 2.26 on ubuntu linux 64 bit.
>> Mike
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



From carsonhh at gmail.com  Thu Aug  2 10:31:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 12:31:37 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <501AA347.2080304@uni-wuerzburg.de>
Message-ID: <CC40233B.102F6%carsonhh@gmail.com>

Thanks Felix.

You may also have to run the fix below on wherever Bioperl is installed as
well, because it contains the same (DB_File GDBM_File NDBM_File SDBM_File)
statement in the Bio::DB::Fasta module in one of the BEGIN statements.

Thanks,
Carson



On 12-08-02 11:56 AM, "Felix Bemm" <felix.bemm at uni-wuerzburg.de> wrote:

>This snippet will fix your problem temporarily:
>
>cd maker/lib
>sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>
>But do it with caution, I still do not know what problems are caused by
>this fix.
>
>Best regards
>Felix
>
>On 02.08.2012 15:49, Carson Holt wrote:
>> Odd.  Have you tried updating the 3 modules I mentioned.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault
>>>when
>>> processing the fasta file.  The strange thing is, when I run maker with
>>> the -debug parameter, I don't get the segmentation fault.  I'm using
>>> maker 2.26 on ubuntu linux 64 bit.
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From kippjohnson at uchicago.edu  Thu Aug  2 13:12:15 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Thu, 2 Aug 2012 14:12:15 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
Message-ID: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The
first time I ran maker, I ignored these and the maker run seemingly
proceeded to the process the contigs correctly. Is this something I should
be worried about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for
each restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/70c4c837/attachment-0001.html>

From alangfan at gmail.com  Thu Aug  2 18:45:17 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Fri, 3 Aug 2012 08:45:17 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CC3FE8DD.102BC%carsonhh@gmail.com>
References: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
	<CC3FE8DD.102BC%carsonhh@gmail.com>
Message-ID: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>

Thanks Carson.

I'm using a single server with centOS6 64 bit.
==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep
-l 'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>

> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
>
> Thanks,
> Carson
>
>
>
> From: "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date: Thursday, 2 August, 2012 1:36 AM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
>
> Dear All,
>
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
>
> Best,
>
> -------command and error message-------------------------------------
> [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
>
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
>
> #-----Protein Homology Evidence
> #protein=protein.fasta
>
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
>
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
> of NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
> of NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
> #location of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
> #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
>
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
> of eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
>
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
> #location of probuild executable (required for genemark)
>
> ============================================
>
> --
>       Alang
>
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/36149b1d/attachment-0001.html>

From carsonhh at gmail.com  Thu Aug  2 20:07:53 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 22:07:53 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>
Message-ID: <CC40AA6F.10327%carsonhh@gmail.com>

Ru with the ?f flag before rerunning in the same directory as your previous
error.  Also collect the STDERR both with and without the -debug flag and
send it to me.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 8:45 PM
To:  Carson Holt <carsonhh at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] Question: maker Segmentation fault (core dumped)

Thanks Carson.

I'm using a single server with centOS6 64 bit.
============================================================================
==
STATUS MAKER 2.26
============================================================================
==
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------
------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep -l
'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>
> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
> 
> Thanks,
> Carson
> 
> 
> 
> From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date:  Thursday, 2 August, 2012 1:36 AM
> To:  <maker-devel at yandell-lab.org>
> Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)
> 
> Dear All,
> 
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
> 
> Best,
> 
> -------command and error message-------------------------------------
>  [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
> 
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
> 
> #-----Protein Homology Evidence
> #protein=protein.fasta
> 
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
> 
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location of
> NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location of
> NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
> of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMaske
> r #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
> 
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location of
> eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
> 
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
> of probuild executable (required for genemark)
> 
> ============================================
> 
> -- 
>       Alang
> 
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak
> er-devel_yandell-lab.org



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------
------------------------------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/9a29692b/attachment-0001.html>

From mike.thon at gmail.com  Thu Aug  2 22:30:15 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 06:30:15 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been started
	in the same directory.
Message-ID: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>

I get this warning when running the mpi version of maker.  Should I be concerned?  the command I use is this:

 mpiexec -n 8 maker

-Mike




From carsonhh at gmail.com  Thu Aug  2 23:11:39 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 01:11:39 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>
Message-ID: <CC40CC6D.10338%carsonhh@gmail.com>

I'm sorry I'm confused.  Which warning? the segmentation fault you
mentioned before?  For now I would say it's best to first try maker once
outside of MPI to verify that the segmentation fault no longer happens
before moving onto the MPI environment.

Make sure to run with the -f flag when retrying after the fix suggested by
Felix, so maker completely restarts from scratch.  That way you can see
that the fasta database is being generated from start to finish (and is
not just being reread from the successful run under the --debug flag).

Make sure that the fix Felix sent is also applied to the Bioperl
Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
fault will not go away otherwise (if it is infact one of perl's DB modules
causing the issue, which it likely is).  Your error sounds similar to an
error Felix and Thomas encountered some time ago
(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
h/000941.html).  Basically one of perl's native DB modules or the database
the module relies on are broken, so the command line snippet is cutting
any calls to those other databases out of maker and Bioperl (which should
be ok if you have at minimum a working Berkley DB and DB_File module on
your system).

Use this command line to get the correct directory for applying the
changes to Bioperl as well -->
perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
s/[^\/]+$//; print "$dir\n"'

Then copy the directory location-->

cd <to_that_directory>
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)



Finally run maker with the -f flag.  If it works let me know, and then you
can give MPI a shot.  Make said yes to the "compile for MPI question"
asked during the maker install process or mpiexec will fail when running
maker.  If it doesn?t work without MPI or if it works and then MPI fails,
make sure to collect the STDERR in a text file and send it to me.  I will
probably have you run it with the --debug flag as well following any
initial failure (just to get the extra status messages about your system
configuration).

Thanks,
Carson



On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I get this warning when running the mpi version of maker.  Should I be
>concerned?  the command I use is this:
>
> mpiexec -n 8 maker
>
>-Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From mike.thon at gmail.com  Fri Aug  3 01:56:34 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 09:56:34 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>

I meant the warning that is in the subject line of my first email:

WARNING: Multiple MAKER processes have been started in the same directory.

I get that warning when I run mpi maker.  For now, I'm running it without mpi.  When the run finishes I can try it again with mpi and make sure I get the same output.

For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.

best
Mike

On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From carsonhh at gmail.com  Fri Aug  3 06:37:24 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 08:37:24 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC413D27.1038C%carsonhh@gmail.com>

That's good to know the seg fault is gone.  Sorry for the confusion, but I
don't seem to have received your other e-mail about the "multiple process
warning".

Could you run '/maker/src/Build status' and send me the output?

Thanks,
Carson

On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 07:11:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:11:03 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC414316.103A1%carsonhh@gmail.com>

One more thing.

There are three locations where this exact line of code are called
--> @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
run_log
ds_utility
Bio::Fasta::DB

Because of the way perl works only the last one to run needs to be fixed
to stop the error (because it overrides the other two calls), but that
means if the line is ever removed from the last loaded module or load
order changes, then the error will come back because it is the last one to
run that fixes the issue.  So to be safe you should fix it in all
instances.

Currently ds_utility loads last, but it really doesn't need AnyDBM_File
any more to perform it's function and the call will likely be removed in
future releases, so changing the line in Bio::Fasta::DB should avoid
potential future errors on your system.

Thanks,
Carson



On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 07:27:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:27:37 -0400
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
Message-ID: <CC414867.103D1%carsonhh@gmail.com>

It means that your GFF3 results will not be integrated.  Does it happen if
you start in a different directory?

The lock error is coming from SQLite, and may be the result of your working
directory being mounted on an NFS filesystem which is a known issue with
SQLite.  If you give maker a location for TMP that is not NFS mounted (I.e.
A locally mounted filesystem), then maker will try and move the SQLite
database off of the NFS mount.

You can check the mount status of the current working directory and the
directory you choose fro TMP by using the Linux df command.

df <directory>

Any NFS mount will follow the format of host:/location.

Example:
152.234.123.234:/data/research

Or 

storage.utah.edu:/data/research

If you are not running on an NFS device let me know if running in a
different directory results in the same error.

Thanks,
Carson

From:  Kipp Johnson <kippjohnson at uchicago.edu>
Date:  Thursday, 2 August, 2012 3:12 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The first
time I ran maker, I ignored these and the maker run seemingly proceeded to
the process the contigs correctly. Is this something I should be worried
about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for each
restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/5517b922/attachment-0001.html>

From cjfields at illinois.edu  Fri Aug  3 07:51:04 2012
From: cjfields at illinois.edu (Fields, Christopher J)
Date: Fri, 3 Aug 2012 13:51:04 +0000
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>

I just want to add in, if there are any fixes that need to be made to Bioperl let me know.  We're trying to move towards an AnyDBM_File approach, but I think this effort has stalled.

chris

On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
 wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug  3 08:28:44 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 10:28:44 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>
Message-ID: <CC414FC9.1040E%carsonhh@gmail.com>

I think I've finally figured it out, as well as a fix to quash this error
for good.  It not strictly a Bioperl issue.  It's an an issue with
AnyDBM_File as well as it's documentation (as it recommends setting
@AnyDBM_File::ISA in code that uses it but never mentions potential
caveats of multiple modules using it).

Basically if you set @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File
SDBM_File) and then call 'use AnyDBM_File', the AnyDBM_File module will
actually preprocess and trim @AnyDBM_File::ISA using this code -->

my $mod;
for $mod (@ISA) {
	if (eval "require $mod") {
		@ISA = ($mod);  # if we leave @ISA alone, warnings abound
		return 1;
	}
}


However if you load any other module that also sets @AnyDBM_File::ISA that
code doesn?t get called on the next 'use AnyDBM_File'.  So you will loose
the preprocessing and basically mess up @AnyDBM_File::ISA if you ever have
two modules loaded at the same time that use AnyDBM_File.

So loading Bio::DB::Fasta before or after a module that also uses
AnyDBM_File has the potential to result in errors.

Example:
use Bio::DB::Fasta;
print join(',', @AnyDBM_File::ISA );

Result --> DB_File


use Bio::DB::Qual;

use Bio::DB::Fasta;

print join(',', @AnyDBM_File::ISA );


Result --> DB_File,GDBM_File,NDBM_File,SDBM_File


The second module messes up @AnyDBM_File::ISA.  It also happens if you
load them in reverse.
Perhaps changing the BEGIN block in Bio::DB::Fasta, Bio::DB::Qual, and any
other any other code using AnyDBM_File to include one of these alternate
BEGIN blocks.

BEGIN {
	@AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
if(!$INC{'AnyDBM_File.pm'});
}


Or 

BEGIN {
	for my $mod (qw(DB_File GDBM_File NDBM_File SDBM_File)) {
		if (eval "require $mod") {
			@AnyDBM_File::ISA = ($mod);
			last;
		}
	}

}


As far as maker goes, I've actually removed all use of AnyDBM_File from
it's modules now, so it won't be an issue there any more, so Bioperl is
the only one left to fix.

Thanks,
Carson



On 12-08-03 9:51 AM, "Fields, Christopher J" <cjfields at illinois.edu> wrote:

>I just want to add in, if there are any fixes that need to be made to
>Bioperl let me know.  We're trying to move towards an AnyDBM_File
>approach, but I think this effort has stalled.
>
>chris
>
>On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>





From kippjohnson at uchicago.edu  Fri Aug  3 10:16:33 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Fri, 3 Aug 2012 11:16:33 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CC414867.103D1%carsonhh@gmail.com>
References: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
	<CC414867.103D1%carsonhh@gmail.com>
Message-ID: <CAGJuV2XTG0GWGi9Dbuk7xdT=OOJodWhbUORsj9XcZJwdz3PYYw@mail.gmail.com>

It does happen if I start in a different directory. My current working
directory is NFS. I changed the tmp directory in maker_opts from the NFS to
a local partition (tmpfs), and I'm still getting this error. Does the
working directory also have to be locally mounted?

On Fri, Aug 3, 2012 at 8:27 AM, Carson Holt <carsonhh at gmail.com> wrote:

> It means that your GFF3 results will not be integrated.  Does it happen if
> you start in a different directory?
>
> The lock error is coming from SQLite, and may be the result of your
> working directory being mounted on an NFS filesystem which is a known issue
> with SQLite.  If you give maker a location for TMP that is not NFS mounted
> (I.e. A locally mounted filesystem), then maker will try and move the
> SQLite database off of the NFS mount.
>
> You can check the mount status of the current working directory and the
> directory you choose fro TMP by using the Linux df command.
>
> df <directory>
>
> Any NFS mount will follow the format of host:/location.
>
> Example:
> 152.234.123.234:/data/research
>
> Or
>
> storage.utah.edu:/data/research
>
> If you are not running on an NFS device let me know if running in a
> different directory results in the same error.
>
> Thanks,
> Carson
>
> From: Kipp Johnson <kippjohnson at uchicago.edu>
> Date: Thursday, 2 August, 2012 3:12 PM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
>
> Hi,
>
>      I am running Maker version 2.26. I have two questions:
>
> 1)
>
>     When I run maker, I get errors like the following printed to STDOUT:
>
> DBD::SQLite::db selectcol_arrayref failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.
>
>      I reinstalled the indicated module via cpanm and restarted Maker from
> the beginning in a new directory, and I am getting the same error. The
> first time I ran maker, I ignored these and the maker run seemingly
> proceeded to the process the contigs correctly. Is this something I should
> be worried about?
>
>
> 2)
>
>     I am running Maker on a single server with 64 processors. Would it be
> faster to use MPICH2, or to simply restart Maker in the same directory ~16
> times, with the # of CPUs option that gets passed to Blast set to 4 for
> each restart?
>
>     I believe that I have MPICH2 correctly compiled and installed; if it
> would be faster to use MPICH2, could someone provide me with a sample
> command line entry to use for Maker? I am confused about how to run MPICH2
> on a single node (without a hostfile?), and searching the maker-dev-list
> archive seems to give me results for older versions of MAKER and MPICH2.
>
> Thank you,
>
>
> Best,
>
>      Kipp Johnson
>      kippjohnson at uchicago.edu
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/b5f2b0cd/attachment-0001.html>

From mike.thon at gmail.com  Fri Aug  3 23:48:52 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 07:48:52 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC413D27.1038C%carsonhh@gmail.com>
References: <CC413D27.1038C%carsonhh@gmail.com>
Message-ID: <E44B4BF9-7D27-4B11-B33F-42A975222F99@gmail.com>

./Build status


==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:	VERIFIED
External Programs:	VERIFIED
External C Libraries:	VERIFIED
MPI SUPPORT:		ENABLED
MWAS Web Interface:	DISABLED
MAKER PACKAGE:		CONFIGURATION OK

...

I was impatient and I ran maker with MPI.  In the master log file I see a lot of failed contigs.  Now I'm running it without MPI to see if those contigs fail again. 

On Aug 3, 2012, at 2:37 PM, Carson Holt <carsonhh at gmail.com> wrote:

> That's good to know the seg fault is gone.  Sorry for the confusion, but I
> don't seem to have received your other e-mail about the "multiple process
> warning".
> 
> Could you run '/maker/src/Build status' and send me the output?
> 
> Thanks,
> Carson
> 
> On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I meant the warning that is in the subject line of my first email:
>> 
>> WARNING: Multiple MAKER processes have been started in the same directory.
>> 
>> I get that warning when I run mpi maker.  For now, I'm running it without
>> mpi.  When the run finishes I can try it again with mpi and make sure I
>> get the same output.
>> 
>> For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>> to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>> 
>> best
>> Mike
>> 
>> On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>>> mentioned before?  For now I would say it's best to first try maker once
>>> outside of MPI to verify that the segmentation fault no longer happens
>>> before moving onto the MPI environment.
>>> 
>>> Make sure to run with the -f flag when retrying after the fix suggested
>>> by
>>> Felix, so maker completely restarts from scratch.  That way you can see
>>> that the fasta database is being generated from start to finish (and is
>>> not just being reread from the successful run under the --debug flag).
>>> 
>>> Make sure that the fix Felix sent is also applied to the Bioperl
>>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>>> fault will not go away otherwise (if it is infact one of perl's DB
>>> modules
>>> causing the issue, which it likely is).  Your error sounds similar to an
>>> error Felix and Thomas encountered some time ago
>>> 
>>> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>> rc
>>> h/000941.html).  Basically one of perl's native DB modules or the
>>> database
>>> the module relies on are broken, so the command line snippet is cutting
>>> any calls to those other databases out of maker and Bioperl (which
>>> should
>>> be ok if you have at minimum a working Berkley DB and DB_File module on
>>> your system).
>>> 
>>> Use this command line to get the correct directory for applying the
>>> changes to Bioperl as well -->
>>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>>> s/[^\/]+$//; print "$dir\n"'
>>> 
>>> Then copy the directory location-->
>>> 
>>> cd <to_that_directory>
>>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>> $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>>> 
>>> 
>>> 
>>> Finally run maker with the -f flag.  If it works let me know, and then
>>> you
>>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>>> asked during the maker install process or mpiexec will fail when running
>>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>> fails,
>>> make sure to collect the STDERR in a text file and send it to me.  I
>>> will
>>> probably have you run it with the --debug flag as well following any
>>> initial failure (just to get the extra status messages about your system
>>> configuration).
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> 
>>> 
>>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>> 
>>>> I get this warning when running the mpi version of maker.  Should I be
>>>> concerned?  the command I use is this:
>>>> 
>>>> mpiexec -n 8 maker
>>>> 
>>>> -Mike
>>>> 
>>>> 
>>>> _______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From mike.thon at gmail.com  Sat Aug  4 05:51:04 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 13:51:04 +0200
Subject: [maker-devel] failed contigs
Message-ID: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>

running maker 2.26 some contigs seem to consistently fail to run, according to XXX_master_datastore_index.log

Here are the last few lines of run.log for one of the failed contigs:

CTL_OPTIONS     run     
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
DIED    RANK    0
DIED    COUNT   1


the contents of run.log.child.1 for the same contig:
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
FINISHED        cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.te_proteins%2Efasta.repeatrunner  
DIED    RANK    0:2:0:1
DIED    COUNT   1

So perhaps its a problem with the repeat masking step.  I already have annotated repeats using RepeatMasker and I have the gff2 file produced by RepeatMasker.  Do I need to convert this to gff3 before adding to to my maker_opts file?  Better yet, does anyone know why these contigs are failing and how to fix it?
Mike




From carsonhh at gmail.com  Sat Aug  4 07:44:25 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Sat, 04 Aug 2012 09:44:25 -0400
Subject: [maker-devel] failed contigs
In-Reply-To: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>
Message-ID: <CC429E5D.104F1%carsonhh@gmail.com>

Do you have STDERR log stored somewhere?  Could you send it to me?
Sometimes the error causing the failure is upstream in the log and then
cascades into other errors.  Also if it is the repeatrunner step failing,
the log will have the command to run the BLAST step outside of MAKER (is
helpful for debugging).  If you don't have the captured STDERR, could you
run again and send it to me.

Thanks,
Carson

On 12-08-04 7:51 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>running maker 2.26 some contigs seem to consistently fail to run,
>according to XXX_master_datastore_index.log
>
>Here are the last few lines of run.log for one of the failed contigs:
>
>CTL_OPTIONS     run
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>DIED    RANK    0
>DIED    COUNT   1
>
>
>the contents of run.log.child.1 for the same contig:
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>FINISHED        
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.te_proteins%2Efasta.repeatrunner
>DIED    RANK    0:2:0:1
>DIED    COUNT   1
>
>So perhaps its a problem with the repeat masking step.  I already have
>annotated repeats using RepeatMasker and I have the gff2 file produced by
>RepeatMasker.  Do I need to convert this to gff3 before adding to to my
>maker_opts file?  Better yet, does anyone know why these contigs are
>failing and how to fix it?
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From barry.moore at genetics.utah.edu  Fri Aug 17 19:05:00 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 17 Aug 2012 19:05:00 -0600
Subject: [maker-devel] Exonerate web access is back
References: <201208160913.q7G9DTbB005179@rabbit.ebi.ac.uk>
Message-ID: <C52C60DC-1052-4FD7-9CAC-2F4830E78827@genetics.utah.edu>



Begin forwarded message:

> From: <datasubs at ebi.ac.uk>
> Subject: SUP/EMBL/ 24308 moved from EBIHELP by apc (rasko) (SUB#818751)
> Date: August 16, 2012 3:13:29 AM MDT
> To: <barry.moore at genetics.utah.edu>
> 
> Dear Barry,
> 
> Access to http://www.ebi.ac.uk/~guy/exonerate/ has been restored.
> 
> Kind Regards,
> Rasko Leinonen 
> European Nucleotide Archive (ENA)
> 

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120817/0a68e03e/attachment-0001.html>

From kippjohnson at uchicago.edu  Tue Aug 21 15:25:49 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Tue, 21 Aug 2012 16:25:49 -0500
Subject: [maker-devel] Maker Re-tries
In-Reply-To: <6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
References: <CC414867.103D1%carsonhh@gmail.com>
	<6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
Message-ID: <1E417039-68CF-4EFE-90BF-964DC86B1864@uchicago.edu>

Hi Carson,

	Simple question: If I start maker and then kill the job before a thread finishes processing a contig, does this failure to complete the contig count in the number of tries maker will try to complete the contig before skipping to another?	

	Basically, does the "tries=2 #number of times to try a contig if there is a failure for some reason" parameter take into account failures resulting from me killing the job?	


	Best,

		Kipp Johnson
		kippjohnson at uchicago.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120821/74551a9f/attachment-0001.html>

From jeremy.semeiks at utsw.edu  Tue Aug 21 10:26:32 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Tue, 21 Aug 2012 11:26:32 -0500
Subject: [maker-devel] maker dying on a contig
Message-ID: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>

Hi,

I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
~36,000 contigs, and per the master log maker FINISHED all of them
except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
run. It didn't report ERROR for that contig in the master log, but it
did output two DIED lines in the contig's run.log and then just hung
for a week until I killed it.

I reran maker on just that contig by deleting the START line from the
master log. maker again died on that contig.

I attach the relevant part of the stderr stream; the rest was just
"--Next Contig--"-type lines. Any ideas?

A related question. Is deleting lines in the master log, as I did
here, the best way to get maker to rerun on just a few contigs? I
often find myself needing to do this, sometimes because of ERROR lines
or other assumed quirks in maker and sometimes just because I needed
to pause a big run for a while. One disadvantage is that when I
restart, maker needs to manually scan over many completed contigs,
which takes a while.

(I'd think the best solution would be for maker to automatically
recognize whether it's really currently running on contigs marked
STARTED but not FINISHED. It doesn't seem to do this now, but I
realize that might be hard to implement.)

Thanks,
Jeremy
Grad student, Grishin lab
UT Southwestern, Dallas TX
510.384.8959
-------------- next part --------------
A non-text attachment was scrubbed...
Name: maker.stderr
Type: application/octet-stream
Size: 11421 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120821/7f29efb2/attachment-0001.obj>

From mike.thon at gmail.com  Fri Aug 24 01:44:13 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 24 Aug 2012 09:44:13 +0200
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
Message-ID: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>

I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.


On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:

> Hi,
> 
> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
> ~36,000 contigs, and per the master log maker FINISHED all of them
> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
> run. It didn't report ERROR for that contig in the master log, but it
> did output two DIED lines in the contig's run.log and then just hung
> for a week until I killed it.
> 
> I reran maker on just that contig by deleting the START line from the
> master log. maker again died on that contig.
> 
> I attach the relevant part of the stderr stream; the rest was just
> "--Next Contig--"-type lines. Any ideas?
> 
> A related question. Is deleting lines in the master log, as I did
> here, the best way to get maker to rerun on just a few contigs? I
> often find myself needing to do this, sometimes because of ERROR lines
> or other assumed quirks in maker and sometimes just because I needed
> to pause a big run for a while. One disadvantage is that when I
> restart, maker needs to manually scan over many completed contigs,
> which takes a while.
> 
> (I'd think the best solution would be for maker to automatically
> recognize whether it's really currently running on contigs marked
> STARTED but not FINISHED. It doesn't seem to do this now, but I
> realize that might be hard to implement.)
> 
> Thanks,
> Jeremy
> Grad student, Grishin lab
> UT Southwestern, Dallas TX
> 510.384.8959
> <maker.stderr>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From kmegy at ebi.ac.uk  Fri Aug 24 02:38:32 2012
From: kmegy at ebi.ac.uk (Karyn Megy)
Date: Fri, 24 Aug 2012 09:38:32 +0100
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
	<BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <463EA6A1-58CC-4358-9812-1B659DED9F13@ebi.ac.uk>

Hi Jeremy, 

Could it be a contig with lots of repeats, or that would hit lots of proteins?

One of my colleague had this issue and he ended splitting the few problematic contigs in sections... but then you have to reconciliate the results because the gene/BLAST coordinates are based in the bits of sub-contigs rather than the initial contig.  

You could try running MAKER on half of this contig and see what happen.

Cheers,
Karyn.

On 24 Aug 2012, at 08:44, Michael Thon wrote:

> I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.
> 
> 
> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:
> 
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug 24 06:24:56 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 08:24:56 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <CC5CEA43.11379%carsonhh@gmail.com>

Could you send the blast version informations for reference purposes? I
can see this being caused by BLAST.

Thanks,
Carson



On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I'm not sure if this is relevant to your problem but I had problems with
>maker failing on some contigs but not others.  I reinstalled maker and
>used the Build script to have it install all of its own dependencies.
>Now maker is running fine, although the culprit seems to have been the
>blast version I was using before.
>
>
>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>wrote:
>
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From jeremy.semeiks at utsw.edu  Fri Aug 24 13:13:39 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 14:13:39 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC5CEA43.11379%carsonhh@gmail.com>
References: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
	<CC5CEA43.11379%carsonhh@gmail.com>
Message-ID: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>

Carson,

$ blastx -version
blastx: 2.2.26+
Package: blast 2.2.26, build Feb 22 2012 16:04:28

- J

On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
> Could you send the blast version informations for reference purposes? I
> can see this being caused by BLAST.
>
> Thanks,
> Carson
>
>
>
> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>>I'm not sure if this is relevant to your problem but I had problems with
>>maker failing on some contigs but not others.  I reinstalled maker and
>>used the Build script to have it install all of its own dependencies.
>>Now maker is running fine, although the culprit seems to have been the
>>blast version I was using before.
>>
>>
>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>> run. It didn't report ERROR for that contig in the master log, but it
>>> did output two DIED lines in the contig's run.log and then just hung
>>> for a week until I killed it.
>>>
>>> I reran maker on just that contig by deleting the START line from the
>>> master log. maker again died on that contig.
>>>
>>> I attach the relevant part of the stderr stream; the rest was just
>>> "--Next Contig--"-type lines. Any ideas?
>>>
>>> A related question. Is deleting lines in the master log, as I did
>>> here, the best way to get maker to rerun on just a few contigs? I
>>> often find myself needing to do this, sometimes because of ERROR lines
>>> or other assumed quirks in maker and sometimes just because I needed
>>> to pause a big run for a while. One disadvantage is that when I
>>> restart, maker needs to manually scan over many completed contigs,
>>> which takes a while.
>>>
>>> (I'd think the best solution would be for maker to automatically
>>> recognize whether it's really currently running on contigs marked
>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>> realize that might be hard to implement.)
>>>
>>> Thanks,
>>> Jeremy
>>> Grad student, Grishin lab
>>> UT Southwestern, Dallas TX
>>> 510.384.8959
>>> <maker.stderr>_______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>_______________________________________________
>>maker-devel mailing list
>>maker-devel at box290.bluehost.com
>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>



From carsonhh at gmail.com  Fri Aug 24 13:17:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 15:17:03 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
Message-ID: <CC5D4ADD.11459%carsonhh@gmail.com>

Let maker install it's own blast (a version I know should work).

cd ./maker/src
./Build blast

Then to finish up your job (this command will just update executable
locations in the maker_exe.ctl file) -->
maker -EXE

Then run maker again with a higher retry count.  Let me know if the contig
still fails.

Thanks,
Carson




On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Carson,
>
>$ blastx -version
>blastx: 2.2.26+
>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>
>- J
>
>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send the blast version informations for reference purposes? I
>> can see this being caused by BLAST.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>>I'm not sure if this is relevant to your problem but I had problems with
>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>used the Build script to have it install all of its own dependencies.
>>>Now maker is running fine, although the culprit seems to have been the
>>>blast version I was using before.
>>>
>>>
>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>> did output two DIED lines in the contig's run.log and then just hung
>>>> for a week until I killed it.
>>>>
>>>> I reran maker on just that contig by deleting the START line from the
>>>> master log. maker again died on that contig.
>>>>
>>>> I attach the relevant part of the stderr stream; the rest was just
>>>> "--Next Contig--"-type lines. Any ideas?
>>>>
>>>> A related question. Is deleting lines in the master log, as I did
>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>> or other assumed quirks in maker and sometimes just because I needed
>>>> to pause a big run for a while. One disadvantage is that when I
>>>> restart, maker needs to manually scan over many completed contigs,
>>>> which takes a while.
>>>>
>>>> (I'd think the best solution would be for maker to automatically
>>>> recognize whether it's really currently running on contigs marked
>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>> realize that might be hard to implement.)
>>>>
>>>> Thanks,
>>>> Jeremy
>>>> Grad student, Grishin lab
>>>> UT Southwestern, Dallas TX
>>>> 510.384.8959
>>>> <maker.stderr>_______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> 
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>>>_______________________________________________
>>>maker-devel mailing list
>>>maker-devel at box290.bluehost.com
>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>





From jeremy.semeiks at utsw.edu  Fri Aug 24 15:11:10 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 16:11:10 -0500
Subject: [maker-devel]  maker dying on a contig
In-Reply-To: <CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
References: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
	<CC5D4ADD.11459%carsonhh@gmail.com>
	<CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
Message-ID: <CABPMbe3nSDGzf_VqDj1Nucd0DPuPUB_tKcqQj64sV-2SyuUAQQ@mail.gmail.com>

After doing this, I still get the same errors as I attached before.
There was also nothing in the log supporting that maker tried this
more than once. (Initially, I had tries=2, but I changed it to
tries=10 for this run. I'm guessing that's what you mean by "retry
count".) It also doesn't matter if I keep the same old contig
directory and master log before rerunning, or if I delete them both.

FWIW, maker's own version of blast is 2.2.26+, which is the same as my
native version.

Thanks,
Jeremy

On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Let maker install it's own blast (a version I know should work).
>
> cd ./maker/src
> ./Build blast
>
> Then to finish up your job (this command will just update executable
> locations in the maker_exe.ctl file) -->
> maker -EXE
>
> Then run maker again with a higher retry count.  Let me know if the contig
> still fails.
>
> Thanks,
> Carson
>
>
>
>
> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Carson,
>>
>>$ blastx -version
>>blastx: 2.2.26+
>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>
>>- J
>>
>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send the blast version informations for reference purposes? I
>>> can see this being caused by BLAST.
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>>
>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>
>>>>I'm not sure if this is relevant to your problem but I had problems with
>>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>>used the Build script to have it install all of its own dependencies.
>>>>Now maker is running fine, although the culprit seems to have been the
>>>>blast version I was using before.
>>>>
>>>>
>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>>> did output two DIED lines in the contig's run.log and then just hung
>>>>> for a week until I killed it.
>>>>>
>>>>> I reran maker on just that contig by deleting the START line from the
>>>>> master log. maker again died on that contig.
>>>>>
>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>
>>>>> A related question. Is deleting lines in the master log, as I did
>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>>> or other assumed quirks in maker and sometimes just because I needed
>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>> which takes a while.
>>>>>
>>>>> (I'd think the best solution would be for maker to automatically
>>>>> recognize whether it's really currently running on contigs marked
>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>> realize that might be hard to implement.)
>>>>>
>>>>> Thanks,
>>>>> Jeremy
>>>>> Grad student, Grishin lab
>>>>> UT Southwestern, Dallas TX
>>>>> 510.384.8959
>>>>> <maker.stderr>_______________________________________________
>>>>> maker-devel mailing list
>>>>> maker-devel at box290.bluehost.com
>>>>>
>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>>
>>>>
>>>>_______________________________________________
>>>>maker-devel mailing list
>>>>maker-devel at box290.bluehost.com
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>
>



From jokelley at stanford.edu  Thu Aug 30 18:18:29 2012
From: jokelley at stanford.edu (Joanna Kelley)
Date: Thu, 30 Aug 2012 17:18:29 -0700
Subject: [maker-devel] convert gff to ncbi table format?
Message-ID: <CA+77YqHuS8CUK+YmDCQX8KMt1FWiMkJ00eXsgwdzHTM1PxsZ1g@mail.gmail.com>

Hello,

I am trying to submit the maker annotations to NCBI and
I was wondering if there are any tools to convert gff to
the NCBI table format for submission?

Thank you,
Joanna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120830/9a5ce895/attachment-0001.html>

From chrisi.hahni at gmail.com  Fri Aug 31 06:43:21 2012
From: chrisi.hahni at gmail.com (Christoph Hahn)
Date: Fri, 31 Aug 2012 14:43:21 +0200
Subject: [maker-devel] keep_preds option?
Message-ID: <5040B169.1030909@gmail.com>

Hello maker users and developers,

I am new to gene prediction and I am trying to use maker 2.25 on a newly 
assembled non-model organisms draft genome. Within maker I use genemark, 
SNAP and Augustus. I have a few questions:

1. I was wondering what the keep_preds option means exactly.

I found two slightly different explanations on the option
#Add unsupported gene prediction to final annotation set (maker2.25)
#Add non-overlapping ab-inito gene prediction to final annotation set 
(found on the net - probably older maker version)

As far as I understood keep_preds=0 only retains gene models for which 
the ab initio predictions agree. But how many, all three? two of three?
keep_preds=1 instead keeps all gene models regardless if the different 
programs agree, right?

In my case I get substantial differences in the number of gene models 
found between the two settings, while with =1 I get a number that is 
close to what we would expect. How would you interpret that? What would 
you recommend me to do? Obiously =0 is the saver option.

2. I tried to use EST data of an alternative organism in altest= 
(#EST/cDNA sequence file in fasta format from an alternate organism). 
The organism is quite distantly related, but its the closest I have so I 
thought I d give it a shot. I ran maker twice with identical settigs 
expect in altest and est2genome=0/1. The number of genes predicted is 
identical with both approaches, so I am not sure whether or not the EST 
data was actually used or its just to distantly related. Any easy way to 
assess this?

3. I am running maker in several passes and after each pass I am 
training SNAP using the result of the previous pass. Then for every pass 
I run maker from scratch. Would you recommend to supply the gff of the 
previous pass in "#-----Re-annotation Using MAKER Derived GFF3
maker_gff= #re-annotate genome based on this gff3 file", instead?

Thanks in advance for any thoughts/advice on these things! I appreciate 
your help!

much obliged,
Christoph



From barry.moore at genetics.utah.edu  Fri Aug 31 10:52:10 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 31 Aug 2012 10:52:10 -0600
Subject: [maker-devel] keep_preds option?
In-Reply-To: <5040B169.1030909@gmail.com>
References: <5040B169.1030909@gmail.com>
Message-ID: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly assembled non-model organisms draft genome. Within maker I use genemark, SNAP and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical evidence.  There are two pieces of information that are required for every MAKER annotation (by default).  MAKER runs the ab initio gene predictors and aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to the genome.  For each locus where one or more gene predictions exist MAKER checks to see if there is any physical evidence for gene expression at that locus (RNA/protein sequence alignments) and if there is (splice EST or protein alignments) evidence overlapping the predictions, MAKER decides which prediction best matches the evidence and promotes that prediction to an annotation.  If there is no evidence overlapping any of the predictions then those predictions are not included in the output annotation file (although they are saved).  If you set keep_preds=1 then for each locus where prediction(s) exist maker keeps one and promotes it to an annotation even though there is no physical evidence.  The description of 'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as evidence and thus in the absence of aligned physical evidence MAKER will not output an annotation even if all three ab initio tools predict a gene at that locus.

> In my case I get substantial differences in the number of gene models found between the two settings, while with =1 I get a number that is close to what we would expect. How would you interpret that? What would you recommend me to do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is too few, you could run MAKER with keep_preds=1.  After the run is finished, use a tool like IPRScan to search all MAKER predictions for protein domain content and push that IPRScan output back into the MAKER GFF file with the ipr_update_gff script.  Then as a final step you can run over the GFF file and remove any gene model that doesn't have either physical evidence (AED < 1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA sequence file in fasta format from an alternate organism). The organism is quite distantly related, but its the closest I have so I thought I d give it a shot. I ran maker twice with identical settigs expect in altest and est2genome=0/1. The number of genes predicted is identical with both approaches, so I am not sure whether or not the EST data was actually used or its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in the way of additional support (compared to just using protein evidence from say Swiss-prot) and this would be especially true if your alt_est is distantly related.  I'm not sure I really understand you alt_est/est2genome combo's to comment in more detail.  I could see four possible combinations there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP using the result of the previous pass. Then for every pass I run maker from scratch. Would you recommend to supply the gff of the previous pass in "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a genome when you want certain parts of the previous run to be passed through unchanged, but with retraining SNAP you want MAKER to re-evaluate each annotation in light of the new predictions made by the retrained SNAP.  MAKER should run really fast in all of the runs after the first one because as long as you haven't changed the evidence files it won't have to redo any of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/8b0671bd/attachment-0001.html>

From carsonhh at gmail.com  Fri Aug 31 13:03:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 15:03:14 -0400
Subject: [maker-devel] keep_preds option?
In-Reply-To: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>
Message-ID: <CC667EFB.117DD%carsonhh@gmail.com>

I concur with everything Barry said.  Also let me add that alt-ESTs do not
get polished around splice sites (exonerate won't handle them).  However
ESTs and proteins do.  This means that the utility of alt-ESTs in adding
UTR, and splice information is zero.  They just function as an anchor to
maintain gene models that might have otherwise been rejected.  This also
means alt_est=some.fasta  together with est2genome=1 will produce virtually
no additional results because est2genome requires that the splice site makes
sense.  You are better off using proteins with protein2genome=1 if you don?t
have ESTs and want partial models for training.  Once you have a trained ab
initio gene predictor, turn the est2genome and protein2genome options off.
Otherwise they will give weird partial models that decrease the quality of
your final annotations. (partial models are ok for training but that's about
it).

If you are getting too low a gene count with keep_preds=0, then you probably
need to add more evidence.  Try adding all proteins from a couple of related
species (the protein= option accepts comma separated lists of files). If
your genome is a fungi, oomycete, or a prokaryote, then setting keep_preds=1
is usually safe.  Theses are genomes with high gene density and simple gene
structure, so ab initio predictors do really well and don't need as much
help from the evidence.  For other organisms, leave it set to 0 or you will
get a lot of false positives (false positives on some genomes with complex
gene structure can outnumber the genes by 10 to 1 if you turn this on).

Thanks,
Carson




From:  Barry Moore <barry.moore at genetics.utah.edu>
Date:  Friday, 31 August, 2012 12:52 PM
To:  Christoph Hahn <chrisi.hahni at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] keep_preds option?

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly
> assembled non-model organisms draft genome. Within maker I use genemark, SNAP
> and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found
> on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical
evidence.  There are two pieces of information that are required for every
MAKER annotation (by default).  MAKER runs the ab initio gene predictors and
aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to
the genome.  For each locus where one or more gene predictions exist MAKER
checks to see if there is any physical evidence for gene expression at that
locus (RNA/protein sequence alignments) and if there is (splice EST or
protein alignments) evidence overlapping the predictions, MAKER decides
which prediction best matches the evidence and promotes that prediction to
an annotation.  If there is no evidence overlapping any of the predictions
then those predictions are not included in the output annotation file
(although they are saved).  If you set keep_preds=1 then for each locus
where prediction(s) exist maker keeps one and promotes it to an annotation
even though there is no physical evidence.  The description of
'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio
predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab
> initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different
> programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as
evidence and thus in the absence of aligned physical evidence MAKER will not
output an annotation even if all three ab initio tools predict a gene at
that locus.

> In my case I get substantial differences in the number of gene models found
> between the two settings, while with =1 I get a number that is close to what
> we would expect. How would you interpret that? What would you recommend me to
> do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is
too few, you could run MAKER with keep_preds=1.  After the run is finished,
use a tool like IPRScan to search all MAKER predictions for protein domain
content and push that IPRScan output back into the MAKER GFF file with the
ipr_update_gff script.  Then as a final step you can run over the GFF file
and remove any gene model that doesn't have either physical evidence (AED <
1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a
script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA
> sequence file in fasta format from an alternate organism). The organism is
> quite distantly related, but its the closest I have so I thought I d give it a
> shot. I ran maker twice with identical settigs expect in altest and
> est2genome=0/1. The number of genes predicted is identical with both
> approaches, so I am not sure whether or not the EST data was actually used or
> its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in
the way of additional support (compared to just using protein evidence from
say Swiss-prot) and this would be especially true if your alt_est is
distantly related.  I'm not sure I really understand you alt_est/est2genome
combo's to comment in more detail.  I could see four possible combinations
there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP
> using the result of the previous pass. Then for every pass I run maker from
> scratch. Would you recommend to supply the gff of the previous pass in
> "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a
genome when you want certain parts of the previous run to be passed through
unchanged, but with retraining SNAP you want MAKER to re-evaluate each
annotation in light of the new predictions made by the retrained SNAP.
MAKER should run really fast in all of the runs after the first one because
as long as you haven't changed the evidence files it won't have to redo any
of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your
> help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/ed670166/attachment-0001.html>

From carsonhh at gmail.com  Fri Aug 31 14:47:33 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 16:47:33 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
Message-ID: <CC66941F.118C8%carsonhh@gmail.com>

Looks like there is a slight format variation in one line of the
RepeatMasker outputs.  Not sure what could have caused it -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          0    109    (0)

The position/compliment information in columns 12-14 give a start position
of 0 for part of the alignment.  A 0 is invalid because repeat masker
doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
asks for the starting position of the alignment in another part of the
code.

This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
wu-blast and then reconfigure RepeatMasker I get this line -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          1    109    (0)

Notice the 0 becomes a 1 and all logic is restored to the world.

>From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
output, then I just turn it into a 1.
I've attached a file that you should use to replace the
./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
I'll also make a note to the RepeatMasker developers.

Thanks,
Carson



On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Attached.
>
>Thanks,
>J
>
>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send me the entire directory for just the failed contig?
>>
>> Thanks,
>> Carson
>>
>>
>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>
>>>After doing this, I still get the same errors as I attached before.
>>>There was also nothing in the log supporting that maker tried this
>>>more than once. (Initially, I had tries=2, but I changed it to
>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>count".) It also doesn't matter if I keep the same old contig
>>>directory and master log before rerunning, or if I delete them both.
>>>
>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>native version.
>>>
>>>Thanks,
>>>Jeremy
>>>
>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Let maker install it's own blast (a version I know should work).
>>>>
>>>> cd ./maker/src
>>>> ./Build blast
>>>>
>>>> Then to finish up your job (this command will just update executable
>>>> locations in the maker_exe.ctl file) -->
>>>> maker -EXE
>>>>
>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>contig
>>>> still fails.
>>>>
>>>> Thanks,
>>>> Carson
>>>>
>>>>
>>>>
>>>>
>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>
>>>>>Carson,
>>>>>
>>>>>$ blastx -version
>>>>>blastx: 2.2.26+
>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>
>>>>>- J
>>>>>
>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>wrote:
>>>>>> Could you send the blast version informations for reference
>>>>>>purposes?
>>>>>>I
>>>>>> can see this being caused by BLAST.
>>>>>>
>>>>>> Thanks,
>>>>>> Carson
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>
>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>with
>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>and
>>>>>>>used the Build script to have it install all of its own
>>>>>>>dependencies.
>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>the
>>>>>>>blast version I was using before.
>>>>>>>
>>>>>>>
>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>have
>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>first
>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>it
>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>hung
>>>>>>>> for a week until I killed it.
>>>>>>>>
>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>the
>>>>>>>> master log. maker again died on that contig.
>>>>>>>>
>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>
>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>lines
>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>needed
>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>> which takes a while.
>>>>>>>>
>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>> realize that might be hard to implement.)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jeremy
>>>>>>>> Grad student, Grishin lab
>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>> 510.384.8959
>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>.o
>>>>>>>>rg
>>>>>>>
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>maker-devel mailing list
>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>or
>>>>>>>g
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: RepeatMasker.pm
Type: text/x-perl-script
Size: 9225 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/c248081c/attachment-0001.bin>

From jeremy.semeiks at utsw.edu  Fri Aug 31 16:48:37 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 31 Aug 2012 17:48:37 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC66941F.118C8%carsonhh@gmail.com>
References: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
	<CC66941F.118C8%carsonhh@gmail.com>
Message-ID: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>

Carson, maker completed on the scaffold without error when I used your
patch. Thanks!

- J

On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Looks like there is a slight format variation in one line of the
> RepeatMasker outputs.  Not sure what could have caused it -->
> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          0    109    (0)
>
> The position/compliment information in columns 12-14 give a start position
> of 0 for part of the alignment.  A 0 is invalid because repeat masker
> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
> asks for the starting position of the alignment in another part of the
> code.
>
> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
> wu-blast and then reconfigure RepeatMasker I get this line -->
> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          1    109    (0)
>
> Notice the 0 becomes a 1 and all logic is restored to the world.
>
> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
> output, then I just turn it into a 1.
> I've attached a file that you should use to replace the
> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
> I'll also make a note to the RepeatMasker developers.
>
> Thanks,
> Carson
>
>
>
> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Attached.
>>
>>Thanks,
>>J
>>
>>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send me the entire directory for just the failed contig?
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>
>>>>After doing this, I still get the same errors as I attached before.
>>>>There was also nothing in the log supporting that maker tried this
>>>>more than once. (Initially, I had tries=2, but I changed it to
>>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>count".) It also doesn't matter if I keep the same old contig
>>>>directory and master log before rerunning, or if I delete them both.
>>>>
>>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>native version.
>>>>
>>>>Thanks,
>>>>Jeremy
>>>>
>>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>> Let maker install it's own blast (a version I know should work).
>>>>>
>>>>> cd ./maker/src
>>>>> ./Build blast
>>>>>
>>>>> Then to finish up your job (this command will just update executable
>>>>> locations in the maker_exe.ctl file) -->
>>>>> maker -EXE
>>>>>
>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>contig
>>>>> still fails.
>>>>>
>>>>> Thanks,
>>>>> Carson
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>
>>>>>>Carson,
>>>>>>
>>>>>>$ blastx -version
>>>>>>blastx: 2.2.26+
>>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>
>>>>>>- J
>>>>>>
>>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>wrote:
>>>>>>> Could you send the blast version informations for reference
>>>>>>>purposes?
>>>>>>>I
>>>>>>> can see this being caused by BLAST.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Carson
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>
>>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>with
>>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>and
>>>>>>>>used the Build script to have it install all of its own
>>>>>>>>dependencies.
>>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>>the
>>>>>>>>blast version I was using before.
>>>>>>>>
>>>>>>>>
>>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>have
>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>first
>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>it
>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>hung
>>>>>>>>> for a week until I killed it.
>>>>>>>>>
>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>the
>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>
>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>
>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>lines
>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>needed
>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>> which takes a while.
>>>>>>>>>
>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jeremy
>>>>>>>>> Grad student, Grishin lab
>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>> 510.384.8959
>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>
>>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>.o
>>>>>>>>>rg
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>maker-devel mailing list
>>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>or
>>>>>>>>g
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>



From carsonhh at gmail.com  Fri Aug 31 17:00:27 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 19:00:27 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>
Message-ID: <CC66BA4B.820D%carsonhh@gmail.com>

Your welcome.  I also submitted a bug report to RepeatMasker.  They said the
issue will go away with the next release since simple repeat detection is
going to be handled by TRF rather than rmblast.

Thanks,
Carson


On 8/31/12 6:48 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

> Carson, maker completed on the scaffold without error when I used your
> patch. Thanks!
> 
> - J
> 
> On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Looks like there is a slight format variation in one line of the
>> RepeatMasker outputs.  Not sure what could have caused it -->
>> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          0    109    (0)
>> 
>> The position/compliment information in columns 12-14 give a start position
>> of 0 for part of the alignment.  A 0 is invalid because repeat masker
>> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
>> asks for the starting position of the alignment in another part of the
>> code.
>> 
>> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
>> wu-blast and then reconfigure RepeatMasker I get this line -->
>> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          1    109    (0)
>> 
>> Notice the 0 becomes a 1 and all logic is restored to the world.
>> 
>> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
>> output, then I just turn it into a 1.
>> I've attached a file that you should use to replace the
>> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
>> I'll also make a note to the RepeatMasker developers.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>> 
>>> Attached.
>>> 
>>> Thanks,
>>> J
>>> 
>>> On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Could you send me the entire directory for just the failed contig?
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> 
>>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>> 
>>>>> After doing this, I still get the same errors as I attached before.
>>>>> There was also nothing in the log supporting that maker tried this
>>>>> more than once. (Initially, I had tries=2, but I changed it to
>>>>> tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>> count".) It also doesn't matter if I keep the same old contig
>>>>> directory and master log before rerunning, or if I delete them both.
>>>>> 
>>>>> FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>> native version.
>>>>> 
>>>>> Thanks,
>>>>> Jeremy
>>>>> 
>>>>> On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>>> Let maker install it's own blast (a version I know should work).
>>>>>> 
>>>>>> cd ./maker/src
>>>>>> ./Build blast
>>>>>> 
>>>>>> Then to finish up your job (this command will just update executable
>>>>>> locations in the maker_exe.ctl file) -->
>>>>>> maker -EXE
>>>>>> 
>>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>> contig
>>>>>> still fails.
>>>>>> 
>>>>>> Thanks,
>>>>>> Carson
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>> 
>>>>>>> Carson,
>>>>>>> 
>>>>>>> $ blastx -version
>>>>>>> blastx: 2.2.26+
>>>>>>> Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>> 
>>>>>>> - J
>>>>>>> 
>>>>>>> On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>> wrote:
>>>>>>>> Could you send the blast version informations for reference
>>>>>>>> purposes?
>>>>>>>> I
>>>>>>>> can see this being caused by BLAST.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Carson
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>> with
>>>>>>>>> maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>> and
>>>>>>>>> used the Build script to have it install all of its own
>>>>>>>>> dependencies.
>>>>>>>>> Now maker is running fine, although the culprit seems to have been
>>>>>>>>> the
>>>>>>>>> blast version I was using before.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>>> <jeremy.semeiks at utsw.edu>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>> have
>>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>> first
>>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>> it
>>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>> hung
>>>>>>>>>> for a week until I killed it.
>>>>>>>>>> 
>>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>> the
>>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>> 
>>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>> 
>>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>> lines
>>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>> needed
>>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>>> which takes a while.
>>>>>>>>>> 
>>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Jeremy
>>>>>>>>>> Grad student, Grishin lab
>>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>>> 510.384.8959
>>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>>> maker-devel mailing list
>>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>> 
>>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>> .o
>>>>>>>>>> rg
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>> or
>>>>>>>>> g
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 





From Sean.Li at csiro.au  Wed Aug  1 19:21:08 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 11:21:08 +1000
Subject: [maker-devel] The est_gff option
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/50cfa71d/attachment-0002.html>

From carsonhh at gmail.com  Wed Aug  1 19:40:09 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 21:40:09 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F4EB7.10283%carsonhh@gmail.com>

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

Thanks,
Carson



From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 9:21 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] The est_gff option

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120801/ba79d284/attachment-0002.html>

From Sean.Li at csiro.au  Wed Aug  1 20:45:10 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 12:45:10 +1000
Subject: [maker-devel] The est_gff option
In-Reply-To: <CC3F4EB7.10283%carsonhh@gmail.com>
References: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
	<CC3F4EB7.10283%carsonhh@gmail.com>
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>

Dear Carson,

Thank you for your explanation! Now I understand how the RNA-Seq data's been interpreted in MAKER.  I check the tag est_gff and only found it in a binary db file: the scaffold_56.db, which should be a SQLite file. I guess this is right? Will this file be only generated when est data's been provided or I turned the alt_splice to 1? As the db file didn't exist for my first run of Maker when no EST flags have been switched on.

Best regards,
Sean

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished because they are already aligned (so no run.log entry is generated), they are just popped into an alignment object in memory and integrated directly.  The run.log file is used primarily to track finished calls to external programs (I.e. to avoid partial results files on failure). So since there is nothing to run, there is no run.log entry.  You will just get a message saying reading GFF3 in STDERR. The GFF3 results are combined with any other results produced within maker and are then used to generate hints for gene predictors you run as to the location of introns/exons.  This is during the step where you see SNAP or augustus running again and again.  They also become part of the AED score for selecting from the pool of gene models, and can be used to add UTR to resulting gene models.  If you specify alt_splice=1 they can even be used to infer alternate splice forms.  For reference purposes, they will end up in the maker results with the tag est_gff:Cufflinks.

Thanks,
Carson



From: <Sean.Li at csiro.au<mailto:Sean.Li at csiro.au>>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: [maker-devel] The est_gff option

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



_______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com<mailto:maker-devel at box290.bluehost.com> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/d8018277/attachment-0002.html>

From carsonhh at gmail.com  Wed Aug  1 20:50:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 22:50:14 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F629F.102AC%carsonhh@gmail.com>

The file is generated any time you provide GFF3 input.  It is just a quick
way to pull out the features for a region via SQLite.  It's generated at the
start of a MAKER run.

Thanks,
Carson

From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 10:45 PM
To:  Carson Holt <carsonhh at gmail.com>, <maker-devel at yandell-lab.org>
Subject:  RE: [maker-devel] The est_gff option

Dear Carson,
 
Thank you for your explanation! Now I understand how the RNA-Seq data?s been
interpreted in MAKER.  I check the tag est_gff and only found it in a binary
db file: the scaffold_56.db, which should be a SQLite file. I guess this is
right? Will this file be only generated when est data?s been provided or I
turned the alt_splice to 1? As the db file didn?t exist for my first run of
Maker when no EST flags have been switched on.
 
Best regards,
Sean   
 

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option
 

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

 

Thanks,

Carson

 

 

 

From: <Sean.Li at csiro.au>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org>
Subject: [maker-devel] The est_gff option

 

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120801/e466dcaa/attachment-0002.html>

From alangfan at gmail.com  Wed Aug  1 23:36:24 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Thu, 2 Aug 2012 13:36:24 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
Message-ID: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
[alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
#location of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
#location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
#location of probuild executable (required for genemark)

============================================

-- 
      Alang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/03155f36/attachment-0002.html>

From carsonhh at gmail.com  Thu Aug  2 06:23:21 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 08:23:21 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
Message-ID: <CC3FE8DD.102BC%carsonhh@gmail.com>

The timing of the fault suggests either a problem with your AnyDBM_File
module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
--debug and send me the output as well as your version of maker?  Also try
updating those 3 modules on your system.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 1:36 AM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
 [alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMas
ker #location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
of probuild executable (required for genemark)

============================================

-- 
      Alang

_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/32122c14/attachment-0002.html>

From mike.thon at gmail.com  Thu Aug  2 06:48:31 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Thu, 2 Aug 2012 14:48:31 +0200
Subject: [maker-devel] segmentation fault
Message-ID: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>

I have the same problem as Fan Wen-Lang.  I get a segmentation fault when processing the fasta file.  The strange thing is, when I run maker with the -debug parameter, I don't get the segmentation fault.  I'm using maker 2.26 on ubuntu linux 64 bit.
Mike




From carsonhh at gmail.com  Thu Aug  2 07:49:47 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 09:49:47 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>
Message-ID: <CC3FFDA2.102D5%carsonhh@gmail.com>

Odd.  Have you tried updating the 3 modules I mentioned.

Thanks,
Carson



On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>processing the fasta file.  The strange thing is, when I run maker with
>the -debug parameter, I don't get the segmentation fault.  I'm using
>maker 2.26 on ubuntu linux 64 bit.
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From felix.bemm at uni-wuerzburg.de  Thu Aug  2 09:56:55 2012
From: felix.bemm at uni-wuerzburg.de (Felix Bemm)
Date: Thu, 02 Aug 2012 17:56:55 +0200
Subject: [maker-devel] segmentation fault
In-Reply-To: <CC3FFDA2.102D5%carsonhh@gmail.com>
References: <CC3FFDA2.102D5%carsonhh@gmail.com>
Message-ID: <501AA347.2080304@uni-wuerzburg.de>

This snippet will fix your problem temporarily:

cd maker/lib
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)

But do it with caution, I still do not know what problems are caused by 
this fix.

Best regards
Felix

On 02.08.2012 15:49, Carson Holt wrote:
> Odd.  Have you tried updating the 3 modules I mentioned.
>
> Thanks,
> Carson
>
>
>
> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>> processing the fasta file.  The strange thing is, when I run maker with
>> the -debug parameter, I don't get the segmentation fault.  I'm using
>> maker 2.26 on ubuntu linux 64 bit.
>> Mike
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



From carsonhh at gmail.com  Thu Aug  2 10:31:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 12:31:37 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <501AA347.2080304@uni-wuerzburg.de>
Message-ID: <CC40233B.102F6%carsonhh@gmail.com>

Thanks Felix.

You may also have to run the fix below on wherever Bioperl is installed as
well, because it contains the same (DB_File GDBM_File NDBM_File SDBM_File)
statement in the Bio::DB::Fasta module in one of the BEGIN statements.

Thanks,
Carson



On 12-08-02 11:56 AM, "Felix Bemm" <felix.bemm at uni-wuerzburg.de> wrote:

>This snippet will fix your problem temporarily:
>
>cd maker/lib
>sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>
>But do it with caution, I still do not know what problems are caused by
>this fix.
>
>Best regards
>Felix
>
>On 02.08.2012 15:49, Carson Holt wrote:
>> Odd.  Have you tried updating the 3 modules I mentioned.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault
>>>when
>>> processing the fasta file.  The strange thing is, when I run maker with
>>> the -debug parameter, I don't get the segmentation fault.  I'm using
>>> maker 2.26 on ubuntu linux 64 bit.
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From kippjohnson at uchicago.edu  Thu Aug  2 13:12:15 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Thu, 2 Aug 2012 14:12:15 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
Message-ID: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The
first time I ran maker, I ignored these and the maker run seemingly
proceeded to the process the contigs correctly. Is this something I should
be worried about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for
each restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/70c4c837/attachment-0002.html>

From alangfan at gmail.com  Thu Aug  2 18:45:17 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Fri, 3 Aug 2012 08:45:17 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CC3FE8DD.102BC%carsonhh@gmail.com>
References: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
	<CC3FE8DD.102BC%carsonhh@gmail.com>
Message-ID: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>

Thanks Carson.

I'm using a single server with centOS6 64 bit.
==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep
-l 'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>

> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
>
> Thanks,
> Carson
>
>
>
> From: "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date: Thursday, 2 August, 2012 1:36 AM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
>
> Dear All,
>
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
>
> Best,
>
> -------command and error message-------------------------------------
> [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
>
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
>
> #-----Protein Homology Evidence
> #protein=protein.fasta
>
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
>
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
> of NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
> of NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
> #location of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
> #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
>
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
> of eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
>
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
> #location of probuild executable (required for genemark)
>
> ============================================
>
> --
>       Alang
>
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/36149b1d/attachment-0002.html>

From carsonhh at gmail.com  Thu Aug  2 20:07:53 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 22:07:53 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>
Message-ID: <CC40AA6F.10327%carsonhh@gmail.com>

Ru with the ?f flag before rerunning in the same directory as your previous
error.  Also collect the STDERR both with and without the -debug flag and
send it to me.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 8:45 PM
To:  Carson Holt <carsonhh at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] Question: maker Segmentation fault (core dumped)

Thanks Carson.

I'm using a single server with centOS6 64 bit.
============================================================================
==
STATUS MAKER 2.26
============================================================================
==
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------
------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep -l
'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>
> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
> 
> Thanks,
> Carson
> 
> 
> 
> From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date:  Thursday, 2 August, 2012 1:36 AM
> To:  <maker-devel at yandell-lab.org>
> Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)
> 
> Dear All,
> 
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
> 
> Best,
> 
> -------command and error message-------------------------------------
>  [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
> 
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
> 
> #-----Protein Homology Evidence
> #protein=protein.fasta
> 
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
> 
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location of
> NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location of
> NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
> of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMaske
> r #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
> 
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location of
> eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
> 
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
> of probuild executable (required for genemark)
> 
> ============================================
> 
> -- 
>       Alang
> 
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak
> er-devel_yandell-lab.org



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------
------------------------------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/9a29692b/attachment-0002.html>

From mike.thon at gmail.com  Thu Aug  2 22:30:15 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 06:30:15 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been started
	in the same directory.
Message-ID: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>

I get this warning when running the mpi version of maker.  Should I be concerned?  the command I use is this:

 mpiexec -n 8 maker

-Mike




From carsonhh at gmail.com  Thu Aug  2 23:11:39 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 01:11:39 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>
Message-ID: <CC40CC6D.10338%carsonhh@gmail.com>

I'm sorry I'm confused.  Which warning? the segmentation fault you
mentioned before?  For now I would say it's best to first try maker once
outside of MPI to verify that the segmentation fault no longer happens
before moving onto the MPI environment.

Make sure to run with the -f flag when retrying after the fix suggested by
Felix, so maker completely restarts from scratch.  That way you can see
that the fasta database is being generated from start to finish (and is
not just being reread from the successful run under the --debug flag).

Make sure that the fix Felix sent is also applied to the Bioperl
Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
fault will not go away otherwise (if it is infact one of perl's DB modules
causing the issue, which it likely is).  Your error sounds similar to an
error Felix and Thomas encountered some time ago
(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
h/000941.html).  Basically one of perl's native DB modules or the database
the module relies on are broken, so the command line snippet is cutting
any calls to those other databases out of maker and Bioperl (which should
be ok if you have at minimum a working Berkley DB and DB_File module on
your system).

Use this command line to get the correct directory for applying the
changes to Bioperl as well -->
perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
s/[^\/]+$//; print "$dir\n"'

Then copy the directory location-->

cd <to_that_directory>
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)



Finally run maker with the -f flag.  If it works let me know, and then you
can give MPI a shot.  Make said yes to the "compile for MPI question"
asked during the maker install process or mpiexec will fail when running
maker.  If it doesn?t work without MPI or if it works and then MPI fails,
make sure to collect the STDERR in a text file and send it to me.  I will
probably have you run it with the --debug flag as well following any
initial failure (just to get the extra status messages about your system
configuration).

Thanks,
Carson



On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I get this warning when running the mpi version of maker.  Should I be
>concerned?  the command I use is this:
>
> mpiexec -n 8 maker
>
>-Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From mike.thon at gmail.com  Fri Aug  3 01:56:34 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 09:56:34 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>

I meant the warning that is in the subject line of my first email:

WARNING: Multiple MAKER processes have been started in the same directory.

I get that warning when I run mpi maker.  For now, I'm running it without mpi.  When the run finishes I can try it again with mpi and make sure I get the same output.

For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.

best
Mike

On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From carsonhh at gmail.com  Fri Aug  3 06:37:24 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 08:37:24 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC413D27.1038C%carsonhh@gmail.com>

That's good to know the seg fault is gone.  Sorry for the confusion, but I
don't seem to have received your other e-mail about the "multiple process
warning".

Could you run '/maker/src/Build status' and send me the output?

Thanks,
Carson

On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 07:11:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:11:03 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC414316.103A1%carsonhh@gmail.com>

One more thing.

There are three locations where this exact line of code are called
--> @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
run_log
ds_utility
Bio::Fasta::DB

Because of the way perl works only the last one to run needs to be fixed
to stop the error (because it overrides the other two calls), but that
means if the line is ever removed from the last loaded module or load
order changes, then the error will come back because it is the last one to
run that fixes the issue.  So to be safe you should fix it in all
instances.

Currently ds_utility loads last, but it really doesn't need AnyDBM_File
any more to perform it's function and the call will likely be removed in
future releases, so changing the line in Bio::Fasta::DB should avoid
potential future errors on your system.

Thanks,
Carson



On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 07:27:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:27:37 -0400
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
Message-ID: <CC414867.103D1%carsonhh@gmail.com>

It means that your GFF3 results will not be integrated.  Does it happen if
you start in a different directory?

The lock error is coming from SQLite, and may be the result of your working
directory being mounted on an NFS filesystem which is a known issue with
SQLite.  If you give maker a location for TMP that is not NFS mounted (I.e.
A locally mounted filesystem), then maker will try and move the SQLite
database off of the NFS mount.

You can check the mount status of the current working directory and the
directory you choose fro TMP by using the Linux df command.

df <directory>

Any NFS mount will follow the format of host:/location.

Example:
152.234.123.234:/data/research

Or 

storage.utah.edu:/data/research

If you are not running on an NFS device let me know if running in a
different directory results in the same error.

Thanks,
Carson

From:  Kipp Johnson <kippjohnson at uchicago.edu>
Date:  Thursday, 2 August, 2012 3:12 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The first
time I ran maker, I ignored these and the maker run seemingly proceeded to
the process the contigs correctly. Is this something I should be worried
about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for each
restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/5517b922/attachment-0002.html>

From cjfields at illinois.edu  Fri Aug  3 07:51:04 2012
From: cjfields at illinois.edu (Fields, Christopher J)
Date: Fri, 3 Aug 2012 13:51:04 +0000
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>

I just want to add in, if there are any fixes that need to be made to Bioperl let me know.  We're trying to move towards an AnyDBM_File approach, but I think this effort has stalled.

chris

On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
 wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug  3 08:28:44 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 10:28:44 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>
Message-ID: <CC414FC9.1040E%carsonhh@gmail.com>

I think I've finally figured it out, as well as a fix to quash this error
for good.  It not strictly a Bioperl issue.  It's an an issue with
AnyDBM_File as well as it's documentation (as it recommends setting
@AnyDBM_File::ISA in code that uses it but never mentions potential
caveats of multiple modules using it).

Basically if you set @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File
SDBM_File) and then call 'use AnyDBM_File', the AnyDBM_File module will
actually preprocess and trim @AnyDBM_File::ISA using this code -->

my $mod;
for $mod (@ISA) {
	if (eval "require $mod") {
		@ISA = ($mod);  # if we leave @ISA alone, warnings abound
		return 1;
	}
}


However if you load any other module that also sets @AnyDBM_File::ISA that
code doesn?t get called on the next 'use AnyDBM_File'.  So you will loose
the preprocessing and basically mess up @AnyDBM_File::ISA if you ever have
two modules loaded at the same time that use AnyDBM_File.

So loading Bio::DB::Fasta before or after a module that also uses
AnyDBM_File has the potential to result in errors.

Example:
use Bio::DB::Fasta;
print join(',', @AnyDBM_File::ISA );

Result --> DB_File


use Bio::DB::Qual;

use Bio::DB::Fasta;

print join(',', @AnyDBM_File::ISA );


Result --> DB_File,GDBM_File,NDBM_File,SDBM_File


The second module messes up @AnyDBM_File::ISA.  It also happens if you
load them in reverse.
Perhaps changing the BEGIN block in Bio::DB::Fasta, Bio::DB::Qual, and any
other any other code using AnyDBM_File to include one of these alternate
BEGIN blocks.

BEGIN {
	@AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
if(!$INC{'AnyDBM_File.pm'});
}


Or 

BEGIN {
	for my $mod (qw(DB_File GDBM_File NDBM_File SDBM_File)) {
		if (eval "require $mod") {
			@AnyDBM_File::ISA = ($mod);
			last;
		}
	}

}


As far as maker goes, I've actually removed all use of AnyDBM_File from
it's modules now, so it won't be an issue there any more, so Bioperl is
the only one left to fix.

Thanks,
Carson



On 12-08-03 9:51 AM, "Fields, Christopher J" <cjfields at illinois.edu> wrote:

>I just want to add in, if there are any fixes that need to be made to
>Bioperl let me know.  We're trying to move towards an AnyDBM_File
>approach, but I think this effort has stalled.
>
>chris
>
>On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>





From kippjohnson at uchicago.edu  Fri Aug  3 10:16:33 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Fri, 3 Aug 2012 11:16:33 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CC414867.103D1%carsonhh@gmail.com>
References: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
	<CC414867.103D1%carsonhh@gmail.com>
Message-ID: <CAGJuV2XTG0GWGi9Dbuk7xdT=OOJodWhbUORsj9XcZJwdz3PYYw@mail.gmail.com>

It does happen if I start in a different directory. My current working
directory is NFS. I changed the tmp directory in maker_opts from the NFS to
a local partition (tmpfs), and I'm still getting this error. Does the
working directory also have to be locally mounted?

On Fri, Aug 3, 2012 at 8:27 AM, Carson Holt <carsonhh at gmail.com> wrote:

> It means that your GFF3 results will not be integrated.  Does it happen if
> you start in a different directory?
>
> The lock error is coming from SQLite, and may be the result of your
> working directory being mounted on an NFS filesystem which is a known issue
> with SQLite.  If you give maker a location for TMP that is not NFS mounted
> (I.e. A locally mounted filesystem), then maker will try and move the
> SQLite database off of the NFS mount.
>
> You can check the mount status of the current working directory and the
> directory you choose fro TMP by using the Linux df command.
>
> df <directory>
>
> Any NFS mount will follow the format of host:/location.
>
> Example:
> 152.234.123.234:/data/research
>
> Or
>
> storage.utah.edu:/data/research
>
> If you are not running on an NFS device let me know if running in a
> different directory results in the same error.
>
> Thanks,
> Carson
>
> From: Kipp Johnson <kippjohnson at uchicago.edu>
> Date: Thursday, 2 August, 2012 3:12 PM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
>
> Hi,
>
>      I am running Maker version 2.26. I have two questions:
>
> 1)
>
>     When I run maker, I get errors like the following printed to STDOUT:
>
> DBD::SQLite::db selectcol_arrayref failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.
>
>      I reinstalled the indicated module via cpanm and restarted Maker from
> the beginning in a new directory, and I am getting the same error. The
> first time I ran maker, I ignored these and the maker run seemingly
> proceeded to the process the contigs correctly. Is this something I should
> be worried about?
>
>
> 2)
>
>     I am running Maker on a single server with 64 processors. Would it be
> faster to use MPICH2, or to simply restart Maker in the same directory ~16
> times, with the # of CPUs option that gets passed to Blast set to 4 for
> each restart?
>
>     I believe that I have MPICH2 correctly compiled and installed; if it
> would be faster to use MPICH2, could someone provide me with a sample
> command line entry to use for Maker? I am confused about how to run MPICH2
> on a single node (without a hostfile?), and searching the maker-dev-list
> archive seems to give me results for older versions of MAKER and MPICH2.
>
> Thank you,
>
>
> Best,
>
>      Kipp Johnson
>      kippjohnson at uchicago.edu
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/b5f2b0cd/attachment-0002.html>

From mike.thon at gmail.com  Fri Aug  3 23:48:52 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 07:48:52 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC413D27.1038C%carsonhh@gmail.com>
References: <CC413D27.1038C%carsonhh@gmail.com>
Message-ID: <E44B4BF9-7D27-4B11-B33F-42A975222F99@gmail.com>

./Build status


==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:	VERIFIED
External Programs:	VERIFIED
External C Libraries:	VERIFIED
MPI SUPPORT:		ENABLED
MWAS Web Interface:	DISABLED
MAKER PACKAGE:		CONFIGURATION OK

...

I was impatient and I ran maker with MPI.  In the master log file I see a lot of failed contigs.  Now I'm running it without MPI to see if those contigs fail again. 

On Aug 3, 2012, at 2:37 PM, Carson Holt <carsonhh at gmail.com> wrote:

> That's good to know the seg fault is gone.  Sorry for the confusion, but I
> don't seem to have received your other e-mail about the "multiple process
> warning".
> 
> Could you run '/maker/src/Build status' and send me the output?
> 
> Thanks,
> Carson
> 
> On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I meant the warning that is in the subject line of my first email:
>> 
>> WARNING: Multiple MAKER processes have been started in the same directory.
>> 
>> I get that warning when I run mpi maker.  For now, I'm running it without
>> mpi.  When the run finishes I can try it again with mpi and make sure I
>> get the same output.
>> 
>> For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>> to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>> 
>> best
>> Mike
>> 
>> On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>>> mentioned before?  For now I would say it's best to first try maker once
>>> outside of MPI to verify that the segmentation fault no longer happens
>>> before moving onto the MPI environment.
>>> 
>>> Make sure to run with the -f flag when retrying after the fix suggested
>>> by
>>> Felix, so maker completely restarts from scratch.  That way you can see
>>> that the fasta database is being generated from start to finish (and is
>>> not just being reread from the successful run under the --debug flag).
>>> 
>>> Make sure that the fix Felix sent is also applied to the Bioperl
>>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>>> fault will not go away otherwise (if it is infact one of perl's DB
>>> modules
>>> causing the issue, which it likely is).  Your error sounds similar to an
>>> error Felix and Thomas encountered some time ago
>>> 
>>> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>> rc
>>> h/000941.html).  Basically one of perl's native DB modules or the
>>> database
>>> the module relies on are broken, so the command line snippet is cutting
>>> any calls to those other databases out of maker and Bioperl (which
>>> should
>>> be ok if you have at minimum a working Berkley DB and DB_File module on
>>> your system).
>>> 
>>> Use this command line to get the correct directory for applying the
>>> changes to Bioperl as well -->
>>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>>> s/[^\/]+$//; print "$dir\n"'
>>> 
>>> Then copy the directory location-->
>>> 
>>> cd <to_that_directory>
>>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>> $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>>> 
>>> 
>>> 
>>> Finally run maker with the -f flag.  If it works let me know, and then
>>> you
>>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>>> asked during the maker install process or mpiexec will fail when running
>>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>> fails,
>>> make sure to collect the STDERR in a text file and send it to me.  I
>>> will
>>> probably have you run it with the --debug flag as well following any
>>> initial failure (just to get the extra status messages about your system
>>> configuration).
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> 
>>> 
>>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>> 
>>>> I get this warning when running the mpi version of maker.  Should I be
>>>> concerned?  the command I use is this:
>>>> 
>>>> mpiexec -n 8 maker
>>>> 
>>>> -Mike
>>>> 
>>>> 
>>>> _______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From mike.thon at gmail.com  Sat Aug  4 05:51:04 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 13:51:04 +0200
Subject: [maker-devel] failed contigs
Message-ID: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>

running maker 2.26 some contigs seem to consistently fail to run, according to XXX_master_datastore_index.log

Here are the last few lines of run.log for one of the failed contigs:

CTL_OPTIONS     run     
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
DIED    RANK    0
DIED    COUNT   1


the contents of run.log.child.1 for the same contig:
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
FINISHED        cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.te_proteins%2Efasta.repeatrunner  
DIED    RANK    0:2:0:1
DIED    COUNT   1

So perhaps its a problem with the repeat masking step.  I already have annotated repeats using RepeatMasker and I have the gff2 file produced by RepeatMasker.  Do I need to convert this to gff3 before adding to to my maker_opts file?  Better yet, does anyone know why these contigs are failing and how to fix it?
Mike




From carsonhh at gmail.com  Sat Aug  4 07:44:25 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Sat, 04 Aug 2012 09:44:25 -0400
Subject: [maker-devel] failed contigs
In-Reply-To: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>
Message-ID: <CC429E5D.104F1%carsonhh@gmail.com>

Do you have STDERR log stored somewhere?  Could you send it to me?
Sometimes the error causing the failure is upstream in the log and then
cascades into other errors.  Also if it is the repeatrunner step failing,
the log will have the command to run the BLAST step outside of MAKER (is
helpful for debugging).  If you don't have the captured STDERR, could you
run again and send it to me.

Thanks,
Carson

On 12-08-04 7:51 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>running maker 2.26 some contigs seem to consistently fail to run,
>according to XXX_master_datastore_index.log
>
>Here are the last few lines of run.log for one of the failed contigs:
>
>CTL_OPTIONS     run
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>DIED    RANK    0
>DIED    COUNT   1
>
>
>the contents of run.log.child.1 for the same contig:
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>FINISHED        
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.te_proteins%2Efasta.repeatrunner
>DIED    RANK    0:2:0:1
>DIED    COUNT   1
>
>So perhaps its a problem with the repeat masking step.  I already have
>annotated repeats using RepeatMasker and I have the gff2 file produced by
>RepeatMasker.  Do I need to convert this to gff3 before adding to to my
>maker_opts file?  Better yet, does anyone know why these contigs are
>failing and how to fix it?
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From barry.moore at genetics.utah.edu  Fri Aug 17 19:05:00 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 17 Aug 2012 19:05:00 -0600
Subject: [maker-devel] Exonerate web access is back
References: <201208160913.q7G9DTbB005179@rabbit.ebi.ac.uk>
Message-ID: <C52C60DC-1052-4FD7-9CAC-2F4830E78827@genetics.utah.edu>



Begin forwarded message:

> From: <datasubs at ebi.ac.uk>
> Subject: SUP/EMBL/ 24308 moved from EBIHELP by apc (rasko) (SUB#818751)
> Date: August 16, 2012 3:13:29 AM MDT
> To: <barry.moore at genetics.utah.edu>
> 
> Dear Barry,
> 
> Access to http://www.ebi.ac.uk/~guy/exonerate/ has been restored.
> 
> Kind Regards,
> Rasko Leinonen 
> European Nucleotide Archive (ENA)
> 

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120817/0a68e03e/attachment-0002.html>

From kippjohnson at uchicago.edu  Tue Aug 21 15:25:49 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Tue, 21 Aug 2012 16:25:49 -0500
Subject: [maker-devel] Maker Re-tries
In-Reply-To: <6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
References: <CC414867.103D1%carsonhh@gmail.com>
	<6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
Message-ID: <1E417039-68CF-4EFE-90BF-964DC86B1864@uchicago.edu>

Hi Carson,

	Simple question: If I start maker and then kill the job before a thread finishes processing a contig, does this failure to complete the contig count in the number of tries maker will try to complete the contig before skipping to another?	

	Basically, does the "tries=2 #number of times to try a contig if there is a failure for some reason" parameter take into account failures resulting from me killing the job?	


	Best,

		Kipp Johnson
		kippjohnson at uchicago.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120821/74551a9f/attachment-0002.html>

From jeremy.semeiks at utsw.edu  Tue Aug 21 10:26:32 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Tue, 21 Aug 2012 11:26:32 -0500
Subject: [maker-devel] maker dying on a contig
Message-ID: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>

Hi,

I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
~36,000 contigs, and per the master log maker FINISHED all of them
except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
run. It didn't report ERROR for that contig in the master log, but it
did output two DIED lines in the contig's run.log and then just hung
for a week until I killed it.

I reran maker on just that contig by deleting the START line from the
master log. maker again died on that contig.

I attach the relevant part of the stderr stream; the rest was just
"--Next Contig--"-type lines. Any ideas?

A related question. Is deleting lines in the master log, as I did
here, the best way to get maker to rerun on just a few contigs? I
often find myself needing to do this, sometimes because of ERROR lines
or other assumed quirks in maker and sometimes just because I needed
to pause a big run for a while. One disadvantage is that when I
restart, maker needs to manually scan over many completed contigs,
which takes a while.

(I'd think the best solution would be for maker to automatically
recognize whether it's really currently running on contigs marked
STARTED but not FINISHED. It doesn't seem to do this now, but I
realize that might be hard to implement.)

Thanks,
Jeremy
Grad student, Grishin lab
UT Southwestern, Dallas TX
510.384.8959
-------------- next part --------------
A non-text attachment was scrubbed...
Name: maker.stderr
Type: application/octet-stream
Size: 11422 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120821/7f29efb2/attachment-0002.obj>

From mike.thon at gmail.com  Fri Aug 24 01:44:13 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 24 Aug 2012 09:44:13 +0200
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
Message-ID: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>

I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.


On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:

> Hi,
> 
> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
> ~36,000 contigs, and per the master log maker FINISHED all of them
> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
> run. It didn't report ERROR for that contig in the master log, but it
> did output two DIED lines in the contig's run.log and then just hung
> for a week until I killed it.
> 
> I reran maker on just that contig by deleting the START line from the
> master log. maker again died on that contig.
> 
> I attach the relevant part of the stderr stream; the rest was just
> "--Next Contig--"-type lines. Any ideas?
> 
> A related question. Is deleting lines in the master log, as I did
> here, the best way to get maker to rerun on just a few contigs? I
> often find myself needing to do this, sometimes because of ERROR lines
> or other assumed quirks in maker and sometimes just because I needed
> to pause a big run for a while. One disadvantage is that when I
> restart, maker needs to manually scan over many completed contigs,
> which takes a while.
> 
> (I'd think the best solution would be for maker to automatically
> recognize whether it's really currently running on contigs marked
> STARTED but not FINISHED. It doesn't seem to do this now, but I
> realize that might be hard to implement.)
> 
> Thanks,
> Jeremy
> Grad student, Grishin lab
> UT Southwestern, Dallas TX
> 510.384.8959
> <maker.stderr>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From kmegy at ebi.ac.uk  Fri Aug 24 02:38:32 2012
From: kmegy at ebi.ac.uk (Karyn Megy)
Date: Fri, 24 Aug 2012 09:38:32 +0100
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
	<BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <463EA6A1-58CC-4358-9812-1B659DED9F13@ebi.ac.uk>

Hi Jeremy, 

Could it be a contig with lots of repeats, or that would hit lots of proteins?

One of my colleague had this issue and he ended splitting the few problematic contigs in sections... but then you have to reconciliate the results because the gene/BLAST coordinates are based in the bits of sub-contigs rather than the initial contig.  

You could try running MAKER on half of this contig and see what happen.

Cheers,
Karyn.

On 24 Aug 2012, at 08:44, Michael Thon wrote:

> I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.
> 
> 
> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:
> 
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug 24 06:24:56 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 08:24:56 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <CC5CEA43.11379%carsonhh@gmail.com>

Could you send the blast version informations for reference purposes? I
can see this being caused by BLAST.

Thanks,
Carson



On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I'm not sure if this is relevant to your problem but I had problems with
>maker failing on some contigs but not others.  I reinstalled maker and
>used the Build script to have it install all of its own dependencies.
>Now maker is running fine, although the culprit seems to have been the
>blast version I was using before.
>
>
>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>wrote:
>
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From jeremy.semeiks at utsw.edu  Fri Aug 24 13:13:39 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 14:13:39 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC5CEA43.11379%carsonhh@gmail.com>
References: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
	<CC5CEA43.11379%carsonhh@gmail.com>
Message-ID: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>

Carson,

$ blastx -version
blastx: 2.2.26+
Package: blast 2.2.26, build Feb 22 2012 16:04:28

- J

On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
> Could you send the blast version informations for reference purposes? I
> can see this being caused by BLAST.
>
> Thanks,
> Carson
>
>
>
> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>>I'm not sure if this is relevant to your problem but I had problems with
>>maker failing on some contigs but not others.  I reinstalled maker and
>>used the Build script to have it install all of its own dependencies.
>>Now maker is running fine, although the culprit seems to have been the
>>blast version I was using before.
>>
>>
>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>> run. It didn't report ERROR for that contig in the master log, but it
>>> did output two DIED lines in the contig's run.log and then just hung
>>> for a week until I killed it.
>>>
>>> I reran maker on just that contig by deleting the START line from the
>>> master log. maker again died on that contig.
>>>
>>> I attach the relevant part of the stderr stream; the rest was just
>>> "--Next Contig--"-type lines. Any ideas?
>>>
>>> A related question. Is deleting lines in the master log, as I did
>>> here, the best way to get maker to rerun on just a few contigs? I
>>> often find myself needing to do this, sometimes because of ERROR lines
>>> or other assumed quirks in maker and sometimes just because I needed
>>> to pause a big run for a while. One disadvantage is that when I
>>> restart, maker needs to manually scan over many completed contigs,
>>> which takes a while.
>>>
>>> (I'd think the best solution would be for maker to automatically
>>> recognize whether it's really currently running on contigs marked
>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>> realize that might be hard to implement.)
>>>
>>> Thanks,
>>> Jeremy
>>> Grad student, Grishin lab
>>> UT Southwestern, Dallas TX
>>> 510.384.8959
>>> <maker.stderr>_______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>_______________________________________________
>>maker-devel mailing list
>>maker-devel at box290.bluehost.com
>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>



From carsonhh at gmail.com  Fri Aug 24 13:17:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 15:17:03 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
Message-ID: <CC5D4ADD.11459%carsonhh@gmail.com>

Let maker install it's own blast (a version I know should work).

cd ./maker/src
./Build blast

Then to finish up your job (this command will just update executable
locations in the maker_exe.ctl file) -->
maker -EXE

Then run maker again with a higher retry count.  Let me know if the contig
still fails.

Thanks,
Carson




On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Carson,
>
>$ blastx -version
>blastx: 2.2.26+
>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>
>- J
>
>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send the blast version informations for reference purposes? I
>> can see this being caused by BLAST.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>>I'm not sure if this is relevant to your problem but I had problems with
>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>used the Build script to have it install all of its own dependencies.
>>>Now maker is running fine, although the culprit seems to have been the
>>>blast version I was using before.
>>>
>>>
>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>> did output two DIED lines in the contig's run.log and then just hung
>>>> for a week until I killed it.
>>>>
>>>> I reran maker on just that contig by deleting the START line from the
>>>> master log. maker again died on that contig.
>>>>
>>>> I attach the relevant part of the stderr stream; the rest was just
>>>> "--Next Contig--"-type lines. Any ideas?
>>>>
>>>> A related question. Is deleting lines in the master log, as I did
>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>> or other assumed quirks in maker and sometimes just because I needed
>>>> to pause a big run for a while. One disadvantage is that when I
>>>> restart, maker needs to manually scan over many completed contigs,
>>>> which takes a while.
>>>>
>>>> (I'd think the best solution would be for maker to automatically
>>>> recognize whether it's really currently running on contigs marked
>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>> realize that might be hard to implement.)
>>>>
>>>> Thanks,
>>>> Jeremy
>>>> Grad student, Grishin lab
>>>> UT Southwestern, Dallas TX
>>>> 510.384.8959
>>>> <maker.stderr>_______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> 
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>>>_______________________________________________
>>>maker-devel mailing list
>>>maker-devel at box290.bluehost.com
>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>





From jeremy.semeiks at utsw.edu  Fri Aug 24 15:11:10 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 16:11:10 -0500
Subject: [maker-devel]  maker dying on a contig
In-Reply-To: <CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
References: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
	<CC5D4ADD.11459%carsonhh@gmail.com>
	<CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
Message-ID: <CABPMbe3nSDGzf_VqDj1Nucd0DPuPUB_tKcqQj64sV-2SyuUAQQ@mail.gmail.com>

After doing this, I still get the same errors as I attached before.
There was also nothing in the log supporting that maker tried this
more than once. (Initially, I had tries=2, but I changed it to
tries=10 for this run. I'm guessing that's what you mean by "retry
count".) It also doesn't matter if I keep the same old contig
directory and master log before rerunning, or if I delete them both.

FWIW, maker's own version of blast is 2.2.26+, which is the same as my
native version.

Thanks,
Jeremy

On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Let maker install it's own blast (a version I know should work).
>
> cd ./maker/src
> ./Build blast
>
> Then to finish up your job (this command will just update executable
> locations in the maker_exe.ctl file) -->
> maker -EXE
>
> Then run maker again with a higher retry count.  Let me know if the contig
> still fails.
>
> Thanks,
> Carson
>
>
>
>
> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Carson,
>>
>>$ blastx -version
>>blastx: 2.2.26+
>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>
>>- J
>>
>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send the blast version informations for reference purposes? I
>>> can see this being caused by BLAST.
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>>
>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>
>>>>I'm not sure if this is relevant to your problem but I had problems with
>>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>>used the Build script to have it install all of its own dependencies.
>>>>Now maker is running fine, although the culprit seems to have been the
>>>>blast version I was using before.
>>>>
>>>>
>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>>> did output two DIED lines in the contig's run.log and then just hung
>>>>> for a week until I killed it.
>>>>>
>>>>> I reran maker on just that contig by deleting the START line from the
>>>>> master log. maker again died on that contig.
>>>>>
>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>
>>>>> A related question. Is deleting lines in the master log, as I did
>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>>> or other assumed quirks in maker and sometimes just because I needed
>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>> which takes a while.
>>>>>
>>>>> (I'd think the best solution would be for maker to automatically
>>>>> recognize whether it's really currently running on contigs marked
>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>> realize that might be hard to implement.)
>>>>>
>>>>> Thanks,
>>>>> Jeremy
>>>>> Grad student, Grishin lab
>>>>> UT Southwestern, Dallas TX
>>>>> 510.384.8959
>>>>> <maker.stderr>_______________________________________________
>>>>> maker-devel mailing list
>>>>> maker-devel at box290.bluehost.com
>>>>>
>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>>
>>>>
>>>>_______________________________________________
>>>>maker-devel mailing list
>>>>maker-devel at box290.bluehost.com
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>
>



From jokelley at stanford.edu  Thu Aug 30 18:18:29 2012
From: jokelley at stanford.edu (Joanna Kelley)
Date: Thu, 30 Aug 2012 17:18:29 -0700
Subject: [maker-devel] convert gff to ncbi table format?
Message-ID: <CA+77YqHuS8CUK+YmDCQX8KMt1FWiMkJ00eXsgwdzHTM1PxsZ1g@mail.gmail.com>

Hello,

I am trying to submit the maker annotations to NCBI and
I was wondering if there are any tools to convert gff to
the NCBI table format for submission?

Thank you,
Joanna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120830/9a5ce895/attachment-0002.html>

From chrisi.hahni at gmail.com  Fri Aug 31 06:43:21 2012
From: chrisi.hahni at gmail.com (Christoph Hahn)
Date: Fri, 31 Aug 2012 14:43:21 +0200
Subject: [maker-devel] keep_preds option?
Message-ID: <5040B169.1030909@gmail.com>

Hello maker users and developers,

I am new to gene prediction and I am trying to use maker 2.25 on a newly 
assembled non-model organisms draft genome. Within maker I use genemark, 
SNAP and Augustus. I have a few questions:

1. I was wondering what the keep_preds option means exactly.

I found two slightly different explanations on the option
#Add unsupported gene prediction to final annotation set (maker2.25)
#Add non-overlapping ab-inito gene prediction to final annotation set 
(found on the net - probably older maker version)

As far as I understood keep_preds=0 only retains gene models for which 
the ab initio predictions agree. But how many, all three? two of three?
keep_preds=1 instead keeps all gene models regardless if the different 
programs agree, right?

In my case I get substantial differences in the number of gene models 
found between the two settings, while with =1 I get a number that is 
close to what we would expect. How would you interpret that? What would 
you recommend me to do? Obiously =0 is the saver option.

2. I tried to use EST data of an alternative organism in altest= 
(#EST/cDNA sequence file in fasta format from an alternate organism). 
The organism is quite distantly related, but its the closest I have so I 
thought I d give it a shot. I ran maker twice with identical settigs 
expect in altest and est2genome=0/1. The number of genes predicted is 
identical with both approaches, so I am not sure whether or not the EST 
data was actually used or its just to distantly related. Any easy way to 
assess this?

3. I am running maker in several passes and after each pass I am 
training SNAP using the result of the previous pass. Then for every pass 
I run maker from scratch. Would you recommend to supply the gff of the 
previous pass in "#-----Re-annotation Using MAKER Derived GFF3
maker_gff= #re-annotate genome based on this gff3 file", instead?

Thanks in advance for any thoughts/advice on these things! I appreciate 
your help!

much obliged,
Christoph



From barry.moore at genetics.utah.edu  Fri Aug 31 10:52:10 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 31 Aug 2012 10:52:10 -0600
Subject: [maker-devel] keep_preds option?
In-Reply-To: <5040B169.1030909@gmail.com>
References: <5040B169.1030909@gmail.com>
Message-ID: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly assembled non-model organisms draft genome. Within maker I use genemark, SNAP and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical evidence.  There are two pieces of information that are required for every MAKER annotation (by default).  MAKER runs the ab initio gene predictors and aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to the genome.  For each locus where one or more gene predictions exist MAKER checks to see if there is any physical evidence for gene expression at that locus (RNA/protein sequence alignments) and if there is (splice EST or protein alignments) evidence overlapping the predictions, MAKER decides which prediction best matches the evidence and promotes that prediction to an annotation.  If there is no evidence overlapping any of the predictions then those predictions are not included in the output annotation file (although they are saved).  If you set keep_preds=1 then for each locus where prediction(s) exist maker keeps one and promotes it to an annotation even though there is no physical evidence.  The description of 'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as evidence and thus in the absence of aligned physical evidence MAKER will not output an annotation even if all three ab initio tools predict a gene at that locus.

> In my case I get substantial differences in the number of gene models found between the two settings, while with =1 I get a number that is close to what we would expect. How would you interpret that? What would you recommend me to do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is too few, you could run MAKER with keep_preds=1.  After the run is finished, use a tool like IPRScan to search all MAKER predictions for protein domain content and push that IPRScan output back into the MAKER GFF file with the ipr_update_gff script.  Then as a final step you can run over the GFF file and remove any gene model that doesn't have either physical evidence (AED < 1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA sequence file in fasta format from an alternate organism). The organism is quite distantly related, but its the closest I have so I thought I d give it a shot. I ran maker twice with identical settigs expect in altest and est2genome=0/1. The number of genes predicted is identical with both approaches, so I am not sure whether or not the EST data was actually used or its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in the way of additional support (compared to just using protein evidence from say Swiss-prot) and this would be especially true if your alt_est is distantly related.  I'm not sure I really understand you alt_est/est2genome combo's to comment in more detail.  I could see four possible combinations there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP using the result of the previous pass. Then for every pass I run maker from scratch. Would you recommend to supply the gff of the previous pass in "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a genome when you want certain parts of the previous run to be passed through unchanged, but with retraining SNAP you want MAKER to re-evaluate each annotation in light of the new predictions made by the retrained SNAP.  MAKER should run really fast in all of the runs after the first one because as long as you haven't changed the evidence files it won't have to redo any of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/8b0671bd/attachment-0002.html>

From carsonhh at gmail.com  Fri Aug 31 13:03:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 15:03:14 -0400
Subject: [maker-devel] keep_preds option?
In-Reply-To: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>
Message-ID: <CC667EFB.117DD%carsonhh@gmail.com>

I concur with everything Barry said.  Also let me add that alt-ESTs do not
get polished around splice sites (exonerate won't handle them).  However
ESTs and proteins do.  This means that the utility of alt-ESTs in adding
UTR, and splice information is zero.  They just function as an anchor to
maintain gene models that might have otherwise been rejected.  This also
means alt_est=some.fasta  together with est2genome=1 will produce virtually
no additional results because est2genome requires that the splice site makes
sense.  You are better off using proteins with protein2genome=1 if you don?t
have ESTs and want partial models for training.  Once you have a trained ab
initio gene predictor, turn the est2genome and protein2genome options off.
Otherwise they will give weird partial models that decrease the quality of
your final annotations. (partial models are ok for training but that's about
it).

If you are getting too low a gene count with keep_preds=0, then you probably
need to add more evidence.  Try adding all proteins from a couple of related
species (the protein= option accepts comma separated lists of files). If
your genome is a fungi, oomycete, or a prokaryote, then setting keep_preds=1
is usually safe.  Theses are genomes with high gene density and simple gene
structure, so ab initio predictors do really well and don't need as much
help from the evidence.  For other organisms, leave it set to 0 or you will
get a lot of false positives (false positives on some genomes with complex
gene structure can outnumber the genes by 10 to 1 if you turn this on).

Thanks,
Carson




From:  Barry Moore <barry.moore at genetics.utah.edu>
Date:  Friday, 31 August, 2012 12:52 PM
To:  Christoph Hahn <chrisi.hahni at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] keep_preds option?

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly
> assembled non-model organisms draft genome. Within maker I use genemark, SNAP
> and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found
> on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical
evidence.  There are two pieces of information that are required for every
MAKER annotation (by default).  MAKER runs the ab initio gene predictors and
aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to
the genome.  For each locus where one or more gene predictions exist MAKER
checks to see if there is any physical evidence for gene expression at that
locus (RNA/protein sequence alignments) and if there is (splice EST or
protein alignments) evidence overlapping the predictions, MAKER decides
which prediction best matches the evidence and promotes that prediction to
an annotation.  If there is no evidence overlapping any of the predictions
then those predictions are not included in the output annotation file
(although they are saved).  If you set keep_preds=1 then for each locus
where prediction(s) exist maker keeps one and promotes it to an annotation
even though there is no physical evidence.  The description of
'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio
predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab
> initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different
> programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as
evidence and thus in the absence of aligned physical evidence MAKER will not
output an annotation even if all three ab initio tools predict a gene at
that locus.

> In my case I get substantial differences in the number of gene models found
> between the two settings, while with =1 I get a number that is close to what
> we would expect. How would you interpret that? What would you recommend me to
> do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is
too few, you could run MAKER with keep_preds=1.  After the run is finished,
use a tool like IPRScan to search all MAKER predictions for protein domain
content and push that IPRScan output back into the MAKER GFF file with the
ipr_update_gff script.  Then as a final step you can run over the GFF file
and remove any gene model that doesn't have either physical evidence (AED <
1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a
script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA
> sequence file in fasta format from an alternate organism). The organism is
> quite distantly related, but its the closest I have so I thought I d give it a
> shot. I ran maker twice with identical settigs expect in altest and
> est2genome=0/1. The number of genes predicted is identical with both
> approaches, so I am not sure whether or not the EST data was actually used or
> its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in
the way of additional support (compared to just using protein evidence from
say Swiss-prot) and this would be especially true if your alt_est is
distantly related.  I'm not sure I really understand you alt_est/est2genome
combo's to comment in more detail.  I could see four possible combinations
there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP
> using the result of the previous pass. Then for every pass I run maker from
> scratch. Would you recommend to supply the gff of the previous pass in
> "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a
genome when you want certain parts of the previous run to be passed through
unchanged, but with retraining SNAP you want MAKER to re-evaluate each
annotation in light of the new predictions made by the retrained SNAP.
MAKER should run really fast in all of the runs after the first one because
as long as you haven't changed the evidence files it won't have to redo any
of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your
> help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/ed670166/attachment-0002.html>

From carsonhh at gmail.com  Fri Aug 31 14:47:33 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 16:47:33 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
Message-ID: <CC66941F.118C8%carsonhh@gmail.com>

Looks like there is a slight format variation in one line of the
RepeatMasker outputs.  Not sure what could have caused it -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          0    109    (0)

The position/compliment information in columns 12-14 give a start position
of 0 for part of the alignment.  A 0 is invalid because repeat masker
doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
asks for the starting position of the alignment in another part of the
code.

This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
wu-blast and then reconfigure RepeatMasker I get this line -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          1    109    (0)

Notice the 0 becomes a 1 and all logic is restored to the world.

>From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
output, then I just turn it into a 1.
I've attached a file that you should use to replace the
./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
I'll also make a note to the RepeatMasker developers.

Thanks,
Carson



On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Attached.
>
>Thanks,
>J
>
>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send me the entire directory for just the failed contig?
>>
>> Thanks,
>> Carson
>>
>>
>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>
>>>After doing this, I still get the same errors as I attached before.
>>>There was also nothing in the log supporting that maker tried this
>>>more than once. (Initially, I had tries=2, but I changed it to
>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>count".) It also doesn't matter if I keep the same old contig
>>>directory and master log before rerunning, or if I delete them both.
>>>
>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>native version.
>>>
>>>Thanks,
>>>Jeremy
>>>
>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Let maker install it's own blast (a version I know should work).
>>>>
>>>> cd ./maker/src
>>>> ./Build blast
>>>>
>>>> Then to finish up your job (this command will just update executable
>>>> locations in the maker_exe.ctl file) -->
>>>> maker -EXE
>>>>
>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>contig
>>>> still fails.
>>>>
>>>> Thanks,
>>>> Carson
>>>>
>>>>
>>>>
>>>>
>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>
>>>>>Carson,
>>>>>
>>>>>$ blastx -version
>>>>>blastx: 2.2.26+
>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>
>>>>>- J
>>>>>
>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>wrote:
>>>>>> Could you send the blast version informations for reference
>>>>>>purposes?
>>>>>>I
>>>>>> can see this being caused by BLAST.
>>>>>>
>>>>>> Thanks,
>>>>>> Carson
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>
>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>with
>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>and
>>>>>>>used the Build script to have it install all of its own
>>>>>>>dependencies.
>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>the
>>>>>>>blast version I was using before.
>>>>>>>
>>>>>>>
>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>have
>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>first
>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>it
>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>hung
>>>>>>>> for a week until I killed it.
>>>>>>>>
>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>the
>>>>>>>> master log. maker again died on that contig.
>>>>>>>>
>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>
>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>lines
>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>needed
>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>> which takes a while.
>>>>>>>>
>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>> realize that might be hard to implement.)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jeremy
>>>>>>>> Grad student, Grishin lab
>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>> 510.384.8959
>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>.o
>>>>>>>>rg
>>>>>>>
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>maker-devel mailing list
>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>or
>>>>>>>g
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: RepeatMasker.pm
Type: text/x-perl-script
Size: 9226 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/c248081c/attachment-0002.bin>

From jeremy.semeiks at utsw.edu  Fri Aug 31 16:48:37 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 31 Aug 2012 17:48:37 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC66941F.118C8%carsonhh@gmail.com>
References: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
	<CC66941F.118C8%carsonhh@gmail.com>
Message-ID: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>

Carson, maker completed on the scaffold without error when I used your
patch. Thanks!

- J

On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Looks like there is a slight format variation in one line of the
> RepeatMasker outputs.  Not sure what could have caused it -->
> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          0    109    (0)
>
> The position/compliment information in columns 12-14 give a start position
> of 0 for part of the alignment.  A 0 is invalid because repeat masker
> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
> asks for the starting position of the alignment in another part of the
> code.
>
> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
> wu-blast and then reconfigure RepeatMasker I get this line -->
> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          1    109    (0)
>
> Notice the 0 becomes a 1 and all logic is restored to the world.
>
> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
> output, then I just turn it into a 1.
> I've attached a file that you should use to replace the
> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
> I'll also make a note to the RepeatMasker developers.
>
> Thanks,
> Carson
>
>
>
> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Attached.
>>
>>Thanks,
>>J
>>
>>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send me the entire directory for just the failed contig?
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>
>>>>After doing this, I still get the same errors as I attached before.
>>>>There was also nothing in the log supporting that maker tried this
>>>>more than once. (Initially, I had tries=2, but I changed it to
>>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>count".) It also doesn't matter if I keep the same old contig
>>>>directory and master log before rerunning, or if I delete them both.
>>>>
>>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>native version.
>>>>
>>>>Thanks,
>>>>Jeremy
>>>>
>>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>> Let maker install it's own blast (a version I know should work).
>>>>>
>>>>> cd ./maker/src
>>>>> ./Build blast
>>>>>
>>>>> Then to finish up your job (this command will just update executable
>>>>> locations in the maker_exe.ctl file) -->
>>>>> maker -EXE
>>>>>
>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>contig
>>>>> still fails.
>>>>>
>>>>> Thanks,
>>>>> Carson
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>
>>>>>>Carson,
>>>>>>
>>>>>>$ blastx -version
>>>>>>blastx: 2.2.26+
>>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>
>>>>>>- J
>>>>>>
>>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>wrote:
>>>>>>> Could you send the blast version informations for reference
>>>>>>>purposes?
>>>>>>>I
>>>>>>> can see this being caused by BLAST.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Carson
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>
>>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>with
>>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>and
>>>>>>>>used the Build script to have it install all of its own
>>>>>>>>dependencies.
>>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>>the
>>>>>>>>blast version I was using before.
>>>>>>>>
>>>>>>>>
>>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>have
>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>first
>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>it
>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>hung
>>>>>>>>> for a week until I killed it.
>>>>>>>>>
>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>the
>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>
>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>
>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>lines
>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>needed
>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>> which takes a while.
>>>>>>>>>
>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jeremy
>>>>>>>>> Grad student, Grishin lab
>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>> 510.384.8959
>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>
>>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>.o
>>>>>>>>>rg
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>maker-devel mailing list
>>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>or
>>>>>>>>g
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>



From carsonhh at gmail.com  Fri Aug 31 17:00:27 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 19:00:27 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>
Message-ID: <CC66BA4B.820D%carsonhh@gmail.com>

Your welcome.  I also submitted a bug report to RepeatMasker.  They said the
issue will go away with the next release since simple repeat detection is
going to be handled by TRF rather than rmblast.

Thanks,
Carson


On 8/31/12 6:48 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

> Carson, maker completed on the scaffold without error when I used your
> patch. Thanks!
> 
> - J
> 
> On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Looks like there is a slight format variation in one line of the
>> RepeatMasker outputs.  Not sure what could have caused it -->
>> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          0    109    (0)
>> 
>> The position/compliment information in columns 12-14 give a start position
>> of 0 for part of the alignment.  A 0 is invalid because repeat masker
>> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
>> asks for the starting position of the alignment in another part of the
>> code.
>> 
>> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
>> wu-blast and then reconfigure RepeatMasker I get this line -->
>> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          1    109    (0)
>> 
>> Notice the 0 becomes a 1 and all logic is restored to the world.
>> 
>> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
>> output, then I just turn it into a 1.
>> I've attached a file that you should use to replace the
>> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
>> I'll also make a note to the RepeatMasker developers.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>> 
>>> Attached.
>>> 
>>> Thanks,
>>> J
>>> 
>>> On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Could you send me the entire directory for just the failed contig?
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> 
>>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>> 
>>>>> After doing this, I still get the same errors as I attached before.
>>>>> There was also nothing in the log supporting that maker tried this
>>>>> more than once. (Initially, I had tries=2, but I changed it to
>>>>> tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>> count".) It also doesn't matter if I keep the same old contig
>>>>> directory and master log before rerunning, or if I delete them both.
>>>>> 
>>>>> FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>> native version.
>>>>> 
>>>>> Thanks,
>>>>> Jeremy
>>>>> 
>>>>> On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>>> Let maker install it's own blast (a version I know should work).
>>>>>> 
>>>>>> cd ./maker/src
>>>>>> ./Build blast
>>>>>> 
>>>>>> Then to finish up your job (this command will just update executable
>>>>>> locations in the maker_exe.ctl file) -->
>>>>>> maker -EXE
>>>>>> 
>>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>> contig
>>>>>> still fails.
>>>>>> 
>>>>>> Thanks,
>>>>>> Carson
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>> 
>>>>>>> Carson,
>>>>>>> 
>>>>>>> $ blastx -version
>>>>>>> blastx: 2.2.26+
>>>>>>> Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>> 
>>>>>>> - J
>>>>>>> 
>>>>>>> On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>> wrote:
>>>>>>>> Could you send the blast version informations for reference
>>>>>>>> purposes?
>>>>>>>> I
>>>>>>>> can see this being caused by BLAST.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Carson
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>> with
>>>>>>>>> maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>> and
>>>>>>>>> used the Build script to have it install all of its own
>>>>>>>>> dependencies.
>>>>>>>>> Now maker is running fine, although the culprit seems to have been
>>>>>>>>> the
>>>>>>>>> blast version I was using before.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>>> <jeremy.semeiks at utsw.edu>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>> have
>>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>> first
>>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>> it
>>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>> hung
>>>>>>>>>> for a week until I killed it.
>>>>>>>>>> 
>>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>> the
>>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>> 
>>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>> 
>>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>> lines
>>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>> needed
>>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>>> which takes a while.
>>>>>>>>>> 
>>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Jeremy
>>>>>>>>>> Grad student, Grishin lab
>>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>>> 510.384.8959
>>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>>> maker-devel mailing list
>>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>> 
>>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>> .o
>>>>>>>>>> rg
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>> or
>>>>>>>>> g
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 





From Sean.Li at csiro.au  Wed Aug  1 19:21:08 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 11:21:08 +1000
Subject: [maker-devel] The est_gff option
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/50cfa71d/attachment-0003.html>

From carsonhh at gmail.com  Wed Aug  1 19:40:09 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 21:40:09 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F4EB7.10283%carsonhh@gmail.com>

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

Thanks,
Carson



From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 9:21 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] The est_gff option

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120801/ba79d284/attachment-0003.html>

From Sean.Li at csiro.au  Wed Aug  1 20:45:10 2012
From: Sean.Li at csiro.au (Sean.Li at csiro.au)
Date: Thu, 2 Aug 2012 12:45:10 +1000
Subject: [maker-devel] The est_gff option
In-Reply-To: <CC3F4EB7.10283%carsonhh@gmail.com>
References: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7A@exvic-mbx04.nexus.csiro.au>
	<CC3F4EB7.10283%carsonhh@gmail.com>
Message-ID: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>

Dear Carson,

Thank you for your explanation! Now I understand how the RNA-Seq data's been interpreted in MAKER.  I check the tag est_gff and only found it in a binary db file: the scaffold_56.db, which should be a SQLite file. I guess this is right? Will this file be only generated when est data's been provided or I turned the alt_splice to 1? As the db file didn't exist for my first run of Maker when no EST flags have been switched on.

Best regards,
Sean

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished because they are already aligned (so no run.log entry is generated), they are just popped into an alignment object in memory and integrated directly.  The run.log file is used primarily to track finished calls to external programs (I.e. to avoid partial results files on failure). So since there is nothing to run, there is no run.log entry.  You will just get a message saying reading GFF3 in STDERR. The GFF3 results are combined with any other results produced within maker and are then used to generate hints for gene predictors you run as to the location of introns/exons.  This is during the step where you see SNAP or augustus running again and again.  They also become part of the AED score for selecting from the pool of gene models, and can be used to add UTR to resulting gene models.  If you specify alt_splice=1 they can even be used to infer alternate splice forms.  For reference purposes, they will end up in the maker results with the tag est_gff:Cufflinks.

Thanks,
Carson



From: <Sean.Li at csiro.au<mailto:Sean.Li at csiro.au>>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: [maker-devel] The est_gff option

Hello,

We are trying to add RNA-Seq data into the Maker run. As they have been properly aligned to our draft scaffolds by Cufflinks, we just gave the appropriate gff file to the est_gff option. The gff file is with the following format:

....
scaffold_56     Cufflinks       match_part      248833  249471  .       +       .       ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +       .       ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Target=1:TCONS_00039698 121526236 121526336 +;
....

All the prediction algorithms have been trained before running Maker. Meanwhile, we provided the protein homology evidence.  By simply checking the run.log file, I couldn't find any records that show how RNA-Seq got involved into the prediction process (no blastn or exonerate for est).  Is there anything I missed so the RNA-Seq evidence didn't include in the prediction? Do I need to turn the est2genome on, since I suspect this option is only used when the algorithms aren't trained properly?  Or, do we need to set the value of pcov_blastn and eval_blastn a bit lower?

Thanks!

Regards,
Sean



_______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com<mailto:maker-devel at box290.bluehost.com> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/d8018277/attachment-0003.html>

From carsonhh at gmail.com  Wed Aug  1 20:50:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Wed, 01 Aug 2012 22:50:14 -0400
Subject: [maker-devel] The est_gff option
In-Reply-To: <D20E25E5F5CCF24A998144DAA6D34BD494E8847E7B@exvic-mbx04.nexus.csiro.au>
Message-ID: <CC3F629F.102AC%carsonhh@gmail.com>

The file is generated any time you provide GFF3 input.  It is just a quick
way to pull out the features for a region via SQLite.  It's generated at the
start of a MAKER run.

Thanks,
Carson

From:  <Sean.Li at csiro.au>
Date:  Wednesday, 1 August, 2012 10:45 PM
To:  Carson Holt <carsonhh at gmail.com>, <maker-devel at yandell-lab.org>
Subject:  RE: [maker-devel] The est_gff option

Dear Carson,
 
Thank you for your explanation! Now I understand how the RNA-Seq data?s been
interpreted in MAKER.  I check the tag est_gff and only found it in a binary
db file: the scaffold_56.db, which should be a SQLite file. I guess this is
right? Will this file be only generated when est data?s been provided or I
turned the alt_splice to 1? As the db file didn?t exist for my first run of
Maker when no EST flags have been switched on.
 
Best regards,
Sean   
 

From: Carson Holt [mailto:carsonhh at gmail.com]
Sent: Thursday, 2 August 2012 11:40 AM
To: Li, Sean (CMIS, Acton); maker-devel at yandell-lab.org
Subject: Re: [maker-devel] The est_gff option
 

The mRANseq reads from GFF3 don't need to be blasted/exonerate polished
because they are already aligned (so no run.log entry is generated), they
are just popped into an alignment object in memory and integrated directly.
The run.log file is used primarily to track finished calls to external
programs (I.e. to avoid partial results files on failure). So since there is
nothing to run, there is no run.log entry.  You will just get a message
saying reading GFF3 in STDERR. The GFF3 results are combined with any other
results produced within maker and are then used to generate hints for gene
predictors you run as to the location of introns/exons.  This is during the
step where you see SNAP or augustus running again and again.  They also
become part of the AED score for selecting from the pool of gene models, and
can be used to add UTR to resulting gene models.  If you specify
alt_splice=1 they can even be used to infer alternate splice forms.  For
reference purposes, they will end up in the maker results with the tag
est_gff:Cufflinks.

 

Thanks,

Carson

 

 

 

From: <Sean.Li at csiro.au>
Date: Wednesday, 1 August, 2012 9:21 PM
To: <maker-devel at yandell-lab.org>
Subject: [maker-devel] The est_gff option

 

Hello,
 
We are trying to add RNA-Seq data into the Maker run. As they have been
properly aligned to our draft scaffolds by Cufflinks, we just gave the
appropriate gff file to the est_gff option. The gff file is with the
following format:
 
?.
scaffold_56     Cufflinks       match_part      248833  249471  .       +
.       
ID=1:TCONS_00039698:exon-1;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121525597 121526235 +;
scaffold_56     Cufflinks       match_part      253262  253362  .       +
.       
ID=1:TCONS_00039698:exon-2;Name=1:TCONS_00039698;Parent=1:TCONS_00039698;Tar
get=1:TCONS_00039698 121526236 121526336 +;
?.
 
All the prediction algorithms have been trained before running Maker.
Meanwhile, we provided the protein homology evidence.  By simply checking
the run.log file, I couldn?t find any records that show how RNA-Seq got
involved into the prediction process (no blastn or exonerate for est).  Is
there anything I missed so the RNA-Seq evidence didn?t include in the
prediction? Do I need to turn the est2genome on, since I suspect this option
is only used when the algorithms aren?t trained properly?  Or, do we need to
set the value of pcov_blastn and eval_blastn a bit lower?
 
Thanks! 
 
Regards,
Sean  
 
 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120801/e466dcaa/attachment-0003.html>

From alangfan at gmail.com  Wed Aug  1 23:36:24 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Thu, 2 Aug 2012 13:36:24 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
Message-ID: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
[alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
#location of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
#location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
#location of probuild executable (required for genemark)

============================================

-- 
      Alang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/03155f36/attachment-0003.html>

From carsonhh at gmail.com  Thu Aug  2 06:23:21 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 08:23:21 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
Message-ID: <CC3FE8DD.102BC%carsonhh@gmail.com>

The timing of the fault suggests either a problem with your AnyDBM_File
module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
--debug and send me the output as well as your version of maker?  Also try
updating those 3 modules on your system.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 1:36 AM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)

Dear All,

I am running Maker and get "Segmentation fault (core dumped)."  Below
description is command line and setting.
Does anyone have same problem or any suggestions?

Best,

-------command and error message-------------------------------------
 [alang at bioinfoX temporary]$ maker -CTL
modify maker_opts.ctl
...........
[alang at bioinfoX temporary]$ maker
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
Segmentation fault (core dumped)
===========================================

------- maker_opts.ctl -------------------------------------
#-----Genome
genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
#-----EST Evidence
est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
#altest=/fastas/alt_est.fasta

#-----Protein Homology Evidence
#protein=protein.fasta

#-----MAKER Specific Options
evaluate=0
max_dna_len=100000
min_contig=1
min_protein=0
split_hit=10000
pred_flank=200
single_exon=0
single_length=250
keep_preds=0
map_forward=0
retry=1
clean_try=0
clean_up=0
============================================

----------maker_exe.ctl---------------------------------------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
#location of NCBI+ makeblastdb executable
blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
of NCBI+ blastn executable
blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
of NCBI+ blastx executable
tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMas
ker #location of RepeatMasker executable
exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
#location of exonerate executable

#-----Ab-initio Gene Prediction Algorithms
snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
executable
gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
#location of augustus executable
fgenesh= #location of fgenesh executable

#-----Other Algorithms
probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
of probuild executable (required for genemark)

============================================

-- 
      Alang

_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/32122c14/attachment-0003.html>

From mike.thon at gmail.com  Thu Aug  2 06:48:31 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Thu, 2 Aug 2012 14:48:31 +0200
Subject: [maker-devel] segmentation fault
Message-ID: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>

I have the same problem as Fan Wen-Lang.  I get a segmentation fault when processing the fasta file.  The strange thing is, when I run maker with the -debug parameter, I don't get the segmentation fault.  I'm using maker 2.26 on ubuntu linux 64 bit.
Mike




From carsonhh at gmail.com  Thu Aug  2 07:49:47 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 09:49:47 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <6F82F3EC-4A0D-461F-9C68-587AAAC980F1@gmail.com>
Message-ID: <CC3FFDA2.102D5%carsonhh@gmail.com>

Odd.  Have you tried updating the 3 modules I mentioned.

Thanks,
Carson



On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>processing the fasta file.  The strange thing is, when I run maker with
>the -debug parameter, I don't get the segmentation fault.  I'm using
>maker 2.26 on ubuntu linux 64 bit.
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From felix.bemm at uni-wuerzburg.de  Thu Aug  2 09:56:55 2012
From: felix.bemm at uni-wuerzburg.de (Felix Bemm)
Date: Thu, 02 Aug 2012 17:56:55 +0200
Subject: [maker-devel] segmentation fault
In-Reply-To: <CC3FFDA2.102D5%carsonhh@gmail.com>
References: <CC3FFDA2.102D5%carsonhh@gmail.com>
Message-ID: <501AA347.2080304@uni-wuerzburg.de>

This snippet will fix your problem temporarily:

cd maker/lib
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)

But do it with caution, I still do not know what problems are caused by 
this fix.

Best regards
Felix

On 02.08.2012 15:49, Carson Holt wrote:
> Odd.  Have you tried updating the 3 modules I mentioned.
>
> Thanks,
> Carson
>
>
>
> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault when
>> processing the fasta file.  The strange thing is, when I run maker with
>> the -debug parameter, I don't get the segmentation fault.  I'm using
>> maker 2.26 on ubuntu linux 64 bit.
>> Mike
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



From carsonhh at gmail.com  Thu Aug  2 10:31:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 12:31:37 -0400
Subject: [maker-devel] segmentation fault
In-Reply-To: <501AA347.2080304@uni-wuerzburg.de>
Message-ID: <CC40233B.102F6%carsonhh@gmail.com>

Thanks Felix.

You may also have to run the fix below on wherever Bioperl is installed as
well, because it contains the same (DB_File GDBM_File NDBM_File SDBM_File)
statement in the Bio::DB::Fasta module in one of the BEGIN statements.

Thanks,
Carson



On 12-08-02 11:56 AM, "Felix Bemm" <felix.bemm at uni-wuerzburg.de> wrote:

>This snippet will fix your problem temporarily:
>
>cd maker/lib
>sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>
>But do it with caution, I still do not know what problems are caused by
>this fix.
>
>Best regards
>Felix
>
>On 02.08.2012 15:49, Carson Holt wrote:
>> Odd.  Have you tried updating the 3 modules I mentioned.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-02 8:48 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>> I have the same problem as Fan Wen-Lang.  I get a segmentation fault
>>>when
>>> processing the fasta file.  The strange thing is, when I run maker with
>>> the -debug parameter, I don't get the segmentation fault.  I'm using
>>> maker 2.26 on ubuntu linux 64 bit.
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From kippjohnson at uchicago.edu  Thu Aug  2 13:12:15 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Thu, 2 Aug 2012 14:12:15 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
Message-ID: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The
first time I ran maker, I ignored these and the maker run seemingly
proceeded to the process the contigs correctly. Is this something I should
be worried about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for
each restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/70c4c837/attachment-0003.html>

From alangfan at gmail.com  Thu Aug  2 18:45:17 2012
From: alangfan at gmail.com (=?Big5?B?RmFuO1dlbi1MYW5nIK1TpOWtpg==?=)
Date: Fri, 3 Aug 2012 08:45:17 +0800
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CC3FE8DD.102BC%carsonhh@gmail.com>
References: <CAF6vX=r-LUKRRZmQ++R-0BJRcKdF8BO13zq3A+OypMzovjP0FA@mail.gmail.com>
	<CC3FE8DD.102BC%carsonhh@gmail.com>
Message-ID: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>

Thanks Carson.

I'm using a single server with centOS6 64 bit.
==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep
-l 'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>

> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
>
> Thanks,
> Carson
>
>
>
> From: "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date: Thursday, 2 August, 2012 1:36 AM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
>
> Dear All,
>
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
>
> Best,
>
> -------command and error message-------------------------------------
> [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
>
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
>
> #-----Protein Homology Evidence
> #protein=protein.fasta
>
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
>
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location
> of NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location
> of NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx
> #location of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMasker
> #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
>
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location
> of eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
>
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild
> #location of probuild executable (required for genemark)
>
> ============================================
>
> --
>       Alang
>
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/36149b1d/attachment-0003.html>

From carsonhh at gmail.com  Thu Aug  2 20:07:53 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Thu, 02 Aug 2012 22:07:53 -0400
Subject: [maker-devel] Question: maker Segmentation fault (core dumped)
In-Reply-To: <CAF6vX=oecAvAW341g70bgkc6LiYZcY+xWKUgfM1ujJpyPxNQHQ@mail.gmail.com>
Message-ID: <CC40AA6F.10327%carsonhh@gmail.com>

Ru with the ?f flag before rerunning in the same directory as your previous
error.  Also collect the STDERR both with and without the -debug flag and
send it to me.

Thanks,
Carson



From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
Date:  Thursday, 2 August, 2012 8:45 PM
To:  Carson Holt <carsonhh at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] Question: maker Segmentation fault (core dumped)

Thanks Carson.

I'm using a single server with centOS6 64 bit.
============================================================================
==
STATUS MAKER 2.26
============================================================================
==
PERL Dependencies:      VERIFIED
External Programs:      VERIFIED
External C Libraries:   VERIFIED
MPI SUPPORT:            DISABLED
MWAS Web Interface:     DISABLED
MAKER PACKAGE:          CONFIGURATION OK
----------------------------------------------------------------------------
------------------------

I have updated/reinstalled the three modules,  AnyDBM_File, DBD::SQLite and
BerkeleyDB. And fix these file mentioned by Felix.
"sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/' $(grep -l
'DB_File GDBM_File NDBM_File SDBM_File' *)"

I get new error messages when I run maker2.26.
==========================================================
...........
...........
Process::MpiTiers::__ANON__('Error::Simple=HASH(0x476f690)',
'SCALAR(0x284f2c0)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 339
        eval {...} called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 329
        Error::subs::run_clauses('HASH(0x476bc98)', 'ERROR: Fasta index
error\x{a} at /home/alang/tools/maker/maker/bi...', undef,
'ARRAY(0x285be30)') called at
/home/alang/tools/maker/maker/bin/../lib/Error.pm line 426
        Error::subs::try('CODE(0x476feb8)', 'HASH(0x476bc98)') called at
/home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line 79
        Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x476c4f0)')
called at /home/alang/tools/maker/maker/bin/../lib/Process/MpiTiers.pm line
56
        Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x4773be8)', 0,
'Process::MpiChunk') calledERROR: Failed while processing contig output
ERROR: Can not load chunks
WARNING: You must always set a rank before running MpiTiers
FATAL: argument `the_void` does not exist in MpiTier object
===========================================================

2012/8/2 Carson Holt <carsonhh at gmail.com>
> The timing of the fault suggests either a problem with your AnyDBM_File
> module, BDB::SQLlite module, or BerkeleyDB module.  Could you run maker
> --debug and send me the output as well as your version of maker?  Also try
> updating those 3 modules on your system.
> 
> Thanks,
> Carson
> 
> 
> 
> From:  "Fan;Wen-Lang ???" <alangfan at gmail.com>
> Date:  Thursday, 2 August, 2012 1:36 AM
> To:  <maker-devel at yandell-lab.org>
> Subject:  [maker-devel] Question: maker Segmentation fault (core dumped)
> 
> Dear All,
> 
> I am running Maker and get "Segmentation fault (core dumped)."  Below
> description is command line and setting.
> Does anyone have same problem or any suggestions?
> 
> Best,
> 
> -------command and error message-------------------------------------
>  [alang at bioinfoX temporary]$ maker -CTL
> modify maker_opts.ctl
> ...........
> [alang at bioinfoX temporary]$ maker
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> Segmentation fault (core dumped)
> ===========================================
> 
> ------- maker_opts.ctl -------------------------------------
> #-----Genome
> genome=/home/alang/tools/maker/maker/temporary/dpp_contig.fasta
> #-----EST Evidence
> est=/home/alang/tools/maker/maker/temporary/dpp_est.fasta
> #altest=/fastas/alt_est.fasta
> 
> #-----Protein Homology Evidence
> #protein=protein.fasta
> 
> #-----MAKER Specific Options
> evaluate=0
> max_dna_len=100000
> min_contig=1
> min_protein=0
> split_hit=10000
> pred_flank=200
> single_exon=0
> single_length=250
> keep_preds=0
> map_forward=0
> retry=1
> clean_try=0
> clean_up=0
> ============================================
> 
> ----------maker_exe.ctl---------------------------------------------
> #-----Location of Executables Used by MAKER/EVALUATOR
> makeblastdb=/home/alang/tools/maker/maker/bin/../exe/blast/bin/makeblastdb
> #location of NCBI+ makeblastdb executable
> blastn=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastn #location of
> NCBI+ blastn executable
> blastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/blastx #location of
> NCBI+ blastx executable
> tblastx=/home/alang/tools/maker/maker/bin/../exe/blast/bin/tblastx #location
> of NCBI+ tblastx executable
> formatdb= #location of NCBI formatdb executable
> blastall= #location of NCBI blastall executable
> xdformat= #location of WUBLAST xdformat executable
> blasta= #location of WUBLAST blasta executable
> RepeatMasker=/home/alang/tools/maker/maker/bin/../exe/RepeatMasker/RepeatMaske
> r #location of RepeatMasker executable
> exonerate=/home/alang/tools/maker/maker/bin/../exe/exonerate/bin/exonerate
> #location of exonerate executable
> 
> #-----Ab-initio Gene Prediction Algorithms
> snap=/home/alang/tools/maker/maker/bin/../exe/snap/snap #location of snap
> executable
> gmhmme3=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/gmhmme3 #location of
> eukaryotic genemark executable
> gmhmmp= #location of prokaryotic genemark executable
> augustus=/home/alang/tools/maker/maker/bin/../exe/augustus/bin/augustus
> #location of augustus executable
> fgenesh= #location of fgenesh executable
> 
> #-----Other Algorithms
> probuild=/home/alang/software/gm_es_bp_linux64_v2.3e/gmes/probuild #location
> of probuild executable (required for genemark)
> 
> ============================================
> 
> -- 
>       Alang
> 
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak
> er-devel_yandell-lab.org



-- 
      Alang
-----
???
Wen-Lang Fan,
Post-doctoral Fellow,
The Genomics Research Center, Academia Sinica. 128 Academia Road,
Section 2,Nankang, Taipei, 115, Taiwan (02) 27871246
----------------------------------------------------------------------------
------------------------------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120802/9a29692b/attachment-0003.html>

From mike.thon at gmail.com  Thu Aug  2 22:30:15 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 06:30:15 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been started
	in the same directory.
Message-ID: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>

I get this warning when running the mpi version of maker.  Should I be concerned?  the command I use is this:

 mpiexec -n 8 maker

-Mike




From carsonhh at gmail.com  Thu Aug  2 23:11:39 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 01:11:39 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <9ABEA0EF-D1AF-447C-B1D4-8DC0EBF2A53E@gmail.com>
Message-ID: <CC40CC6D.10338%carsonhh@gmail.com>

I'm sorry I'm confused.  Which warning? the segmentation fault you
mentioned before?  For now I would say it's best to first try maker once
outside of MPI to verify that the segmentation fault no longer happens
before moving onto the MPI environment.

Make sure to run with the -f flag when retrying after the fix suggested by
Felix, so maker completely restarts from scratch.  That way you can see
that the fasta database is being generated from start to finish (and is
not just being reread from the successful run under the --debug flag).

Make sure that the fix Felix sent is also applied to the Bioperl
Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
fault will not go away otherwise (if it is infact one of perl's DB modules
causing the issue, which it likely is).  Your error sounds similar to an
error Felix and Thomas encountered some time ago
(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
h/000941.html).  Basically one of perl's native DB modules or the database
the module relies on are broken, so the command line snippet is cutting
any calls to those other databases out of maker and Bioperl (which should
be ok if you have at minimum a working Berkley DB and DB_File module on
your system).

Use this command line to get the correct directory for applying the
changes to Bioperl as well -->
perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
s/[^\/]+$//; print "$dir\n"'

Then copy the directory location-->

cd <to_that_directory>
sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)



Finally run maker with the -f flag.  If it works let me know, and then you
can give MPI a shot.  Make said yes to the "compile for MPI question"
asked during the maker install process or mpiexec will fail when running
maker.  If it doesn?t work without MPI or if it works and then MPI fails,
make sure to collect the STDERR in a text file and send it to me.  I will
probably have you run it with the --debug flag as well following any
initial failure (just to get the extra status messages about your system
configuration).

Thanks,
Carson



On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I get this warning when running the mpi version of maker.  Should I be
>concerned?  the command I use is this:
>
> mpiexec -n 8 maker
>
>-Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From mike.thon at gmail.com  Fri Aug  3 01:56:34 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 3 Aug 2012 09:56:34 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>

I meant the warning that is in the subject line of my first email:

WARNING: Multiple MAKER processes have been started in the same directory.

I get that warning when I run mpi maker.  For now, I'm running it without mpi.  When the run finishes I can try it again with mpi and make sure I get the same output.

For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.

best
Mike

On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From carsonhh at gmail.com  Fri Aug  3 06:37:24 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 08:37:24 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC413D27.1038C%carsonhh@gmail.com>

That's good to know the seg fault is gone.  Sorry for the confusion, but I
don't seem to have received your other e-mail about the "multiple process
warning".

Could you run '/maker/src/Build status' and send me the output?

Thanks,
Carson

On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 07:11:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:11:03 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <58972E91-ABBD-4F2F-836A-0C20A9493BF8@gmail.com>
Message-ID: <CC414316.103A1%carsonhh@gmail.com>

One more thing.

There are three locations where this exact line of code are called
--> @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
run_log
ds_utility
Bio::Fasta::DB

Because of the way perl works only the last one to run needs to be fixed
to stop the error (because it overrides the other two calls), but that
means if the line is ever removed from the last loaded module or load
order changes, then the error will come back because it is the last one to
run that fixes the issue.  So to be safe you should fix it in all
instances.

Currently ds_utility loads last, but it really doesn't need AnyDBM_File
any more to perform it's function and the call will likely be removed in
future releases, so changing the line in Bio::Fasta::DB should avoid
potential future errors on your system.

Thanks,
Carson



On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I meant the warning that is in the subject line of my first email:
>
>WARNING: Multiple MAKER processes have been started in the same directory.
>
>I get that warning when I run mpi maker.  For now, I'm running it without
>mpi.  When the run finishes I can try it again with mpi and make sure I
>get the same output.
>
>For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>
>best
>Mike
>
>On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From carsonhh at gmail.com  Fri Aug  3 07:27:37 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 09:27:37 -0400
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
Message-ID: <CC414867.103D1%carsonhh@gmail.com>

It means that your GFF3 results will not be integrated.  Does it happen if
you start in a different directory?

The lock error is coming from SQLite, and may be the result of your working
directory being mounted on an NFS filesystem which is a known issue with
SQLite.  If you give maker a location for TMP that is not NFS mounted (I.e.
A locally mounted filesystem), then maker will try and move the SQLite
database off of the NFS mount.

You can check the mount status of the current working directory and the
directory you choose fro TMP by using the Linux df command.

df <directory>

Any NFS mount will follow the format of host:/location.

Example:
152.234.123.234:/data/research

Or 

storage.utah.edu:/data/research

If you are not running on an NFS device let me know if running in a
different directory results in the same error.

Thanks,
Carson

From:  Kipp Johnson <kippjohnson at uchicago.edu>
Date:  Thursday, 2 August, 2012 3:12 PM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help

Hi,

     I am running Maker version 2.26. I have two questions:

1)

    When I run maker, I get errors like the following printed to STDOUT:

DBD::SQLite::db selectcol_arrayref failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
...
DBD::SQLite::db do failed: database is locked at
/home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.

     I reinstalled the indicated module via cpanm and restarted Maker from
the beginning in a new directory, and I am getting the same error. The first
time I ran maker, I ignored these and the maker run seemingly proceeded to
the process the contigs correctly. Is this something I should be worried
about?


2)

    I am running Maker on a single server with 64 processors. Would it be
faster to use MPICH2, or to simply restart Maker in the same directory ~16
times, with the # of CPUs option that gets passed to Blast set to 4 for each
restart?

    I believe that I have MPICH2 correctly compiled and installed; if it
would be faster to use MPICH2, could someone provide me with a sample
command line entry to use for Maker? I am confused about how to run MPICH2
on a single node (without a hostfile?), and searching the maker-dev-list
archive seems to give me results for older versions of MAKER and MPICH2.

Thank you,


Best,

     Kipp Johnson
     kippjohnson at uchicago.edu
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/5517b922/attachment-0003.html>

From cjfields at illinois.edu  Fri Aug  3 07:51:04 2012
From: cjfields at illinois.edu (Fields, Christopher J)
Date: Fri, 3 Aug 2012 13:51:04 +0000
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <CC40CC6D.10338%carsonhh@gmail.com>
References: <CC40CC6D.10338%carsonhh@gmail.com>
Message-ID: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>

I just want to add in, if there are any fixes that need to be made to Bioperl let me know.  We're trying to move towards an AnyDBM_File approach, but I think this effort has stalled.

chris

On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
 wrote:

> I'm sorry I'm confused.  Which warning? the segmentation fault you
> mentioned before?  For now I would say it's best to first try maker once
> outside of MPI to verify that the segmentation fault no longer happens
> before moving onto the MPI environment.
> 
> Make sure to run with the -f flag when retrying after the fix suggested by
> Felix, so maker completely restarts from scratch.  That way you can see
> that the fasta database is being generated from start to finish (and is
> not just being reread from the successful run under the --debug flag).
> 
> Make sure that the fix Felix sent is also applied to the Bioperl
> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
> fault will not go away otherwise (if it is infact one of perl's DB modules
> causing the issue, which it likely is).  Your error sounds similar to an
> error Felix and Thomas encountered some time ago
> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Marc
> h/000941.html).  Basically one of perl's native DB modules or the database
> the module relies on are broken, so the command line snippet is cutting
> any calls to those other databases out of maker and Bioperl (which should
> be ok if you have at minimum a working Berkley DB and DB_File module on
> your system).
> 
> Use this command line to get the correct directory for applying the
> changes to Bioperl as well -->
> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
> s/[^\/]+$//; print "$dir\n"'
> 
> Then copy the directory location-->
> 
> cd <to_that_directory>
> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
> 
> 
> 
> Finally run maker with the -f flag.  If it works let me know, and then you
> can give MPI a shot.  Make said yes to the "compile for MPI question"
> asked during the maker install process or mpiexec will fail when running
> maker.  If it doesn?t work without MPI or if it works and then MPI fails,
> make sure to collect the STDERR in a text file and send it to me.  I will
> probably have you run it with the --debug flag as well following any
> initial failure (just to get the extra status messages about your system
> configuration).
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I get this warning when running the mpi version of maker.  Should I be
>> concerned?  the command I use is this:
>> 
>> mpiexec -n 8 maker
>> 
>> -Mike
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug  3 08:28:44 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 03 Aug 2012 10:28:44 -0400
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
 started in the same directory.
In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF33B5FB68@CITESMBX5.ad.uillinois.edu>
Message-ID: <CC414FC9.1040E%carsonhh@gmail.com>

I think I've finally figured it out, as well as a fix to quash this error
for good.  It not strictly a Bioperl issue.  It's an an issue with
AnyDBM_File as well as it's documentation (as it recommends setting
@AnyDBM_File::ISA in code that uses it but never mentions potential
caveats of multiple modules using it).

Basically if you set @AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File
SDBM_File) and then call 'use AnyDBM_File', the AnyDBM_File module will
actually preprocess and trim @AnyDBM_File::ISA using this code -->

my $mod;
for $mod (@ISA) {
	if (eval "require $mod") {
		@ISA = ($mod);  # if we leave @ISA alone, warnings abound
		return 1;
	}
}


However if you load any other module that also sets @AnyDBM_File::ISA that
code doesn?t get called on the next 'use AnyDBM_File'.  So you will loose
the preprocessing and basically mess up @AnyDBM_File::ISA if you ever have
two modules loaded at the same time that use AnyDBM_File.

So loading Bio::DB::Fasta before or after a module that also uses
AnyDBM_File has the potential to result in errors.

Example:
use Bio::DB::Fasta;
print join(',', @AnyDBM_File::ISA );

Result --> DB_File


use Bio::DB::Qual;

use Bio::DB::Fasta;

print join(',', @AnyDBM_File::ISA );


Result --> DB_File,GDBM_File,NDBM_File,SDBM_File


The second module messes up @AnyDBM_File::ISA.  It also happens if you
load them in reverse.
Perhaps changing the BEGIN block in Bio::DB::Fasta, Bio::DB::Qual, and any
other any other code using AnyDBM_File to include one of these alternate
BEGIN blocks.

BEGIN {
	@AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File SDBM_File)
if(!$INC{'AnyDBM_File.pm'});
}


Or 

BEGIN {
	for my $mod (qw(DB_File GDBM_File NDBM_File SDBM_File)) {
		if (eval "require $mod") {
			@AnyDBM_File::ISA = ($mod);
			last;
		}
	}

}


As far as maker goes, I've actually removed all use of AnyDBM_File from
it's modules now, so it won't be an issue there any more, so Bioperl is
the only one left to fix.

Thanks,
Carson



On 12-08-03 9:51 AM, "Fields, Christopher J" <cjfields at illinois.edu> wrote:

>I just want to add in, if there are any fixes that need to be made to
>Bioperl let me know.  We're trying to move towards an AnyDBM_File
>approach, but I think this effort has stalled.
>
>chris
>
>On Aug 3, 2012, at 12:11 AM, Carson Holt <carsonhh at gmail.com>
> wrote:
>
>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>> mentioned before?  For now I would say it's best to first try maker once
>> outside of MPI to verify that the segmentation fault no longer happens
>> before moving onto the MPI environment.
>> 
>> Make sure to run with the -f flag when retrying after the fix suggested
>>by
>> Felix, so maker completely restarts from scratch.  That way you can see
>> that the fasta database is being generated from start to finish (and is
>> not just being reread from the successful run under the --debug flag).
>> 
>> Make sure that the fix Felix sent is also applied to the Bioperl
>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>> fault will not go away otherwise (if it is infact one of perl's DB
>>modules
>> causing the issue, which it likely is).  Your error sounds similar to an
>> error Felix and Thomas encountered some time ago
>> 
>>(http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>rc
>> h/000941.html).  Basically one of perl's native DB modules or the
>>database
>> the module relies on are broken, so the command line snippet is cutting
>> any calls to those other databases out of maker and Bioperl (which
>>should
>> be ok if you have at minimum a working Berkley DB and DB_File module on
>> your system).
>> 
>> Use this command line to get the correct directory for applying the
>> changes to Bioperl as well -->
>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>> s/[^\/]+$//; print "$dir\n"'
>> 
>> Then copy the directory location-->
>> 
>> cd <to_that_directory>
>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>  $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>> 
>> 
>> 
>> Finally run maker with the -f flag.  If it works let me know, and then
>>you
>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>> asked during the maker install process or mpiexec will fail when running
>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>fails,
>> make sure to collect the STDERR in a text file and send it to me.  I
>>will
>> probably have you run it with the --debug flag as well following any
>> initial failure (just to get the extra status messages about your system
>> configuration).
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>> 
>>> I get this warning when running the mpi version of maker.  Should I be
>>> concerned?  the command I use is this:
>>> 
>>> mpiexec -n 8 maker
>>> 
>>> -Mike
>>> 
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>





From kippjohnson at uchicago.edu  Fri Aug  3 10:16:33 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Fri, 3 Aug 2012 11:16:33 -0500
Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
In-Reply-To: <CC414867.103D1%carsonhh@gmail.com>
References: <CAGJuV2WFcaSHwbNRTfPnKowA=5=fhWh-UxPBf8c9T7Pq7LY10Q@mail.gmail.com>
	<CC414867.103D1%carsonhh@gmail.com>
Message-ID: <CAGJuV2XTG0GWGi9Dbuk7xdT=OOJodWhbUORsj9XcZJwdz3PYYw@mail.gmail.com>

It does happen if I start in a different directory. My current working
directory is NFS. I changed the tmp directory in maker_opts from the NFS to
a local partition (tmpfs), and I'm still getting this error. Does the
working directory also have to be locally mounted?

On Fri, Aug 3, 2012 at 8:27 AM, Carson Holt <carsonhh at gmail.com> wrote:

> It means that your GFF3 results will not be integrated.  Does it happen if
> you start in a different directory?
>
> The lock error is coming from SQLite, and may be the result of your
> working directory being mounted on an NFS filesystem which is a known issue
> with SQLite.  If you give maker a location for TMP that is not NFS mounted
> (I.e. A locally mounted filesystem), then maker will try and move the
> SQLite database off of the NFS mount.
>
> You can check the mount status of the current working directory and the
> directory you choose fro TMP by using the Linux df command.
>
> df <directory>
>
> Any NFS mount will follow the format of host:/location.
>
> Example:
> 152.234.123.234:/data/research
>
> Or
>
> storage.utah.edu:/data/research
>
> If you are not running on an NFS device let me know if running in a
> different directory results in the same error.
>
> Thanks,
> Carson
>
> From: Kipp Johnson <kippjohnson at uchicago.edu>
> Date: Thursday, 2 August, 2012 3:12 PM
> To: <maker-devel at yandell-lab.org>
> Subject: [maker-devel] "DBD::SQLite::db locked" error; MPICH2 help
>
> Hi,
>
>      I am running Maker version 2.26. I have two questions:
>
> 1)
>
>     When I run maker, I get errors like the following printed to STDOUT:
>
> DBD::SQLite::db selectcol_arrayref failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 107.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 109.
> ...
> DBD::SQLite::db do failed: database is locked at
> /home/zjiang/.mnt/kwj/maker/bin/../lib/GFFDB.pm line 497, <$IN> line 2.
>
>      I reinstalled the indicated module via cpanm and restarted Maker from
> the beginning in a new directory, and I am getting the same error. The
> first time I ran maker, I ignored these and the maker run seemingly
> proceeded to the process the contigs correctly. Is this something I should
> be worried about?
>
>
> 2)
>
>     I am running Maker on a single server with 64 processors. Would it be
> faster to use MPICH2, or to simply restart Maker in the same directory ~16
> times, with the # of CPUs option that gets passed to Blast set to 4 for
> each restart?
>
>     I believe that I have MPICH2 correctly compiled and installed; if it
> would be faster to use MPICH2, could someone provide me with a sample
> command line entry to use for Maker? I am confused about how to run MPICH2
> on a single node (without a hostfile?), and searching the maker-dev-list
> archive seems to give me results for older versions of MAKER and MPICH2.
>
> Thank you,
>
>
> Best,
>
>      Kipp Johnson
>      kippjohnson at uchicago.edu
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120803/b5f2b0cd/attachment-0003.html>

From mike.thon at gmail.com  Fri Aug  3 23:48:52 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 07:48:52 +0200
Subject: [maker-devel] WARNING: Multiple MAKER processes have been
	started in the same directory.
In-Reply-To: <CC413D27.1038C%carsonhh@gmail.com>
References: <CC413D27.1038C%carsonhh@gmail.com>
Message-ID: <E44B4BF9-7D27-4B11-B33F-42A975222F99@gmail.com>

./Build status


==============================================================================
STATUS MAKER 2.26
==============================================================================
PERL Dependencies:	VERIFIED
External Programs:	VERIFIED
External C Libraries:	VERIFIED
MPI SUPPORT:		ENABLED
MWAS Web Interface:	DISABLED
MAKER PACKAGE:		CONFIGURATION OK

...

I was impatient and I ran maker with MPI.  In the master log file I see a lot of failed contigs.  Now I'm running it without MPI to see if those contigs fail again. 

On Aug 3, 2012, at 2:37 PM, Carson Holt <carsonhh at gmail.com> wrote:

> That's good to know the seg fault is gone.  Sorry for the confusion, but I
> don't seem to have received your other e-mail about the "multiple process
> warning".
> 
> Could you run '/maker/src/Build status' and send me the output?
> 
> Thanks,
> Carson
> 
> On 12-08-03 3:56 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
> 
>> I meant the warning that is in the subject line of my first email:
>> 
>> WARNING: Multiple MAKER processes have been started in the same directory.
>> 
>> I get that warning when I run mpi maker.  For now, I'm running it without
>> mpi.  When the run finishes I can try it again with mpi and make sure I
>> get the same output.
>> 
>> For me, just editing  ds_utility.pm and runlog.pm in maker/lib was enough
>> to fix the seg fault.  I didn't have to change anything in Bio::DB::Fasta.
>> 
>> best
>> Mike
>> 
>> On Aug 3, 2012, at 7:11 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> 
>>> I'm sorry I'm confused.  Which warning? the segmentation fault you
>>> mentioned before?  For now I would say it's best to first try maker once
>>> outside of MPI to verify that the segmentation fault no longer happens
>>> before moving onto the MPI environment.
>>> 
>>> Make sure to run with the -f flag when retrying after the fix suggested
>>> by
>>> Felix, so maker completely restarts from scratch.  That way you can see
>>> that the fasta database is being generated from start to finish (and is
>>> not just being reread from the successful run under the --debug flag).
>>> 
>>> Make sure that the fix Felix sent is also applied to the Bioperl
>>> Bio::DB::Fasta module and not just the /maker/lib/ directory or the seg
>>> fault will not go away otherwise (if it is infact one of perl's DB
>>> modules
>>> causing the issue, which it likely is).  Your error sounds similar to an
>>> error Felix and Thomas encountered some time ago
>>> 
>>> (http://box290.bluehost.com/pipermail/maker-devel_yandell-lab.org/2012-Ma
>>> rc
>>> h/000941.html).  Basically one of perl's native DB modules or the
>>> database
>>> the module relies on are broken, so the command line snippet is cutting
>>> any calls to those other databases out of maker and Bioperl (which
>>> should
>>> be ok if you have at minimum a working Berkley DB and DB_File module on
>>> your system).
>>> 
>>> Use this command line to get the correct directory for applying the
>>> changes to Bioperl as well -->
>>> perl -e 'use Bio::DB::Fasta; ($dir = $INC{"Bio/DB/Fasta.pm"}) =~
>>> s/[^\/]+$//; print "$dir\n"'
>>> 
>>> Then copy the directory location-->
>>> 
>>> cd <to_that_directory>
>>> sed -i 's/qw(DB_File GDBM_File NDBM_File SDBM_File)/qw(DB_File)/'\
>>> $(grep -l 'DB_File GDBM_File NDBM_File SDBM_File' *)
>>> 
>>> 
>>> 
>>> Finally run maker with the -f flag.  If it works let me know, and then
>>> you
>>> can give MPI a shot.  Make said yes to the "compile for MPI question"
>>> asked during the maker install process or mpiexec will fail when running
>>> maker.  If it doesn?t work without MPI or if it works and then MPI
>>> fails,
>>> make sure to collect the STDERR in a text file and send it to me.  I
>>> will
>>> probably have you run it with the --debug flag as well following any
>>> initial failure (just to get the extra status messages about your system
>>> configuration).
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> 
>>> 
>>> On 12-08-03 12:30 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>> 
>>>> I get this warning when running the mpi version of maker.  Should I be
>>>> concerned?  the command I use is this:
>>>> 
>>>> mpiexec -n 8 maker
>>>> 
>>>> -Mike
>>>> 
>>>> 
>>>> _______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 




From mike.thon at gmail.com  Sat Aug  4 05:51:04 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Sat, 4 Aug 2012 13:51:04 +0200
Subject: [maker-devel] failed contigs
Message-ID: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>

running maker 2.26 some contigs seem to consistently fail to run, according to XXX_master_datastore_index.log

Here are the last few lines of run.log for one of the failed contigs:

CTL_OPTIONS     run     
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.0       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
LOGCHILD        /home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/run.log.child.1       
DIED    RANK    0
DIED    COUNT   1


the contents of run.log.child.1 for the same contig:
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
FINISHED        cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out 
STARTED cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/supercont1%2E6.1.te_proteins%2Efasta.repeatrunner  
DIED    RANK    0:2:0:1
DIED    COUNT   1

So perhaps its a problem with the repeat masking step.  I already have annotated repeats using RepeatMasker and I have the gff2 file produced by RepeatMasker.  Do I need to convert this to gff3 before adding to to my maker_opts file?  Better yet, does anyone know why these contigs are failing and how to fix it?
Mike




From carsonhh at gmail.com  Sat Aug  4 07:44:25 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Sat, 04 Aug 2012 09:44:25 -0400
Subject: [maker-devel] failed contigs
In-Reply-To: <28D83426-5182-4785-B1ED-C4CE357DD42F@gmail.com>
Message-ID: <CC429E5D.104F1%carsonhh@gmail.com>

Do you have STDERR log stored somewhere?  Could you send it to me?
Sometimes the error causing the failure is upstream in the log and then
cascades into other errors.  Also if it is the repeatrunner step failing,
the log will have the command to run the BLAST step outside of MAKER (is
helpful for debugging).  If you don't have the captured STDERR, could you
run again and send it to me.

Thanks,
Carson

On 12-08-04 7:51 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>running maker 2.26 some contigs seem to consistently fail to run,
>according to XXX_master_datastore_index.log
>
>Here are the last few lines of run.log for one of the failed contigs:
>
>CTL_OPTIONS     run
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.0
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>LOGCHILD        
>/home/mike/projects/cg_maker/cgram.maker.output/cgram_datastore/supercont1
>.6/theVoid.supercont1.6/run.log.child.1
>DIED    RANK    0
>DIED    COUNT   1
>
>
>the contents of run.log.child.1 for the same contig:
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>FINISHED        
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.Repeat_Type_Elements%2Efasta.specific.out
>STARTED 
>cgram.maker.output/cgram_datastore/supercont1.6/theVoid.supercont1.6/super
>cont1%2E6.1.te_proteins%2Efasta.repeatrunner
>DIED    RANK    0:2:0:1
>DIED    COUNT   1
>
>So perhaps its a problem with the repeat masking step.  I already have
>annotated repeats using RepeatMasker and I have the gff2 file produced by
>RepeatMasker.  Do I need to convert this to gff3 before adding to to my
>maker_opts file?  Better yet, does anyone know why these contigs are
>failing and how to fix it?
>Mike
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From barry.moore at genetics.utah.edu  Fri Aug 17 19:05:00 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 17 Aug 2012 19:05:00 -0600
Subject: [maker-devel] Exonerate web access is back
References: <201208160913.q7G9DTbB005179@rabbit.ebi.ac.uk>
Message-ID: <C52C60DC-1052-4FD7-9CAC-2F4830E78827@genetics.utah.edu>



Begin forwarded message:

> From: <datasubs at ebi.ac.uk>
> Subject: SUP/EMBL/ 24308 moved from EBIHELP by apc (rasko) (SUB#818751)
> Date: August 16, 2012 3:13:29 AM MDT
> To: <barry.moore at genetics.utah.edu>
> 
> Dear Barry,
> 
> Access to http://www.ebi.ac.uk/~guy/exonerate/ has been restored.
> 
> Kind Regards,
> Rasko Leinonen 
> European Nucleotide Archive (ENA)
> 

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120817/0a68e03e/attachment-0003.html>

From kippjohnson at uchicago.edu  Tue Aug 21 15:25:49 2012
From: kippjohnson at uchicago.edu (Kipp Johnson)
Date: Tue, 21 Aug 2012 16:25:49 -0500
Subject: [maker-devel] Maker Re-tries
In-Reply-To: <6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
References: <CC414867.103D1%carsonhh@gmail.com>
	<6F32139D-2D27-4658-B343-21B8A8929BED@gmail.com>
Message-ID: <1E417039-68CF-4EFE-90BF-964DC86B1864@uchicago.edu>

Hi Carson,

	Simple question: If I start maker and then kill the job before a thread finishes processing a contig, does this failure to complete the contig count in the number of tries maker will try to complete the contig before skipping to another?	

	Basically, does the "tries=2 #number of times to try a contig if there is a failure for some reason" parameter take into account failures resulting from me killing the job?	


	Best,

		Kipp Johnson
		kippjohnson at uchicago.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120821/74551a9f/attachment-0003.html>

From jeremy.semeiks at utsw.edu  Tue Aug 21 10:26:32 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Tue, 21 Aug 2012 11:26:32 -0500
Subject: [maker-devel] maker dying on a contig
Message-ID: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>

Hi,

I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
~36,000 contigs, and per the master log maker FINISHED all of them
except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
run. It didn't report ERROR for that contig in the master log, but it
did output two DIED lines in the contig's run.log and then just hung
for a week until I killed it.

I reran maker on just that contig by deleting the START line from the
master log. maker again died on that contig.

I attach the relevant part of the stderr stream; the rest was just
"--Next Contig--"-type lines. Any ideas?

A related question. Is deleting lines in the master log, as I did
here, the best way to get maker to rerun on just a few contigs? I
often find myself needing to do this, sometimes because of ERROR lines
or other assumed quirks in maker and sometimes just because I needed
to pause a big run for a while. One disadvantage is that when I
restart, maker needs to manually scan over many completed contigs,
which takes a while.

(I'd think the best solution would be for maker to automatically
recognize whether it's really currently running on contigs marked
STARTED but not FINISHED. It doesn't seem to do this now, but I
realize that might be hard to implement.)

Thanks,
Jeremy
Grad student, Grishin lab
UT Southwestern, Dallas TX
510.384.8959
-------------- next part --------------
A non-text attachment was scrubbed...
Name: maker.stderr
Type: application/octet-stream
Size: 11422 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120821/7f29efb2/attachment-0003.obj>

From mike.thon at gmail.com  Fri Aug 24 01:44:13 2012
From: mike.thon at gmail.com (Michael Thon)
Date: Fri, 24 Aug 2012 09:44:13 +0200
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
Message-ID: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>

I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.


On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:

> Hi,
> 
> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
> ~36,000 contigs, and per the master log maker FINISHED all of them
> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
> run. It didn't report ERROR for that contig in the master log, but it
> did output two DIED lines in the contig's run.log and then just hung
> for a week until I killed it.
> 
> I reran maker on just that contig by deleting the START line from the
> master log. maker again died on that contig.
> 
> I attach the relevant part of the stderr stream; the rest was just
> "--Next Contig--"-type lines. Any ideas?
> 
> A related question. Is deleting lines in the master log, as I did
> here, the best way to get maker to rerun on just a few contigs? I
> often find myself needing to do this, sometimes because of ERROR lines
> or other assumed quirks in maker and sometimes just because I needed
> to pause a big run for a while. One disadvantage is that when I
> restart, maker needs to manually scan over many completed contigs,
> which takes a while.
> 
> (I'd think the best solution would be for maker to automatically
> recognize whether it's really currently running on contigs marked
> STARTED but not FINISHED. It doesn't seem to do this now, but I
> realize that might be hard to implement.)
> 
> Thanks,
> Jeremy
> Grad student, Grishin lab
> UT Southwestern, Dallas TX
> 510.384.8959
> <maker.stderr>_______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From kmegy at ebi.ac.uk  Fri Aug 24 02:38:32 2012
From: kmegy at ebi.ac.uk (Karyn Megy)
Date: Fri, 24 Aug 2012 09:38:32 +0100
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
References: <CABPMbe121uhxGd6zehX+xHzuu_04sbYP+7TTERAHZXM3a7TVrQ@mail.gmail.com>
	<BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <463EA6A1-58CC-4358-9812-1B659DED9F13@ebi.ac.uk>

Hi Jeremy, 

Could it be a contig with lots of repeats, or that would hit lots of proteins?

One of my colleague had this issue and he ended splitting the few problematic contigs in sections... but then you have to reconciliate the results because the gene/BLAST coordinates are based in the bits of sub-contigs rather than the initial contig.  

You could try running MAKER on half of this contig and see what happen.

Cheers,
Karyn.

On 24 Aug 2012, at 08:44, Michael Thon wrote:

> I'm not sure if this is relevant to your problem but I had problems with maker failing on some contigs but not others.  I reinstalled maker and used the Build script to have it install all of its own dependencies.  Now maker is running fine, although the culprit seems to have been the blast version I was using before.
> 
> 
> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu> wrote:
> 
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org




From carsonhh at gmail.com  Fri Aug 24 06:24:56 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 08:24:56 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
Message-ID: <CC5CEA43.11379%carsonhh@gmail.com>

Could you send the blast version informations for reference purposes? I
can see this being caused by BLAST.

Thanks,
Carson



On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:

>I'm not sure if this is relevant to your problem but I had problems with
>maker failing on some contigs but not others.  I reinstalled maker and
>used the Build script to have it install all of its own dependencies.
>Now maker is running fine, although the culprit seems to have been the
>blast version I was using before.
>
>
>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>wrote:
>
>> Hi,
>> 
>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>> ~36,000 contigs, and per the master log maker FINISHED all of them
>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>> run. It didn't report ERROR for that contig in the master log, but it
>> did output two DIED lines in the contig's run.log and then just hung
>> for a week until I killed it.
>> 
>> I reran maker on just that contig by deleting the START line from the
>> master log. maker again died on that contig.
>> 
>> I attach the relevant part of the stderr stream; the rest was just
>> "--Next Contig--"-type lines. Any ideas?
>> 
>> A related question. Is deleting lines in the master log, as I did
>> here, the best way to get maker to rerun on just a few contigs? I
>> often find myself needing to do this, sometimes because of ERROR lines
>> or other assumed quirks in maker and sometimes just because I needed
>> to pause a big run for a while. One disadvantage is that when I
>> restart, maker needs to manually scan over many completed contigs,
>> which takes a while.
>> 
>> (I'd think the best solution would be for maker to automatically
>> recognize whether it's really currently running on contigs marked
>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>> realize that might be hard to implement.)
>> 
>> Thanks,
>> Jeremy
>> Grad student, Grishin lab
>> UT Southwestern, Dallas TX
>> 510.384.8959
>> <maker.stderr>_______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>
>_______________________________________________
>maker-devel mailing list
>maker-devel at box290.bluehost.com
>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org





From jeremy.semeiks at utsw.edu  Fri Aug 24 13:13:39 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 14:13:39 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC5CEA43.11379%carsonhh@gmail.com>
References: <BF6D5904-4723-45F5-8591-E3A52B876BF4@gmail.com>
	<CC5CEA43.11379%carsonhh@gmail.com>
Message-ID: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>

Carson,

$ blastx -version
blastx: 2.2.26+
Package: blast 2.2.26, build Feb 22 2012 16:04:28

- J

On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
> Could you send the blast version informations for reference purposes? I
> can see this being caused by BLAST.
>
> Thanks,
> Carson
>
>
>
> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>
>>I'm not sure if this is relevant to your problem but I had problems with
>>maker failing on some contigs but not others.  I reinstalled maker and
>>used the Build script to have it install all of its own dependencies.
>>Now maker is running fine, although the culprit seems to have been the
>>blast version I was using before.
>>
>>
>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>> run. It didn't report ERROR for that contig in the master log, but it
>>> did output two DIED lines in the contig's run.log and then just hung
>>> for a week until I killed it.
>>>
>>> I reran maker on just that contig by deleting the START line from the
>>> master log. maker again died on that contig.
>>>
>>> I attach the relevant part of the stderr stream; the rest was just
>>> "--Next Contig--"-type lines. Any ideas?
>>>
>>> A related question. Is deleting lines in the master log, as I did
>>> here, the best way to get maker to rerun on just a few contigs? I
>>> often find myself needing to do this, sometimes because of ERROR lines
>>> or other assumed quirks in maker and sometimes just because I needed
>>> to pause a big run for a while. One disadvantage is that when I
>>> restart, maker needs to manually scan over many completed contigs,
>>> which takes a while.
>>>
>>> (I'd think the best solution would be for maker to automatically
>>> recognize whether it's really currently running on contigs marked
>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>> realize that might be hard to implement.)
>>>
>>> Thanks,
>>> Jeremy
>>> Grad student, Grishin lab
>>> UT Southwestern, Dallas TX
>>> 510.384.8959
>>> <maker.stderr>_______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>
>>_______________________________________________
>>maker-devel mailing list
>>maker-devel at box290.bluehost.com
>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>



From carsonhh at gmail.com  Fri Aug 24 13:17:03 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 24 Aug 2012 15:17:03 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
Message-ID: <CC5D4ADD.11459%carsonhh@gmail.com>

Let maker install it's own blast (a version I know should work).

cd ./maker/src
./Build blast

Then to finish up your job (this command will just update executable
locations in the maker_exe.ctl file) -->
maker -EXE

Then run maker again with a higher retry count.  Let me know if the contig
still fails.

Thanks,
Carson




On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Carson,
>
>$ blastx -version
>blastx: 2.2.26+
>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>
>- J
>
>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send the blast version informations for reference purposes? I
>> can see this being caused by BLAST.
>>
>> Thanks,
>> Carson
>>
>>
>>
>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>
>>>I'm not sure if this is relevant to your problem but I had problems with
>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>used the Build script to have it install all of its own dependencies.
>>>Now maker is running fine, although the culprit seems to have been the
>>>blast version I was using before.
>>>
>>>
>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>> did output two DIED lines in the contig's run.log and then just hung
>>>> for a week until I killed it.
>>>>
>>>> I reran maker on just that contig by deleting the START line from the
>>>> master log. maker again died on that contig.
>>>>
>>>> I attach the relevant part of the stderr stream; the rest was just
>>>> "--Next Contig--"-type lines. Any ideas?
>>>>
>>>> A related question. Is deleting lines in the master log, as I did
>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>> or other assumed quirks in maker and sometimes just because I needed
>>>> to pause a big run for a while. One disadvantage is that when I
>>>> restart, maker needs to manually scan over many completed contigs,
>>>> which takes a while.
>>>>
>>>> (I'd think the best solution would be for maker to automatically
>>>> recognize whether it's really currently running on contigs marked
>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>> realize that might be hard to implement.)
>>>>
>>>> Thanks,
>>>> Jeremy
>>>> Grad student, Grishin lab
>>>> UT Southwestern, Dallas TX
>>>> 510.384.8959
>>>> <maker.stderr>_______________________________________________
>>>> maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> 
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>>>_______________________________________________
>>>maker-devel mailing list
>>>maker-devel at box290.bluehost.com
>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>
>>





From jeremy.semeiks at utsw.edu  Fri Aug 24 15:11:10 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 24 Aug 2012 16:11:10 -0500
Subject: [maker-devel]  maker dying on a contig
In-Reply-To: <CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
References: <CABPMbe28Y87-NCu2ABpiTL8LDtxBoxJfE1gX+9rrabDYXfwAfQ@mail.gmail.com>
	<CC5D4ADD.11459%carsonhh@gmail.com>
	<CABPMbe0h=TmXbBEC5siaaTCwe_xAea3FaT+GW9GZaD0bVEVjOw@mail.gmail.com>
Message-ID: <CABPMbe3nSDGzf_VqDj1Nucd0DPuPUB_tKcqQj64sV-2SyuUAQQ@mail.gmail.com>

After doing this, I still get the same errors as I attached before.
There was also nothing in the log supporting that maker tried this
more than once. (Initially, I had tries=2, but I changed it to
tries=10 for this run. I'm guessing that's what you mean by "retry
count".) It also doesn't matter if I keep the same old contig
directory and master log before rerunning, or if I delete them both.

FWIW, maker's own version of blast is 2.2.26+, which is the same as my
native version.

Thanks,
Jeremy

On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Let maker install it's own blast (a version I know should work).
>
> cd ./maker/src
> ./Build blast
>
> Then to finish up your job (this command will just update executable
> locations in the maker_exe.ctl file) -->
> maker -EXE
>
> Then run maker again with a higher retry count.  Let me know if the contig
> still fails.
>
> Thanks,
> Carson
>
>
>
>
> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Carson,
>>
>>$ blastx -version
>>blastx: 2.2.26+
>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>
>>- J
>>
>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send the blast version informations for reference purposes? I
>>> can see this being caused by BLAST.
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>>
>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>
>>>>I'm not sure if this is relevant to your problem but I had problems with
>>>>maker failing on some contigs but not others.  I reinstalled maker and
>>>>used the Build script to have it install all of its own dependencies.
>>>>Now maker is running fine, although the culprit seems to have been the
>>>>blast version I was using before.
>>>>
>>>>
>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks <jeremy.semeiks at utsw.edu>
>>>>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I have
>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the first
>>>>> run. It didn't report ERROR for that contig in the master log, but it
>>>>> did output two DIED lines in the contig's run.log and then just hung
>>>>> for a week until I killed it.
>>>>>
>>>>> I reran maker on just that contig by deleting the START line from the
>>>>> master log. maker again died on that contig.
>>>>>
>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>
>>>>> A related question. Is deleting lines in the master log, as I did
>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>> often find myself needing to do this, sometimes because of ERROR lines
>>>>> or other assumed quirks in maker and sometimes just because I needed
>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>> which takes a while.
>>>>>
>>>>> (I'd think the best solution would be for maker to automatically
>>>>> recognize whether it's really currently running on contigs marked
>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>> realize that might be hard to implement.)
>>>>>
>>>>> Thanks,
>>>>> Jeremy
>>>>> Grad student, Grishin lab
>>>>> UT Southwestern, Dallas TX
>>>>> 510.384.8959
>>>>> <maker.stderr>_______________________________________________
>>>>> maker-devel mailing list
>>>>> maker-devel at box290.bluehost.com
>>>>>
>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>>
>>>>
>>>>_______________________________________________
>>>>maker-devel mailing list
>>>>maker-devel at box290.bluehost.com
>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>
>



From jokelley at stanford.edu  Thu Aug 30 18:18:29 2012
From: jokelley at stanford.edu (Joanna Kelley)
Date: Thu, 30 Aug 2012 17:18:29 -0700
Subject: [maker-devel] convert gff to ncbi table format?
Message-ID: <CA+77YqHuS8CUK+YmDCQX8KMt1FWiMkJ00eXsgwdzHTM1PxsZ1g@mail.gmail.com>

Hello,

I am trying to submit the maker annotations to NCBI and
I was wondering if there are any tools to convert gff to
the NCBI table format for submission?

Thank you,
Joanna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120830/9a5ce895/attachment-0003.html>

From chrisi.hahni at gmail.com  Fri Aug 31 06:43:21 2012
From: chrisi.hahni at gmail.com (Christoph Hahn)
Date: Fri, 31 Aug 2012 14:43:21 +0200
Subject: [maker-devel] keep_preds option?
Message-ID: <5040B169.1030909@gmail.com>

Hello maker users and developers,

I am new to gene prediction and I am trying to use maker 2.25 on a newly 
assembled non-model organisms draft genome. Within maker I use genemark, 
SNAP and Augustus. I have a few questions:

1. I was wondering what the keep_preds option means exactly.

I found two slightly different explanations on the option
#Add unsupported gene prediction to final annotation set (maker2.25)
#Add non-overlapping ab-inito gene prediction to final annotation set 
(found on the net - probably older maker version)

As far as I understood keep_preds=0 only retains gene models for which 
the ab initio predictions agree. But how many, all three? two of three?
keep_preds=1 instead keeps all gene models regardless if the different 
programs agree, right?

In my case I get substantial differences in the number of gene models 
found between the two settings, while with =1 I get a number that is 
close to what we would expect. How would you interpret that? What would 
you recommend me to do? Obiously =0 is the saver option.

2. I tried to use EST data of an alternative organism in altest= 
(#EST/cDNA sequence file in fasta format from an alternate organism). 
The organism is quite distantly related, but its the closest I have so I 
thought I d give it a shot. I ran maker twice with identical settigs 
expect in altest and est2genome=0/1. The number of genes predicted is 
identical with both approaches, so I am not sure whether or not the EST 
data was actually used or its just to distantly related. Any easy way to 
assess this?

3. I am running maker in several passes and after each pass I am 
training SNAP using the result of the previous pass. Then for every pass 
I run maker from scratch. Would you recommend to supply the gff of the 
previous pass in "#-----Re-annotation Using MAKER Derived GFF3
maker_gff= #re-annotate genome based on this gff3 file", instead?

Thanks in advance for any thoughts/advice on these things! I appreciate 
your help!

much obliged,
Christoph



From barry.moore at genetics.utah.edu  Fri Aug 31 10:52:10 2012
From: barry.moore at genetics.utah.edu (Barry Moore)
Date: Fri, 31 Aug 2012 10:52:10 -0600
Subject: [maker-devel] keep_preds option?
In-Reply-To: <5040B169.1030909@gmail.com>
References: <5040B169.1030909@gmail.com>
Message-ID: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly assembled non-model organisms draft genome. Within maker I use genemark, SNAP and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical evidence.  There are two pieces of information that are required for every MAKER annotation (by default).  MAKER runs the ab initio gene predictors and aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to the genome.  For each locus where one or more gene predictions exist MAKER checks to see if there is any physical evidence for gene expression at that locus (RNA/protein sequence alignments) and if there is (splice EST or protein alignments) evidence overlapping the predictions, MAKER decides which prediction best matches the evidence and promotes that prediction to an annotation.  If there is no evidence overlapping any of the predictions then those predictions are not included in the output annotation file (although they are saved).  If you set keep_preds=1 then for each locus where prediction(s) exist maker keeps one and promotes it to an annotation even though there is no physical evidence.  The description of 'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as evidence and thus in the absence of aligned physical evidence MAKER will not output an annotation even if all three ab initio tools predict a gene at that locus.

> In my case I get substantial differences in the number of gene models found between the two settings, while with =1 I get a number that is close to what we would expect. How would you interpret that? What would you recommend me to do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is too few, you could run MAKER with keep_preds=1.  After the run is finished, use a tool like IPRScan to search all MAKER predictions for protein domain content and push that IPRScan output back into the MAKER GFF file with the ipr_update_gff script.  Then as a final step you can run over the GFF file and remove any gene model that doesn't have either physical evidence (AED < 1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA sequence file in fasta format from an alternate organism). The organism is quite distantly related, but its the closest I have so I thought I d give it a shot. I ran maker twice with identical settigs expect in altest and est2genome=0/1. The number of genes predicted is identical with both approaches, so I am not sure whether or not the EST data was actually used or its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in the way of additional support (compared to just using protein evidence from say Swiss-prot) and this would be especially true if your alt_est is distantly related.  I'm not sure I really understand you alt_est/est2genome combo's to comment in more detail.  I could see four possible combinations there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP using the result of the previous pass. Then for every pass I run maker from scratch. Would you recommend to supply the gff of the previous pass in "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a genome when you want certain parts of the previous run to be passed through unchanged, but with retraining SNAP you want MAKER to re-evaluate each annotation in light of the new predictions made by the retrained SNAP.  MAKER should run really fast in all of the runs after the first one because as long as you haven't changed the evidence files it won't have to redo any of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/8b0671bd/attachment-0003.html>

From carsonhh at gmail.com  Fri Aug 31 13:03:14 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 15:03:14 -0400
Subject: [maker-devel] keep_preds option?
In-Reply-To: <B2ED2F80-0122-4938-A9B9-67885DC631C7@genetics.utah.edu>
Message-ID: <CC667EFB.117DD%carsonhh@gmail.com>

I concur with everything Barry said.  Also let me add that alt-ESTs do not
get polished around splice sites (exonerate won't handle them).  However
ESTs and proteins do.  This means that the utility of alt-ESTs in adding
UTR, and splice information is zero.  They just function as an anchor to
maintain gene models that might have otherwise been rejected.  This also
means alt_est=some.fasta  together with est2genome=1 will produce virtually
no additional results because est2genome requires that the splice site makes
sense.  You are better off using proteins with protein2genome=1 if you don?t
have ESTs and want partial models for training.  Once you have a trained ab
initio gene predictor, turn the est2genome and protein2genome options off.
Otherwise they will give weird partial models that decrease the quality of
your final annotations. (partial models are ok for training but that's about
it).

If you are getting too low a gene count with keep_preds=0, then you probably
need to add more evidence.  Try adding all proteins from a couple of related
species (the protein= option accepts comma separated lists of files). If
your genome is a fungi, oomycete, or a prokaryote, then setting keep_preds=1
is usually safe.  Theses are genomes with high gene density and simple gene
structure, so ab initio predictors do really well and don't need as much
help from the evidence.  For other organisms, leave it set to 0 or you will
get a lot of false positives (false positives on some genomes with complex
gene structure can outnumber the genes by 10 to 1 if you turn this on).

Thanks,
Carson




From:  Barry Moore <barry.moore at genetics.utah.edu>
Date:  Friday, 31 August, 2012 12:52 PM
To:  Christoph Hahn <chrisi.hahni at gmail.com>
Cc:  <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] keep_preds option?

Hi Christopher,

Comments below:

On Aug 31, 2012, at 6:43 AM, Christoph Hahn wrote:

> Hello maker users and developers,
> 
> I am new to gene prediction and I am trying to use maker 2.25 on a newly
> assembled non-model organisms draft genome. Within maker I use genemark, SNAP
> and Augustus. I have a few questions:
> 

Welcome!

> 1. I was wondering what the keep_preds option means exactly.
> 
> I found two slightly different explanations on the option
> #Add unsupported gene prediction to final annotation set (maker2.25)
> #Add non-overlapping ab-inito gene prediction to final annotation set (found
> on the net - probably older maker version)
> 

It means to keep ab initio gene predictions for which there is no physical
evidence.  There are two pieces of information that are required for every
MAKER annotation (by default).  MAKER runs the ab initio gene predictors and
aligns transcript (EST/cDNA/mRNASeq transcripts) and protein sequences to
the genome.  For each locus where one or more gene predictions exist MAKER
checks to see if there is any physical evidence for gene expression at that
locus (RNA/protein sequence alignments) and if there is (splice EST or
protein alignments) evidence overlapping the predictions, MAKER decides
which prediction best matches the evidence and promotes that prediction to
an annotation.  If there is no evidence overlapping any of the predictions
then those predictions are not included in the output annotation file
(although they are saved).  If you set keep_preds=1 then for each locus
where prediction(s) exist maker keeps one and promotes it to an annotation
even though there is no physical evidence.  The description of
'non-overlapping ab-initio'  means that MAKER has clustered all ab-initio
predictions at one locus and chose one representative transcript to output.

> As far as I understood keep_preds=0 only retains gene models for which the ab
> initio predictions agree. But how many, all three? two of three?
> keep_preds=1 instead keeps all gene models regardless if the different
> programs agree, right?
> 

MAKER does not take the presence of multiple ab initio predictions as
evidence and thus in the absence of aligned physical evidence MAKER will not
output an annotation even if all three ab initio tools predict a gene at
that locus.

> In my case I get substantial differences in the number of gene models found
> between the two settings, while with =1 I get a number that is close to what
> we would expect. How would you interpret that? What would you recommend me to
> do? Obiously =0 is the saver option.

If you think that the number of genes you are getting from a MAKER run is
too few, you could run MAKER with keep_preds=1.  After the run is finished,
use a tool like IPRScan to search all MAKER predictions for protein domain
content and push that IPRScan output back into the MAKER GFF file with the
ipr_update_gff script.  Then as a final step you can run over the GFF file
and remove any gene model that doesn't have either physical evidence (AED <
1) or protein domain content (Dbxref=PFAM:XXX etc?) sorry there's not a
script prepackaged with MAKER for that yet.

> 
> 2. I tried to use EST data of an alternative organism in altest= (#EST/cDNA
> sequence file in fasta format from an alternate organism). The organism is
> quite distantly related, but its the closest I have so I thought I d give it a
> shot. I ran maker twice with identical settigs expect in altest and
> est2genome=0/1. The number of genes predicted is identical with both
> approaches, so I am not sure whether or not the EST data was actually used or
> its just to distantly related. Any easy way to assess this?

Typically EST evidence from another organism with alt_est will add little in
the way of additional support (compared to just using protein evidence from
say Swiss-prot) and this would be especially true if your alt_est is
distantly related.  I'm not sure I really understand you alt_est/est2genome
combo's to comment in more detail.  I could see four possible combinations
there: which two gave identical results?

> 
> 3. I am running maker in several passes and after each pass I am training SNAP
> using the result of the previous pass. Then for every pass I run maker from
> scratch. Would you recommend to supply the gff of the previous pass in
> "#-----Re-annotation Using MAKER Derived GFF3
> maker_gff= #re-annotate genome based on this gff3 file", instead?
> 

No, 'Re-annotation using MAKER Derived GFF3' is used for re-annotation of a
genome when you want certain parts of the previous run to be passed through
unchanged, but with retraining SNAP you want MAKER to re-evaluate each
annotation in light of the new predictions made by the retrained SNAP.
MAKER should run really fast in all of the runs after the first one because
as long as you haven't changed the evidence files it won't have to redo any
of the alignments.


B

> Thanks in advance for any thoughts/advice on these things! I appreciate your
> help!
> 
> much obliged,
> Christoph
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

Barry Moore
Research Scientist
Dept. of Human Genetics
University of Utah
Salt Lake City, UT 84112
--------------------------------------------
(801) 585-3543




_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/ed670166/attachment-0003.html>

From carsonhh at gmail.com  Fri Aug 31 14:47:33 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 16:47:33 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
Message-ID: <CC66941F.118C8%carsonhh@gmail.com>

Looks like there is a slight format variation in one line of the
RepeatMasker outputs.  Not sure what could have caused it -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          0    109    (0)

The position/compliment information in columns 12-14 give a start position
of 0 for part of the alignment.  A 0 is invalid because repeat masker
doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
asks for the starting position of the alignment in another part of the
code.

This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
wu-blast and then reconfigure RepeatMasker I get this line -->
249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
Simple_repeat          1    109    (0)

Notice the 0 becomes a 1 and all logic is restored to the world.

>From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
output, then I just turn it into a 1.
I've attached a file that you should use to replace the
./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
I'll also make a note to the RepeatMasker developers.

Thanks,
Carson



On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

>Attached.
>
>Thanks,
>J
>
>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Could you send me the entire directory for just the failed contig?
>>
>> Thanks,
>> Carson
>>
>>
>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>
>>>After doing this, I still get the same errors as I attached before.
>>>There was also nothing in the log supporting that maker tried this
>>>more than once. (Initially, I had tries=2, but I changed it to
>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>count".) It also doesn't matter if I keep the same old contig
>>>directory and master log before rerunning, or if I delete them both.
>>>
>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>native version.
>>>
>>>Thanks,
>>>Jeremy
>>>
>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Let maker install it's own blast (a version I know should work).
>>>>
>>>> cd ./maker/src
>>>> ./Build blast
>>>>
>>>> Then to finish up your job (this command will just update executable
>>>> locations in the maker_exe.ctl file) -->
>>>> maker -EXE
>>>>
>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>contig
>>>> still fails.
>>>>
>>>> Thanks,
>>>> Carson
>>>>
>>>>
>>>>
>>>>
>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>
>>>>>Carson,
>>>>>
>>>>>$ blastx -version
>>>>>blastx: 2.2.26+
>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>
>>>>>- J
>>>>>
>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>wrote:
>>>>>> Could you send the blast version informations for reference
>>>>>>purposes?
>>>>>>I
>>>>>> can see this being caused by BLAST.
>>>>>>
>>>>>> Thanks,
>>>>>> Carson
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>
>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>with
>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>and
>>>>>>>used the Build script to have it install all of its own
>>>>>>>dependencies.
>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>the
>>>>>>>blast version I was using before.
>>>>>>>
>>>>>>>
>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>have
>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>first
>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>it
>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>hung
>>>>>>>> for a week until I killed it.
>>>>>>>>
>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>the
>>>>>>>> master log. maker again died on that contig.
>>>>>>>>
>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>
>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>lines
>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>needed
>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>> which takes a while.
>>>>>>>>
>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>> realize that might be hard to implement.)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jeremy
>>>>>>>> Grad student, Grishin lab
>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>> 510.384.8959
>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>> maker-devel mailing list
>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>.o
>>>>>>>>rg
>>>>>>>
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>maker-devel mailing list
>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>or
>>>>>>>g
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: RepeatMasker.pm
Type: text/x-perl-script
Size: 9226 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20120831/c248081c/attachment-0003.bin>

From jeremy.semeiks at utsw.edu  Fri Aug 31 16:48:37 2012
From: jeremy.semeiks at utsw.edu (Jeremy Semeiks)
Date: Fri, 31 Aug 2012 17:48:37 -0500
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CC66941F.118C8%carsonhh@gmail.com>
References: <CABPMbe1BbQxaq0Tp2cbwCEW8h0uUJz3twfFPjYD7P5LGamYcsQ@mail.gmail.com>
	<CC66941F.118C8%carsonhh@gmail.com>
Message-ID: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>

Carson, maker completed on the scaffold without error when I used your
patch. Thanks!

- J

On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
> Looks like there is a slight format variation in one line of the
> RepeatMasker outputs.  Not sure what could have caused it -->
> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          0    109    (0)
>
> The position/compliment information in columns 12-14 give a start position
> of 0 for part of the alignment.  A 0 is invalid because repeat masker
> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
> asks for the starting position of the alignment in another part of the
> code.
>
> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
> wu-blast and then reconfigure RepeatMasker I get this line -->
> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
> Simple_repeat          1    109    (0)
>
> Notice the 0 becomes a 1 and all logic is restored to the world.
>
> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
> output, then I just turn it into a 1.
> I've attached a file that you should use to replace the
> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
> I'll also make a note to the RepeatMasker developers.
>
> Thanks,
> Carson
>
>
>
> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>
>>Attached.
>>
>>Thanks,
>>J
>>
>>On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Could you send me the entire directory for just the failed contig?
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>
>>>>After doing this, I still get the same errors as I attached before.
>>>>There was also nothing in the log supporting that maker tried this
>>>>more than once. (Initially, I had tries=2, but I changed it to
>>>>tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>count".) It also doesn't matter if I keep the same old contig
>>>>directory and master log before rerunning, or if I delete them both.
>>>>
>>>>FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>native version.
>>>>
>>>>Thanks,
>>>>Jeremy
>>>>
>>>>On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>> Let maker install it's own blast (a version I know should work).
>>>>>
>>>>> cd ./maker/src
>>>>> ./Build blast
>>>>>
>>>>> Then to finish up your job (this command will just update executable
>>>>> locations in the maker_exe.ctl file) -->
>>>>> maker -EXE
>>>>>
>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>contig
>>>>> still fails.
>>>>>
>>>>> Thanks,
>>>>> Carson
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>
>>>>>>Carson,
>>>>>>
>>>>>>$ blastx -version
>>>>>>blastx: 2.2.26+
>>>>>>Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>
>>>>>>- J
>>>>>>
>>>>>>On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>wrote:
>>>>>>> Could you send the blast version informations for reference
>>>>>>>purposes?
>>>>>>>I
>>>>>>> can see this being caused by BLAST.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Carson
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>
>>>>>>>>I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>with
>>>>>>>>maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>and
>>>>>>>>used the Build script to have it install all of its own
>>>>>>>>dependencies.
>>>>>>>>Now maker is running fine, although the culprit seems to have been
>>>>>>>>the
>>>>>>>>blast version I was using before.
>>>>>>>>
>>>>>>>>
>>>>>>>>On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>><jeremy.semeiks at utsw.edu>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>have
>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>first
>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>it
>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>hung
>>>>>>>>> for a week until I killed it.
>>>>>>>>>
>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>the
>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>
>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>
>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>lines
>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>needed
>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>> which takes a while.
>>>>>>>>>
>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jeremy
>>>>>>>>> Grad student, Grishin lab
>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>> 510.384.8959
>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>
>>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>.o
>>>>>>>>>rg
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>maker-devel mailing list
>>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>or
>>>>>>>>g
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>



From carsonhh at gmail.com  Fri Aug 31 17:00:27 2012
From: carsonhh at gmail.com (Carson Holt)
Date: Fri, 31 Aug 2012 19:00:27 -0400
Subject: [maker-devel] maker dying on a contig
In-Reply-To: <CABPMbe2G8jB1R-+0H8agoEOaYVgSFtYLcoZ7Umy4hW81OpMn_A@mail.gmail.com>
Message-ID: <CC66BA4B.820D%carsonhh@gmail.com>

Your welcome.  I also submitted a bug report to RepeatMasker.  They said the
issue will go away with the next release since simple repeat detection is
going to be handled by TRF rather than rmblast.

Thanks,
Carson


On 8/31/12 6:48 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:

> Carson, maker completed on the scaffold without error when I used your
> patch. Thanks!
> 
> - J
> 
> On Fri, Aug 31, 2012 at 3:47 PM, Carson Holt <carsonhh at gmail.com> wrote:
>> Looks like there is a slight format variation in one line of the
>> RepeatMasker outputs.  Not sure what could have caused it -->
>> 249 30.8 1.9 0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          0    109    (0)
>> 
>> The position/compliment information in columns 12-14 give a start position
>> of 0 for part of the alignment.  A 0 is invalid because repeat masker
>> doesn't use 0 based coordinates.  This causes a bug downstream when MAKER
>> asks for the starting position of the alignment in another part of the
>> code.
>> 
>> This may be a RepeatMasker or rmblast bug.  When I replace rmblast with
>> wu-blast and then reconfigure RepeatMasker I get this line -->
>> 249   30.8  1.9  0.9  scaffold_10105  22320 22372 (34998) + (TCCA)n
>> Simple_repeat          1    109    (0)
>> 
>> Notice the 0 becomes a 1 and all logic is restored to the world.
>> 
>> From my end the fix is simple.  If I see a 0 when reading RepeatMaskers
>> output, then I just turn it into a 1.
>> I've attached a file that you should use to replace the
>> ./maker/lib/Widget/RepeatMasker.pm file.  It will fix the failure issue.
>> I'll also make a note to the RepeatMasker developers.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 12-08-25 12:21 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>> 
>>> Attached.
>>> 
>>> Thanks,
>>> J
>>> 
>>> On Fri, Aug 24, 2012 at 5:23 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>> Could you send me the entire directory for just the failed contig?
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> 
>>>> On 12-08-24 5:09 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>> 
>>>>> After doing this, I still get the same errors as I attached before.
>>>>> There was also nothing in the log supporting that maker tried this
>>>>> more than once. (Initially, I had tries=2, but I changed it to
>>>>> tries=10 for this run. I'm guessing that's what you mean by "retry
>>>>> count".) It also doesn't matter if I keep the same old contig
>>>>> directory and master log before rerunning, or if I delete them both.
>>>>> 
>>>>> FWIW, maker's own version of blast is 2.2.26+, which is the same as my
>>>>> native version.
>>>>> 
>>>>> Thanks,
>>>>> Jeremy
>>>>> 
>>>>> On Fri, Aug 24, 2012 at 2:17 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>>>> Let maker install it's own blast (a version I know should work).
>>>>>> 
>>>>>> cd ./maker/src
>>>>>> ./Build blast
>>>>>> 
>>>>>> Then to finish up your job (this command will just update executable
>>>>>> locations in the maker_exe.ctl file) -->
>>>>>> maker -EXE
>>>>>> 
>>>>>> Then run maker again with a higher retry count.  Let me know if the
>>>>>> contig
>>>>>> still fails.
>>>>>> 
>>>>>> Thanks,
>>>>>> Carson
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 12-08-24 3:13 PM, "Jeremy Semeiks" <jeremy.semeiks at utsw.edu> wrote:
>>>>>> 
>>>>>>> Carson,
>>>>>>> 
>>>>>>> $ blastx -version
>>>>>>> blastx: 2.2.26+
>>>>>>> Package: blast 2.2.26, build Feb 22 2012 16:04:28
>>>>>>> 
>>>>>>> - J
>>>>>>> 
>>>>>>> On Fri, Aug 24, 2012 at 7:24 AM, Carson Holt <carsonhh at gmail.com>
>>>>>>> wrote:
>>>>>>>> Could you send the blast version informations for reference
>>>>>>>> purposes?
>>>>>>>> I
>>>>>>>> can see this being caused by BLAST.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Carson
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 12-08-24 3:44 AM, "Michael Thon" <mike.thon at gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> I'm not sure if this is relevant to your problem but I had problems
>>>>>>>>> with
>>>>>>>>> maker failing on some contigs but not others.  I reinstalled maker
>>>>>>>>> and
>>>>>>>>> used the Build script to have it install all of its own
>>>>>>>>> dependencies.
>>>>>>>>> Now maker is running fine, although the culprit seems to have been
>>>>>>>>> the
>>>>>>>>> blast version I was using before.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Aug 21, 2012, at 6:26 PM, Jeremy Semeiks
>>>>>>>>> <jeremy.semeiks at utsw.edu>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm trying to annotate a mammalian genome with maker-2.26-beta. I
>>>>>>>>>> have
>>>>>>>>>> ~36,000 contigs, and per the master log maker FINISHED all of them
>>>>>>>>>> except one (scaffold_10105, ~60 kbp, not obviously weird) on the
>>>>>>>>>> first
>>>>>>>>>> run. It didn't report ERROR for that contig in the master log, but
>>>>>>>>>> it
>>>>>>>>>> did output two DIED lines in the contig's run.log and then just
>>>>>>>>>> hung
>>>>>>>>>> for a week until I killed it.
>>>>>>>>>> 
>>>>>>>>>> I reran maker on just that contig by deleting the START line from
>>>>>>>>>> the
>>>>>>>>>> master log. maker again died on that contig.
>>>>>>>>>> 
>>>>>>>>>> I attach the relevant part of the stderr stream; the rest was just
>>>>>>>>>> "--Next Contig--"-type lines. Any ideas?
>>>>>>>>>> 
>>>>>>>>>> A related question. Is deleting lines in the master log, as I did
>>>>>>>>>> here, the best way to get maker to rerun on just a few contigs? I
>>>>>>>>>> often find myself needing to do this, sometimes because of ERROR
>>>>>>>>>> lines
>>>>>>>>>> or other assumed quirks in maker and sometimes just because I
>>>>>>>>>> needed
>>>>>>>>>> to pause a big run for a while. One disadvantage is that when I
>>>>>>>>>> restart, maker needs to manually scan over many completed contigs,
>>>>>>>>>> which takes a while.
>>>>>>>>>> 
>>>>>>>>>> (I'd think the best solution would be for maker to automatically
>>>>>>>>>> recognize whether it's really currently running on contigs marked
>>>>>>>>>> STARTED but not FINISHED. It doesn't seem to do this now, but I
>>>>>>>>>> realize that might be hard to implement.)
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Jeremy
>>>>>>>>>> Grad student, Grishin lab
>>>>>>>>>> UT Southwestern, Dallas TX
>>>>>>>>>> 510.384.8959
>>>>>>>>>> <maker.stderr>_______________________________________________
>>>>>>>>>> maker-devel mailing list
>>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>>> 
>>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>> .o
>>>>>>>>>> rg
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> maker-devel mailing list
>>>>>>>>> maker-devel at box290.bluehost.com
>>>>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.
>>>>>>>>> or
>>>>>>>>> g
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>