[maker-devel] Repeatmasker error?

Carson Holt carsonhh at gmail.com
Mon Apr 22 14:25:06 MDT 2013 

Just forwarding this e-mail chain to the devel list for archiving.

--Carson

From:  "Freeman, Robert M." <Bob_Freeman at hms.harvard.edu>
Date:  Monday, 22 April, 2013 4:16 PM
To:  Carson Holt <carsonhh at gmail.com>
Subject:  Re: [maker-devel] Repeatmasker error?

Already looks better ... been checking stderr and it looks error-free so far
(knock on wood). Thanks for the help, and sorry for the bother!

Oh, should I fall back to the 2.27beta release that you announced on the
list??

-b

On Apr 22, 2013, at 4:09 PM, Carson Holt wrote:

> Let me know you still get problems.  Redirecting TMP back locally will also
> give a big performance boost.
> 
> Thanks,
> Carson
> 
> 
> From:  "Freeman, Robert M." <Bob_Freeman at hms.harvard.edu>
> Date:  Monday, 22 April, 2013 4:01 PM
> To:  Carson Holt <carsonhh at gmail.com>
> Subject:  Re: [maker-devel] Repeatmasker error?
> 
> (chuckle) wow, always something new to learn -- been working with IT systems
> for > 20 years, and HPC > 8, and no one has ever explained this to me.
> 
> Have directed TMP to /scratch, which also turns out to be an Isilon-related
> mount. Will re-direct all this to /tmp to see if this eliminates the problems.
> 
> -b
> 
> On Apr 22, 2013, at 3:43 PM, Carson Holt wrote:
> 
>> The missing file is part of the GFF3 output, the fasta sequence to be
>> specific.  Sometimes on NFS (network mounted file systems), they can return
>> status 'success' even though the IO event really has not succeeded yet (this
>> is called asynchronous IO).  The result is a certain speed gain but it also
>> means that you can write a file, then immediately try and open it, and the
>> system will say that it doesn't exist.  On some systems you get weird files
>> starting with the name '.nfs000' when these types of errors occur.  NFS type
>> errors are more common when you use many cpus or other jobs on the cluster
>> (not just maker) are using a large amount of IO.  To avoid this, MAKER tries
>> to do as much work as possible in the directory specified by TMP in the
>> control files.  By default this is /tmp, and if you set it to something else,
>> make sure that the location is locally mounted and not NFS mounted (otherwise
>> it can't perform it's purpose of bypassing NFS for certain quick read/write
>> operations).  The newest version of MAKER unloads exonerate and even most
>> gene prediction operations into TMP in addition to other steps that were
>> already unloaded there in other versions of the pipeline, and I've been able
>> to scale up to > 1500 cpus.
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> From:  "Freeman, Robert M." <Bob_Freeman at hms.harvard.edu>
>> Date:  Monday, 22 April, 2013 3:24 PM
>> To:  Carson Holt <carsonhh at gmail.com>
>> Subject:  Re: [maker-devel] Repeatmasker error?
>> 
>> Thanks, Carson. I'll give this a try.
>> 
>> Randomly? Not sure ... I'll have to go back thru the logs to see if this is
>> happening consistently or not. Right now, this log is close to 1 GB in size.
>> When I saw it getting this large, I stopped the run as I knew errors were
>> getting spewed into the log file.
>> 
>> Thought it might be filesystem as well, but unlikely -- the location for the
>> MAKER runs is on our Isilon, and these problems appear only with MAKER.
>> 
>> Other files seem to be present...
>> 
>>> % ls -alt
>>> drwxrwx--- 3 rmf1 SYSTEMBIO_klab_genome     236 Apr 21 14:50 ..
>>> drwxrwx--- 2 rmf1 SYSTEMBIO_klab_genome   48225 Apr 21 12:38 .
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome       0 Apr 21 12:34 run.log.child.0
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome 1055922 Apr 21 12:34
>>> contig_157.0.final.section
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome 1055922 Apr 21 12:34
>>> contig_157.0.raw.section
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome  388512 Apr 21 12:34 evidence_0.gff
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome    7269 Apr 21 12:34
>>> contig_157.102049-103030.gi%7C145478069%7Cref%7CXP_001425057%
>>> 2E1%7C.p_exonerate
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome    4561 Apr 21 12:34
>>> contig_157.101950-103090.gi%7C145514179%7Cref%7CXP_001443000%
>>> 2E1%7C.p_exonerate
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome    7088 Apr 21 12:34
>>> contig_157.101956-103435.gi%7C145505343%7Cref%7CXP_001438638%
>>> 2E1%7C.p_exonerate
>>> ....
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome 7184469 Apr 21 12:33
>>> contig_157.0.sequences_r5%2Efasta.blastx
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome       0 Apr 21 12:25
>>> query.masked.gff.def
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome    9885 Apr 21 12:25
>>> query.masked.gff.ann
>>> -rw-r--r-- 1 rmf1 SYSTEMBIO_klab_genome   49152 Apr 21 12:25
>>> query.masked.fasta.index
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome  106522 Apr 21 12:25
>>> query.masked.fasta
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome    9991 Apr 21 12:25
>>> query.masked.gff
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome   28002 Apr 21 12:25
>>> contig_157.0.te_proteins%2Efasta.repeatrunner
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome    3589 Apr 21 12:24
>>> contig_157.0.all.rb.out
>>> -rwxrwx--- 1 rmf1 SYSTEMBIO_klab_genome  106522 Apr 21 12:23 query.fasta
>>> 
>> It's just that the one output file MAKER is looking for isn't there.
>> 
>> I guess the other question I should ask: as there are exonerate sequences
>> there, does it appear that the pipeline is running OK, and just ignore these
>> errors (somehow)?
>> 
>> -b
>> 
>> On Apr 22, 2013, at 12:39 PM, Carson Holt wrote:
>> 
>>> Could you give the devel version a try to see if it experiences the same
>>> failure, as it's easier to debug off of the most current code.
>>> 
>>> Type this on the command line to download-->
>>> **************************
>>> 
>>> user: *******
>>> password: *******
>>> 
>>> The error appears to be filesystem related though.  Does it appear to happen
>>> randomly?
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> From:  "Freeman, Robert M." <Bob_Freeman at hms.harvard.edu>
>>> Date:  Monday, 22 April, 2013 10:09 AM
>>> To:  "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
>>> Subject:  [maker-devel] Repeatmasker error?
>>> 
>>> Greetings,
>>> 
>>> Am using MAKER 2.27b to annotate a ciliate genome and am finding that my log
>>> files are growing to GB sizes. When looking more carefully, an error seems
>>> to be occurring around the Repeatmasker stage:
>>> 
>>>> ....
>>>> Now starting the contig!!
>>>> --
>>>> 
>>>> setting up GFF3 output and fasta chunks
>>>> doing repeat masking
>>>> doing blastx repeats
>>>> doing blastx of proteins
>>>> doing blastx of proteins
>>>> doing blastx of proteins
>>>> doing blastx repeats
>>>> collecting blastx repeatmasking
>>>> processing all repeats
>>>> ERROR: Can't open seq file:
>>>> /files/.retain-snapshots.d14d-w60d/SysBio/klab_genome/maker/stentor/run_cur
>>>> rent_r3/soapPrice1.cycle7.maker.output/soapPrice1.cycle7_datastore/03/EF/co
>>>> ntig_157//theVoid.contig_157/query.masked.gff.seq
>>>> No such file or directory
>>>> 
>>>>  at /groups/acornworm/opt/maker-2.27-beta/bin/../lib/Dumper/GFF/GFFV3.pm
>>>> line 182
>>>>         Dumper::GFF::GFFV3::finalize('Dumper::GFF::GFFV3=HASH(0x50547f8)')
>>>> called at 
>>>> /groups/acornworm/opt/maker-2.27-beta/bin/../lib/Process/MpiChunk.pm line
>>>> 691
>>>>         Process::MpiChunk::__ANON__() called at
>>>> /groups/acornworm/opt/maker-2.27-beta/bin/../lib/Error.pm line 415
>>>>         eval {...} called at
>>>> /groups/acornworm/opt/maker-2.27-beta/bin/../lib/Error.pm line 407
>>>>         Error::subs::try('CODE(0x5b859c0)', 'HASH(0x11ea63f0)') called at
>>>> /groups/acornworm/opt/maker-2.27-beta/bin/../lib
>>>> ...
>>> 
>>> I don't seem to have this problem when I fall back to the 2.25b version
>>> (though I start having major DBD:SQLite issues).
>>> 
>>> I'm doing this on a cluster, running this under MPI with 50 cores.
>>> 
>>> Any help/suggestions would be appreciated!
>>> 
>>> -Bob
>>> 
>>> _______________________________________________ maker-devel mailing list
>>> maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/m
>>> aker-devel_yandell-lab.org
>> 
>> 

-----------------------------------------------------
Bob Freeman, Ph.D.
Acorn Worm Informatics, Kirschner lab
Dept of Systems Biology, Alpert 524
Harvard Medical School
200 Longwood Avenue
Boston, MA  02115
617/432.2294, vox

"Sorry I'm late. Oh, God, that sounded insincere. I'm late."
-- Karen Walker, from Will and Grace





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130422/e3af4618/attachment-0003.html>

More information about the maker-devel mailing list