[maker-devel] Strange error at blastn step
Daniel Standage
daniel.standage at gmail.com
Mon Oct 29 07:43:46 MDT 2012
I just ran svn update and tried again with the development version of
Maker. Same long list of errors as in the last message.
--
Daniel S. Standage
Ph.D. Candidate
Bioinformatics and Computational Biology Program
Department of Genetics, Development, and Cell Biology
Iowa State University
On Mon, Oct 29, 2012 at 9:39 AM, Daniel Standage
<daniel.standage at gmail.com>wrote:
> Carson,
>
> I was able to run the first svn update before the test job ran, but I
> didn't get the message about this second update until after it has already
> executed and failed. I got about 372 lines of such.
>
> STATUS: Parsing control files...
> STATUS: Processing and indexing input FASTA files...
> STATUS: Setting up database for any GFF3 input...
> A data structure will be created for you at:
>
> /N/dc/scratch/dstandag/PdomGenomic/Annotation/output/DELETEME.maker.pdom.1.mason.maker.output/DELETEME.maker.pdom.1.mason_datastore
>
> To access files for individual sequences use the datastore index:
>
> /N/dc/scratch/dstandag/PdomGenomic/Annotation/output/DELETEME.maker.pdom.1.mason.maker.output/DELETEME.maker.pdom.1.mason_master_datastore_index.log
>
> STATUS: Now running MAKER...
> WARNING: Cannot find >scaffold_0, trying to re-index the fasta.
> stop here: scaffold_0
> ERROR: Fasta index error
> at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiChunk.pm
> line 239.
> Process::MpiChunk::_prepare('Process::MpiChunk=HASH(0x3c8b300)',
> 'HASH(0x3c80820)', 0) called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiTiers.pm
> line 73
> Process::MpiTiers::__ANON__() called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Error.pm line 415
> eval {...} called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Error.pm line 407
> Error::subs::try('CODE(0x3c808b0)', 'HASH(0x3c8ec40)') called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiTiers.pm
> line 79
> Process::MpiTiers::_prepare('Process::MpiTiers=HASH(0x3c71f30)')
> called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiTiers.pm
> line 56
> Process::MpiTiers::new('Process::MpiTiers', 'HASH(0x3c80640)', 0,
> 'Process::MpiChunk') called at
> /N/u/dstandag/Mason/local/src/maker-dev/bin/maker line 627
> --> rank=NA, hostname=c4
> ERROR: Failed in tier preparation
> WARNING: You must always set a rank before running MpiTiers
> FATAL: argument `seq_id` does not exist in MpiTier object
> at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiChunk.pm
> line 86.
>
> Process::MpiChunk::_initialize_vars('Process::MpiChunk=HASH(0x3cc93d0)',
> 'HASH(0x3cc9400)') called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiChunk.pm
> line 47
> Process::MpiChunk::new('Process::MpiChunk', 'HASH(0x3c80820)', 0,
> 0) called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiChunk.pm
> line 407
> Process::MpiChunk::__ANON__() called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Error.pm line 415
> eval {...} called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Error.pm line 407
> Error::subs::try('CODE(0x3c8f498)', 'HASH(0x3cc8f20)') called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiChunk.pm
> line 3811
> Process::MpiChunk::_go('Process::MpiChunk=HASH(0x3c8b300)',
> 'load', 'HASH(0x3c80820)', 0, 0) called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiChunk.pm
> line 310
> Process::MpiChunk::_loader('Process::MpiChunk=HASH(0x3c8b300)',
> 'HASH(0x3c80820)', 0, 0, 'Process::MpiTiers=HASH(0x3c71f30)') called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiTiers.pm
> line 364
> Process::MpiTiers::__ANON__() called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Error.pm line 415
> eval {...} called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Error.pm line 407
> Error::subs::try('CODE(0x3c8f9c0)', 'HASH(0x3c8fb28)') called at
> /N/hd01/dstandag/Mason/local/src/maker-dev/bin/../lib/Process/MpiTiers.pm
> line 375
>
>
> I'll try the next update and run the test again. I see a lot of references
> in there to MPI, and just thought I would make it clear that although I am
> running in a cluster environment, I am using the default serial version of
> Maker, not the parallel MPI version.
>
> Also, after this failed, I tried changing the TMP directory so that it was
> located on the same NFS mount as the scratch disk to which the output was
> written. This did not seem to have any affect, and I saw the same issues
> with the EST sequences unable to be found.
>
>
> --
> Daniel S. Standage
> Ph.D. Candidate
> Bioinformatics and Computational Biology Program
> Department of Genetics, Development, and Cell Biology
> Iowa State University
>
>
>
> On Fri, Oct 26, 2012 at 6:59 PM, Carson Holt <carsonhh at gmail.com> wrote:
>
>>
>>
>> From: Carson Holt <carsonhh at gmail.com>
>> Date: Friday, 26 October, 2012 6:10 PM
>> To: Daniel Standage <daniel.standage at gmail.com>
>> Cc: "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
>>
>> Subject: Re: [maker-devel] Strange error at blastn step
>>
>> I've been going over the indexing code using different scenarios and I
>> may have isolated a candidate for what is causing this. Could you do one
>> more 'svn update' inside the maker devel directory before running a test
>> job?
>>
>> Thanks,
>> Carson
>>
>> From: Daniel Standage <daniel.standage at gmail.com>
>> Date: Friday, 26 October, 2012 2:29 PM
>> To: Carson Holt <carsonhh at gmail.com>
>> Cc: "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
>>
>> Subject: Re: [maker-devel] Strange error at blastn step
>>
>> Got this from the compute node. Looks like native disk space to me.
>>
>> [dstandag at mason ~] df /tmp
>> Filesystem 1K-blocks Used Available Use% Mounted on
>> /dev/sda1 478573472 12319684 441943620 3% /tmp
>>
>> Installing a bundle of Perl prereqs for development version, will try
>> that soon.
>>
>> --
>> Daniel S. Standage
>> Ph.D. Candidate
>> Bioinformatics and Computational Biology Program
>> Department of Genetics, Development, and Cell Biology
>> Iowa State University
>>
>>
>>
>> On Fri, Oct 26, 2012 at 2:24 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>
>>> Ok. Try the developer release and see if it still happens.
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>> From: Daniel Standage <daniel.standage at gmail.com>
>>> Date: Friday, 26 October, 2012 2:19 PM
>>>
>>> To: Carson Holt <carsonhh at gmail.com>
>>> Cc: "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
>>>
>>> Subject: Re: [maker-devel] Strange error at blastn step
>>>
>>> Unfortunately, the job is no longer running and as a result I cannot
>>> connect to the compute nodes as I could while it was running. On the
>>> interactive node, it looks like it's real disk, although it looks like
>>> there are some tmpfs mounts.
>>>
>>> [dstandag at mason src] df /tmp
>>> Filesystem 1K-blocks Used Available Use% Mounted on
>>> /dev/sdb2 462824304 180235660 259078476 42% /tmp
>>> [dstandag at mason src] df
>>> Filesystem 1K-blocks Used Available Use% Mounted on
>>> login_x86_64 16497564 3077352 13420212 19% /
>>> tmpfs 16497564 0 16497564 0% /dev/shm
>>> tmpfs 10240 0 10240 0% /var/tmp
>>> /dev/sdb2 462824304 180235660 259078476 42% /tmp
>>> AFS 9000000 0 9000000 0% /afs
>>> bl-nas1:/vol/hd00 3435973856 1775658144 1660315712 52% /N/hd00
>>> bl-nas1:/vol/hd01 3435973856 1684116928 1751856928 50% /N/hd01
>>> bl-nas2:/vol/hd02 3435973856 1856598656 1579375200 55% /N/hd02
>>> bl-nas2:/vol/hd03 3435973856 2747626240 688347616 80% /N/hd03
>>> bl-nas1:/vol/hdln 81920 3424 78496 5% /N/u
>>> bl-nas2:/vol/soft 1258291200 837003424 421287776 67% /N/soft
>>> bl-nas1:/vol/logs 419430400 67163328 352267072 17% /N/logs
>>> ...
>>> ...
>>>
>>> I'll see if I can launch another short job and verify this on the
>>> compute nodes.
>>>
>>> --
>>> Daniel S. Standage
>>> Ph.D. Candidate
>>> Bioinformatics and Computational Biology Program
>>> Department of Genetics, Development, and Cell Biology
>>> Iowa State University
>>>
>>>
>>>
>>> On Fri, Oct 26, 2012 at 2:14 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>>
>>>> The command 'df /tmp' will tell you whether /tmp is a tmpfs mount
>>>>
>>>> Thanks,
>>>> Carson
>>>>
>>>>
>>>> From: Daniel Standage <daniel.standage at gmail.com>
>>>> Date: Friday, 26 October, 2012 2:12 PM
>>>> To: Carson Holt <carsonhh at gmail.com>
>>>> Cc: "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
>>>>
>>>> Subject: Re: [maker-devel] Strange error at blastn step
>>>>
>>>> It looks like /tmp is indeed being used: the files I played with were
>>>> under */tmp/maker_1YQF9o*.
>>>>
>>>> --
>>>> Daniel S. Standage
>>>> Ph.D. Candidate
>>>> Bioinformatics and Computational Biology Program
>>>> Department of Genetics, Development, and Cell Biology
>>>> Iowa State University
>>>>
>>>>
>>>>
>>>> On Fri, Oct 26, 2012 at 2:09 PM, Carson Holt <carsonhh at gmail.com>wrote:
>>>>
>>>>> Check to see where /tmp is located? Some clusters have it set up as a
>>>>> tmpfs directory and I have had problems with fasta indexes running from
>>>>> tmpfs mounts in the past.
>>>>>
>>>>> --Carson
>>>>>
>>>>>
>>>>> From: Daniel Standage <daniel.standage at gmail.com>
>>>>> Date: Friday, 26 October, 2012 2:05 PM
>>>>> To: Carson Holt <carsonhh at gmail.com>
>>>>>
>>>>> Subject: Re: [maker-devel] Strange error at blastn step
>>>>>
>>>>> The maker working directory is in a cluster environment with shared
>>>>> scratch space (I'm guessing NFS-mounted). I didn't change the temp
>>>>> directory setting, so it should be the local default (/tmp).
>>>>>
>>>>> I'll give the dev version a shot. Thanks.
>>>>>
>>>>> --
>>>>> Daniel S. Standage
>>>>> Ph.D. Candidate
>>>>> Bioinformatics and Computational Biology Program
>>>>> Department of Genetics, Development, and Cell Biology
>>>>> Iowa State University
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 26, 2012 at 1:57 PM, Carson Holt <carsonhh at gmail.com>wrote:
>>>>>
>>>>>> Could you try this development version and tell me if the error still
>>>>>> happens?
>>>>>>
>>>>>> Use this command to download -->
>>>>>> <<blanked out from the mailing list>>
>>>>>>
>>>>>> Username: <<blanked out from the mailing list>>
>>>>>> Password: <<blanked out from the mailing list>>
>>>>>>
>>>>>> Are you running in an NFS mounted directory or are you resetting TMP
>>>>>> to a different location?
>>>>>>
>>>>>> Thanks,
>>>>>> Carson
>>>>>>
>>>>>>
>>>>>> From: Daniel Standage <daniel.standage at gmail.com>
>>>>>> Date: Friday, 26 October, 2012 1:52 PM
>>>>>> To: Maker Mailing List <maker-devel at yandell-lab.org>
>>>>>> Subject: Re: [maker-devel] Strange error at blastn step
>>>>>>
>>>>>> I have since installed Maker on a different machine and tried it out.
>>>>>> The test run completed successfully, but as I commenced with the full
>>>>>> genome annotation, I have noticed the following error popping up frequently.
>>>>>>
>>>>>> formating database...
>>>>>> #--------- command -------------#
>>>>>> Widget::formater:
>>>>>> /N/u/dstandag/Mason/local/bin/makeblastdb -dbtype prot -in
>>>>>> /tmp/maker_1YQF9o/Amel3%2E2_Dmel5%2E47%2Efaa.mpi.10.8
>>>>>> #-------------------------------#
>>>>>> running blast search.
>>>>>> #--------- command -------------#
>>>>>> Widget::blastx:
>>>>>> /N/u/dstandag/Mason/local/bin/blastx -db
>>>>>> /tmp/maker_1YQF9o/Amel3%2E2_Dmel5%2E47%2Efaa.mpi.10.8 -query
>>>>>> /tmp/maker_1YQF9o/rank0/scaffold_0.0 -num_alignments 100000
>>>>>> -num_descriptions 100000 -evalue 1e-06 -dbsize 300 -searchsp 500000000
>>>>>> -num_threads 16 -seg yes -soft_masking true -lcase_masking -show_gis -out
>>>>>> /N/dc/scratch/dstandag/PdomGenomic/Annotation/output/maker.pdom.1.mason.maker.output/maker.pdom.1.mason_datastore/scaffold_0/theVoid.scaffold_0/scaffold_0.0.Amel3%2E2_Dmel5%2E47%2Efaa.blastx.temp_dir/Amel3%2E2_Dmel5%2E47%2Efaa.mpi.10.8.blastx
>>>>>> #-------------------------------#
>>>>>> deleted:-10 hits
>>>>>> formating database...
>>>>>> #--------- command -------------#
>>>>>> Widget::formater:
>>>>>> /N/u/dstandag/Mason/local/bin/makeblastdb -dbtype prot -in
>>>>>> /tmp/maker_1YQF9o/Amel3%2E2_Dmel5%2E47%2Efaa.mpi.10.9
>>>>>> #-------------------------------#
>>>>>> running blast search.
>>>>>> #--------- command -------------#
>>>>>> Widget::blastx:
>>>>>> /N/u/dstandag/Mason/local/bin/blastx -db
>>>>>> /tmp/maker_1YQF9o/Amel3%2E2_Dmel5%2E47%2Efaa.mpi.10.9 -query
>>>>>> /tmp/maker_1YQF9o/rank0/scaffold_0.0 -num_alignments 100000
>>>>>> -num_descriptions 100000 -evalue 1e-06 -dbsize 300 -searchsp 500000000
>>>>>> -num_threads 16 -seg yes -soft_masking true -lcase_masking -show_gis -out
>>>>>> /N/dc/scratch/dstandag/PdomGenomic/Annotation/output/maker.pdom.1.mason.maker.output/maker.pdom.1.mason_datastore/scaffold_0/theVoid.scaffold_0/scaffold_0.0.Amel3%2E2_Dmel5%2E47%2Efaa.blastx.temp_dir/Amel3%2E2_Dmel5%2E47%2Efaa.mpi.10.9.blastx
>>>>>> #-------------------------------#
>>>>>> deleted:-6 hits
>>>>>> WARNING: Cannot find> comp59088_c1_seq7, trying to re-index the fasta.
>>>>>> stop here:comp59088_c1_seq7
>>>>>> ERROR: Fasta index error
>>>>>>
>>>>>> FATAL ERROR
>>>>>> ERROR: Failed while polishig ESTs!!
>>>>>>
>>>>>> ERROR: Chunk failed at level 14
>>>>>> !!
>>>>>> FAILED CONTIG:scaffold_0
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --Next Contig--
>>>>>>
>>>>>> #---------------------------------------------------------------------
>>>>>> Now starting the contig!!
>>>>>> SeqID: scaffold_1
>>>>>> Length: 5805686
>>>>>> #---------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> My first thought based on the message is that *blastdbcmd* could not
>>>>>> find the sequence in the database. I verified this was the case--I could
>>>>>> not extract sequence *comp59088_c1_seq7* from the database Maker had
>>>>>> created under /tmp. However, after removing the index files and re-running
>>>>>> *makeblastdb* with the *-parse_seqids* option set, *blastdbcmd* successfully
>>>>>> extracted the sequence.
>>>>>>
>>>>>> I was initially happy with this finding, but upon closer inspection
>>>>>> it looks like Maker does not use *blastdbcmd* to extract sequences,
>>>>>> but rather its own internal code. Therefore I'm still not sure where the
>>>>>> problem is and how I might fix it. Any insights?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> --
>>>>>> Daniel S. Standage
>>>>>> Ph.D. Candidate
>>>>>> Bioinformatics and Computational Biology Program
>>>>>> Department of Genetics, Development, and Cell Biology
>>>>>> Iowa State University
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 25, 2012 at 3:30 PM, Daniel Standage <
>>>>>> daniel.standage at gmail.com> wrote:
>>>>>>
>>>>>>> Greetings!
>>>>>>>
>>>>>>> I am doing a test run of my Maker setup on a new machine, annotating
>>>>>>> a pretty short contig (about 3kb). However, there seems to be a hiccup
>>>>>>> during the blastn stage. This is the terminal message.
>>>>>>>
>>>>>>> #--------- command -------------#
>>>>>>> Widget::blastn:
>>>>>>> /share/home/01854/standage/local/bin/blastn -db
>>>>>>> /tmp/2881473.1.development/maker_i6ZSJi/Pdom%2ETrinity%2ETrimmomatic%2Efasta.mpi.10.7
>>>>>>> -query /tmp/2881473.1.development/maker_i6ZSJi/rank0/scaffold_866.0
>>>>>>> -num_alignments 100000 -num_descriptions 100000 -evalue 1e-10 -gapextend 3
>>>>>>> -word_size 15 -reward 1 -penalty -3 -gapopen 3 -dbsize 1000
>>>>>>> -searchsp 500000000 -num_threads 16 -lcase_masking -dust yes -soft_masking
>>>>>>> true -show_gis -out
>>>>>>> /scratch/01854/standage/PdomGenomic/Annotation/scripts/maker.bogus.maker.output/maker.bogus_datastore/scaffold_866/theVoid.scaff
>>>>>>> old_866/scaffold_866.0.Pdom%2ETrinity%2ETrimmomatic%2Efasta.blastn.temp_dir/Pdom%2ETrinity%2ETrimmomatic%2Efasta.mpi.10.7.blastn
>>>>>>> #-------------------------------#
>>>>>>> deleted:0 hits
>>>>>>> ERROR: Could not obtain lock to format database
>>>>>>>
>>>>>>>
>>>>>>> FATAL ERROR
>>>>>>> ERROR: Failed while doing blastn of ESTs!!
>>>>>>>
>>>>>>> ERROR: Chunk failed at level 8
>>>>>>> !!
>>>>>>> FAILED CONTIG:scaffold_866
>>>>>>>
>>>>>>>
>>>>>>> Several blastn steps appeared to have completed successfully to this
>>>>>>> one failing. Any ideas what could be causing this?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --
>>>>>>> Daniel S. Standage
>>>>>>> Ph.D. Candidate
>>>>>>> Bioinformatics and Computational Biology Program
>>>>>>> Department of Genetics, Development, and Cell Biology
>>>>>>> Iowa State University
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________ maker-devel mailing
>>>>>> list maker-devel at box290.bluehost.com
>>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20121029/c675dfdb/attachment-0003.html>
More information about the maker-devel
mailing list