[maker-devel] DBD::SQLite::db do failed: database is locked (GFFDB.pm lines 407, 408, 497) error - Using MAKER 2.27b MPI
Carson Holt
carsonhh at gmail.com
Thu Jan 24 14:39:15 MST 2013
Done.
Thanks,
Carson
From: "Krishnakumar, Vivek" <vkrishnakumar at jcvi.org>
Date: Thursday, 24 January, 2013 4:29 PM
To: Carson Holt <carsonhh at gmail.com>
Cc: "mcampbell at genetics.utah.edu" <mcampbell at genetics.utah.edu>
Subject: Re: [maker-devel] DBD::SQLite::db do failed: database is locked
(GFFDB.pm lines 407, 408, 497) error - Using MAKER 2.27b MPI
Ok. Seems to be working fine. I pointed the TMP parameter to our scratch
area.
Thanks for your suggestion.
Should I forward this conversation thread to the maker-devel list ?
Vivek
On 01/24/2013 04:08 PM, Carson Holt wrote:
>
> SQLlite will choke almost immediately if it's not set up with the correct
> flags, so you will know right away.
>
>
>
>
> --Carson
>
>
>
>
> From: "Krishnakumar, Vivek" <vkrishnakumar at jcvi.org>
> Date: Thursday, 24 January, 2013 3:35 PM
> To: Carson Holt <carsonhh at gmail.com>
> Cc: "mcampbell at genetics.utah.edu" <mcampbell at genetics.utah.edu>
> Subject: Re: [maker-devel] DBD::SQLite::db do failed: database is locked
> (GFFDB.pm lines 407, 408, 497) error - Using MAKER 2.27b MPI
>
>
>
>
>
>
> Okay. I will try doing that. But how can I find out that my NFS mounted area
> has -nolock set? The filesystem areas that we work on are managed by our IT
> department.
>
> Thanks.
> Vivek
>
>
> On 01/24/2013 03:27 PM, Carson Holt wrote:
>
>
>>
>> If you don't have locally mounted space big enough, then just use an NFS
>> mounted /TMP. You will take a slight performance hit, but it may still work
>> if it is mounted with -nolock set so SQLite stops complaining.
>>
>>
>>
>>
>> --Carson
>>
>>
>>
>>
>> From: "Krishnakumar, Vivek" <vkrishnakumar at jcvi.org>
>> Date: Thursday, 24 January, 2013 3:24 PM
>> To: Carson Holt <carsonhh at gmail.com>
>> Cc: "mcampbell at genetics.utah.edu" <mcampbell at genetics.utah.edu>
>> Subject: Re: [maker-devel] DBD::SQLite::db do failed: database is locked
>> (GFFDB.pm lines 407, 408, 497) error - Using MAKER 2.27b MPI
>>
>>
>>
>>
>>
>>
>> I no longer see that warning.
>>
>> The only thing I see is that MAKER now complains that the DB is corrupt, and
>> that is because I only have 5GB of /tmp space on the server I currently have
>> access to and that is not enough for the whole genome FASTA + est2genome +
>> protein2genome + model_gff (EVM).
>>
>> Thanks,
>> Vivek
>>
>>
>>
>> On 01/24/2013 03:20 PM, Carson Holt wrote:
>>
>>
>>>
>>> You should be able to run on an NFS drive. After creating the initial
>>> SQLite database, maker now gives each node it's own copy that is stored in
>>> the TMP directory (should be local to each node). Could you send me the
>>> GFF3 file that gives the warning, so I can look at it.
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Carson
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: "Krishnakumar, Vivek" <vkrishnakumar at jcvi.org>
>>> Date: Thursday, 24 January, 2013 3:17 PM
>>> To: Carson Holt <carsonhh at gmail.com>
>>> Cc: "mcampbell at genetics.utah.edu" <mcampbell at genetics.utah.edu>
>>> Subject: Re: [maker-devel] DBD::SQLite::db do failed: database is locked
>>> (GFFDB.pm lines 407, 408, 497) error - Using MAKER 2.27b MPI
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi Carson,
>>>
>>> I did make sure that every time I reran the pipeline, I deleted the
>>> maker.output directory and the temporary files in /tmp. That way I know that
>>> the DBs get created fresh.
>>>
>>> When it was openMPI based, I noticed that irrespective of whether I enabled
>>> or disabled the `-TMP` parameter, I would get the database locked error. At
>>> this point, mpicc was openMPI based and mpiexec (hydra) was MPICH2 based.
>>>
>>> On installing MPICH2 as a non-root user and rebuilding `maker`, this issue
>>> vanished. Just an observation.
>>>
>>> The issue I have now with scaling up is that I do not currently have access
>>> to a server that has a large amount of non-NFS mounted storage space (we do
>>> have such a server but it has been reserved for a different project by a
>>> group in our institute). Thus, the creation of the SQLite db is halted mid
>>> way and now MAKER complains that the database is corrupt. I can either run
>>> the chromosomes individually (serially) and then the scaffolds as one group
>>> or wait to get a time slot of that server.
>>>
>>> Will let you know if I have any more issues.
>>>
>>> Thank you.
>>> Vivek
>>>
>>>
>>>
>>> On 01/24/2013 03:02 PM, Carson Holt wrote:
>>>
>>>
>>>>
>>>> If you still see issue, let me know. I have a hard time seeing how just
>>>> moving to MPICH2 would have solved the issue. It may also have been an
>>>> initial problem with the first GFF# db being created and running on a new
>>>> job recreated it correctly. Either way, let me know if it comes up on your
>>>> bigger run.
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Carson
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: "Krishnakumar, Vivek" <vkrishnakumar at jcvi.org>
>>>> Date: Thursday, 24 January, 2013 2:49 PM
>>>> To: <maker-devel at yandell-lab.org>, <mcampbell at genetics.utah.edu>
>>>> Subject: Re: [maker-devel] DBD::SQLite::db do failed: database is locked
>>>> (GFFDB.pm lines 407, 408, 497) error - Using MAKER 2.27b MPI
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi Everybody,
>>>>
>>>> Sorry for jumping the gun and writing to the mailing list before
>>>> completely troubleshooting this issue. I was able to figure out the problem
>>>> and get MAKER 2.27b working fine in MPI mode.
>>>>
>>>> The issue was with the incorrect version of mpicc (it was openMPI based
>>>> and not mpich2). I was able to successfully run through the steps described
>>>> below on one chromosome in under a few hours (~2). Now I'm scaling it to
>>>> the whole genome (8 chromosomes and ~2200 unanchored scaffolds).
>>>>
>>>> Thank you.
>>>> Vivek
>>>>
>>>>
>>>> On 01/23/2013 02:42 PM, Krishnakumar, Vivek wrote:
>>>>
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to use the MAKER pipeline to calculate AED scores for the gene
>>>>> models coming out of our annotation pipeline (EVidence Modeler). We then
>>>>> want to compare these computed AED scores with those computed for the gene
>>>>> models coming out of the MAKER2 pipeline.
>>>>>
>>>>> For this purpose, I configured the maker_opts.ctl file like so:
>>>>> 1) Specify gff file coming from our annotation pipeline using `model_gff`
>>>>> param
>>>>> 2) Specify gff file containing est2genome results (computed using MAKER2)
>>>>> using the `est_gff` param
>>>>> 3) Specify gff file containing protein2genome results (computed using
>>>>> MAKER2) using the `protein_gff` param
>>>>> 4) Specify the genome fasta sequence file using the `genome` param
>>>>>
>>>>> I recently configured MAKER 2.27b to work using MPI and I tried launching
>>>>> one contig (chromosome) on a compute node spawning 12 processes like so:
>>>>> mpiexec -n 12 maker -R -TMP /tmp/maker maker_opts.ctl maker_bopts.ctl
>>>>> maker_exe.ctl 2>&1 | tee maker.err
>>>>>
>>>>> Side note: Without MPI, it works fine but takes very very long to
>>>>> complete running on one chromosome (since it is going through the chunks
>>>>> serially).
>>>>>
>>>>> On inspecting the `maker.err` file, I see server error messages that say
>>>>> `DBD::SQLite::db do failed: database is locked`. On referring to a
>>>>> previous email thread on the `maker-devel` mailing list, I see that Carson
>>>>> had replied to the person with a similar issue asking them to check if the
>>>>> working directory and TMP directory are being mounted on an NFS filesystem
>>>>> because this is a potential problem with SQLite databases.
>>>>>
>>>>> Based on that suggestion, I switched the TMP directory to use a local
>>>>> filesytem but the working directory is still NFS mounted. Despite that, I
>>>>> still get this error. The previous email thread had no resolution for this
>>>>> issue. My question is, should the working directory be local as well? I
>>>>> see that despite setting the `TMP` variable, the SQLite DB is getting
>>>>> created in the working directory.
>>>>>
>>>>> What might be the issue?
>>>>>
>>>>> Also, apart from this one error, I notice another error:
>>>>> ERROR: Non-unique top level ID for chr5:hit:1454:4_0
>>>>> While this is technically legal in GFF3, it usually
>>>>> indicates a poorly fomatted GFF3 file (perhaps you
>>>>> tried to merge two GFF3 files without accounting for
>>>>> unique IDs). MAKER will not handle these correctly.
>>>>>
>>>>> On inspecting the input GFF file (protein2genome), I see that this ID is
>>>>> never repeated. So I have no idea whats going on here either. Any
>>>>> suggestions?
>>>>>
>>>>> Below is my `maker.err` file from the run. (Dropbox link)
>>>>>
>>>>> Thank you very much.
>>>>> Best,
>>>>>
>>>>> Vivek
>>>>>
>>>>> --
>>>>> Vivek Krishnakumar
>>>>> vkrishnakumar at jcvi.org
>>>>>
>>>>>
>>>>>
>>>>> I've linked 1 file to this email:
>>>>>
>>>>>
>>>>> maker.err <http://db.tt/2tYyKCD7> (97.7 MB)Dropbox
>>>>> <https://www.dropbox.com/> http://db.tt/2tYyKCD7
>>>>>
>>>>>
>>>>> Mozilla Thunderbird <http://www.getthunderbird.com> makes it easy to
>>>>> share large files over email.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________ maker-devel mailing list
>>>> maker-devel at box290.bluehost.com
>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130124/fff51928/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Preview.app Document
Type: image/png
Size: 641 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130124/fff51928/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Preview.app Document
Type: image/png
Size: 1205 bytes
Desc: not available
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130124/fff51928/attachment-0007.png>
More information about the maker-devel
mailing list