[maker-devel] errors in final gff

[Anurag Priyam]

Mmm ... I didn't use -nolock option. But I did launch some 10 MAKER
processes in the same directory.

I feel it's unlikely that my file system doesn't allow hardlinks
because a few processes quit earlier than the others, saying something
to the tune of "Another MAKER process is processing this scaffold
already."

I remember one process in particular had _just_ crashed. I don't
remember how: I might have Ctrl-C'ed by mistake instead of detaching
screen? admin killed it? temporary system glitch? Could this have
caused the same issue?

-- Priyam


On Wed, Jun 25, 2014 at 1:35 AM, Carson Holt <carsonhh at gmail.com> wrote:
> Thanks. For the first two --> scaffold00002:hit:1026:1.3.0.12
>
> The value 1026 is held in a global iterator, so it cannot repeat the same
> value during the life of the process. And 1.3.0.12 is generated from the
> point in the code the ID is being generated.  This means that two distinct
> processses had to write to the same file at the same point in the code,
> which should normally be impossible.
>
> However, there are ways to make this happen.  First if you turn file locks
> off (-nolock) option and then run MAKER multiple times on the same dataset
> you can get process collisions (because you disabled the locks that stop
> this).  If your NFS file system does not support hard links (FhGFS for
> example) then you cannot lock the files (which is the same as setting
> -nolock).  Or you have other serious IO failures over NFS. Note that NFS
> is your Network Mounted Storage.
>
> The last example you give shows the preceding line being truncated.  This
> suggests that two processes are trying to write to the same file
> simultaneously (inserting lines in between other lines), or serious IO
> failures are occurring where writes are not completing but true is being
> returned for the operations (can happen on unreliable NFS implementations).
>
> So in summary either your NFS storage implementation is giving IO errors,
> you have run MAKER with -nolock set and then started MAKER multiple times
> in the same directory (process collisions), or your NFS implementation
> doesn't support hardlinks and won't allow MAKER to lock files (process
> collisions).  If it is one of the latter two, you will have to make sure
> you never start MAKER more than once simultaneously on the same dataset.
> You can still run via MPI fro parallelization, but you won't be able to
> start a second MPI process while the first one is still running.
>
> Thanks,
> Carson
>
>
> On 6/24/14, 12:56 PM, "Anurag Priyam" <a.priyam at qmul.ac.uk> wrote:
>
>>I am sorry. I have updated the gist -
>>https://gist.github.com/yeban/ffaf5cd419639dd073a7.
>>1. The first two chunks contain the annotations with duplicate ids. (4
>>rows)
>>2. The last chunk contains the annotations that refer to a
>>non-existent parent. And what looks like an incomplete line of
>>annotation (I forgot to state this in my original email).
>>
>>No, I didn't use est_forward. I am not passing in any old data via GFF3.
>>
>>-- Priyam
>>
>>On Sat, Jun 21, 2014 at 3:26 AM, Carson Holt <carsonhh at gmail.com> wrote:
>>> Also note that ID= must be unique. Name= does not have to be, and won't
>>>be
>>> if the same protein or repeat element aligns to more than one location
>>>for
>>> example.
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>> On 6/20/14, 3:50 PM, "Carson Holt" <carsonhh at gmail.com> wrote:
>>>
>>>>did you use est_forward?  Also in the example you showed all the IDs are
>>>>unique (one says hit and the other hsp in the ID, so they are
>>>>different)?
>>>>Could you find the non-uunique IDs causing the error?
>>>>
>>>>--Carson
>>>>
>>>>
>>>>On 6/19/14, 2:05 AM, "Anurag Priyam" <anurag08priyam at gmail.com> wrote:
>>>>
>>>>>I used est_gff= option, which refers to a GFF file generated by
>>>>>cufflinks2gff3. The erroneous annotations didn't come from this GFF.
>>>>>
>>>>>-- Priyam
>>>>>
>>>>>On Thu, Jun 19, 2014 at 3:03 AM, Carson Holt <carsonhh at gmail.com>
>>>>>wrote:
>>>>>> Are you passing in old data via GFF3?
>>>>>>
>>>>>> --Carson
>>>>>>
>>>>>>
>>>>>> On 6/18/14, 12:15 PM, "Anurag Priyam" <anurag08priyam at gmail.com>
>>>>>>wrote:
>>>>>>
>>>>>>>It's version 2.31.
>>>>>>>
>>>>>>>-- Priyam
>>>>>>>
>>>>>>>On Wed, Jun 18, 2014 at 11:41 PM, Carson Holt <carsonhh at gmail.com>
>>>>>>>wrote:
>>>>>>>> What MAKER version are you using?
>>>>>>>>
>>>>>>>> --Carson
>>>>>>>>
>>>>>>>>
>>>>>>>> On 6/18/14, 11:44 AM, "Anurag Priyam" <a.priyam at qmul.ac.uk> wrote:
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>I compiled all annotations generated by MAKER into a single GFF
>>>>>>>>>file
>>>>>>>>>using the gff3_merge script distributed with MAKER. While
>>>>>>>>>formatting
>>>>>>>>>this GFF for use with JBrowse, I found a few errors:
>>>>>>>>>
>>>>>>>>>1. Three instances where two features were assigned the same id.
>>>>>>>>>2. One instance where a group of three subfeatures refer to a
>>>>>>>>>non-existent parent.
>>>>>>>>>
>>>>>>>>>Here is the relevant portion of the GFF file:
>>>>>>>>>https://gist.github.com/yeban/ffaf5cd419639dd073a7
>>>>>>>>>
>>>>>>>>>I worked around the issue temporarily for the job at hand, but I am
>>>>>>>>>left wondering why would these errors creep in.
>>>>>>>>>
>>>>>>>>>-- Priyam
>>>>>>>>>
>>>>>>>>>_______________________________________________
>>>>>>>>>maker-devel mailing list
>>>>>>>>>maker-devel at box290.bluehost.com
>>>>>>>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab
>>>>>>>>>.o
>>>>>>>>>r
>>>>>>>>>g
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>
>