[maker-devel] thread terminated, causing all processes to fail

[Carson Holt]

Try running maker outside of with the ­a flag after deleting mpi_blastdb.

Does it still happen.

Also if you try again with MPI with the ­a flag and having deleted
mpi_blastdb, does it fail the same every time?

Could you also check for background maker processes that may be trying to
work in the same directory that you may not have realized were running.

Thanks,
Carson



From:  Ramón Fallon <ramonfallon at gmail.com>
Date:  Thursday, 7 March, 2013 12:47 PM
To:  Carson Holt <carsonhh at gmail.com>
Cc:  "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
Subject:  Re: [maker-devel] thread terminated, causing all processes to fail

This is a standalone machine and no NFS at all. "df" gives a healthy amount
of disk space, so there should be no problem there.

Yes that file does exist although it has the nominal 12288 bytes size, which
appears to be the minimum for a DB_file tie.

As I mentioned the dpp_contig.fa example set does work so part of my
investigation is looking at how.

I can do some trivial unit tests on the Bioperl stat-before-tied-hashes
situation and see what comes up.

So I'll attempt to clear that up and then revert.

Many thanks! / Ramón.


On Thu, Mar 7, 2013 at 5:44 PM, Carson Holt <carsonhh at gmail.com> wrote:
> That is extremely odd.  It fails to even generate the indexes. Could you check
> the drive space of your working directory and your /tmp directory?
> 
> It is odd because Bioperl uses the stat command to check on the file right
> before making a tied hash.  So it was there for the stat but not the tie,
> which is immediately following.
> 
> If you check manually does it exist now? -->
> /home/ramonf/makertrials/mgallocut7/sca29310_8.maker.output/mpi_blastdb/sca293
> 10_8%2Efa.mpi.1/sca29310_8%2Efa.mpi.1.0.index
> 
> Are you running in an NFS mounted directory?
> 
> --Carson
> 
> 
> From:  Ramón Fallon <ramonfallon at gmail.com>
> Date:  Thursday, 7 March, 2013 9:40 AM
> 
> To:  Carson Holt <carson.holt at oicr.on.ca>
> Cc:  "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
> Subject:  Re: [maker-devel] thread terminated, causing all processes to fail
> 
> Hi Carson,
> 
> I send you a zip of the text file of my repeated maker session, this time
> having deleted the mpi_blastdb dir and with the -a flag added to "mpiexec -n 8
> maker -debug". Command line.
> 
> Cheers / Ramón.
> 
> 
> On Wed, Mar 6, 2013 at 7:49 PM, Ramón Fallon <ramonfallon at gmail.com> wrote:
>> OK, will do. 
>> 
>> Will get back to you tomorrow on it.
>> 
>> Many thanks!
>>  
>> 
>> On Wed, Mar 6, 2013 at 7:22 PM, Carson Holt <Carson.Holt at oicr.on.ca> wrote:
>>> Could you delete your ../*maker.output/mpi_blastdb directory, and then when
>>> rerunning maker, run with the ­a flag.
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> 
>>> From: Ramón Fallon <ramonfallon at gmail.com>
>>> Date: Wednesday, 6 March, 2013 1:15 PM
>>> To: Carson Holt <carson.holt at oicr.on.ca>
>>> Cc: "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
>>> 
>>> Subject: Re: thread terminated, causing all processes to fail
>>> 
>>> OK great, here goes .. many thanks!
>>> 
>>> 
>>> 
>>> On Wed, Mar 6, 2013 at 7:04 PM, Carson Holt <Carson.Holt at oicr.on.ca> wrote:
>>>> If you do reply all to this message, I should get the attachment.  It will
>>>> be stripped from the one going to the list though.
>>>> 
>>>> Thanks,
>>>> Carson
>>>> 
>>>> 
>>>> 
>>>> From: Ramón Fallon <ramonfallon at gmail.com>
>>>> Date: Wednesday, 6 March, 2013 12:57 PM
>>>> To: <maker-devel at yandell-lab.org>
>>>> Subject: Re: thread terminated, causing all processes to fail
>>>> 
>>>> Hi, 
>>>> 
>>>> Many thanks for your quick reply and hint.
>>>> 
>>>> Yes, you're right .. further up there is indeed
>>>> 
>>>> Calling FastaDB::new at /opt/src/maker_svn/bin/../lib/FastaSeq.pm line 148
>>>> thread 1.
>>>> Thread 1 terminated abnormally: ERROR: Could not reestablish DB to thaw
>>>> FastaSeq for Storable
>>>> --> rank=5, hostname=fatnode, at /opt/src/maker_svn/bin/maker line 1457
>>>> thread 1.
>>>> 
>>>> I run a "script" session and have maker on -debug so I have everything in
>>>> one file. Do you prefer to have it attached to a post to this mailing list
>>>> (if it accepts txt attachments)
>>>> 
>>>> Cheers.
>>>> 
>>>> 
>>>> On Wed, Mar 6, 2013 at 6:34 PM, Ramón Fallon <ramonfallon at gmail.com> wrote:
>>>>> Hi, 
>>>>> 
>>>>> I'm using the maker_svn rev 995 version and hand-compiled MPICH2 on a
>>>>> single multicore machine.
>>>>> 
>>>>> I've successfully run the dpp_contig.fasta (MPI/8 processes) example but
>>>>> am having trouble with larger contigs fasta files of my own, which are
>>>>> well formed.
>>>>> 
>>>>> I've run into a problem whereby an mpiexec run of 8 processes will stop
>>>>> due to a perl-thread related problem which says
>>>>> 
>>>>> FATAL: Thread terminated, causing all processes to fail
>>>>>  
>>>>> this corresponds to line 924 in the maker executable (which is for the
>>>>> secondary/worker threads), and is the result of a test on !$thr OR'd with
>>>>> !$thr->is_running, so clearly one of these is failing.
>>>>> 
>>>>> $thr itself is a threads->new(\&$node_thread, $gdbfile). Despite being a
>>>>> programmer, I've only recently started to look at the code and have not
>>>>> got the hang of the parallelisation setup here, though I gather the master
>>>>> must use threads to initially generate the parallel instances which then
>>>>> use the message passing. Of course threads don't have message passing
>>>>> ability, so I guess something clever is going on and will take some time
>>>>> for me to understand.
>>>>> 
>>>>> Clearly however, it has worked before on dpp_contigs, so it may be is
>>>>> something wrong with my datafile or the way I am carrying out the
>>>>> analysis.
>>>>> 
>>>>> Any clues that can be put my way are welcome.
>>>>> 
>>>>> Thank you!
>>>> 
>>> 
>> 
> 
> _______________________________________________ maker-devel mailing list
> maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak
> er-devel_yandell-lab.org



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130307/5eebba85/attachment-0003.html>