[maker-devel] maker leaving large numbers of defunct zombies

[Mikael Brandström Durling]

I saw the same problem with the latest MPICH2 using hydra too, so it might boil down to perl/openmpi interactions. I didn't see this problem with the debian supplied perl 5.10, but then I had intermittent segfaults with in DB_File and libpthread. Seemed to be some interaction with the LD_PRELOADed libmpi. That requirement for preloading libmpi was easiest solved by compiling openmpi with --disable-dlopen.

Thanks,
Mikael

31 maj 2012 kl. 15:17 skrev Carson Holt:

> MAKER uses IPC::Open3 to open almost all external applications, including
> a helper script called every once in a while that helps check file locks
> on NFS.
> 
> MAKER then calls waitpid to reap the processes, as IPC::Open3 doesn't
> auto-reap.  The only time previously I've seen issues with zombie
> accumulation was with MPICH2 when it moved from the MPD process manager to
> Hydra.  Hydra had certain broken signal handling issues that I had to bug
> the MPICH2 developers about and they fixed it.  It is possible that the
> issue you are having may be with OpenMPI or with perl 5.16.  I currently
> use perl 5.12.  Perl instituted something called safe signals in either
> 5.6 or 5.8 and there may be some updates in 5.16 where they've been
> changing those around again.
> 
> I can try installing a copy of 5.16 to test with and OpenMPI to see if I
> can replicate the error.
> 
> Thanks,
> Carson
> 
> 
> 
> On 12-05-31 8:25 AM, "Mikael Brandström Durling" <mikael.durling at slu.se>
> wrote:
> 
>> Hello,
>> 
>> I've been working lately to set up maker for annotation work on a few
>> fungal genomes. I've got mpi maker up and running now, however, I notice
>> that maker is leaving a lot of <defunct> perl processes behind. This
>> happens to the extent that the process table on the system gets filled up
>> after a few hours run time. Right now the process tree after three hours
>> running looks like this:
>> 
>>   |-sge_execd-+-sge_shepherd---bash---mpirun-+-maker-+-maker
>>    |           |                              |       `-perl
>>    |           |                              |-maker-+-maker
>>    |           |                              |
>> |-maker---1371*[perl]
>>    |           |                              |       `-perl
>>    |           |                              |-maker-+-maker
>>    |           |                              |
>> |-maker---1348*[perl]
>>    |           |                              |       `-perl
>>    |           |                              |-maker-+-maker
>>    |           |                              |
>> |-maker---1384*[perl]
>>    |           |                              |       `-perl
>> 
>> ...and so on for all mpi processes, except for the controlling processes.
>> 
>> What perl programs is maker calling, that might end up as zombies? I've
>> had a brief look at the source to no avail, but would be happy to dig
>> further with some pointers for where to look.
>> 
>> This is run with the 2.25-beta from the web page, perl 5.16.0 and openmpi
>> 1.4.5. 
>> 
>> Thanks,
>> Mikael
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> -------------------------------------
>> Mikael Brandström Durling, PhD
>> Assistant Professor
>> 
>> Sveriges lantbruksuniversitet
>> Swedish University of Agricultural Sciences
>> 
>> Uppsala BioCenter
>> Dept of Forest Mycology and Plant Pathology
>> Box 7026, 75007 Uppsala
>> Visiting address: Almas Allé 5
>> Telefon: 018-671512
>> mikael.durling at slu.se, www.slu.se/mykopat
>> 
>> 
>> 
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
>