[maker-devel] MAKER and large number of 'ps' processes
Timothy Stitt (TGAC)
Timothy.Stitt at tgac.ac.uk
Fri Sep 5 01:58:59 MDT 2014
Thanks Carson. That seemed to do the trick!
I'm now running our large case again and the 'ps' processes are definitely suppressed. On a very small test it looked like this new version completed quicker as well. I assume you would expect better performance from avoiding use of 'ps' and directly accessing the process table? Are there any disadvantages to this approach which is why it isn't default in the code?
Much appreciated,
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>
From: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Date: Thursday, 4 September 2014 20:42
To: Timothy Stitt <timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
I think I found what to do to get around the issue, since you are trying to force the use of 'Proc::ProcessTable' instead of using the systems 'ps'. Replace the get_proc_by_id subroutine in .../maker/lib/Proc/Signal.pm with the following one -->
sub get_proc_by_id {
my $id = shift;
my $select;
my $obj = new Proc::ProcessTable_simple;
if(ref($obj) eq "Proc::ProcessTable"){
my ($p) = grep {$_->pid eq $id} @{$obj->table};
return $p;
}
else{
return $obj->get_proc_by_id($id);
}
}
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk<mailto:Timothy.Stitt at tgac.ac.uk>>
Date: Thursday, September 4, 2014 at 1:24 PM
To: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Sorry Carson. Not much luck with that either. I'm building afresh each time and then just running 'maker –h' and the error appears. I meant to say I'm using ActivePerl v5.18.2. I'm assuming that shouldn't make any difference.
Do you have any other suggestions to get the ProcessTable working directly? We are using 128 MPI processes for a large MAKER run and the 'ps' processes are overloading our servers.
Cheers,
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>
From: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Date: Thursday, 4 September 2014 19:52
To: Timothy Stitt <timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Try changing --> eval 'require Proc::ProcessTable';
to --> use Proc::ProcessTable;
in .../maker/lib/Proc/ProcessTable_simple.pm.
That way it forces the perls import method to run incase explicitly exports something for it to function properly.
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk<mailto:Timothy.Stitt at tgac.ac.uk>>
Date: Thursday, September 4, 2014 at 12:45 PM
To: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Thanks Carson.
I downloaded a couple of different versions of Proc::ProcessTable (v0.50 and v0.48). In each case they compiled successfully. I've copied snippets of the 'make test' below to confirm. I've scoured the source and build directories and don't see the .al files. Nothing seems to indicate that they are generated.
I notice that the error occurs at line #143 in ../lib/Proc/Signal.pm of the MAKER source according to the diagnostics:
#142 my $obj = new Proc::ProcessTable_simple;
#143 return $obj->get_proc_by_id($id);
Is there a possibility that the issue is caused by $obj not having the attribute that is being referenced in line $143? I'm not a Perl expert so just throwing out ideas here. If not, how do I get the *.al files to be generated if the build says everything built and tested ok?
> make test
make[1]: Entering directory `/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Process'
make[1]: Leaving directory `/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Process'
PERL_DL_NONLAZY=1 /tgac/software/testing/perl_activeperl/5.16.3.1603/x86_64/bin/perl-static "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/process.t .. --------------------------------
uid: 10344
gid: 11995
…
cmndline: /tgac/software/testing/perl_activeperl/5.16.3.1603/x86_64/bin/perl-static t/process.t
exec: /tgac/software/testing/perl_activeperl/5.16.3.1603/x86_64/bin/perl-static
cwd: /usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50
t/process.t .. ok
All tests successful.
Files=1, Tests=3, 0 wallclock secs ( 0.04 usr 0.02 sys + 0.08 cusr 0.07 csys = 0.21 CPU)
Result: PASS
make[1]: Entering directory `/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Process'
No tests defined for Proc::ProcessTable::Process extension.
make[1]: Leaving directory `/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Process'
Thanks,
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>
From: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Date: Thursday, 4 September 2014 15:25
To: Timothy Stitt <timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
You can also try an older version from http://search.cpan.org if you think that is the issue, but I'd try checking the directories and installation locations first.
--Carson
From: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Date: Thursday, September 4, 2014 at 8:22 AM
To: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk<mailto:Timothy.Stitt at tgac.ac.uk>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
The error means Proc:ProcessTable didn't install and compile correctly. Any *.al files should be created during installation of Proc::ProcessTable.
Go through these directories one at a time and check for the existence of ./Proc/ProcessTable.pm and then ./auto/Proc/ProcessTable/. If they are not there, then you installed Proc::ProcessTable somewhere else and you need to see what is wrong with your CPAN configuration. If they are there then you may need to manually delete both before attempting to reinstall.
/usr/users/TGAC_ga007/stittt/Software/Perl/5.18/site/lib
/usr/users/TGAC_ga007/stittt/Software/Perl/5.18/lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../perl/lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../src/inc/perl/lib
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk<mailto:Timothy.Stitt at tgac.ac.uk>>
Date: Thursday, September 4, 2014 at 5:38 AM
To: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Hi Carson,
I tried the –nolock option and it didn't have much effect. I then installed Proc:ProcessTable (which built successfully via cpan). Running MAKER though I get the following error:
Can't locate auto/Proc/ProcessTable/get_proc_by.al in @INC (@INC contains: /usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../perl/lib /usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../lib /usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../src/inc/perl/lib /usr/users/TGAC_ga007/stittt/Software/Perl/5.18/site/lib /usr/users/TGAC_ga007/stittt/Software/Perl/5.18/lib .) at /usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../lib/Proc/Signal.pm line 143.
I looked within the directories of the ProcessTable build but I don't see the get_proc_by.al file. Should I be using an older version of ProcessTable? The one that was installed is v0.50.
Thanks in advance for any further help with this.
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>
From: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Date: Thursday, 21 August 2014 21:17
To: Timothy Stitt <timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>>, "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
MAKER uses 'ps' every so often to check on certain processes to make sure they haven't failed or become zombies. On your system these 'ps' calls may be hanging which would cause them to build up over time.
You can try and run MAKER with the '-nolock' flag, since it is the NFS file locking that requires these process checks.
Alternatively you can edit .../maker/lib/Proc/ProcessTable_simple.pm and change it as follows.
Find the 'new' subroutine and change it from this -->
sub new {
if($PS){
my $self = {};
my $class = shift;
bless($self, $class);
return $self;
}
else{
eval 'require Proc::ProcessTable';
return Proc::ProcessTable->new(@_);
}
}
to this -->
sub new {
eval 'require Proc::ProcessTable';
return Proc::ProcessTable->new(@_);
}
This will access the process table directly rather than through 'ps', but it may experience the same hang as 'ps' is experiencing. Also you will need to install 'Proc::ProcessTable' via CPAN for it to work, and that particular module may not install on some Linux systems.
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk<mailto:Timothy.Stitt at tgac.ac.uk>>
Date: Thursday, August 21, 2014 at 2:05 PM
To: "maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>" <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: [maker-devel] MAKER and large number of 'ps' processes
Dear MAKER developers,
One of my users is running MAKER on our large shared-memory SGI UV2000 system (with over 2000 cores) and the application appears to be generating large amounts of 'ps' processes that are overwhelming the system and causing the system to be unusable for other users.
Can you confirm that MAKER would be generating this behaviour and if so, is there a way to prevent the application from running 'ps' repeatedly?
Thanks in advance,
Tim.
—
Timothy Stitt PhD | Head of Scientific Computing
+44 1603 450378 | timothy.stitt at tgac.ac.uk<mailto:timothy.stitt at tgac.ac.uk>
The Genome Analysis Centre (TGAC)
Norwich Research Park, Norwich, NR4 7UH, UK | http://www.tgac.ac.uk<http://www.tgac.ac.uk/>
_______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com<mailto:maker-devel at box290.bluehost.com> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20140905/b4195a2a/attachment-0003.html>
More information about the maker-devel
mailing list