[maker-devel] MAKER and large number of 'ps' processes
Carson Holt
carsonhh at gmail.com
Fri Sep 5 09:17:45 MDT 2014
I'm glad the work around is working for you. Proc::ProcessTable being faster
than 'ps' is actually very very atypical. It is likely there is an issue
with your system which is suggested by the fact 'ps' is hanging and
accumulating processes which is also very atypical (ps should return in a
fraction of a second). We actually switched from Proc::ProcessTable to 'ps'
some time ago because 'ps' is several fold faster, and Proc::ProcessTable
won't compile on about 10-15% of system architectures.
Thanks,
Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk>
Date: Friday, September 5, 2014 at 1:58 AM
To: Carson Holt <carsonhh at gmail.com>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Thanks Carson. That seemed to do the trick!
I'm now running our large case again and the 'ps' processes are definitely
suppressed. On a very small test it looked like this new version completed
quicker as well. I assume you would expect better performance from avoiding
use of 'ps' and directly accessing the process table? Are there any
disadvantages to this approach which is why it isn't default in the code?
Much appreciated,
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk
From: Carson Holt <carsonhh at gmail.com>
Date: Thursday, 4 September 2014 20:42
To: Timothy Stitt <timothy.stitt at tgac.ac.uk>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
I think I found what to do to get around the issue, since you are trying to
force the use of 'Proc::ProcessTable' instead of using the systems 'ps'.
Replace the get_proc_by_id subroutine in .../maker/lib/Proc/Signal.pm with
the following one -->
sub get_proc_by_id {
my $id = shift;
my $select;
my $obj = new Proc::ProcessTable_simple;
if(ref($obj) eq "Proc::ProcessTable"){
my ($p) = grep {$_->pid eq $id} @{$obj->table};
return $p;
}
else{
return $obj->get_proc_by_id($id);
}
}
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk>
Date: Thursday, September 4, 2014 at 1:24 PM
To: Carson Holt <carsonhh at gmail.com>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Sorry Carson. Not much luck with that either. I'm building afresh each time
and then just running 'maker –h' and the error appears. I meant to say I'm
using ActivePerl v5.18.2. I'm assuming that shouldn't make any difference.
Do you have any other suggestions to get the ProcessTable working directly?
We are using 128 MPI processes for a large MAKER run and the 'ps' processes
are overloading our servers.
Cheers,
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk
From: Carson Holt <carsonhh at gmail.com>
Date: Thursday, 4 September 2014 19:52
To: Timothy Stitt <timothy.stitt at tgac.ac.uk>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Try changing --> eval 'require Proc::ProcessTable';
to --> use Proc::ProcessTable;
in .../maker/lib/Proc/ProcessTable_simple.pm.
That way it forces the perls import method to run incase explicitly exports
something for it to function properly.
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk>
Date: Thursday, September 4, 2014 at 12:45 PM
To: Carson Holt <carsonhh at gmail.com>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Thanks Carson.
I downloaded a couple of different versions of Proc::ProcessTable (v0.50 and
v0.48). In each case they compiled successfully. I've copied snippets of the
'make test' below to confirm. I've scoured the source and build directories
and don't see the .al files. Nothing seems to indicate that they are
generated.
I notice that the error occurs at line #143 in ../lib/Proc/Signal.pm of the
MAKER source according to the diagnostics:
#142 my $obj = new Proc::ProcessTable_simple;
#143 return $obj->get_proc_by_id($id);
Is there a possibility that the issue is caused by $obj not having the
attribute that is being referenced in line $143? I'm not a Perl expert so
just throwing out ideas here. If not, how do I get the *.al files to be
generated if the build says everything built and tested ok?
> make test
make[1]: Entering directory
`/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Proce
ss'
make[1]: Leaving directory
`/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Proce
ss'
PERL_DL_NONLAZY=1
/tgac/software/testing/perl_activeperl/5.16.3.1603/x86_64/bin/perl-static
"-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')"
t/*.t
t/process.t .. --------------------------------
uid: 10344
gid: 11995
…
cmndline:
/tgac/software/testing/perl_activeperl/5.16.3.1603/x86_64/bin/perl-static
t/process.t
exec:
/tgac/software/testing/perl_activeperl/5.16.3.1603/x86_64/bin/perl-static
cwd: /usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50
t/process.t .. ok
All tests successful.
Files=1, Tests=3, 0 wallclock secs ( 0.04 usr 0.02 sys + 0.08 cusr 0.07
csys = 0.21 CPU)
Result: PASS
make[1]: Entering directory
`/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Proce
ss'
No tests defined for Proc::ProcessTable::Process extension.
make[1]: Leaving directory
`/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/Proc-ProcessTable-0.50/Proce
ss'
Thanks,
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk
From: Carson Holt <carsonhh at gmail.com>
Date: Thursday, 4 September 2014 15:25
To: Timothy Stitt <timothy.stitt at tgac.ac.uk>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
You can also try an older version from http://search.cpan.org if you think
that is the issue, but I'd try checking the directories and installation
locations first.
--Carson
From: Carson Holt <carsonhh at gmail.com>
Date: Thursday, September 4, 2014 at 8:22 AM
To: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk>,
"maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
The error means Proc:ProcessTable didn't install and compile correctly. Any
*.al files should be created during installation of Proc::ProcessTable.
Go through these directories one at a time and check for the existence of
./Proc/ProcessTable.pm and then ./auto/Proc/ProcessTable/. If they are not
there, then you installed Proc::ProcessTable somewhere else and you need to
see what is wrong with your CPAN configuration. If they are there then you
may need to manually delete both before attempting to reinstall.
/usr/users/TGAC_ga007/stittt/Software/Perl/5.18/site/lib
/usr/users/TGAC_ga007/stittt/Software/Perl/5.18/lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../perl/lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../src/inc/perl/lib
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk>
Date: Thursday, September 4, 2014 at 5:38 AM
To: Carson Holt <carsonhh at gmail.com>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
Hi Carson,
I tried the –nolock option and it didn't have much effect. I then installed
Proc:ProcessTable (which built successfully via cpan). Running MAKER though
I get the following error:
Can't locate auto/Proc/ProcessTable/get_proc_by.al in @INC (@INC contains:
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../perl/lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../lib
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../src/inc/perl/lib
/usr/users/TGAC_ga007/stittt/Software/Perl/5.18/site/lib
/usr/users/TGAC_ga007/stittt/Software/Perl/5.18/lib .) at
/usr/users/TGAC_ga007/stittt/Software/MAKER/UV/maker/bin/../lib/Proc/Signal.
pm line 143.
I looked within the directories of the ProcessTable build but I don't see
the get_proc_by.al file. Should I be using an older version of ProcessTable?
The one that was installed is v0.50.
Thanks in advance for any further help with this.
Tim.
---
Timothy Stitt PhD / Head of Scientific Computing
The Genome Analysis Centre (TGAC)
http://www.tgac.ac.uk/
p: +44 1603 450378
e: timothy.stitt at tgac.ac.uk
From: Carson Holt <carsonhh at gmail.com>
Date: Thursday, 21 August 2014 21:17
To: Timothy Stitt <timothy.stitt at tgac.ac.uk>, "maker-devel at yandell-lab.org"
<maker-devel at yandell-lab.org>
Subject: Re: [maker-devel] MAKER and large number of 'ps' processes
MAKER uses 'ps' every so often to check on certain processes to make sure
they haven't failed or become zombies. On your system these 'ps' calls may
be hanging which would cause them to build up over time.
You can try and run MAKER with the '-nolock' flag, since it is the NFS file
locking that requires these process checks.
Alternatively you can edit .../maker/lib/Proc/ProcessTable_simple.pm and
change it as follows.
Find the 'new' subroutine and change it from this -->
sub new {
if($PS){
my $self = {};
my $class = shift;
bless($self, $class);
return $self;
}
else{
eval 'require Proc::ProcessTable';
return Proc::ProcessTable->new(@_);
}
}
to this -->
sub new {
eval 'require Proc::ProcessTable';
return Proc::ProcessTable->new(@_);
}
This will access the process table directly rather than through 'ps', but it
may experience the same hang as 'ps' is experiencing. Also you will need to
install 'Proc::ProcessTable' via CPAN for it to work, and that particular
module may not install on some Linux systems.
--Carson
From: "Timothy Stitt (TGAC)" <Timothy.Stitt at tgac.ac.uk>
Date: Thursday, August 21, 2014 at 2:05 PM
To: "maker-devel at yandell-lab.org" <maker-devel at yandell-lab.org>
Subject: [maker-devel] MAKER and large number of 'ps' processes
Dear MAKER developers,
One of my users is running MAKER on our large shared-memory SGI UV2000
system (with over 2000 cores) and the application appears to be generating
large amounts of 'ps' processes that are overwhelming the system and causing
the system to be unusable for other users.
Can you confirm that MAKER would be generating this behaviour and if so, is
there a way to prevent the application from running 'ps' repeatedly?
Thanks in advance,
Tim.
—
Timothy Stitt PhD | Head of Scientific Computing
+44 1603 450378 | timothy.stitt at tgac.ac.uk
The Genome Analysis Centre (TGAC)
Norwich Research Park, Norwich, NR4 7UH, UK | http://www.tgac.ac.uk
<http://www.tgac.ac.uk/>
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20140905/cbc37e78/attachment-0003.html>
More information about the maker-devel
mailing list