From bushleyk at science.oregonstate.edu Mon Feb 4 17:48:32 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Mon, 04 Feb 2013 15:48:32 -0800 Subject: [maker-devel] maker MPI Message-ID: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Greetings, We are reinstalling maker after a system upgrade and having some issues with getting MPI version running. One question, will maker only run in MPI-CH2 or will it also run in LAM-MPI? When trying to run in MPI-CH2, we get this error: /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt cd /pseudospace2/bushleyk/MAKER/Testrun; /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 '/local/cluster/MAKER_2.10/bin/mpi_maker' /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr make: *** [maker] Error 127 which maybe means it's not finding the right perl? When trying to run in LAM-MPI we get this error, was wondering what the MAKER/EVALUATOR is? ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR running repeat masker. #--------- command -------------# Widget::RepeatMasker: /local/cluster/spatafora/RepeatMasker/RepeatMasker /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-500-500.0.all.rb -species all -dir /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa 1 #-------------------------------# ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM make: *** [maker_new] Error 252 Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 From mikael.durling at slu.se Tue Feb 5 09:01:32 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Tue, 5 Feb 2013 15:01:32 +0000 Subject: [maker-devel] Handling of strandedness of est_gff Message-ID: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Hi, while looking into the annotations created by maker I find that there are cases where the ab initio prediction sits on one strand of the reference, while the est_gff evidence from cufflinks sits on the other strand. In this case the aed is rather low, and the predictions seems quite off. Is maker taking the strand information of the est_gff into account? If so, is it possible to tell maker that the ests are unstranded and only use the structural information rather than structure and strand? Thanks for any advice, Mikael From carsonhh at gmail.com Tue Feb 5 09:17:40 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 10:17:40 -0500 Subject: [maker-devel] Handling of strandedness of est_gff In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Message-ID: MAKER assumes the data in the est_gff is correctly stranded. If not there is no real way for MAKER to adjust for that as I can show you several cases where I can get coding models that match the est alignment well on both strands. MAKER uses exonerate to polish alignments when generated internally because in cases such as the ones I just described, one strand will match better only when you account for all splice sites in the model, but I can't do that type of polishing on gff3 derived alignments, and in the case of cufflinks, it should already be taking splice sites into account. Of course if these are single exon alignments then cufflinks doesn't know which strand to assign them to and I would recommend not including single exon alignments in the est_gff. Also consider using trinity over cufflinks, as I've seen more positive results from its alignments/assemblies of mRNAseq. Thanks, Carson On 13-02-05 10:01 AM, "Mikael Brandstr?m Durling" wrote: >Hi, > >while looking into the annotations created by maker I find that there are >cases where the ab initio prediction sits on one strand of the reference, >while the est_gff evidence from cufflinks sits on the other strand. In >this case the aed is rather low, and the predictions seems quite off. Is >maker taking the strand information of the est_gff into account? If so, >is it possible to tell maker that the ests are unstranded and only use >the structural information rather than structure and strand? > >Thanks for any advice, >Mikael > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Tue Feb 5 14:23:36 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 15:23:36 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Message-ID: LAMMPI is pretty much dead, even their website recommends using OpenMPI instead. For installing MAKER with OpenMPI you need to set a couple of environmental variables before MAKER setup. export OMPI_MCA_mpi_warn_on_fork 0 export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD These not only need to be set before compilation, but also before any run (so add them to you ~/.bashrc or ~/.bash_profile or any module load scripts thanks). The LD_PRELOAD statement needs to be set for any program using OpenMPI's shared libraries and not just MAKER, so it's normally a good idea to have that set system wide for all users. The detail can be Found in the OpenMPI documentation. Note sometimes system library updates can break OpenMPI's shared libraries while not breaking OpenMPI itself, so you might also need to recompile OpenMPI if it has broken shared libraries. Thanks, Carson On 13-02-04 6:48 PM, "Kathryn E. Bushley" wrote: >Greetings, > >We are reinstalling maker after a system upgrade and having some >issues with getting MPI version running. One question, will maker >only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >run in MPI-CH2, we get this error: > >/local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >cd /pseudospace2/bushleyk/MAKER/Testrun; >/local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >'/local/cluster/MAKER_2.10/bin/mpi_maker' >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >make: *** [maker] Error 127 > >which maybe means it's not finding the right perl? > > >When trying to run in LAM-MPI we get this error, was wondering what >the MAKER/EVALUATOR is? > >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >running repeat masker. >#--------- command -------------# >Widget::RepeatMasker: >/local/cluster/spatafora/RepeatMasker/RepeatMasker >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >0-500.0.all.rb -species all -dir >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >1 >#-------------------------------# >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >make: *** [maker_new] Error 252 > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > > > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From dence at genetics.utah.edu Mon Feb 11 10:23:28 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Mon, 11 Feb 2013 16:23:28 +0000 Subject: [maker-devel] Help on MAKER In-Reply-To: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> References: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> Message-ID: Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________ From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Mon Feb 11 10:25:58 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 11 Feb 2013 11:25:58 -0500 Subject: [maker-devel] Help on MAKER In-Reply-To: Message-ID: When the "./Build " script is created then "./Build installexes" can be used to get external executables and "./Build installdeps" can be used to get perl prerequisite modules. Thanks, Carson From: Daniel Ence Date: Monday, 11 February, 2013 11:23 AM To: "maker-devel at yandell-lab.org" Cc: Hud Hud Subject: Re: [maker-devel] Help on MAKER Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.campbell1 at gmail.com Mon Feb 11 10:50:04 2013 From: michael.s.campbell1 at gmail.com (Michael Campbell) Date: Mon, 11 Feb 2013 09:50:04 -0700 Subject: [maker-devel] Help on Maker In-Reply-To: <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> References: <1360546957.87948.YahooMailNeo@web164902.mail.bf1.yahoo.com> <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> Message-ID: Hi Huda, I'm forwarding this to the MAKER dev list to get some more eyes on it. The link to the Korf website works for me. Where are you trying to connect form? Mike On Sun, Feb 10, 2013 at 10:10 PM, Hud Hud wrote: > Hi Mike, > Thnks 4 ur reply..ive tried the ./Build blast command..I prtscrn the last > part of the configuration as attached. Is there anything that I've > missed?....And for the SNAP, I tried to download manually from the website, > but i cant get access to the korflab.ucdavis.edu/software website.. > > Thanks.. > ------------------------------ > *From:* Michael Campbell > *To:* "?? Li, Qing" > *Cc:* Hud Hud > *Sent:* Monday, February 11, 2013 11:42 AM > *Subject:* Re: Help on Maker > > Hi Huda, > > If you have unpacked the maker tar ball and moved to the src directory and > have run the "perl Build.PL " command you can run the > next command "./Build status" and that should give a report that looks like > this. > > > ============================================================================== > STATUS MAKER 2.25 > > ============================================================================== > PERL Dependencies: VERIFIED > External Programs: VERIFIED > External C Libraries: VERIFIED > MPI SUPPORT: DISABLED > MWAS Web Interface: DISABLED > MAKER PACKAGE: CONFIGURATION OK > > > Important Commands: > ./Build installdeps #installs missing PERL dependencies > ./Build installexes #installs all missing external programs > ./Build install #installs MAKER > ./Build status #Shows this status menu > > Other Commands: > ./Build repeatmasker #installs RepeatMasker (asks for RepBase) > ./Build blast #installs BLAST (NCBI BLAST+) > ./Build exonerate #installs Exonerate (v2 on UNIX / v1 on Mac OSX) > ./Build snap #installs SNAP > ./Build augustus #installs Augustus > ./Build apollo #installs Apollo > ./Build gbrowse #installs GBrowse (must be root) > ./Build jbrowse #installs JBrowse (MAKER copy, not web accecible) > ./Build mpich2 #installs MPICH2 (but manual install recommended) > > > I've highlighted the commands above that you can run to have MAKER install > snap and blast for you. Here is a url for a gmod tutorial for installing > and running MAKER http://gmod.org/wiki/MAKER_Tutorial_2012. > > I haven't actually used cygwin so I am interested in hearing how MAKER > runs in that environment. I will ask a few other people and see if they > have any experience with cygwin. > > Let me know if you get stuck or have any more questions, > Mike > > > On Sun, Feb 10, 2013 at 6:52 PM, ?? Li, Qing wrote: > > Hello Huda, > No worries:) I am cp your question to the expert maker usr in our group! > > Mike, would you think you could answer Huda's question and help him, or > would you think we need to ask Carlson for help? > Best, > Qing > > > 2013/2/10 Hud Hud > > Hello Hi, Im Huda...Im having problem during the installation of > MAKER...I'm not sure whether im asking the right person, I just pick anyone > from the list of Yandell People page...Hofefully im not troubling u with > this email... > Back to the problem...I try to install Maker but I've noticed that im > missing some external program that are Blast and SNAP...how to solve this > problem. FYI im very new in this field and not very familiar with > programming or scripting. Im using cygwin. Thanks for your help > > > > > > -- > Michael Campbell MS, RD. > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ph:585-3543 > > > > -- Michael Campbell MS, RD. Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ph:585-3543 -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Fri Feb 15 11:15:53 2013 From: carsonhh at gmail.com (Carson Holt) Date: Fri, 15 Feb 2013 12:15:53 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Message-ID: It cannot establish the communication ring on LAMMPI, so it is acting like multiple independent instances of MAKER up to the number of processors specified to MPI. So it gives the warning "MAKER already running" when it detects the other job's file locks. I'd recommend to just use MPICH2, as I don't have any idea what might happen with LAMMPI. --Carson On 13-02-15 12:10 PM, "Kathryn E. Bushley" wrote: >Hello Carson, > >Thanks. We currently have either LAMMPI or MPICH2 up and running on >our cluster so are trying to make it work in either one of those. >We're able to get it run in LAMMPI but get this error about the MAKER >EVALUATOR already running, any idea what that might be? It did work >previously in MPICH2 so maybe we need to go that route. > >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt > >LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University > >[bushleyk at fungi0 Test]$ lamnodes >n0 fungi0.cgrb.oregonstate.local:4:origin,this_node >[bushleyk at fungi0 Test]$ ps -aef | grep lam >bushleyk 11818 1 0 08:56 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 >bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam >sullichr 13795 1 0 Feb14 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 >/local/cluster/maker/bin/mpi_maker >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >Received SIGTERM > > > > > > > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > >Quoting Carson Holt : > >> LAMMPI is pretty much dead, even their website recommends using OpenMPI >> instead. >> >> For installing MAKER with OpenMPI you need to set a couple of >> environmental variables before MAKER setup. >> >> export OMPI_MCA_mpi_warn_on_fork 0 >> export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD >> >> These not only need to be set before compilation, but also before any >>run >> (so add them to you ~/.bashrc or ~/.bash_profile or any module load >> scripts thanks). The LD_PRELOAD statement needs to be set for any >>program >> using OpenMPI's shared libraries and not just MAKER, so it's normally a >> good idea to have that set system wide for all users. The detail can be >> Found in the OpenMPI documentation. Note sometimes system library >>updates >> can break OpenMPI's shared libraries while not breaking OpenMPI itself, >>so >> you might also need to recompile OpenMPI if it has broken shared >>libraries. >> >> Thanks, >> Carson >> >> >> On 13-02-04 6:48 PM, "Kathryn E. Bushley" >> wrote: >> >>> Greetings, >>> >>> We are reinstalling maker after a system upgrade and having some >>> issues with getting MPI version running. One question, will maker >>> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >>> run in MPI-CH2, we get this error: >>> >>> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >>> cd /pseudospace2/bushleyk/MAKER/Testrun; >>> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >>> '/local/cluster/MAKER_2.10/bin/mpi_maker' >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> make: *** [maker] Error 127 >>> >>> which maybe means it's not finding the right perl? >>> >>> >>> When trying to run in LAM-MPI we get this error, was wondering what >>> the MAKER/EVALUATOR is? >>> >>> ERROR: Cannot start process. MAKER/EVALUATOR already running >>> with different settings in this same directory. >>> >>> >>> FATAL ERROR >>> running repeat masker. >>> #--------- command -------------# >>> Widget::RepeatMasker: >>> /local/cluster/spatafora/RepeatMasker/RepeatMasker >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> >>>tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp- >>>50 >>> 0-500.0.all.rb -species all -dir >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >>> 1 >>> #-------------------------------# >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> It seems that [at least] one of the processes that was started with >>> mpirun did not invoke MPI_INIT before quitting (it is possible that >>> more than one process did not invoke MPI_INIT -- mpirun was only >>> notified of the first one, which was on node n0). >>> >>> mpirun can *only* be used with MPI programs (i.e., programs that >>> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >>> to run non-MPI programs over the lambooted nodes. >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> Received SIGTERM >>> make: *** [maker_new] Error 252 >>> >>> >>> >>> >>> >>> >>> >>> >>> Best, >>> >>> Kathryn >>> >>> >>> "To err is human...to really foul up requires the root password." >>> >>> >>> Kathryn E. Bushley >>> Postdoctoral Researcher >>> 2082 Cordley Hall >>> Department of Botany and Plant Pathology >>> Oregon State University >>> Corvallis, OR 97331 >>> office: (541) 737-5284 >>> cell: (541) 908-0116 >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> > > From bushleyk at science.oregonstate.edu Fri Feb 15 11:10:23 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Fri, 15 Feb 2013 09:10:23 -0800 Subject: [maker-devel] maker MPI In-Reply-To: References: Message-ID: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Hello Carson, Thanks. We currently have either LAMMPI or MPICH2 up and running on our cluster so are trying to make it work in either one of those. We're able to get it run in LAMMPI but get this error about the MAKER EVALUATOR already running, any idea what that might be? It did work previously in MPICH2 so maybe we need to go that route. [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University [bushleyk at fungi0 Test]$ lamnodes n0 fungi0.cgrb.oregonstate.local:4:origin,this_node [bushleyk at fungi0 Test]$ ps -aef | grep lam bushleyk 11818 1 0 08:56 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam sullichr 13795 1 0 Feb14 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 /local/cluster/maker/bin/mpi_maker ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM Received SIGTERM Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 Quoting Carson Holt : > LAMMPI is pretty much dead, even their website recommends using OpenMPI > instead. > > For installing MAKER with OpenMPI you need to set a couple of > environmental variables before MAKER setup. > > export OMPI_MCA_mpi_warn_on_fork 0 > export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD > > These not only need to be set before compilation, but also before any run > (so add them to you ~/.bashrc or ~/.bash_profile or any module load > scripts thanks). The LD_PRELOAD statement needs to be set for any program > using OpenMPI's shared libraries and not just MAKER, so it's normally a > good idea to have that set system wide for all users. The detail can be > Found in the OpenMPI documentation. Note sometimes system library updates > can break OpenMPI's shared libraries while not breaking OpenMPI itself, so > you might also need to recompile OpenMPI if it has broken shared libraries. > > Thanks, > Carson > > > On 13-02-04 6:48 PM, "Kathryn E. Bushley" > wrote: > >> Greetings, >> >> We are reinstalling maker after a system upgrade and having some >> issues with getting MPI version running. One question, will maker >> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >> run in MPI-CH2, we get this error: >> >> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >> cd /pseudospace2/bushleyk/MAKER/Testrun; >> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >> '/local/cluster/MAKER_2.10/bin/mpi_maker' >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> make: *** [maker] Error 127 >> >> which maybe means it's not finding the right perl? >> >> >> When trying to run in LAM-MPI we get this error, was wondering what >> the MAKER/EVALUATOR is? >> >> ERROR: Cannot start process. MAKER/EVALUATOR already running >> with different settings in this same directory. >> >> >> FATAL ERROR >> running repeat masker. >> #--------- command -------------# >> Widget::RepeatMasker: >> /local/cluster/spatafora/RepeatMasker/RepeatMasker >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >> 0-500.0.all.rb -species all -dir >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >> 1 >> #-------------------------------# >> -------------------------------------------------------------------------- >> --- >> It seems that [at least] one of the processes that was started with >> mpirun did not invoke MPI_INIT before quitting (it is possible that >> more than one process did not invoke MPI_INIT -- mpirun was only >> notified of the first one, which was on node n0). >> >> mpirun can *only* be used with MPI programs (i.e., programs that >> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >> to run non-MPI programs over the lambooted nodes. >> -------------------------------------------------------------------------- >> --- >> Received SIGTERM >> make: *** [maker_new] Error 252 >> >> >> >> >> >> >> >> >> Best, >> >> Kathryn >> >> >> "To err is human...to really foul up requires the root password." >> >> >> Kathryn E. Bushley >> Postdoctoral Researcher >> 2082 Cordley Hall >> Department of Botany and Plant Pathology >> Oregon State University >> Corvallis, OR 97331 >> office: (541) 737-5284 >> cell: (541) 908-0116 >> >> >> >> >> >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > > From mikael.durling at slu.se Wed Feb 20 07:12:08 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Wed, 20 Feb 2013 13:12:08 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael From dence at genetics.utah.edu Wed Feb 20 10:29:21 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Wed, 20 Feb 2013 16:29:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. How much of the genome has maker completed when the processes start doing this? Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Wednesday, February 20, 2013 6:12 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] Running MAKER on highly fragmented assembly Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Thu Feb 21 09:39:37 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Thu, 21 Feb 2013 15:39:37 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Hi Daniel, the genomes I work with are in the order of 30-60 Mb. Other assemblies have been quick jobs for maker without any problems. If I run maker with -debugmpi, I get sets of debug printouts from the different ranks now and then: COMM INITIALIZATION | SEND | who_I_am | 3 --> 0 | 46731 COMM INITIALIZATION | SEND | what_I_want | 3 --> 0 | 46732 COMM INITIALIZATION | RECV | what_I_want | 0 <-- 3 | 312850 COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 --> 3 | 312851 HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 <-- 3 | 312852 COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 <-- 0 | 46733 HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 --> 0 | 46734 HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 --> 3 | 312853 HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 <-- 0 | 46735 COMM INITIALIZATION | RECV | who_I_am | 0 <-- ANY | 312854 and then they seem to stay waiting while a single rank continues to run the normal analysis. I have filtered the assembly for contigs shorter than then minimum length set in maker_opts.ctl. I did some strace:ing of the ranks that do nothing, and it seems they loop over running a subprocess that basically does a process listing. I might be completely off in my guesses of what the problem might be. I'm sort of afraid that I'm bitten by some NFS related problem as I have been quite a few times by know. I will soon try to reannotate a genome sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure that make behaves as expected with that genome. Mikael 20 feb 2013 kl. 17:29 skrev Daniel Ence : > Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. > > How much of the genome has maker completed when the processes start doing this? > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Wednesday, February 20, 2013 6:12 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] Running MAKER on highly fragmented assembly > > Hi, > > I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). > > cheers, > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Thu Feb 21 09:58:10 2013 From: carsonhh at gmail.com (Carson Holt) Date: Thu, 21 Feb 2013 10:58:10 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Message-ID: Which version of MAKER are you using? We recently deployed MAKER on the large NIH TACC computer cluster with some additional modifications for very large MPI jobs (> 1500 cpus). Some of the modifications focus on reannotating very large contigs as opposed to the small contig de novo annotation that MAKER already works well on. The modifications will be merged back into the MAKER downloadable release soon, but I could give you access to test with especially if you are running large MPI jobs or on very large multi-megabase contigs. --Carson On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" wrote: >Hi Daniel, >the genomes I work with are in the order of 30-60 Mb. Other assemblies >have been quick jobs for maker without any problems. > >If I run maker with -debugmpi, I get sets of debug printouts from the >different ranks now and then: > >COMM INITIALIZATION | SEND | who_I_am | 3 >--> 0 | 46731 >COMM INITIALIZATION | SEND | what_I_want | 3 >--> 0 | 46732 >COMM INITIALIZATION | RECV | what_I_want | 0 ><-- 3 | 312850 >COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >--> 3 | 312851 >HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 ><-- 3 | 312852 >COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 ><-- 0 | 46733 >HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >--> 0 | 46734 >HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >--> 3 | 312853 >HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 ><-- 0 | 46735 >COMM INITIALIZATION | RECV | who_I_am | 0 ><-- ANY | 312854 > >and then they seem to stay waiting while a single rank continues to run >the normal analysis. I have filtered the assembly for contigs shorter >than then minimum length set in maker_opts.ctl. > >I did some strace:ing of the ranks that do nothing, and it seems they >loop over running a subprocess that basically does a process listing. > >I might be completely off in my guesses of what the problem might be. I'm >sort of afraid that I'm bitten by some NFS related problem as I have been >quite a few times by know. I will soon try to reannotate a genome >sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >that make behaves as expected with that genome. >Mikael > > >20 feb 2013 kl. 17:29 skrev Daniel Ence : > >> Hi Mikael, Depending on the genome size, the assembly you've described >>shouldn't be too difficult to work with. The process activity that >>you're describing sounds more like a race condition, where one process >>is hogging all the work and all the other processes keep trying to find >>work, but keep getting in each others' way. >> >> How much of the genome has maker completed when the processes start >>doing this? >> >> Thanks, >> Daniel >> >> Daniel Ence >> Graduate Student >> Eccles Institute of Human Genetics >> University of Utah >> 15 North 2030 East, Room 2100 >> Salt Lake City, UT 84112-5330 >> ________________________________________ >> From: maker-devel-bounces at yandell-lab.org >>[maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>Durling [mikael.durling at slu.se] >> Sent: Wednesday, February 20, 2013 6:12 AM >> To: maker-devel at yandell-lab.org >> Subject: [maker-devel] Running MAKER on highly fragmented assembly >> >> Hi, >> >> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>not optimal, as I will most likely miss a substantial part of the gene >>complement due to the fragmentation. Disregarding this, my question is >>if there are other problems with running maker on these kinds of genomes >>with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>with MPI (mpich2) behaves rather in a rather strange way, with basically >>one of the ranks staying at 100% cpu, and the others lingering at about >>0%. Now and then I see a burst of activity in the other ranks before >>they get back to low activity. Could this be a result of the >>fragmentation level, or should I look for other problems? (Like the all >>to common problems of running over NFS with locking etc). >> >> cheers, >> Mikael >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 07:35:31 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:35:31 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C61403544E@exchange2-2> I have been trying both with 2.27 (r968) and svn r992. Both behave similarly on the fragmented NGS assemblies I try to annotate now. However, I just run a highly contiguous assembly (the one mentioned below with 32 Mbp in 15 scaffolds) that went through MAKER r968 without any problems at all. Based on this it seems the problem is not NFS related. Is there some other debug information I can provide? Nevertheless, I I have seen that the MAKER have seemed to be better at utilising cpu resources for the slightly more fragmented assemblies (1000 scaffolds or so with N50 in the 300-700 kb range) than this single sample of a highly contiguous assembly. Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From mikael.durling at slu.se Fri Feb 22 07:40:25 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:40:25 +0000 Subject: [maker-devel] failing contigs Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael From dence at genetics.utah.edu Fri Feb 22 15:40:54 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Fri, 22 Feb 2013 21:40:54 +0000 Subject: [maker-devel] failing contigs In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Message-ID: Hi Mikael, Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Friday, February 22, 2013 6:40 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] failing contigs Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 16:21:16 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 22:21:16 +0000 Subject: [maker-devel] failing contigs In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2>, Message-ID: <16A5EBD9-1CAC-4D4C-9C6E-9BDAF6E79B24@slu.se> You are most likely right. When you mention that I remember that I stumbled across that particular problem in another analysis pipeline based on blast I wrote long ago. Interesting that such an arbitrary limit has not been lifted. Thanks for the hint. I will try mangling the fasta ids to something shorter. Mikael --- Sent from a crippled computer (a.k.a a phone) 22 feb 2013 kl. 22:40 skrev "Daniel Ence" : > Hi Mikael, > > Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Friday, February 22, 2013 6:40 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] failing contigs > > Hi, > > while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: > > WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. > [?] > stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA > doing blastx repeats > ERROR: Failed while collecting blastn reports > ERROR: Chunk failed at level:1, tier_type:3 > FAILED CONTIG:scaffold_01 > > polishig ESTs > ERROR: Chunk failed at level:4, tier_type:0 > FAILED CONTIG:scaffold_01 > > If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. > > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Mon Feb 25 04:57:21 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Mon, 25 Feb 2013 10:57:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Hi Carson and Daniel, It seems that this issue might be related to the other issue about maker not finding sequences in the protein and est fasta files due to too long sequence ids. For those annotation setups where I have truncated the ids to short unique names, it seems maker runs fine on all cpu:s allocated. While on the other sets, it seems maker ranks stall once they can't polish an est or protein alignment. (I have no firm evidence of this, as i havn't found a way to separate the logs from the different ranks.) Anyhow, now that I have truncated the names, I have not seen any messages of failed polishing (yet). Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From carsonhh at gmail.com Mon Feb 25 09:05:34 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 25 Feb 2013 10:05:34 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Message-ID: Good to know it is working so far. I know that long names can also wrap around in output files causing the BioPerl parser to not splice the name pieces back together (so you get truncated names). For splitting the reports, I think there is a way to get MPI itself to tell you who each message originated from (--tag-output in OpenMPI). Alternatively you could try hacking the SUB{__WARN__} and SUB{__DIE__} entries at the top of the maker script to print the ranks out before all warning or failures. --Carson On 13-02-25 5:57 AM, "Mikael Brandstr?m Durling" wrote: >Hi Carson and Daniel, > >It seems that this issue might be related to the other issue about maker >not finding sequences in the protein and est fasta files due to too long >sequence ids. For those annotation setups where I have truncated the ids >to short unique names, it seems maker runs fine on all cpu:s allocated. >While on the other sets, it seems maker ranks stall once they can't >polish an est or protein alignment. (I have no firm evidence of this, as >i havn't found a way to separate the logs from the different ranks.) > >Anyhow, now that I have truncated the names, I have not seen any messages >of failed polishing (yet). > >Mikael > > > >21 feb 2013 kl. 16:58 skrev Carson Holt : > >> Which version of MAKER are you using? We recently deployed MAKER on the >> large NIH TACC computer cluster with some additional modifications for >> very large MPI jobs (> 1500 cpus). Some of the modifications focus on >> reannotating very large contigs as opposed to the small contig de novo >> annotation that MAKER already works well on. The modifications will be >> merged back into the MAKER downloadable release soon, but I could give >>you >> access to test with especially if you are running large MPI jobs or on >> very large multi-megabase contigs. >> >> --Carson >> >> >> On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" >> >> wrote: >> >>> Hi Daniel, >>> the genomes I work with are in the order of 30-60 Mb. Other assemblies >>> have been quick jobs for maker without any problems. >>> >>> If I run maker with -debugmpi, I get sets of debug printouts from the >>> different ranks now and then: >>> >>> COMM INITIALIZATION | SEND | who_I_am | 3 >>> --> 0 | 46731 >>> COMM INITIALIZATION | SEND | what_I_want | 3 >>> --> 0 | 46732 >>> COMM INITIALIZATION | RECV | what_I_want | 0 >>> <-- 3 | 312850 >>> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >>> --> 3 | 312851 >>> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >>> <-- 3 | 312852 >>> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >>> <-- 0 | 46733 >>> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >>> --> 0 | 46734 >>> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >>> --> 3 | 312853 >>> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >>> <-- 0 | 46735 >>> COMM INITIALIZATION | RECV | who_I_am | 0 >>> <-- ANY | 312854 >>> >>> and then they seem to stay waiting while a single rank continues to run >>> the normal analysis. I have filtered the assembly for contigs shorter >>> than then minimum length set in maker_opts.ctl. >>> >>> I did some strace:ing of the ranks that do nothing, and it seems they >>> loop over running a subprocess that basically does a process listing. >>> >>> I might be completely off in my guesses of what the problem might be. >>>I'm >>> sort of afraid that I'm bitten by some NFS related problem as I have >>>been >>> quite a few times by know. I will soon try to reannotate a genome >>> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make >>>sure >>> that make behaves as expected with that genome. >>> Mikael >>> >>> >>> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >>> >>>> Hi Mikael, Depending on the genome size, the assembly you've described >>>> shouldn't be too difficult to work with. The process activity that >>>> you're describing sounds more like a race condition, where one process >>>> is hogging all the work and all the other processes keep trying to >>>>find >>>> work, but keep getting in each others' way. >>>> >>>> How much of the genome has maker completed when the processes start >>>> doing this? >>>> >>>> Thanks, >>>> Daniel >>>> >>>> Daniel Ence >>>> Graduate Student >>>> Eccles Institute of Human Genetics >>>> University of Utah >>>> 15 North 2030 East, Room 2100 >>>> Salt Lake City, UT 84112-5330 >>>> ________________________________________ >>>> From: maker-devel-bounces at yandell-lab.org >>>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>>> Durling [mikael.durling at slu.se] >>>> Sent: Wednesday, February 20, 2013 6:12 AM >>>> To: maker-devel at yandell-lab.org >>>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>>> >>>> Hi, >>>> >>>> I'm trying to run MAKER on a rather fragmented assembly. I know this >>>>is >>>> not optimal, as I will most likely miss a substantial part of the gene >>>> complement due to the fragmentation. Disregarding this, my question is >>>> if there are other problems with running maker on these kinds of >>>>genomes >>>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, >>>>run >>>> with MPI (mpich2) behaves rather in a rather strange way, with >>>>basically >>>> one of the ranks staying at 100% cpu, and the others lingering at >>>>about >>>> 0%. Now and then I see a burst of activity in the other ranks before >>>> they get back to low activity. Could this be a result of the >>>> fragmentation level, or should I look for other problems? (Like the >>>>all >>>> to common problems of running over NFS with locking etc). >>>> >>>> cheers, >>>> Mikael >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>>> >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> > From jokelley at stanford.edu Tue Feb 26 16:11:26 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 14:11:26 -0800 Subject: [maker-devel] perl Message-ID: Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 19:47:23 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 20:47:23 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: The shebang line (#!) should use whatever perl was used in the "perl Build.PL" step of the configuration. Then "Build install" will overwrite the existing scripts with the ones using a different perl. --Carson From: Joanna Kelley Date: Tuesday, 26 February, 2013 5:11 PM To: Subject: [maker-devel] perl Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlee.nakasugi at sydney.edu.au Tue Feb 26 22:09:34 2013 From: kenlee.nakasugi at sydney.edu.au (Kenlee Nakasugi) Date: Wed, 27 Feb 2013 15:09:34 +1100 Subject: [maker-devel] resuming maker Message-ID: <1361938174.2353.28.camel@waterhouse874-8> Hi, When resuming maker on the same dataset, is the 'theVoid' folder and all its contents required to allow it to start out where it stopped? i.e. is this what is used to let maker know that the scaffold has already be calculated? thanks, Ken -- Kenlee Nakasugi | Research Fellow School of Molecular Bioscience Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 T: +61 2 9114 1321 E: kenlee.nakasugi at sydney.edu.au From carsonhh at gmail.com Tue Feb 26 22:10:55 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:10:55 -0500 Subject: [maker-devel] resuming maker In-Reply-To: <1361938174.2353.28.camel@waterhouse874-8> Message-ID: Yes. theVoid directory contains raw analysis reports and log files. --Carson On 13-02-26 11:09 PM, "Kenlee Nakasugi" wrote: >Hi, > >When resuming maker on the same dataset, is the 'theVoid' folder and all >its contents required to allow it to start out where it stopped? >i.e. is this what is used to let maker know that the scaffold has >already be calculated? > >thanks, >Ken > >-- >Kenlee Nakasugi | Research Fellow >School of Molecular Bioscience >Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 >T: +61 2 9114 1321 >E: kenlee.nakasugi at sydney.edu.au > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From ranjani at uga.edu Tue Feb 26 22:22:43 2013 From: ranjani at uga.edu (Sivaranjani Namasivayam) Date: Wed, 27 Feb 2013 04:22:43 +0000 Subject: [maker-devel] MAKER prediction- evidence weightage Message-ID: Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 22:35:28 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:35:28 -0500 Subject: [maker-devel] MAKER prediction- evidence weightage In-Reply-To: Message-ID: Not so much "weight" as much as different evidence types help infer different parts of a gene model. ESTs can be used to infer introns, UTR, and splice sites. Proteins help infer exons and CDS. They become "hints" to the gene predictors for the location of these sub-features. Then if ESTs have matching exons that extend beyond the new model produced by SNAP or Augustus using the hints, then they can be added on directly as UTR. That error can happen in the older version 2.10, but should not happen with 2.27. To get rid of it you have to edit several lines of code in maker and BioPerl to remove every mention of SDBM_File. It really doesn't effect anything, as the module in question ends up using DB_File regardless. Thanks, Carson From: Sivaranjani Namasivayam Date: Tuesday, 26 February, 2013 11:22 PM To: "maker-devel at yandell-lab.org" Subject: [maker-devel] MAKER prediction- evidence weightage Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jokelley at stanford.edu Tue Feb 26 23:53:52 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 21:53:52 -0800 Subject: [maker-devel] perl In-Reply-To: References: Message-ID: Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl > Build.PL" step of the configuration. Then "Build install" will overwrite > the existing scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Wed Feb 27 00:14:48 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 01:14:48 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: Did you run './Build install' after running '/opt/perl/bin/perl Build.PL ' to overwrite the old scripts? You may have to delete the /maker/bin directory before running './Build install' to guarantee that they are overwritten. PS. I'm glad the shebang line could brighten your day. Thanks, Carson From: Joanna Kelley Date: Wednesday, 27 February, 2013 12:53 AM To: Carson Holt Cc: Subject: Re: [maker-devel] perl Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl Build.PL" > step of the configuration. Then "Build install" will overwrite the existing > scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak > er-devel_yandell-lab.org -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjfields at illinois.edu Wed Feb 27 15:05:47 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Wed, 27 Feb 2013 21:05:47 +0000 Subject: [maker-devel] Preformatted BLAST databases? Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> All, We have a number of pre-formatted BLAST databases set up locally (taxonomically separated, for example) that we would like to use for annotation of specific assemblies. We would also really like to save space where possible and not copy auto-generated BLAST databases where possible. These don't currently seem to be supported however (at least, when I attempt to use them via the config they don't work). Is this possible via MAKER currently, or are there any plans to support this in the future? chris From carsonhh at gmail.com Wed Feb 27 15:15:33 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:15:33 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> Message-ID: MAKER has to make separate databases (min 10) for each input fasta, so that it can split the work up more efficiently (not just one big database per fasta). For space savinf purposes, you could set a different location for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be shared across multiple MAKER instances and not be regenerated constantly. --Carson On 13-02-27 4:05 PM, "Fields, Christopher J" wrote: >All, > >We have a number of pre-formatted BLAST databases set up locally >(taxonomically separated, for example) that we would like to use for >annotation of specific assemblies. We would also really like to save >space where possible and not copy auto-generated BLAST databases where >possible. > >These don't currently seem to be supported however (at least, when I >attempt to use them via the config they don't work). Is this possible >via MAKER currently, or are there any plans to support this in the >future? > >chris >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Wed Feb 27 15:17:13 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:17:13 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: Message-ID: FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not already there. --Carson On 13-02-27 4:15 PM, "Carson Holt" wrote: >MAKER has to make separate databases (min 10) for each input fasta, so >that it can split the work up more efficiently (not just one big database >per fasta). For space savinf purposes, you could set a different >location >for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >shared across multiple MAKER instances and not be regenerated constantly. > >--Carson > > >On 13-02-27 4:05 PM, "Fields, Christopher J" >wrote: > >>All, >> >>We have a number of pre-formatted BLAST databases set up locally >>(taxonomically separated, for example) that we would like to use for >>annotation of specific assemblies. We would also really like to save >>space where possible and not copy auto-generated BLAST databases where >>possible. >> >>These don't currently seem to be supported however (at least, when I >>attempt to use them via the config they don't work). Is this possible >>via MAKER currently, or are there any plans to support this in the >>future? >> >>chris >>_______________________________________________ >>maker-devel mailing list >>maker-devel at box290.bluehost.com >>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From cjfields at illinois.edu Thu Feb 28 00:18:20 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Thu, 28 Feb 2013 06:18:20 +0000 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: References: Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E7062FF@CHIMBX5.ad.uillinois.edu> Thanks Carson, that should work. chris On Feb 27, 2013, at 3:17 PM, Carson Holt wrote: > FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not > already there. > > --Carson > > > On 13-02-27 4:15 PM, "Carson Holt" wrote: > >> MAKER has to make separate databases (min 10) for each input fasta, so >> that it can split the work up more efficiently (not just one big database >> per fasta). For space savinf purposes, you could set a different >> location >> for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >> shared across multiple MAKER instances and not be regenerated constantly. >> >> --Carson >> >> >> On 13-02-27 4:05 PM, "Fields, Christopher J" >> wrote: >> >>> All, >>> >>> We have a number of pre-formatted BLAST databases set up locally >>> (taxonomically separated, for example) that we would like to use for >>> annotation of specific assemblies. We would also really like to save >>> space where possible and not copy auto-generated BLAST databases where >>> possible. >>> >>> These don't currently seem to be supported however (at least, when I >>> attempt to use them via the config they don't work). Is this possible >>> via MAKER currently, or are there any plans to support this in the >>> future? >>> >>> chris >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From bushleyk at science.oregonstate.edu Mon Feb 4 16:48:32 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Mon, 04 Feb 2013 15:48:32 -0800 Subject: [maker-devel] maker MPI Message-ID: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Greetings, We are reinstalling maker after a system upgrade and having some issues with getting MPI version running. One question, will maker only run in MPI-CH2 or will it also run in LAM-MPI? When trying to run in MPI-CH2, we get this error: /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt cd /pseudospace2/bushleyk/MAKER/Testrun; /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 '/local/cluster/MAKER_2.10/bin/mpi_maker' /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr make: *** [maker] Error 127 which maybe means it's not finding the right perl? When trying to run in LAM-MPI we get this error, was wondering what the MAKER/EVALUATOR is? ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR running repeat masker. #--------- command -------------# Widget::RepeatMasker: /local/cluster/spatafora/RepeatMasker/RepeatMasker /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-500-500.0.all.rb -species all -dir /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa 1 #-------------------------------# ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM make: *** [maker_new] Error 252 Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 From mikael.durling at slu.se Tue Feb 5 08:01:32 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Tue, 5 Feb 2013 15:01:32 +0000 Subject: [maker-devel] Handling of strandedness of est_gff Message-ID: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Hi, while looking into the annotations created by maker I find that there are cases where the ab initio prediction sits on one strand of the reference, while the est_gff evidence from cufflinks sits on the other strand. In this case the aed is rather low, and the predictions seems quite off. Is maker taking the strand information of the est_gff into account? If so, is it possible to tell maker that the ests are unstranded and only use the structural information rather than structure and strand? Thanks for any advice, Mikael From carsonhh at gmail.com Tue Feb 5 08:17:40 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 10:17:40 -0500 Subject: [maker-devel] Handling of strandedness of est_gff In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Message-ID: MAKER assumes the data in the est_gff is correctly stranded. If not there is no real way for MAKER to adjust for that as I can show you several cases where I can get coding models that match the est alignment well on both strands. MAKER uses exonerate to polish alignments when generated internally because in cases such as the ones I just described, one strand will match better only when you account for all splice sites in the model, but I can't do that type of polishing on gff3 derived alignments, and in the case of cufflinks, it should already be taking splice sites into account. Of course if these are single exon alignments then cufflinks doesn't know which strand to assign them to and I would recommend not including single exon alignments in the est_gff. Also consider using trinity over cufflinks, as I've seen more positive results from its alignments/assemblies of mRNAseq. Thanks, Carson On 13-02-05 10:01 AM, "Mikael Brandstr?m Durling" wrote: >Hi, > >while looking into the annotations created by maker I find that there are >cases where the ab initio prediction sits on one strand of the reference, >while the est_gff evidence from cufflinks sits on the other strand. In >this case the aed is rather low, and the predictions seems quite off. Is >maker taking the strand information of the est_gff into account? If so, >is it possible to tell maker that the ests are unstranded and only use >the structural information rather than structure and strand? > >Thanks for any advice, >Mikael > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Tue Feb 5 13:23:36 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 15:23:36 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Message-ID: LAMMPI is pretty much dead, even their website recommends using OpenMPI instead. For installing MAKER with OpenMPI you need to set a couple of environmental variables before MAKER setup. export OMPI_MCA_mpi_warn_on_fork 0 export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD These not only need to be set before compilation, but also before any run (so add them to you ~/.bashrc or ~/.bash_profile or any module load scripts thanks). The LD_PRELOAD statement needs to be set for any program using OpenMPI's shared libraries and not just MAKER, so it's normally a good idea to have that set system wide for all users. The detail can be Found in the OpenMPI documentation. Note sometimes system library updates can break OpenMPI's shared libraries while not breaking OpenMPI itself, so you might also need to recompile OpenMPI if it has broken shared libraries. Thanks, Carson On 13-02-04 6:48 PM, "Kathryn E. Bushley" wrote: >Greetings, > >We are reinstalling maker after a system upgrade and having some >issues with getting MPI version running. One question, will maker >only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >run in MPI-CH2, we get this error: > >/local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >cd /pseudospace2/bushleyk/MAKER/Testrun; >/local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >'/local/cluster/MAKER_2.10/bin/mpi_maker' >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >make: *** [maker] Error 127 > >which maybe means it's not finding the right perl? > > >When trying to run in LAM-MPI we get this error, was wondering what >the MAKER/EVALUATOR is? > >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >running repeat masker. >#--------- command -------------# >Widget::RepeatMasker: >/local/cluster/spatafora/RepeatMasker/RepeatMasker >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >0-500.0.all.rb -species all -dir >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >1 >#-------------------------------# >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >make: *** [maker_new] Error 252 > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > > > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From dence at genetics.utah.edu Mon Feb 11 09:23:28 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Mon, 11 Feb 2013 16:23:28 +0000 Subject: [maker-devel] Help on MAKER In-Reply-To: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> References: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> Message-ID: Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________ From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Mon Feb 11 09:25:58 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 11 Feb 2013 11:25:58 -0500 Subject: [maker-devel] Help on MAKER In-Reply-To: Message-ID: When the "./Build " script is created then "./Build installexes" can be used to get external executables and "./Build installdeps" can be used to get perl prerequisite modules. Thanks, Carson From: Daniel Ence Date: Monday, 11 February, 2013 11:23 AM To: "maker-devel at yandell-lab.org" Cc: Hud Hud Subject: Re: [maker-devel] Help on MAKER Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.campbell1 at gmail.com Mon Feb 11 09:50:04 2013 From: michael.s.campbell1 at gmail.com (Michael Campbell) Date: Mon, 11 Feb 2013 09:50:04 -0700 Subject: [maker-devel] Help on Maker In-Reply-To: <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> References: <1360546957.87948.YahooMailNeo@web164902.mail.bf1.yahoo.com> <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> Message-ID: Hi Huda, I'm forwarding this to the MAKER dev list to get some more eyes on it. The link to the Korf website works for me. Where are you trying to connect form? Mike On Sun, Feb 10, 2013 at 10:10 PM, Hud Hud wrote: > Hi Mike, > Thnks 4 ur reply..ive tried the ./Build blast command..I prtscrn the last > part of the configuration as attached. Is there anything that I've > missed?....And for the SNAP, I tried to download manually from the website, > but i cant get access to the korflab.ucdavis.edu/software website.. > > Thanks.. > ------------------------------ > *From:* Michael Campbell > *To:* "?? Li, Qing" > *Cc:* Hud Hud > *Sent:* Monday, February 11, 2013 11:42 AM > *Subject:* Re: Help on Maker > > Hi Huda, > > If you have unpacked the maker tar ball and moved to the src directory and > have run the "perl Build.PL " command you can run the > next command "./Build status" and that should give a report that looks like > this. > > > ============================================================================== > STATUS MAKER 2.25 > > ============================================================================== > PERL Dependencies: VERIFIED > External Programs: VERIFIED > External C Libraries: VERIFIED > MPI SUPPORT: DISABLED > MWAS Web Interface: DISABLED > MAKER PACKAGE: CONFIGURATION OK > > > Important Commands: > ./Build installdeps #installs missing PERL dependencies > ./Build installexes #installs all missing external programs > ./Build install #installs MAKER > ./Build status #Shows this status menu > > Other Commands: > ./Build repeatmasker #installs RepeatMasker (asks for RepBase) > ./Build blast #installs BLAST (NCBI BLAST+) > ./Build exonerate #installs Exonerate (v2 on UNIX / v1 on Mac OSX) > ./Build snap #installs SNAP > ./Build augustus #installs Augustus > ./Build apollo #installs Apollo > ./Build gbrowse #installs GBrowse (must be root) > ./Build jbrowse #installs JBrowse (MAKER copy, not web accecible) > ./Build mpich2 #installs MPICH2 (but manual install recommended) > > > I've highlighted the commands above that you can run to have MAKER install > snap and blast for you. Here is a url for a gmod tutorial for installing > and running MAKER http://gmod.org/wiki/MAKER_Tutorial_2012. > > I haven't actually used cygwin so I am interested in hearing how MAKER > runs in that environment. I will ask a few other people and see if they > have any experience with cygwin. > > Let me know if you get stuck or have any more questions, > Mike > > > On Sun, Feb 10, 2013 at 6:52 PM, ?? Li, Qing wrote: > > Hello Huda, > No worries:) I am cp your question to the expert maker usr in our group! > > Mike, would you think you could answer Huda's question and help him, or > would you think we need to ask Carlson for help? > Best, > Qing > > > 2013/2/10 Hud Hud > > Hello Hi, Im Huda...Im having problem during the installation of > MAKER...I'm not sure whether im asking the right person, I just pick anyone > from the list of Yandell People page...Hofefully im not troubling u with > this email... > Back to the problem...I try to install Maker but I've noticed that im > missing some external program that are Blast and SNAP...how to solve this > problem. FYI im very new in this field and not very familiar with > programming or scripting. Im using cygwin. Thanks for your help > > > > > > -- > Michael Campbell MS, RD. > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ph:585-3543 > > > > -- Michael Campbell MS, RD. Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ph:585-3543 -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Fri Feb 15 10:15:53 2013 From: carsonhh at gmail.com (Carson Holt) Date: Fri, 15 Feb 2013 12:15:53 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Message-ID: It cannot establish the communication ring on LAMMPI, so it is acting like multiple independent instances of MAKER up to the number of processors specified to MPI. So it gives the warning "MAKER already running" when it detects the other job's file locks. I'd recommend to just use MPICH2, as I don't have any idea what might happen with LAMMPI. --Carson On 13-02-15 12:10 PM, "Kathryn E. Bushley" wrote: >Hello Carson, > >Thanks. We currently have either LAMMPI or MPICH2 up and running on >our cluster so are trying to make it work in either one of those. >We're able to get it run in LAMMPI but get this error about the MAKER >EVALUATOR already running, any idea what that might be? It did work >previously in MPICH2 so maybe we need to go that route. > >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt > >LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University > >[bushleyk at fungi0 Test]$ lamnodes >n0 fungi0.cgrb.oregonstate.local:4:origin,this_node >[bushleyk at fungi0 Test]$ ps -aef | grep lam >bushleyk 11818 1 0 08:56 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 >bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam >sullichr 13795 1 0 Feb14 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 >/local/cluster/maker/bin/mpi_maker >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >Received SIGTERM > > > > > > > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > >Quoting Carson Holt : > >> LAMMPI is pretty much dead, even their website recommends using OpenMPI >> instead. >> >> For installing MAKER with OpenMPI you need to set a couple of >> environmental variables before MAKER setup. >> >> export OMPI_MCA_mpi_warn_on_fork 0 >> export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD >> >> These not only need to be set before compilation, but also before any >>run >> (so add them to you ~/.bashrc or ~/.bash_profile or any module load >> scripts thanks). The LD_PRELOAD statement needs to be set for any >>program >> using OpenMPI's shared libraries and not just MAKER, so it's normally a >> good idea to have that set system wide for all users. The detail can be >> Found in the OpenMPI documentation. Note sometimes system library >>updates >> can break OpenMPI's shared libraries while not breaking OpenMPI itself, >>so >> you might also need to recompile OpenMPI if it has broken shared >>libraries. >> >> Thanks, >> Carson >> >> >> On 13-02-04 6:48 PM, "Kathryn E. Bushley" >> wrote: >> >>> Greetings, >>> >>> We are reinstalling maker after a system upgrade and having some >>> issues with getting MPI version running. One question, will maker >>> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >>> run in MPI-CH2, we get this error: >>> >>> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >>> cd /pseudospace2/bushleyk/MAKER/Testrun; >>> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >>> '/local/cluster/MAKER_2.10/bin/mpi_maker' >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> make: *** [maker] Error 127 >>> >>> which maybe means it's not finding the right perl? >>> >>> >>> When trying to run in LAM-MPI we get this error, was wondering what >>> the MAKER/EVALUATOR is? >>> >>> ERROR: Cannot start process. MAKER/EVALUATOR already running >>> with different settings in this same directory. >>> >>> >>> FATAL ERROR >>> running repeat masker. >>> #--------- command -------------# >>> Widget::RepeatMasker: >>> /local/cluster/spatafora/RepeatMasker/RepeatMasker >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> >>>tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp- >>>50 >>> 0-500.0.all.rb -species all -dir >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >>> 1 >>> #-------------------------------# >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> It seems that [at least] one of the processes that was started with >>> mpirun did not invoke MPI_INIT before quitting (it is possible that >>> more than one process did not invoke MPI_INIT -- mpirun was only >>> notified of the first one, which was on node n0). >>> >>> mpirun can *only* be used with MPI programs (i.e., programs that >>> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >>> to run non-MPI programs over the lambooted nodes. >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> Received SIGTERM >>> make: *** [maker_new] Error 252 >>> >>> >>> >>> >>> >>> >>> >>> >>> Best, >>> >>> Kathryn >>> >>> >>> "To err is human...to really foul up requires the root password." >>> >>> >>> Kathryn E. Bushley >>> Postdoctoral Researcher >>> 2082 Cordley Hall >>> Department of Botany and Plant Pathology >>> Oregon State University >>> Corvallis, OR 97331 >>> office: (541) 737-5284 >>> cell: (541) 908-0116 >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> > > From bushleyk at science.oregonstate.edu Fri Feb 15 10:10:23 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Fri, 15 Feb 2013 09:10:23 -0800 Subject: [maker-devel] maker MPI In-Reply-To: References: Message-ID: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Hello Carson, Thanks. We currently have either LAMMPI or MPICH2 up and running on our cluster so are trying to make it work in either one of those. We're able to get it run in LAMMPI but get this error about the MAKER EVALUATOR already running, any idea what that might be? It did work previously in MPICH2 so maybe we need to go that route. [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University [bushleyk at fungi0 Test]$ lamnodes n0 fungi0.cgrb.oregonstate.local:4:origin,this_node [bushleyk at fungi0 Test]$ ps -aef | grep lam bushleyk 11818 1 0 08:56 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam sullichr 13795 1 0 Feb14 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 /local/cluster/maker/bin/mpi_maker ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM Received SIGTERM Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 Quoting Carson Holt : > LAMMPI is pretty much dead, even their website recommends using OpenMPI > instead. > > For installing MAKER with OpenMPI you need to set a couple of > environmental variables before MAKER setup. > > export OMPI_MCA_mpi_warn_on_fork 0 > export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD > > These not only need to be set before compilation, but also before any run > (so add them to you ~/.bashrc or ~/.bash_profile or any module load > scripts thanks). The LD_PRELOAD statement needs to be set for any program > using OpenMPI's shared libraries and not just MAKER, so it's normally a > good idea to have that set system wide for all users. The detail can be > Found in the OpenMPI documentation. Note sometimes system library updates > can break OpenMPI's shared libraries while not breaking OpenMPI itself, so > you might also need to recompile OpenMPI if it has broken shared libraries. > > Thanks, > Carson > > > On 13-02-04 6:48 PM, "Kathryn E. Bushley" > wrote: > >> Greetings, >> >> We are reinstalling maker after a system upgrade and having some >> issues with getting MPI version running. One question, will maker >> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >> run in MPI-CH2, we get this error: >> >> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >> cd /pseudospace2/bushleyk/MAKER/Testrun; >> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >> '/local/cluster/MAKER_2.10/bin/mpi_maker' >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> make: *** [maker] Error 127 >> >> which maybe means it's not finding the right perl? >> >> >> When trying to run in LAM-MPI we get this error, was wondering what >> the MAKER/EVALUATOR is? >> >> ERROR: Cannot start process. MAKER/EVALUATOR already running >> with different settings in this same directory. >> >> >> FATAL ERROR >> running repeat masker. >> #--------- command -------------# >> Widget::RepeatMasker: >> /local/cluster/spatafora/RepeatMasker/RepeatMasker >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >> 0-500.0.all.rb -species all -dir >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >> 1 >> #-------------------------------# >> -------------------------------------------------------------------------- >> --- >> It seems that [at least] one of the processes that was started with >> mpirun did not invoke MPI_INIT before quitting (it is possible that >> more than one process did not invoke MPI_INIT -- mpirun was only >> notified of the first one, which was on node n0). >> >> mpirun can *only* be used with MPI programs (i.e., programs that >> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >> to run non-MPI programs over the lambooted nodes. >> -------------------------------------------------------------------------- >> --- >> Received SIGTERM >> make: *** [maker_new] Error 252 >> >> >> >> >> >> >> >> >> Best, >> >> Kathryn >> >> >> "To err is human...to really foul up requires the root password." >> >> >> Kathryn E. Bushley >> Postdoctoral Researcher >> 2082 Cordley Hall >> Department of Botany and Plant Pathology >> Oregon State University >> Corvallis, OR 97331 >> office: (541) 737-5284 >> cell: (541) 908-0116 >> >> >> >> >> >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > > From mikael.durling at slu.se Wed Feb 20 06:12:08 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Wed, 20 Feb 2013 13:12:08 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael From dence at genetics.utah.edu Wed Feb 20 09:29:21 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Wed, 20 Feb 2013 16:29:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. How much of the genome has maker completed when the processes start doing this? Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Wednesday, February 20, 2013 6:12 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] Running MAKER on highly fragmented assembly Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Thu Feb 21 08:39:37 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Thu, 21 Feb 2013 15:39:37 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Hi Daniel, the genomes I work with are in the order of 30-60 Mb. Other assemblies have been quick jobs for maker without any problems. If I run maker with -debugmpi, I get sets of debug printouts from the different ranks now and then: COMM INITIALIZATION | SEND | who_I_am | 3 --> 0 | 46731 COMM INITIALIZATION | SEND | what_I_want | 3 --> 0 | 46732 COMM INITIALIZATION | RECV | what_I_want | 0 <-- 3 | 312850 COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 --> 3 | 312851 HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 <-- 3 | 312852 COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 <-- 0 | 46733 HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 --> 0 | 46734 HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 --> 3 | 312853 HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 <-- 0 | 46735 COMM INITIALIZATION | RECV | who_I_am | 0 <-- ANY | 312854 and then they seem to stay waiting while a single rank continues to run the normal analysis. I have filtered the assembly for contigs shorter than then minimum length set in maker_opts.ctl. I did some strace:ing of the ranks that do nothing, and it seems they loop over running a subprocess that basically does a process listing. I might be completely off in my guesses of what the problem might be. I'm sort of afraid that I'm bitten by some NFS related problem as I have been quite a few times by know. I will soon try to reannotate a genome sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure that make behaves as expected with that genome. Mikael 20 feb 2013 kl. 17:29 skrev Daniel Ence : > Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. > > How much of the genome has maker completed when the processes start doing this? > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Wednesday, February 20, 2013 6:12 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] Running MAKER on highly fragmented assembly > > Hi, > > I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). > > cheers, > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Thu Feb 21 08:58:10 2013 From: carsonhh at gmail.com (Carson Holt) Date: Thu, 21 Feb 2013 10:58:10 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Message-ID: Which version of MAKER are you using? We recently deployed MAKER on the large NIH TACC computer cluster with some additional modifications for very large MPI jobs (> 1500 cpus). Some of the modifications focus on reannotating very large contigs as opposed to the small contig de novo annotation that MAKER already works well on. The modifications will be merged back into the MAKER downloadable release soon, but I could give you access to test with especially if you are running large MPI jobs or on very large multi-megabase contigs. --Carson On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" wrote: >Hi Daniel, >the genomes I work with are in the order of 30-60 Mb. Other assemblies >have been quick jobs for maker without any problems. > >If I run maker with -debugmpi, I get sets of debug printouts from the >different ranks now and then: > >COMM INITIALIZATION | SEND | who_I_am | 3 >--> 0 | 46731 >COMM INITIALIZATION | SEND | what_I_want | 3 >--> 0 | 46732 >COMM INITIALIZATION | RECV | what_I_want | 0 ><-- 3 | 312850 >COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >--> 3 | 312851 >HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 ><-- 3 | 312852 >COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 ><-- 0 | 46733 >HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >--> 0 | 46734 >HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >--> 3 | 312853 >HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 ><-- 0 | 46735 >COMM INITIALIZATION | RECV | who_I_am | 0 ><-- ANY | 312854 > >and then they seem to stay waiting while a single rank continues to run >the normal analysis. I have filtered the assembly for contigs shorter >than then minimum length set in maker_opts.ctl. > >I did some strace:ing of the ranks that do nothing, and it seems they >loop over running a subprocess that basically does a process listing. > >I might be completely off in my guesses of what the problem might be. I'm >sort of afraid that I'm bitten by some NFS related problem as I have been >quite a few times by know. I will soon try to reannotate a genome >sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >that make behaves as expected with that genome. >Mikael > > >20 feb 2013 kl. 17:29 skrev Daniel Ence : > >> Hi Mikael, Depending on the genome size, the assembly you've described >>shouldn't be too difficult to work with. The process activity that >>you're describing sounds more like a race condition, where one process >>is hogging all the work and all the other processes keep trying to find >>work, but keep getting in each others' way. >> >> How much of the genome has maker completed when the processes start >>doing this? >> >> Thanks, >> Daniel >> >> Daniel Ence >> Graduate Student >> Eccles Institute of Human Genetics >> University of Utah >> 15 North 2030 East, Room 2100 >> Salt Lake City, UT 84112-5330 >> ________________________________________ >> From: maker-devel-bounces at yandell-lab.org >>[maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>Durling [mikael.durling at slu.se] >> Sent: Wednesday, February 20, 2013 6:12 AM >> To: maker-devel at yandell-lab.org >> Subject: [maker-devel] Running MAKER on highly fragmented assembly >> >> Hi, >> >> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>not optimal, as I will most likely miss a substantial part of the gene >>complement due to the fragmentation. Disregarding this, my question is >>if there are other problems with running maker on these kinds of genomes >>with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>with MPI (mpich2) behaves rather in a rather strange way, with basically >>one of the ranks staying at 100% cpu, and the others lingering at about >>0%. Now and then I see a burst of activity in the other ranks before >>they get back to low activity. Could this be a result of the >>fragmentation level, or should I look for other problems? (Like the all >>to common problems of running over NFS with locking etc). >> >> cheers, >> Mikael >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 06:35:31 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:35:31 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C61403544E@exchange2-2> I have been trying both with 2.27 (r968) and svn r992. Both behave similarly on the fragmented NGS assemblies I try to annotate now. However, I just run a highly contiguous assembly (the one mentioned below with 32 Mbp in 15 scaffolds) that went through MAKER r968 without any problems at all. Based on this it seems the problem is not NFS related. Is there some other debug information I can provide? Nevertheless, I I have seen that the MAKER have seemed to be better at utilising cpu resources for the slightly more fragmented assemblies (1000 scaffolds or so with N50 in the 300-700 kb range) than this single sample of a highly contiguous assembly. Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From mikael.durling at slu.se Fri Feb 22 06:40:25 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:40:25 +0000 Subject: [maker-devel] failing contigs Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael From dence at genetics.utah.edu Fri Feb 22 14:40:54 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Fri, 22 Feb 2013 21:40:54 +0000 Subject: [maker-devel] failing contigs In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Message-ID: Hi Mikael, Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Friday, February 22, 2013 6:40 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] failing contigs Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 15:21:16 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 22:21:16 +0000 Subject: [maker-devel] failing contigs In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2>, Message-ID: <16A5EBD9-1CAC-4D4C-9C6E-9BDAF6E79B24@slu.se> You are most likely right. When you mention that I remember that I stumbled across that particular problem in another analysis pipeline based on blast I wrote long ago. Interesting that such an arbitrary limit has not been lifted. Thanks for the hint. I will try mangling the fasta ids to something shorter. Mikael --- Sent from a crippled computer (a.k.a a phone) 22 feb 2013 kl. 22:40 skrev "Daniel Ence" : > Hi Mikael, > > Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Friday, February 22, 2013 6:40 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] failing contigs > > Hi, > > while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: > > WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. > [?] > stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA > doing blastx repeats > ERROR: Failed while collecting blastn reports > ERROR: Chunk failed at level:1, tier_type:3 > FAILED CONTIG:scaffold_01 > > polishig ESTs > ERROR: Chunk failed at level:4, tier_type:0 > FAILED CONTIG:scaffold_01 > > If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. > > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Mon Feb 25 03:57:21 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Mon, 25 Feb 2013 10:57:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Hi Carson and Daniel, It seems that this issue might be related to the other issue about maker not finding sequences in the protein and est fasta files due to too long sequence ids. For those annotation setups where I have truncated the ids to short unique names, it seems maker runs fine on all cpu:s allocated. While on the other sets, it seems maker ranks stall once they can't polish an est or protein alignment. (I have no firm evidence of this, as i havn't found a way to separate the logs from the different ranks.) Anyhow, now that I have truncated the names, I have not seen any messages of failed polishing (yet). Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From carsonhh at gmail.com Mon Feb 25 08:05:34 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 25 Feb 2013 10:05:34 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Message-ID: Good to know it is working so far. I know that long names can also wrap around in output files causing the BioPerl parser to not splice the name pieces back together (so you get truncated names). For splitting the reports, I think there is a way to get MPI itself to tell you who each message originated from (--tag-output in OpenMPI). Alternatively you could try hacking the SUB{__WARN__} and SUB{__DIE__} entries at the top of the maker script to print the ranks out before all warning or failures. --Carson On 13-02-25 5:57 AM, "Mikael Brandstr?m Durling" wrote: >Hi Carson and Daniel, > >It seems that this issue might be related to the other issue about maker >not finding sequences in the protein and est fasta files due to too long >sequence ids. For those annotation setups where I have truncated the ids >to short unique names, it seems maker runs fine on all cpu:s allocated. >While on the other sets, it seems maker ranks stall once they can't >polish an est or protein alignment. (I have no firm evidence of this, as >i havn't found a way to separate the logs from the different ranks.) > >Anyhow, now that I have truncated the names, I have not seen any messages >of failed polishing (yet). > >Mikael > > > >21 feb 2013 kl. 16:58 skrev Carson Holt : > >> Which version of MAKER are you using? We recently deployed MAKER on the >> large NIH TACC computer cluster with some additional modifications for >> very large MPI jobs (> 1500 cpus). Some of the modifications focus on >> reannotating very large contigs as opposed to the small contig de novo >> annotation that MAKER already works well on. The modifications will be >> merged back into the MAKER downloadable release soon, but I could give >>you >> access to test with especially if you are running large MPI jobs or on >> very large multi-megabase contigs. >> >> --Carson >> >> >> On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" >> >> wrote: >> >>> Hi Daniel, >>> the genomes I work with are in the order of 30-60 Mb. Other assemblies >>> have been quick jobs for maker without any problems. >>> >>> If I run maker with -debugmpi, I get sets of debug printouts from the >>> different ranks now and then: >>> >>> COMM INITIALIZATION | SEND | who_I_am | 3 >>> --> 0 | 46731 >>> COMM INITIALIZATION | SEND | what_I_want | 3 >>> --> 0 | 46732 >>> COMM INITIALIZATION | RECV | what_I_want | 0 >>> <-- 3 | 312850 >>> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >>> --> 3 | 312851 >>> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >>> <-- 3 | 312852 >>> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >>> <-- 0 | 46733 >>> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >>> --> 0 | 46734 >>> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >>> --> 3 | 312853 >>> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >>> <-- 0 | 46735 >>> COMM INITIALIZATION | RECV | who_I_am | 0 >>> <-- ANY | 312854 >>> >>> and then they seem to stay waiting while a single rank continues to run >>> the normal analysis. I have filtered the assembly for contigs shorter >>> than then minimum length set in maker_opts.ctl. >>> >>> I did some strace:ing of the ranks that do nothing, and it seems they >>> loop over running a subprocess that basically does a process listing. >>> >>> I might be completely off in my guesses of what the problem might be. >>>I'm >>> sort of afraid that I'm bitten by some NFS related problem as I have >>>been >>> quite a few times by know. I will soon try to reannotate a genome >>> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make >>>sure >>> that make behaves as expected with that genome. >>> Mikael >>> >>> >>> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >>> >>>> Hi Mikael, Depending on the genome size, the assembly you've described >>>> shouldn't be too difficult to work with. The process activity that >>>> you're describing sounds more like a race condition, where one process >>>> is hogging all the work and all the other processes keep trying to >>>>find >>>> work, but keep getting in each others' way. >>>> >>>> How much of the genome has maker completed when the processes start >>>> doing this? >>>> >>>> Thanks, >>>> Daniel >>>> >>>> Daniel Ence >>>> Graduate Student >>>> Eccles Institute of Human Genetics >>>> University of Utah >>>> 15 North 2030 East, Room 2100 >>>> Salt Lake City, UT 84112-5330 >>>> ________________________________________ >>>> From: maker-devel-bounces at yandell-lab.org >>>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>>> Durling [mikael.durling at slu.se] >>>> Sent: Wednesday, February 20, 2013 6:12 AM >>>> To: maker-devel at yandell-lab.org >>>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>>> >>>> Hi, >>>> >>>> I'm trying to run MAKER on a rather fragmented assembly. I know this >>>>is >>>> not optimal, as I will most likely miss a substantial part of the gene >>>> complement due to the fragmentation. Disregarding this, my question is >>>> if there are other problems with running maker on these kinds of >>>>genomes >>>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, >>>>run >>>> with MPI (mpich2) behaves rather in a rather strange way, with >>>>basically >>>> one of the ranks staying at 100% cpu, and the others lingering at >>>>about >>>> 0%. Now and then I see a burst of activity in the other ranks before >>>> they get back to low activity. Could this be a result of the >>>> fragmentation level, or should I look for other problems? (Like the >>>>all >>>> to common problems of running over NFS with locking etc). >>>> >>>> cheers, >>>> Mikael >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>>> >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> > From jokelley at stanford.edu Tue Feb 26 15:11:26 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 14:11:26 -0800 Subject: [maker-devel] perl Message-ID: Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 18:47:23 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 20:47:23 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: The shebang line (#!) should use whatever perl was used in the "perl Build.PL" step of the configuration. Then "Build install" will overwrite the existing scripts with the ones using a different perl. --Carson From: Joanna Kelley Date: Tuesday, 26 February, 2013 5:11 PM To: Subject: [maker-devel] perl Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlee.nakasugi at sydney.edu.au Tue Feb 26 21:09:34 2013 From: kenlee.nakasugi at sydney.edu.au (Kenlee Nakasugi) Date: Wed, 27 Feb 2013 15:09:34 +1100 Subject: [maker-devel] resuming maker Message-ID: <1361938174.2353.28.camel@waterhouse874-8> Hi, When resuming maker on the same dataset, is the 'theVoid' folder and all its contents required to allow it to start out where it stopped? i.e. is this what is used to let maker know that the scaffold has already be calculated? thanks, Ken -- Kenlee Nakasugi | Research Fellow School of Molecular Bioscience Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 T: +61 2 9114 1321 E: kenlee.nakasugi at sydney.edu.au From carsonhh at gmail.com Tue Feb 26 21:10:55 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:10:55 -0500 Subject: [maker-devel] resuming maker In-Reply-To: <1361938174.2353.28.camel@waterhouse874-8> Message-ID: Yes. theVoid directory contains raw analysis reports and log files. --Carson On 13-02-26 11:09 PM, "Kenlee Nakasugi" wrote: >Hi, > >When resuming maker on the same dataset, is the 'theVoid' folder and all >its contents required to allow it to start out where it stopped? >i.e. is this what is used to let maker know that the scaffold has >already be calculated? > >thanks, >Ken > >-- >Kenlee Nakasugi | Research Fellow >School of Molecular Bioscience >Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 >T: +61 2 9114 1321 >E: kenlee.nakasugi at sydney.edu.au > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From ranjani at uga.edu Tue Feb 26 21:22:43 2013 From: ranjani at uga.edu (Sivaranjani Namasivayam) Date: Wed, 27 Feb 2013 04:22:43 +0000 Subject: [maker-devel] MAKER prediction- evidence weightage Message-ID: Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 21:35:28 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:35:28 -0500 Subject: [maker-devel] MAKER prediction- evidence weightage In-Reply-To: Message-ID: Not so much "weight" as much as different evidence types help infer different parts of a gene model. ESTs can be used to infer introns, UTR, and splice sites. Proteins help infer exons and CDS. They become "hints" to the gene predictors for the location of these sub-features. Then if ESTs have matching exons that extend beyond the new model produced by SNAP or Augustus using the hints, then they can be added on directly as UTR. That error can happen in the older version 2.10, but should not happen with 2.27. To get rid of it you have to edit several lines of code in maker and BioPerl to remove every mention of SDBM_File. It really doesn't effect anything, as the module in question ends up using DB_File regardless. Thanks, Carson From: Sivaranjani Namasivayam Date: Tuesday, 26 February, 2013 11:22 PM To: "maker-devel at yandell-lab.org" Subject: [maker-devel] MAKER prediction- evidence weightage Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jokelley at stanford.edu Tue Feb 26 22:53:52 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 21:53:52 -0800 Subject: [maker-devel] perl In-Reply-To: References: Message-ID: Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl > Build.PL" step of the configuration. Then "Build install" will overwrite > the existing scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 23:14:48 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 01:14:48 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: Did you run './Build install' after running '/opt/perl/bin/perl Build.PL ' to overwrite the old scripts? You may have to delete the /maker/bin directory before running './Build install' to guarantee that they are overwritten. PS. I'm glad the shebang line could brighten your day. Thanks, Carson From: Joanna Kelley Date: Wednesday, 27 February, 2013 12:53 AM To: Carson Holt Cc: Subject: Re: [maker-devel] perl Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl Build.PL" > step of the configuration. Then "Build install" will overwrite the existing > scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak > er-devel_yandell-lab.org -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjfields at illinois.edu Wed Feb 27 14:05:47 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Wed, 27 Feb 2013 21:05:47 +0000 Subject: [maker-devel] Preformatted BLAST databases? Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> All, We have a number of pre-formatted BLAST databases set up locally (taxonomically separated, for example) that we would like to use for annotation of specific assemblies. We would also really like to save space where possible and not copy auto-generated BLAST databases where possible. These don't currently seem to be supported however (at least, when I attempt to use them via the config they don't work). Is this possible via MAKER currently, or are there any plans to support this in the future? chris From carsonhh at gmail.com Wed Feb 27 14:15:33 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:15:33 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> Message-ID: MAKER has to make separate databases (min 10) for each input fasta, so that it can split the work up more efficiently (not just one big database per fasta). For space savinf purposes, you could set a different location for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be shared across multiple MAKER instances and not be regenerated constantly. --Carson On 13-02-27 4:05 PM, "Fields, Christopher J" wrote: >All, > >We have a number of pre-formatted BLAST databases set up locally >(taxonomically separated, for example) that we would like to use for >annotation of specific assemblies. We would also really like to save >space where possible and not copy auto-generated BLAST databases where >possible. > >These don't currently seem to be supported however (at least, when I >attempt to use them via the config they don't work). Is this possible >via MAKER currently, or are there any plans to support this in the >future? > >chris >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Wed Feb 27 14:17:13 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:17:13 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: Message-ID: FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not already there. --Carson On 13-02-27 4:15 PM, "Carson Holt" wrote: >MAKER has to make separate databases (min 10) for each input fasta, so >that it can split the work up more efficiently (not just one big database >per fasta). For space savinf purposes, you could set a different >location >for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >shared across multiple MAKER instances and not be regenerated constantly. > >--Carson > > >On 13-02-27 4:05 PM, "Fields, Christopher J" >wrote: > >>All, >> >>We have a number of pre-formatted BLAST databases set up locally >>(taxonomically separated, for example) that we would like to use for >>annotation of specific assemblies. We would also really like to save >>space where possible and not copy auto-generated BLAST databases where >>possible. >> >>These don't currently seem to be supported however (at least, when I >>attempt to use them via the config they don't work). Is this possible >>via MAKER currently, or are there any plans to support this in the >>future? >> >>chris >>_______________________________________________ >>maker-devel mailing list >>maker-devel at box290.bluehost.com >>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From cjfields at illinois.edu Wed Feb 27 23:18:20 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Thu, 28 Feb 2013 06:18:20 +0000 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: References: Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E7062FF@CHIMBX5.ad.uillinois.edu> Thanks Carson, that should work. chris On Feb 27, 2013, at 3:17 PM, Carson Holt wrote: > FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not > already there. > > --Carson > > > On 13-02-27 4:15 PM, "Carson Holt" wrote: > >> MAKER has to make separate databases (min 10) for each input fasta, so >> that it can split the work up more efficiently (not just one big database >> per fasta). For space savinf purposes, you could set a different >> location >> for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >> shared across multiple MAKER instances and not be regenerated constantly. >> >> --Carson >> >> >> On 13-02-27 4:05 PM, "Fields, Christopher J" >> wrote: >> >>> All, >>> >>> We have a number of pre-formatted BLAST databases set up locally >>> (taxonomically separated, for example) that we would like to use for >>> annotation of specific assemblies. We would also really like to save >>> space where possible and not copy auto-generated BLAST databases where >>> possible. >>> >>> These don't currently seem to be supported however (at least, when I >>> attempt to use them via the config they don't work). Is this possible >>> via MAKER currently, or are there any plans to support this in the >>> future? >>> >>> chris >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From bushleyk at science.oregonstate.edu Mon Feb 4 16:48:32 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Mon, 04 Feb 2013 15:48:32 -0800 Subject: [maker-devel] maker MPI Message-ID: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Greetings, We are reinstalling maker after a system upgrade and having some issues with getting MPI version running. One question, will maker only run in MPI-CH2 or will it also run in LAM-MPI? When trying to run in MPI-CH2, we get this error: /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt cd /pseudospace2/bushleyk/MAKER/Testrun; /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 '/local/cluster/MAKER_2.10/bin/mpi_maker' /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr make: *** [maker] Error 127 which maybe means it's not finding the right perl? When trying to run in LAM-MPI we get this error, was wondering what the MAKER/EVALUATOR is? ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR running repeat masker. #--------- command -------------# Widget::RepeatMasker: /local/cluster/spatafora/RepeatMasker/RepeatMasker /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-500-500.0.all.rb -species all -dir /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa 1 #-------------------------------# ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM make: *** [maker_new] Error 252 Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 From mikael.durling at slu.se Tue Feb 5 08:01:32 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Tue, 5 Feb 2013 15:01:32 +0000 Subject: [maker-devel] Handling of strandedness of est_gff Message-ID: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Hi, while looking into the annotations created by maker I find that there are cases where the ab initio prediction sits on one strand of the reference, while the est_gff evidence from cufflinks sits on the other strand. In this case the aed is rather low, and the predictions seems quite off. Is maker taking the strand information of the est_gff into account? If so, is it possible to tell maker that the ests are unstranded and only use the structural information rather than structure and strand? Thanks for any advice, Mikael From carsonhh at gmail.com Tue Feb 5 08:17:40 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 10:17:40 -0500 Subject: [maker-devel] Handling of strandedness of est_gff In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Message-ID: MAKER assumes the data in the est_gff is correctly stranded. If not there is no real way for MAKER to adjust for that as I can show you several cases where I can get coding models that match the est alignment well on both strands. MAKER uses exonerate to polish alignments when generated internally because in cases such as the ones I just described, one strand will match better only when you account for all splice sites in the model, but I can't do that type of polishing on gff3 derived alignments, and in the case of cufflinks, it should already be taking splice sites into account. Of course if these are single exon alignments then cufflinks doesn't know which strand to assign them to and I would recommend not including single exon alignments in the est_gff. Also consider using trinity over cufflinks, as I've seen more positive results from its alignments/assemblies of mRNAseq. Thanks, Carson On 13-02-05 10:01 AM, "Mikael Brandstr?m Durling" wrote: >Hi, > >while looking into the annotations created by maker I find that there are >cases where the ab initio prediction sits on one strand of the reference, >while the est_gff evidence from cufflinks sits on the other strand. In >this case the aed is rather low, and the predictions seems quite off. Is >maker taking the strand information of the est_gff into account? If so, >is it possible to tell maker that the ests are unstranded and only use >the structural information rather than structure and strand? > >Thanks for any advice, >Mikael > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Tue Feb 5 13:23:36 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 15:23:36 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Message-ID: LAMMPI is pretty much dead, even their website recommends using OpenMPI instead. For installing MAKER with OpenMPI you need to set a couple of environmental variables before MAKER setup. export OMPI_MCA_mpi_warn_on_fork 0 export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD These not only need to be set before compilation, but also before any run (so add them to you ~/.bashrc or ~/.bash_profile or any module load scripts thanks). The LD_PRELOAD statement needs to be set for any program using OpenMPI's shared libraries and not just MAKER, so it's normally a good idea to have that set system wide for all users. The detail can be Found in the OpenMPI documentation. Note sometimes system library updates can break OpenMPI's shared libraries while not breaking OpenMPI itself, so you might also need to recompile OpenMPI if it has broken shared libraries. Thanks, Carson On 13-02-04 6:48 PM, "Kathryn E. Bushley" wrote: >Greetings, > >We are reinstalling maker after a system upgrade and having some >issues with getting MPI version running. One question, will maker >only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >run in MPI-CH2, we get this error: > >/local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >cd /pseudospace2/bushleyk/MAKER/Testrun; >/local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >'/local/cluster/MAKER_2.10/bin/mpi_maker' >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >make: *** [maker] Error 127 > >which maybe means it's not finding the right perl? > > >When trying to run in LAM-MPI we get this error, was wondering what >the MAKER/EVALUATOR is? > >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >running repeat masker. >#--------- command -------------# >Widget::RepeatMasker: >/local/cluster/spatafora/RepeatMasker/RepeatMasker >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >0-500.0.all.rb -species all -dir >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >1 >#-------------------------------# >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >make: *** [maker_new] Error 252 > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > > > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From dence at genetics.utah.edu Mon Feb 11 09:23:28 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Mon, 11 Feb 2013 16:23:28 +0000 Subject: [maker-devel] Help on MAKER In-Reply-To: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> References: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> Message-ID: Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________ From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Mon Feb 11 09:25:58 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 11 Feb 2013 11:25:58 -0500 Subject: [maker-devel] Help on MAKER In-Reply-To: Message-ID: When the "./Build " script is created then "./Build installexes" can be used to get external executables and "./Build installdeps" can be used to get perl prerequisite modules. Thanks, Carson From: Daniel Ence Date: Monday, 11 February, 2013 11:23 AM To: "maker-devel at yandell-lab.org" Cc: Hud Hud Subject: Re: [maker-devel] Help on MAKER Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.campbell1 at gmail.com Mon Feb 11 09:50:04 2013 From: michael.s.campbell1 at gmail.com (Michael Campbell) Date: Mon, 11 Feb 2013 09:50:04 -0700 Subject: [maker-devel] Help on Maker In-Reply-To: <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> References: <1360546957.87948.YahooMailNeo@web164902.mail.bf1.yahoo.com> <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> Message-ID: Hi Huda, I'm forwarding this to the MAKER dev list to get some more eyes on it. The link to the Korf website works for me. Where are you trying to connect form? Mike On Sun, Feb 10, 2013 at 10:10 PM, Hud Hud wrote: > Hi Mike, > Thnks 4 ur reply..ive tried the ./Build blast command..I prtscrn the last > part of the configuration as attached. Is there anything that I've > missed?....And for the SNAP, I tried to download manually from the website, > but i cant get access to the korflab.ucdavis.edu/software website.. > > Thanks.. > ------------------------------ > *From:* Michael Campbell > *To:* "?? Li, Qing" > *Cc:* Hud Hud > *Sent:* Monday, February 11, 2013 11:42 AM > *Subject:* Re: Help on Maker > > Hi Huda, > > If you have unpacked the maker tar ball and moved to the src directory and > have run the "perl Build.PL " command you can run the > next command "./Build status" and that should give a report that looks like > this. > > > ============================================================================== > STATUS MAKER 2.25 > > ============================================================================== > PERL Dependencies: VERIFIED > External Programs: VERIFIED > External C Libraries: VERIFIED > MPI SUPPORT: DISABLED > MWAS Web Interface: DISABLED > MAKER PACKAGE: CONFIGURATION OK > > > Important Commands: > ./Build installdeps #installs missing PERL dependencies > ./Build installexes #installs all missing external programs > ./Build install #installs MAKER > ./Build status #Shows this status menu > > Other Commands: > ./Build repeatmasker #installs RepeatMasker (asks for RepBase) > ./Build blast #installs BLAST (NCBI BLAST+) > ./Build exonerate #installs Exonerate (v2 on UNIX / v1 on Mac OSX) > ./Build snap #installs SNAP > ./Build augustus #installs Augustus > ./Build apollo #installs Apollo > ./Build gbrowse #installs GBrowse (must be root) > ./Build jbrowse #installs JBrowse (MAKER copy, not web accecible) > ./Build mpich2 #installs MPICH2 (but manual install recommended) > > > I've highlighted the commands above that you can run to have MAKER install > snap and blast for you. Here is a url for a gmod tutorial for installing > and running MAKER http://gmod.org/wiki/MAKER_Tutorial_2012. > > I haven't actually used cygwin so I am interested in hearing how MAKER > runs in that environment. I will ask a few other people and see if they > have any experience with cygwin. > > Let me know if you get stuck or have any more questions, > Mike > > > On Sun, Feb 10, 2013 at 6:52 PM, ?? Li, Qing wrote: > > Hello Huda, > No worries:) I am cp your question to the expert maker usr in our group! > > Mike, would you think you could answer Huda's question and help him, or > would you think we need to ask Carlson for help? > Best, > Qing > > > 2013/2/10 Hud Hud > > Hello Hi, Im Huda...Im having problem during the installation of > MAKER...I'm not sure whether im asking the right person, I just pick anyone > from the list of Yandell People page...Hofefully im not troubling u with > this email... > Back to the problem...I try to install Maker but I've noticed that im > missing some external program that are Blast and SNAP...how to solve this > problem. FYI im very new in this field and not very familiar with > programming or scripting. Im using cygwin. Thanks for your help > > > > > > -- > Michael Campbell MS, RD. > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ph:585-3543 > > > > -- Michael Campbell MS, RD. Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ph:585-3543 -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Fri Feb 15 10:15:53 2013 From: carsonhh at gmail.com (Carson Holt) Date: Fri, 15 Feb 2013 12:15:53 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Message-ID: It cannot establish the communication ring on LAMMPI, so it is acting like multiple independent instances of MAKER up to the number of processors specified to MPI. So it gives the warning "MAKER already running" when it detects the other job's file locks. I'd recommend to just use MPICH2, as I don't have any idea what might happen with LAMMPI. --Carson On 13-02-15 12:10 PM, "Kathryn E. Bushley" wrote: >Hello Carson, > >Thanks. We currently have either LAMMPI or MPICH2 up and running on >our cluster so are trying to make it work in either one of those. >We're able to get it run in LAMMPI but get this error about the MAKER >EVALUATOR already running, any idea what that might be? It did work >previously in MPICH2 so maybe we need to go that route. > >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt > >LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University > >[bushleyk at fungi0 Test]$ lamnodes >n0 fungi0.cgrb.oregonstate.local:4:origin,this_node >[bushleyk at fungi0 Test]$ ps -aef | grep lam >bushleyk 11818 1 0 08:56 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 >bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam >sullichr 13795 1 0 Feb14 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 >/local/cluster/maker/bin/mpi_maker >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >Received SIGTERM > > > > > > > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > >Quoting Carson Holt : > >> LAMMPI is pretty much dead, even their website recommends using OpenMPI >> instead. >> >> For installing MAKER with OpenMPI you need to set a couple of >> environmental variables before MAKER setup. >> >> export OMPI_MCA_mpi_warn_on_fork 0 >> export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD >> >> These not only need to be set before compilation, but also before any >>run >> (so add them to you ~/.bashrc or ~/.bash_profile or any module load >> scripts thanks). The LD_PRELOAD statement needs to be set for any >>program >> using OpenMPI's shared libraries and not just MAKER, so it's normally a >> good idea to have that set system wide for all users. The detail can be >> Found in the OpenMPI documentation. Note sometimes system library >>updates >> can break OpenMPI's shared libraries while not breaking OpenMPI itself, >>so >> you might also need to recompile OpenMPI if it has broken shared >>libraries. >> >> Thanks, >> Carson >> >> >> On 13-02-04 6:48 PM, "Kathryn E. Bushley" >> wrote: >> >>> Greetings, >>> >>> We are reinstalling maker after a system upgrade and having some >>> issues with getting MPI version running. One question, will maker >>> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >>> run in MPI-CH2, we get this error: >>> >>> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >>> cd /pseudospace2/bushleyk/MAKER/Testrun; >>> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >>> '/local/cluster/MAKER_2.10/bin/mpi_maker' >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> make: *** [maker] Error 127 >>> >>> which maybe means it's not finding the right perl? >>> >>> >>> When trying to run in LAM-MPI we get this error, was wondering what >>> the MAKER/EVALUATOR is? >>> >>> ERROR: Cannot start process. MAKER/EVALUATOR already running >>> with different settings in this same directory. >>> >>> >>> FATAL ERROR >>> running repeat masker. >>> #--------- command -------------# >>> Widget::RepeatMasker: >>> /local/cluster/spatafora/RepeatMasker/RepeatMasker >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> >>>tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp- >>>50 >>> 0-500.0.all.rb -species all -dir >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >>> 1 >>> #-------------------------------# >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> It seems that [at least] one of the processes that was started with >>> mpirun did not invoke MPI_INIT before quitting (it is possible that >>> more than one process did not invoke MPI_INIT -- mpirun was only >>> notified of the first one, which was on node n0). >>> >>> mpirun can *only* be used with MPI programs (i.e., programs that >>> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >>> to run non-MPI programs over the lambooted nodes. >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> Received SIGTERM >>> make: *** [maker_new] Error 252 >>> >>> >>> >>> >>> >>> >>> >>> >>> Best, >>> >>> Kathryn >>> >>> >>> "To err is human...to really foul up requires the root password." >>> >>> >>> Kathryn E. Bushley >>> Postdoctoral Researcher >>> 2082 Cordley Hall >>> Department of Botany and Plant Pathology >>> Oregon State University >>> Corvallis, OR 97331 >>> office: (541) 737-5284 >>> cell: (541) 908-0116 >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> > > From bushleyk at science.oregonstate.edu Fri Feb 15 10:10:23 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Fri, 15 Feb 2013 09:10:23 -0800 Subject: [maker-devel] maker MPI In-Reply-To: References: Message-ID: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Hello Carson, Thanks. We currently have either LAMMPI or MPICH2 up and running on our cluster so are trying to make it work in either one of those. We're able to get it run in LAMMPI but get this error about the MAKER EVALUATOR already running, any idea what that might be? It did work previously in MPICH2 so maybe we need to go that route. [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University [bushleyk at fungi0 Test]$ lamnodes n0 fungi0.cgrb.oregonstate.local:4:origin,this_node [bushleyk at fungi0 Test]$ ps -aef | grep lam bushleyk 11818 1 0 08:56 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam sullichr 13795 1 0 Feb14 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 /local/cluster/maker/bin/mpi_maker ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM Received SIGTERM Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 Quoting Carson Holt : > LAMMPI is pretty much dead, even their website recommends using OpenMPI > instead. > > For installing MAKER with OpenMPI you need to set a couple of > environmental variables before MAKER setup. > > export OMPI_MCA_mpi_warn_on_fork 0 > export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD > > These not only need to be set before compilation, but also before any run > (so add them to you ~/.bashrc or ~/.bash_profile or any module load > scripts thanks). The LD_PRELOAD statement needs to be set for any program > using OpenMPI's shared libraries and not just MAKER, so it's normally a > good idea to have that set system wide for all users. The detail can be > Found in the OpenMPI documentation. Note sometimes system library updates > can break OpenMPI's shared libraries while not breaking OpenMPI itself, so > you might also need to recompile OpenMPI if it has broken shared libraries. > > Thanks, > Carson > > > On 13-02-04 6:48 PM, "Kathryn E. Bushley" > wrote: > >> Greetings, >> >> We are reinstalling maker after a system upgrade and having some >> issues with getting MPI version running. One question, will maker >> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >> run in MPI-CH2, we get this error: >> >> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >> cd /pseudospace2/bushleyk/MAKER/Testrun; >> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >> '/local/cluster/MAKER_2.10/bin/mpi_maker' >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> make: *** [maker] Error 127 >> >> which maybe means it's not finding the right perl? >> >> >> When trying to run in LAM-MPI we get this error, was wondering what >> the MAKER/EVALUATOR is? >> >> ERROR: Cannot start process. MAKER/EVALUATOR already running >> with different settings in this same directory. >> >> >> FATAL ERROR >> running repeat masker. >> #--------- command -------------# >> Widget::RepeatMasker: >> /local/cluster/spatafora/RepeatMasker/RepeatMasker >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >> 0-500.0.all.rb -species all -dir >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >> 1 >> #-------------------------------# >> -------------------------------------------------------------------------- >> --- >> It seems that [at least] one of the processes that was started with >> mpirun did not invoke MPI_INIT before quitting (it is possible that >> more than one process did not invoke MPI_INIT -- mpirun was only >> notified of the first one, which was on node n0). >> >> mpirun can *only* be used with MPI programs (i.e., programs that >> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >> to run non-MPI programs over the lambooted nodes. >> -------------------------------------------------------------------------- >> --- >> Received SIGTERM >> make: *** [maker_new] Error 252 >> >> >> >> >> >> >> >> >> Best, >> >> Kathryn >> >> >> "To err is human...to really foul up requires the root password." >> >> >> Kathryn E. Bushley >> Postdoctoral Researcher >> 2082 Cordley Hall >> Department of Botany and Plant Pathology >> Oregon State University >> Corvallis, OR 97331 >> office: (541) 737-5284 >> cell: (541) 908-0116 >> >> >> >> >> >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > > From mikael.durling at slu.se Wed Feb 20 06:12:08 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Wed, 20 Feb 2013 13:12:08 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael From dence at genetics.utah.edu Wed Feb 20 09:29:21 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Wed, 20 Feb 2013 16:29:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. How much of the genome has maker completed when the processes start doing this? Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Wednesday, February 20, 2013 6:12 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] Running MAKER on highly fragmented assembly Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Thu Feb 21 08:39:37 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Thu, 21 Feb 2013 15:39:37 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Hi Daniel, the genomes I work with are in the order of 30-60 Mb. Other assemblies have been quick jobs for maker without any problems. If I run maker with -debugmpi, I get sets of debug printouts from the different ranks now and then: COMM INITIALIZATION | SEND | who_I_am | 3 --> 0 | 46731 COMM INITIALIZATION | SEND | what_I_want | 3 --> 0 | 46732 COMM INITIALIZATION | RECV | what_I_want | 0 <-- 3 | 312850 COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 --> 3 | 312851 HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 <-- 3 | 312852 COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 <-- 0 | 46733 HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 --> 0 | 46734 HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 --> 3 | 312853 HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 <-- 0 | 46735 COMM INITIALIZATION | RECV | who_I_am | 0 <-- ANY | 312854 and then they seem to stay waiting while a single rank continues to run the normal analysis. I have filtered the assembly for contigs shorter than then minimum length set in maker_opts.ctl. I did some strace:ing of the ranks that do nothing, and it seems they loop over running a subprocess that basically does a process listing. I might be completely off in my guesses of what the problem might be. I'm sort of afraid that I'm bitten by some NFS related problem as I have been quite a few times by know. I will soon try to reannotate a genome sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure that make behaves as expected with that genome. Mikael 20 feb 2013 kl. 17:29 skrev Daniel Ence : > Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. > > How much of the genome has maker completed when the processes start doing this? > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Wednesday, February 20, 2013 6:12 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] Running MAKER on highly fragmented assembly > > Hi, > > I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). > > cheers, > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Thu Feb 21 08:58:10 2013 From: carsonhh at gmail.com (Carson Holt) Date: Thu, 21 Feb 2013 10:58:10 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Message-ID: Which version of MAKER are you using? We recently deployed MAKER on the large NIH TACC computer cluster with some additional modifications for very large MPI jobs (> 1500 cpus). Some of the modifications focus on reannotating very large contigs as opposed to the small contig de novo annotation that MAKER already works well on. The modifications will be merged back into the MAKER downloadable release soon, but I could give you access to test with especially if you are running large MPI jobs or on very large multi-megabase contigs. --Carson On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" wrote: >Hi Daniel, >the genomes I work with are in the order of 30-60 Mb. Other assemblies >have been quick jobs for maker without any problems. > >If I run maker with -debugmpi, I get sets of debug printouts from the >different ranks now and then: > >COMM INITIALIZATION | SEND | who_I_am | 3 >--> 0 | 46731 >COMM INITIALIZATION | SEND | what_I_want | 3 >--> 0 | 46732 >COMM INITIALIZATION | RECV | what_I_want | 0 ><-- 3 | 312850 >COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >--> 3 | 312851 >HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 ><-- 3 | 312852 >COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 ><-- 0 | 46733 >HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >--> 0 | 46734 >HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >--> 3 | 312853 >HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 ><-- 0 | 46735 >COMM INITIALIZATION | RECV | who_I_am | 0 ><-- ANY | 312854 > >and then they seem to stay waiting while a single rank continues to run >the normal analysis. I have filtered the assembly for contigs shorter >than then minimum length set in maker_opts.ctl. > >I did some strace:ing of the ranks that do nothing, and it seems they >loop over running a subprocess that basically does a process listing. > >I might be completely off in my guesses of what the problem might be. I'm >sort of afraid that I'm bitten by some NFS related problem as I have been >quite a few times by know. I will soon try to reannotate a genome >sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >that make behaves as expected with that genome. >Mikael > > >20 feb 2013 kl. 17:29 skrev Daniel Ence : > >> Hi Mikael, Depending on the genome size, the assembly you've described >>shouldn't be too difficult to work with. The process activity that >>you're describing sounds more like a race condition, where one process >>is hogging all the work and all the other processes keep trying to find >>work, but keep getting in each others' way. >> >> How much of the genome has maker completed when the processes start >>doing this? >> >> Thanks, >> Daniel >> >> Daniel Ence >> Graduate Student >> Eccles Institute of Human Genetics >> University of Utah >> 15 North 2030 East, Room 2100 >> Salt Lake City, UT 84112-5330 >> ________________________________________ >> From: maker-devel-bounces at yandell-lab.org >>[maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>Durling [mikael.durling at slu.se] >> Sent: Wednesday, February 20, 2013 6:12 AM >> To: maker-devel at yandell-lab.org >> Subject: [maker-devel] Running MAKER on highly fragmented assembly >> >> Hi, >> >> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>not optimal, as I will most likely miss a substantial part of the gene >>complement due to the fragmentation. Disregarding this, my question is >>if there are other problems with running maker on these kinds of genomes >>with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>with MPI (mpich2) behaves rather in a rather strange way, with basically >>one of the ranks staying at 100% cpu, and the others lingering at about >>0%. Now and then I see a burst of activity in the other ranks before >>they get back to low activity. Could this be a result of the >>fragmentation level, or should I look for other problems? (Like the all >>to common problems of running over NFS with locking etc). >> >> cheers, >> Mikael >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 06:35:31 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:35:31 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C61403544E@exchange2-2> I have been trying both with 2.27 (r968) and svn r992. Both behave similarly on the fragmented NGS assemblies I try to annotate now. However, I just run a highly contiguous assembly (the one mentioned below with 32 Mbp in 15 scaffolds) that went through MAKER r968 without any problems at all. Based on this it seems the problem is not NFS related. Is there some other debug information I can provide? Nevertheless, I I have seen that the MAKER have seemed to be better at utilising cpu resources for the slightly more fragmented assemblies (1000 scaffolds or so with N50 in the 300-700 kb range) than this single sample of a highly contiguous assembly. Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From mikael.durling at slu.se Fri Feb 22 06:40:25 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:40:25 +0000 Subject: [maker-devel] failing contigs Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael From dence at genetics.utah.edu Fri Feb 22 14:40:54 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Fri, 22 Feb 2013 21:40:54 +0000 Subject: [maker-devel] failing contigs In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Message-ID: Hi Mikael, Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Friday, February 22, 2013 6:40 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] failing contigs Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 15:21:16 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 22:21:16 +0000 Subject: [maker-devel] failing contigs In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2>, Message-ID: <16A5EBD9-1CAC-4D4C-9C6E-9BDAF6E79B24@slu.se> You are most likely right. When you mention that I remember that I stumbled across that particular problem in another analysis pipeline based on blast I wrote long ago. Interesting that such an arbitrary limit has not been lifted. Thanks for the hint. I will try mangling the fasta ids to something shorter. Mikael --- Sent from a crippled computer (a.k.a a phone) 22 feb 2013 kl. 22:40 skrev "Daniel Ence" : > Hi Mikael, > > Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Friday, February 22, 2013 6:40 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] failing contigs > > Hi, > > while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: > > WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. > [?] > stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA > doing blastx repeats > ERROR: Failed while collecting blastn reports > ERROR: Chunk failed at level:1, tier_type:3 > FAILED CONTIG:scaffold_01 > > polishig ESTs > ERROR: Chunk failed at level:4, tier_type:0 > FAILED CONTIG:scaffold_01 > > If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. > > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Mon Feb 25 03:57:21 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Mon, 25 Feb 2013 10:57:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Hi Carson and Daniel, It seems that this issue might be related to the other issue about maker not finding sequences in the protein and est fasta files due to too long sequence ids. For those annotation setups where I have truncated the ids to short unique names, it seems maker runs fine on all cpu:s allocated. While on the other sets, it seems maker ranks stall once they can't polish an est or protein alignment. (I have no firm evidence of this, as i havn't found a way to separate the logs from the different ranks.) Anyhow, now that I have truncated the names, I have not seen any messages of failed polishing (yet). Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From carsonhh at gmail.com Mon Feb 25 08:05:34 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 25 Feb 2013 10:05:34 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Message-ID: Good to know it is working so far. I know that long names can also wrap around in output files causing the BioPerl parser to not splice the name pieces back together (so you get truncated names). For splitting the reports, I think there is a way to get MPI itself to tell you who each message originated from (--tag-output in OpenMPI). Alternatively you could try hacking the SUB{__WARN__} and SUB{__DIE__} entries at the top of the maker script to print the ranks out before all warning or failures. --Carson On 13-02-25 5:57 AM, "Mikael Brandstr?m Durling" wrote: >Hi Carson and Daniel, > >It seems that this issue might be related to the other issue about maker >not finding sequences in the protein and est fasta files due to too long >sequence ids. For those annotation setups where I have truncated the ids >to short unique names, it seems maker runs fine on all cpu:s allocated. >While on the other sets, it seems maker ranks stall once they can't >polish an est or protein alignment. (I have no firm evidence of this, as >i havn't found a way to separate the logs from the different ranks.) > >Anyhow, now that I have truncated the names, I have not seen any messages >of failed polishing (yet). > >Mikael > > > >21 feb 2013 kl. 16:58 skrev Carson Holt : > >> Which version of MAKER are you using? We recently deployed MAKER on the >> large NIH TACC computer cluster with some additional modifications for >> very large MPI jobs (> 1500 cpus). Some of the modifications focus on >> reannotating very large contigs as opposed to the small contig de novo >> annotation that MAKER already works well on. The modifications will be >> merged back into the MAKER downloadable release soon, but I could give >>you >> access to test with especially if you are running large MPI jobs or on >> very large multi-megabase contigs. >> >> --Carson >> >> >> On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" >> >> wrote: >> >>> Hi Daniel, >>> the genomes I work with are in the order of 30-60 Mb. Other assemblies >>> have been quick jobs for maker without any problems. >>> >>> If I run maker with -debugmpi, I get sets of debug printouts from the >>> different ranks now and then: >>> >>> COMM INITIALIZATION | SEND | who_I_am | 3 >>> --> 0 | 46731 >>> COMM INITIALIZATION | SEND | what_I_want | 3 >>> --> 0 | 46732 >>> COMM INITIALIZATION | RECV | what_I_want | 0 >>> <-- 3 | 312850 >>> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >>> --> 3 | 312851 >>> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >>> <-- 3 | 312852 >>> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >>> <-- 0 | 46733 >>> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >>> --> 0 | 46734 >>> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >>> --> 3 | 312853 >>> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >>> <-- 0 | 46735 >>> COMM INITIALIZATION | RECV | who_I_am | 0 >>> <-- ANY | 312854 >>> >>> and then they seem to stay waiting while a single rank continues to run >>> the normal analysis. I have filtered the assembly for contigs shorter >>> than then minimum length set in maker_opts.ctl. >>> >>> I did some strace:ing of the ranks that do nothing, and it seems they >>> loop over running a subprocess that basically does a process listing. >>> >>> I might be completely off in my guesses of what the problem might be. >>>I'm >>> sort of afraid that I'm bitten by some NFS related problem as I have >>>been >>> quite a few times by know. I will soon try to reannotate a genome >>> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make >>>sure >>> that make behaves as expected with that genome. >>> Mikael >>> >>> >>> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >>> >>>> Hi Mikael, Depending on the genome size, the assembly you've described >>>> shouldn't be too difficult to work with. The process activity that >>>> you're describing sounds more like a race condition, where one process >>>> is hogging all the work and all the other processes keep trying to >>>>find >>>> work, but keep getting in each others' way. >>>> >>>> How much of the genome has maker completed when the processes start >>>> doing this? >>>> >>>> Thanks, >>>> Daniel >>>> >>>> Daniel Ence >>>> Graduate Student >>>> Eccles Institute of Human Genetics >>>> University of Utah >>>> 15 North 2030 East, Room 2100 >>>> Salt Lake City, UT 84112-5330 >>>> ________________________________________ >>>> From: maker-devel-bounces at yandell-lab.org >>>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>>> Durling [mikael.durling at slu.se] >>>> Sent: Wednesday, February 20, 2013 6:12 AM >>>> To: maker-devel at yandell-lab.org >>>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>>> >>>> Hi, >>>> >>>> I'm trying to run MAKER on a rather fragmented assembly. I know this >>>>is >>>> not optimal, as I will most likely miss a substantial part of the gene >>>> complement due to the fragmentation. Disregarding this, my question is >>>> if there are other problems with running maker on these kinds of >>>>genomes >>>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, >>>>run >>>> with MPI (mpich2) behaves rather in a rather strange way, with >>>>basically >>>> one of the ranks staying at 100% cpu, and the others lingering at >>>>about >>>> 0%. Now and then I see a burst of activity in the other ranks before >>>> they get back to low activity. Could this be a result of the >>>> fragmentation level, or should I look for other problems? (Like the >>>>all >>>> to common problems of running over NFS with locking etc). >>>> >>>> cheers, >>>> Mikael >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>>> >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> > From jokelley at stanford.edu Tue Feb 26 15:11:26 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 14:11:26 -0800 Subject: [maker-devel] perl Message-ID: Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 18:47:23 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 20:47:23 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: The shebang line (#!) should use whatever perl was used in the "perl Build.PL" step of the configuration. Then "Build install" will overwrite the existing scripts with the ones using a different perl. --Carson From: Joanna Kelley Date: Tuesday, 26 February, 2013 5:11 PM To: Subject: [maker-devel] perl Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlee.nakasugi at sydney.edu.au Tue Feb 26 21:09:34 2013 From: kenlee.nakasugi at sydney.edu.au (Kenlee Nakasugi) Date: Wed, 27 Feb 2013 15:09:34 +1100 Subject: [maker-devel] resuming maker Message-ID: <1361938174.2353.28.camel@waterhouse874-8> Hi, When resuming maker on the same dataset, is the 'theVoid' folder and all its contents required to allow it to start out where it stopped? i.e. is this what is used to let maker know that the scaffold has already be calculated? thanks, Ken -- Kenlee Nakasugi | Research Fellow School of Molecular Bioscience Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 T: +61 2 9114 1321 E: kenlee.nakasugi at sydney.edu.au From carsonhh at gmail.com Tue Feb 26 21:10:55 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:10:55 -0500 Subject: [maker-devel] resuming maker In-Reply-To: <1361938174.2353.28.camel@waterhouse874-8> Message-ID: Yes. theVoid directory contains raw analysis reports and log files. --Carson On 13-02-26 11:09 PM, "Kenlee Nakasugi" wrote: >Hi, > >When resuming maker on the same dataset, is the 'theVoid' folder and all >its contents required to allow it to start out where it stopped? >i.e. is this what is used to let maker know that the scaffold has >already be calculated? > >thanks, >Ken > >-- >Kenlee Nakasugi | Research Fellow >School of Molecular Bioscience >Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 >T: +61 2 9114 1321 >E: kenlee.nakasugi at sydney.edu.au > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From ranjani at uga.edu Tue Feb 26 21:22:43 2013 From: ranjani at uga.edu (Sivaranjani Namasivayam) Date: Wed, 27 Feb 2013 04:22:43 +0000 Subject: [maker-devel] MAKER prediction- evidence weightage Message-ID: Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 21:35:28 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:35:28 -0500 Subject: [maker-devel] MAKER prediction- evidence weightage In-Reply-To: Message-ID: Not so much "weight" as much as different evidence types help infer different parts of a gene model. ESTs can be used to infer introns, UTR, and splice sites. Proteins help infer exons and CDS. They become "hints" to the gene predictors for the location of these sub-features. Then if ESTs have matching exons that extend beyond the new model produced by SNAP or Augustus using the hints, then they can be added on directly as UTR. That error can happen in the older version 2.10, but should not happen with 2.27. To get rid of it you have to edit several lines of code in maker and BioPerl to remove every mention of SDBM_File. It really doesn't effect anything, as the module in question ends up using DB_File regardless. Thanks, Carson From: Sivaranjani Namasivayam Date: Tuesday, 26 February, 2013 11:22 PM To: "maker-devel at yandell-lab.org" Subject: [maker-devel] MAKER prediction- evidence weightage Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jokelley at stanford.edu Tue Feb 26 22:53:52 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 21:53:52 -0800 Subject: [maker-devel] perl In-Reply-To: References: Message-ID: Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl > Build.PL" step of the configuration. Then "Build install" will overwrite > the existing scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 23:14:48 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 01:14:48 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: Did you run './Build install' after running '/opt/perl/bin/perl Build.PL ' to overwrite the old scripts? You may have to delete the /maker/bin directory before running './Build install' to guarantee that they are overwritten. PS. I'm glad the shebang line could brighten your day. Thanks, Carson From: Joanna Kelley Date: Wednesday, 27 February, 2013 12:53 AM To: Carson Holt Cc: Subject: Re: [maker-devel] perl Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl Build.PL" > step of the configuration. Then "Build install" will overwrite the existing > scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak > er-devel_yandell-lab.org -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjfields at illinois.edu Wed Feb 27 14:05:47 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Wed, 27 Feb 2013 21:05:47 +0000 Subject: [maker-devel] Preformatted BLAST databases? Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> All, We have a number of pre-formatted BLAST databases set up locally (taxonomically separated, for example) that we would like to use for annotation of specific assemblies. We would also really like to save space where possible and not copy auto-generated BLAST databases where possible. These don't currently seem to be supported however (at least, when I attempt to use them via the config they don't work). Is this possible via MAKER currently, or are there any plans to support this in the future? chris From carsonhh at gmail.com Wed Feb 27 14:15:33 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:15:33 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> Message-ID: MAKER has to make separate databases (min 10) for each input fasta, so that it can split the work up more efficiently (not just one big database per fasta). For space savinf purposes, you could set a different location for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be shared across multiple MAKER instances and not be regenerated constantly. --Carson On 13-02-27 4:05 PM, "Fields, Christopher J" wrote: >All, > >We have a number of pre-formatted BLAST databases set up locally >(taxonomically separated, for example) that we would like to use for >annotation of specific assemblies. We would also really like to save >space where possible and not copy auto-generated BLAST databases where >possible. > >These don't currently seem to be supported however (at least, when I >attempt to use them via the config they don't work). Is this possible >via MAKER currently, or are there any plans to support this in the >future? > >chris >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Wed Feb 27 14:17:13 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:17:13 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: Message-ID: FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not already there. --Carson On 13-02-27 4:15 PM, "Carson Holt" wrote: >MAKER has to make separate databases (min 10) for each input fasta, so >that it can split the work up more efficiently (not just one big database >per fasta). For space savinf purposes, you could set a different >location >for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >shared across multiple MAKER instances and not be regenerated constantly. > >--Carson > > >On 13-02-27 4:05 PM, "Fields, Christopher J" >wrote: > >>All, >> >>We have a number of pre-formatted BLAST databases set up locally >>(taxonomically separated, for example) that we would like to use for >>annotation of specific assemblies. We would also really like to save >>space where possible and not copy auto-generated BLAST databases where >>possible. >> >>These don't currently seem to be supported however (at least, when I >>attempt to use them via the config they don't work). Is this possible >>via MAKER currently, or are there any plans to support this in the >>future? >> >>chris >>_______________________________________________ >>maker-devel mailing list >>maker-devel at box290.bluehost.com >>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From cjfields at illinois.edu Wed Feb 27 23:18:20 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Thu, 28 Feb 2013 06:18:20 +0000 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: References: Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E7062FF@CHIMBX5.ad.uillinois.edu> Thanks Carson, that should work. chris On Feb 27, 2013, at 3:17 PM, Carson Holt wrote: > FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not > already there. > > --Carson > > > On 13-02-27 4:15 PM, "Carson Holt" wrote: > >> MAKER has to make separate databases (min 10) for each input fasta, so >> that it can split the work up more efficiently (not just one big database >> per fasta). For space savinf purposes, you could set a different >> location >> for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >> shared across multiple MAKER instances and not be regenerated constantly. >> >> --Carson >> >> >> On 13-02-27 4:05 PM, "Fields, Christopher J" >> wrote: >> >>> All, >>> >>> We have a number of pre-formatted BLAST databases set up locally >>> (taxonomically separated, for example) that we would like to use for >>> annotation of specific assemblies. We would also really like to save >>> space where possible and not copy auto-generated BLAST databases where >>> possible. >>> >>> These don't currently seem to be supported however (at least, when I >>> attempt to use them via the config they don't work). Is this possible >>> via MAKER currently, or are there any plans to support this in the >>> future? >>> >>> chris >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From bushleyk at science.oregonstate.edu Mon Feb 4 16:48:32 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Mon, 04 Feb 2013 15:48:32 -0800 Subject: [maker-devel] maker MPI Message-ID: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Greetings, We are reinstalling maker after a system upgrade and having some issues with getting MPI version running. One question, will maker only run in MPI-CH2 or will it also run in LAM-MPI? When trying to run in MPI-CH2, we get this error: /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt cd /pseudospace2/bushleyk/MAKER/Testrun; /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 '/local/cluster/MAKER_2.10/bin/mpi_maker' /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr /usr/bin/perl: symbol lookup error: /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: undefined symbol: Perl_Tstack_sp_ptr make: *** [maker] Error 127 which maybe means it's not finding the right perl? When trying to run in LAM-MPI we get this error, was wondering what the MAKER/EVALUATOR is? ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR running repeat masker. #--------- command -------------# Widget::RepeatMasker: /local/cluster/spatafora/RepeatMasker/RepeatMasker /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-500-500.0.all.rb -species all -dir /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_datastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa 1 #-------------------------------# ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM make: *** [maker_new] Error 252 Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 From mikael.durling at slu.se Tue Feb 5 08:01:32 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Tue, 5 Feb 2013 15:01:32 +0000 Subject: [maker-devel] Handling of strandedness of est_gff Message-ID: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Hi, while looking into the annotations created by maker I find that there are cases where the ab initio prediction sits on one strand of the reference, while the est_gff evidence from cufflinks sits on the other strand. In this case the aed is rather low, and the predictions seems quite off. Is maker taking the strand information of the est_gff into account? If so, is it possible to tell maker that the ests are unstranded and only use the structural information rather than structure and strand? Thanks for any advice, Mikael From carsonhh at gmail.com Tue Feb 5 08:17:40 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 10:17:40 -0500 Subject: [maker-devel] Handling of strandedness of est_gff In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614011146@exchange2-2> Message-ID: MAKER assumes the data in the est_gff is correctly stranded. If not there is no real way for MAKER to adjust for that as I can show you several cases where I can get coding models that match the est alignment well on both strands. MAKER uses exonerate to polish alignments when generated internally because in cases such as the ones I just described, one strand will match better only when you account for all splice sites in the model, but I can't do that type of polishing on gff3 derived alignments, and in the case of cufflinks, it should already be taking splice sites into account. Of course if these are single exon alignments then cufflinks doesn't know which strand to assign them to and I would recommend not including single exon alignments in the est_gff. Also consider using trinity over cufflinks, as I've seen more positive results from its alignments/assemblies of mRNAseq. Thanks, Carson On 13-02-05 10:01 AM, "Mikael Brandstr?m Durling" wrote: >Hi, > >while looking into the annotations created by maker I find that there are >cases where the ab initio prediction sits on one strand of the reference, >while the est_gff evidence from cufflinks sits on the other strand. In >this case the aed is rather low, and the predictions seems quite off. Is >maker taking the strand information of the est_gff into account? If so, >is it possible to tell maker that the ests are unstranded and only use >the structural information rather than structure and strand? > >Thanks for any advice, >Mikael > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Tue Feb 5 13:23:36 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 05 Feb 2013 15:23:36 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130204154832.10613sll5r8l8z40@webmail.oregonstate.edu> Message-ID: LAMMPI is pretty much dead, even their website recommends using OpenMPI instead. For installing MAKER with OpenMPI you need to set a couple of environmental variables before MAKER setup. export OMPI_MCA_mpi_warn_on_fork 0 export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD These not only need to be set before compilation, but also before any run (so add them to you ~/.bashrc or ~/.bash_profile or any module load scripts thanks). The LD_PRELOAD statement needs to be set for any program using OpenMPI's shared libraries and not just MAKER, so it's normally a good idea to have that set system wide for all users. The detail can be Found in the OpenMPI documentation. Note sometimes system library updates can break OpenMPI's shared libraries while not breaking OpenMPI itself, so you might also need to recompile OpenMPI if it has broken shared libraries. Thanks, Carson On 13-02-04 6:48 PM, "Kathryn E. Bushley" wrote: >Greetings, > >We are reinstalling maker after a system upgrade and having some >issues with getting MPI version running. One question, will maker >only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >run in MPI-CH2, we get this error: > >/local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >cd /pseudospace2/bushleyk/MAKER/Testrun; >/local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >'/local/cluster/MAKER_2.10/bin/mpi_maker' >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >/usr/bin/perl: symbol lookup error: >/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: > >undefined symbol: Perl_Tstack_sp_ptr >make: *** [maker] Error 127 > >which maybe means it's not finding the right perl? > > >When trying to run in LAM-MPI we get this error, was wondering what >the MAKER/EVALUATOR is? > >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >running repeat masker. >#--------- command -------------# >Widget::RepeatMasker: >/local/cluster/spatafora/RepeatMasker/RepeatMasker >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >0-500.0.all.rb -species all -dir >/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >1 >#-------------------------------# >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >make: *** [maker_new] Error 252 > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > > > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From dence at genetics.utah.edu Mon Feb 11 09:23:28 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Mon, 11 Feb 2013 16:23:28 +0000 Subject: [maker-devel] Help on MAKER In-Reply-To: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> References: <1360546737.56104.YahooMailNeo@web164905.mail.bf1.yahoo.com> Message-ID: Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________ From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Mon Feb 11 09:25:58 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 11 Feb 2013 11:25:58 -0500 Subject: [maker-devel] Help on MAKER In-Reply-To: Message-ID: When the "./Build " script is created then "./Build installexes" can be used to get external executables and "./Build installdeps" can be used to get perl prerequisite modules. Thanks, Carson From: Daniel Ence Date: Monday, 11 February, 2013 11:23 AM To: "maker-devel at yandell-lab.org" Cc: Hud Hud Subject: Re: [maker-devel] Help on MAKER Hi Huda, So, the basic process of install maker is as follows: 1) Download and extract the tgz make file. 2) Go to the "src" directory. 3) Run the command "perl Build.PL". (also its helpful, probably necessary, to have sudo permissions on the machine that you're install 4) This will run some configuration steps for maker and will create a perl script call "Build" (this is different from perl Build.PL). The remaining installation steps will run through this new Build script. 5) Run the command "./Build status" This will give you a report of what prerequisite programs and perl modules the Build scripts were able to find along with a list of commands for installing the various prereqs that you mentioned. If you have any more questions, please reply to the maker-devel. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 From: Hud Hud [hudarul at yahoo.com] Sent: Sunday, February 10, 2013 6:38 PM To: Daniel Ence Subject: Help on MAKER Hello Hi, Im Huda...Im having problem during the installation of MAKER...I'm not sure whether im asking the right person, I just pick anyone from the list of Yandell People page...Hofefully im not troubling u with this email... Back to the problem...I try to install Maker but I've noticed that im missing some external program that are Blast and SNAP...how to solve this problem. FYI im very new in this field and not very familiar with programming or scripting. Im using cygwin. Thanks for your help _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.campbell1 at gmail.com Mon Feb 11 09:50:04 2013 From: michael.s.campbell1 at gmail.com (Michael Campbell) Date: Mon, 11 Feb 2013 09:50:04 -0700 Subject: [maker-devel] Help on Maker In-Reply-To: <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> References: <1360546957.87948.YahooMailNeo@web164902.mail.bf1.yahoo.com> <1360559457.16203.YahooMailNeo@web164903.mail.bf1.yahoo.com> Message-ID: Hi Huda, I'm forwarding this to the MAKER dev list to get some more eyes on it. The link to the Korf website works for me. Where are you trying to connect form? Mike On Sun, Feb 10, 2013 at 10:10 PM, Hud Hud wrote: > Hi Mike, > Thnks 4 ur reply..ive tried the ./Build blast command..I prtscrn the last > part of the configuration as attached. Is there anything that I've > missed?....And for the SNAP, I tried to download manually from the website, > but i cant get access to the korflab.ucdavis.edu/software website.. > > Thanks.. > ------------------------------ > *From:* Michael Campbell > *To:* "?? Li, Qing" > *Cc:* Hud Hud > *Sent:* Monday, February 11, 2013 11:42 AM > *Subject:* Re: Help on Maker > > Hi Huda, > > If you have unpacked the maker tar ball and moved to the src directory and > have run the "perl Build.PL " command you can run the > next command "./Build status" and that should give a report that looks like > this. > > > ============================================================================== > STATUS MAKER 2.25 > > ============================================================================== > PERL Dependencies: VERIFIED > External Programs: VERIFIED > External C Libraries: VERIFIED > MPI SUPPORT: DISABLED > MWAS Web Interface: DISABLED > MAKER PACKAGE: CONFIGURATION OK > > > Important Commands: > ./Build installdeps #installs missing PERL dependencies > ./Build installexes #installs all missing external programs > ./Build install #installs MAKER > ./Build status #Shows this status menu > > Other Commands: > ./Build repeatmasker #installs RepeatMasker (asks for RepBase) > ./Build blast #installs BLAST (NCBI BLAST+) > ./Build exonerate #installs Exonerate (v2 on UNIX / v1 on Mac OSX) > ./Build snap #installs SNAP > ./Build augustus #installs Augustus > ./Build apollo #installs Apollo > ./Build gbrowse #installs GBrowse (must be root) > ./Build jbrowse #installs JBrowse (MAKER copy, not web accecible) > ./Build mpich2 #installs MPICH2 (but manual install recommended) > > > I've highlighted the commands above that you can run to have MAKER install > snap and blast for you. Here is a url for a gmod tutorial for installing > and running MAKER http://gmod.org/wiki/MAKER_Tutorial_2012. > > I haven't actually used cygwin so I am interested in hearing how MAKER > runs in that environment. I will ask a few other people and see if they > have any experience with cygwin. > > Let me know if you get stuck or have any more questions, > Mike > > > On Sun, Feb 10, 2013 at 6:52 PM, ?? Li, Qing wrote: > > Hello Huda, > No worries:) I am cp your question to the expert maker usr in our group! > > Mike, would you think you could answer Huda's question and help him, or > would you think we need to ask Carlson for help? > Best, > Qing > > > 2013/2/10 Hud Hud > > Hello Hi, Im Huda...Im having problem during the installation of > MAKER...I'm not sure whether im asking the right person, I just pick anyone > from the list of Yandell People page...Hofefully im not troubling u with > this email... > Back to the problem...I try to install Maker but I've noticed that im > missing some external program that are Blast and SNAP...how to solve this > problem. FYI im very new in this field and not very familiar with > programming or scripting. Im using cygwin. Thanks for your help > > > > > > -- > Michael Campbell MS, RD. > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ph:585-3543 > > > > -- Michael Campbell MS, RD. Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ph:585-3543 -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Fri Feb 15 10:15:53 2013 From: carsonhh at gmail.com (Carson Holt) Date: Fri, 15 Feb 2013 12:15:53 -0500 Subject: [maker-devel] maker MPI In-Reply-To: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Message-ID: It cannot establish the communication ring on LAMMPI, so it is acting like multiple independent instances of MAKER up to the number of processors specified to MPI. So it gives the warning "MAKER already running" when it detects the other job's file locks. I'd recommend to just use MPICH2, as I don't have any idea what might happen with LAMMPI. --Carson On 13-02-15 12:10 PM, "Kathryn E. Bushley" wrote: >Hello Carson, > >Thanks. We currently have either LAMMPI or MPICH2 up and running on >our cluster so are trying to make it work in either one of those. >We're able to get it run in LAMMPI but get this error about the MAKER >EVALUATOR already running, any idea what that might be? It did work >previously in MPICH2 so maybe we need to go that route. > >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot >/home/bpp/bushleyk/MPI2/fungi0_machinefile.txt > >LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University > >[bushleyk at fungi0 Test]$ lamnodes >n0 fungi0.cgrb.oregonstate.local:4:origin,this_node >[bushleyk at fungi0 Test]$ ps -aef | grep lam >bushleyk 11818 1 0 08:56 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 >bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam >sullichr 13795 1 0 Feb14 ? 00:00:00 >/usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 >[bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 >/local/cluster/maker/bin/mpi_maker >ERROR: Cannot start process. MAKER/EVALUATOR already running >with different settings in this same directory. > > >FATAL ERROR >-------------------------------------------------------------------------- >--- >It seems that [at least] one of the processes that was started with >mpirun did not invoke MPI_INIT before quitting (it is possible that >more than one process did not invoke MPI_INIT -- mpirun was only >notified of the first one, which was on node n0). > >mpirun can *only* be used with MPI programs (i.e., programs that >invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >to run non-MPI programs over the lambooted nodes. >-------------------------------------------------------------------------- >--- >Received SIGTERM >Received SIGTERM > > > > > > > > > > > > > > >Best, > >Kathryn > > >"To err is human...to really foul up requires the root password." > > >Kathryn E. Bushley >Postdoctoral Researcher >2082 Cordley Hall >Department of Botany and Plant Pathology >Oregon State University >Corvallis, OR 97331 >office: (541) 737-5284 >cell: (541) 908-0116 > > > > >Quoting Carson Holt : > >> LAMMPI is pretty much dead, even their website recommends using OpenMPI >> instead. >> >> For installing MAKER with OpenMPI you need to set a couple of >> environmental variables before MAKER setup. >> >> export OMPI_MCA_mpi_warn_on_fork 0 >> export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD >> >> These not only need to be set before compilation, but also before any >>run >> (so add them to you ~/.bashrc or ~/.bash_profile or any module load >> scripts thanks). The LD_PRELOAD statement needs to be set for any >>program >> using OpenMPI's shared libraries and not just MAKER, so it's normally a >> good idea to have that set system wide for all users. The detail can be >> Found in the OpenMPI documentation. Note sometimes system library >>updates >> can break OpenMPI's shared libraries while not breaking OpenMPI itself, >>so >> you might also need to recompile OpenMPI if it has broken shared >>libraries. >> >> Thanks, >> Carson >> >> >> On 13-02-04 6:48 PM, "Kathryn E. Bushley" >> wrote: >> >>> Greetings, >>> >>> We are reinstalling maker after a system upgrade and having some >>> issues with getting MPI version running. One question, will maker >>> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >>> run in MPI-CH2, we get this error: >>> >>> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >>> cd /pseudospace2/bushleyk/MAKER/Testrun; >>> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >>> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >>> '/local/cluster/MAKER_2.10/bin/mpi_maker' >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> /usr/bin/perl: symbol lookup error: >>> >>>/local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so >>>: >>> >>> undefined symbol: Perl_Tstack_sp_ptr >>> make: *** [maker] Error 127 >>> >>> which maybe means it's not finding the right perl? >>> >>> >>> When trying to run in LAM-MPI we get this error, was wondering what >>> the MAKER/EVALUATOR is? >>> >>> ERROR: Cannot start process. MAKER/EVALUATOR already running >>> with different settings in this same directory. >>> >>> >>> FATAL ERROR >>> running repeat masker. >>> #--------- command -------------# >>> Widget::RepeatMasker: >>> /local/cluster/spatafora/RepeatMasker/RepeatMasker >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> >>>tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp- >>>50 >>> 0-500.0.all.rb -species all -dir >>> >>>/pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_ >>>da >>> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >>> 1 >>> #-------------------------------# >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> It seems that [at least] one of the processes that was started with >>> mpirun did not invoke MPI_INIT before quitting (it is possible that >>> more than one process did not invoke MPI_INIT -- mpirun was only >>> notified of the first one, which was on node n0). >>> >>> mpirun can *only* be used with MPI programs (i.e., programs that >>> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >>> to run non-MPI programs over the lambooted nodes. >>> >>>------------------------------------------------------------------------ >>>-- >>> --- >>> Received SIGTERM >>> make: *** [maker_new] Error 252 >>> >>> >>> >>> >>> >>> >>> >>> >>> Best, >>> >>> Kathryn >>> >>> >>> "To err is human...to really foul up requires the root password." >>> >>> >>> Kathryn E. Bushley >>> Postdoctoral Researcher >>> 2082 Cordley Hall >>> Department of Botany and Plant Pathology >>> Oregon State University >>> Corvallis, OR 97331 >>> office: (541) 737-5284 >>> cell: (541) 908-0116 >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> > > From bushleyk at science.oregonstate.edu Fri Feb 15 10:10:23 2013 From: bushleyk at science.oregonstate.edu (Kathryn E. Bushley) Date: Fri, 15 Feb 2013 09:10:23 -0800 Subject: [maker-devel] maker MPI In-Reply-To: References: Message-ID: <20130215091023.21176sncxav25s84@webmail.oregonstate.edu> Hello Carson, Thanks. We currently have either LAMMPI or MPICH2 up and running on our cluster so are trying to make it work in either one of those. We're able to get it run in LAMMPI but get this error about the MAKER EVALUATOR already running, any idea what that might be? It did work previously in MPICH2 so maybe we need to go that route. [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/lamboot /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt LAM 7.1.4/MPI 2 C++/ROMIO - Indiana University [bushleyk at fungi0 Test]$ lamnodes n0 fungi0.cgrb.oregonstate.local:4:origin,this_node [bushleyk at fungi0 Test]$ ps -aef | grep lam bushleyk 11818 1 0 08:56 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 36112 -n 0 -o 0 bushleyk 11821 11262 0 08:56 pts/1 00:00:00 grep lam sullichr 13795 1 0 Feb14 ? 00:00:00 /usr/lib64/lam/bin/lamd -H 192.168.1.136 -P 57579 -n 0 -o 0 [bushleyk at fungi0 Test]$ /usr/lib64/lam/bin/mpirun -np 4 /local/cluster/maker/bin/mpi_maker ERROR: Cannot start process. MAKER/EVALUATOR already running with different settings in this same directory. FATAL ERROR ----------------------------------------------------------------------------- It seems that [at least] one of the processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- Received SIGTERM Received SIGTERM Best, Kathryn "To err is human...to really foul up requires the root password." Kathryn E. Bushley Postdoctoral Researcher 2082 Cordley Hall Department of Botany and Plant Pathology Oregon State University Corvallis, OR 97331 office: (541) 737-5284 cell: (541) 908-0116 Quoting Carson Holt : > LAMMPI is pretty much dead, even their website recommends using OpenMPI > instead. > > For installing MAKER with OpenMPI you need to set a couple of > environmental variables before MAKER setup. > > export OMPI_MCA_mpi_warn_on_fork 0 > export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD > > These not only need to be set before compilation, but also before any run > (so add them to you ~/.bashrc or ~/.bash_profile or any module load > scripts thanks). The LD_PRELOAD statement needs to be set for any program > using OpenMPI's shared libraries and not just MAKER, so it's normally a > good idea to have that set system wide for all users. The detail can be > Found in the OpenMPI documentation. Note sometimes system library updates > can break OpenMPI's shared libraries while not breaking OpenMPI itself, so > you might also need to recompile OpenMPI if it has broken shared libraries. > > Thanks, > Carson > > > On 13-02-04 6:48 PM, "Kathryn E. Bushley" > wrote: > >> Greetings, >> >> We are reinstalling maker after a system upgrade and having some >> issues with getting MPI version running. One question, will maker >> only run in MPI-CH2 or will it also run in LAM-MPI? When trying to >> run in MPI-CH2, we get this error: >> >> /local/cluster/mpich2-1.2.1p1/bin/mpdboot -f >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt >> cd /pseudospace2/bushleyk/MAKER/Testrun; >> /local/cluster/mpich2-1.2.1p1/bin/mpiexec -machinefile >> /home/bpp/bushleyk/MPI2/fungi0_machinefile.txt -n 4 >> '/local/cluster/MAKER_2.10/bin/mpi_maker' >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> /usr/bin/perl: symbol lookup error: >> /local/cluster/MAKER_2.10/bin/../perl/lib/auto/Parallel/MPIcar/MPIcar.so: >> >> undefined symbol: Perl_Tstack_sp_ptr >> make: *** [maker] Error 127 >> >> which maybe means it's not finding the right perl? >> >> >> When trying to run in LAM-MPI we get this error, was wondering what >> the MAKER/EVALUATOR is? >> >> ERROR: Cannot start process. MAKER/EVALUATOR already running >> with different settings in this same directory. >> >> >> FATAL ERROR >> running repeat masker. >> #--------- command -------------# >> Widget::RepeatMasker: >> /local/cluster/spatafora/RepeatMasker/RepeatMasker >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500/contig-dpp-50 >> 0-500.0.all.rb -species all -dir >> /pseudospace2/bushleyk/MAKER/Testrun/dpp_contig.maker.output/dpp_contig_da >> tastore/05/1F/contig-dpp-500-500//theVoid.contig-dpp-500-500 -pa >> 1 >> #-------------------------------# >> -------------------------------------------------------------------------- >> --- >> It seems that [at least] one of the processes that was started with >> mpirun did not invoke MPI_INIT before quitting (it is possible that >> more than one process did not invoke MPI_INIT -- mpirun was only >> notified of the first one, which was on node n0). >> >> mpirun can *only* be used with MPI programs (i.e., programs that >> invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program >> to run non-MPI programs over the lambooted nodes. >> -------------------------------------------------------------------------- >> --- >> Received SIGTERM >> make: *** [maker_new] Error 252 >> >> >> >> >> >> >> >> >> Best, >> >> Kathryn >> >> >> "To err is human...to really foul up requires the root password." >> >> >> Kathryn E. Bushley >> Postdoctoral Researcher >> 2082 Cordley Hall >> Department of Botany and Plant Pathology >> Oregon State University >> Corvallis, OR 97331 >> office: (541) 737-5284 >> cell: (541) 908-0116 >> >> >> >> >> >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > > From mikael.durling at slu.se Wed Feb 20 06:12:08 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Wed, 20 Feb 2013 13:12:08 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael From dence at genetics.utah.edu Wed Feb 20 09:29:21 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Wed, 20 Feb 2013 16:29:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. How much of the genome has maker completed when the processes start doing this? Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Wednesday, February 20, 2013 6:12 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] Running MAKER on highly fragmented assembly Hi, I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). cheers, Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Thu Feb 21 08:39:37 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Thu, 21 Feb 2013 15:39:37 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140344A0@exchange2-2> Message-ID: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Hi Daniel, the genomes I work with are in the order of 30-60 Mb. Other assemblies have been quick jobs for maker without any problems. If I run maker with -debugmpi, I get sets of debug printouts from the different ranks now and then: COMM INITIALIZATION | SEND | who_I_am | 3 --> 0 | 46731 COMM INITIALIZATION | SEND | what_I_want | 3 --> 0 | 46732 COMM INITIALIZATION | RECV | what_I_want | 0 <-- 3 | 312850 COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 --> 3 | 312851 HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 <-- 3 | 312852 COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 <-- 0 | 46733 HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 --> 0 | 46734 HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 --> 3 | 312853 HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 <-- 0 | 46735 COMM INITIALIZATION | RECV | who_I_am | 0 <-- ANY | 312854 and then they seem to stay waiting while a single rank continues to run the normal analysis. I have filtered the assembly for contigs shorter than then minimum length set in maker_opts.ctl. I did some strace:ing of the ranks that do nothing, and it seems they loop over running a subprocess that basically does a process listing. I might be completely off in my guesses of what the problem might be. I'm sort of afraid that I'm bitten by some NFS related problem as I have been quite a few times by know. I will soon try to reannotate a genome sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure that make behaves as expected with that genome. Mikael 20 feb 2013 kl. 17:29 skrev Daniel Ence : > Hi Mikael, Depending on the genome size, the assembly you've described shouldn't be too difficult to work with. The process activity that you're describing sounds more like a race condition, where one process is hogging all the work and all the other processes keep trying to find work, but keep getting in each others' way. > > How much of the genome has maker completed when the processes start doing this? > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Wednesday, February 20, 2013 6:12 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] Running MAKER on highly fragmented assembly > > Hi, > > I'm trying to run MAKER on a rather fragmented assembly. I know this is not optimal, as I will most likely miss a substantial part of the gene complement due to the fragmentation. Disregarding this, my question is if there are other problems with running maker on these kinds of genomes with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run with MPI (mpich2) behaves rather in a rather strange way, with basically one of the ranks staying at 100% cpu, and the others lingering at about 0%. Now and then I see a burst of activity in the other ranks before they get back to low activity. Could this be a result of the fragmentation level, or should I look for other problems? (Like the all to common problems of running over NFS with locking etc). > > cheers, > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Thu Feb 21 08:58:10 2013 From: carsonhh at gmail.com (Carson Holt) Date: Thu, 21 Feb 2013 10:58:10 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C614034DDC@exchange2-2> Message-ID: Which version of MAKER are you using? We recently deployed MAKER on the large NIH TACC computer cluster with some additional modifications for very large MPI jobs (> 1500 cpus). Some of the modifications focus on reannotating very large contigs as opposed to the small contig de novo annotation that MAKER already works well on. The modifications will be merged back into the MAKER downloadable release soon, but I could give you access to test with especially if you are running large MPI jobs or on very large multi-megabase contigs. --Carson On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" wrote: >Hi Daniel, >the genomes I work with are in the order of 30-60 Mb. Other assemblies >have been quick jobs for maker without any problems. > >If I run maker with -debugmpi, I get sets of debug printouts from the >different ranks now and then: > >COMM INITIALIZATION | SEND | who_I_am | 3 >--> 0 | 46731 >COMM INITIALIZATION | SEND | what_I_want | 3 >--> 0 | 46732 >COMM INITIALIZATION | RECV | what_I_want | 0 ><-- 3 | 312850 >COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >--> 3 | 312851 >HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 ><-- 3 | 312852 >COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 ><-- 0 | 46733 >HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >--> 0 | 46734 >HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >--> 3 | 312853 >HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 ><-- 0 | 46735 >COMM INITIALIZATION | RECV | who_I_am | 0 ><-- ANY | 312854 > >and then they seem to stay waiting while a single rank continues to run >the normal analysis. I have filtered the assembly for contigs shorter >than then minimum length set in maker_opts.ctl. > >I did some strace:ing of the ranks that do nothing, and it seems they >loop over running a subprocess that basically does a process listing. > >I might be completely off in my guesses of what the problem might be. I'm >sort of afraid that I'm bitten by some NFS related problem as I have been >quite a few times by know. I will soon try to reannotate a genome >sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >that make behaves as expected with that genome. >Mikael > > >20 feb 2013 kl. 17:29 skrev Daniel Ence : > >> Hi Mikael, Depending on the genome size, the assembly you've described >>shouldn't be too difficult to work with. The process activity that >>you're describing sounds more like a race condition, where one process >>is hogging all the work and all the other processes keep trying to find >>work, but keep getting in each others' way. >> >> How much of the genome has maker completed when the processes start >>doing this? >> >> Thanks, >> Daniel >> >> Daniel Ence >> Graduate Student >> Eccles Institute of Human Genetics >> University of Utah >> 15 North 2030 East, Room 2100 >> Salt Lake City, UT 84112-5330 >> ________________________________________ >> From: maker-devel-bounces at yandell-lab.org >>[maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>Durling [mikael.durling at slu.se] >> Sent: Wednesday, February 20, 2013 6:12 AM >> To: maker-devel at yandell-lab.org >> Subject: [maker-devel] Running MAKER on highly fragmented assembly >> >> Hi, >> >> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>not optimal, as I will most likely miss a substantial part of the gene >>complement due to the fragmentation. Disregarding this, my question is >>if there are other problems with running maker on these kinds of genomes >>with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>with MPI (mpich2) behaves rather in a rather strange way, with basically >>one of the ranks staying at 100% cpu, and the others lingering at about >>0%. Now and then I see a burst of activity in the other ranks before >>they get back to low activity. Could this be a result of the >>fragmentation level, or should I look for other problems? (Like the all >>to common problems of running over NFS with locking etc). >> >> cheers, >> Mikael >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 06:35:31 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:35:31 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C61403544E@exchange2-2> I have been trying both with 2.27 (r968) and svn r992. Both behave similarly on the fragmented NGS assemblies I try to annotate now. However, I just run a highly contiguous assembly (the one mentioned below with 32 Mbp in 15 scaffolds) that went through MAKER r968 without any problems at all. Based on this it seems the problem is not NFS related. Is there some other debug information I can provide? Nevertheless, I I have seen that the MAKER have seemed to be better at utilising cpu resources for the slightly more fragmented assemblies (1000 scaffolds or so with N50 in the 300-700 kb range) than this single sample of a highly contiguous assembly. Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From mikael.durling at slu.se Fri Feb 22 06:40:25 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 13:40:25 +0000 Subject: [maker-devel] failing contigs Message-ID: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael From dence at genetics.utah.edu Fri Feb 22 14:40:54 2013 From: dence at genetics.utah.edu (Daniel Ence) Date: Fri, 22 Feb 2013 21:40:54 +0000 Subject: [maker-devel] failing contigs In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2> Message-ID: Hi Mikael, Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. Thanks, Daniel Daniel Ence Graduate Student Eccles Institute of Human Genetics University of Utah 15 North 2030 East, Room 2100 Salt Lake City, UT 84112-5330 ________________________________________ From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] Sent: Friday, February 22, 2013 6:40 AM To: maker-devel at yandell-lab.org Subject: [maker-devel] failing contigs Hi, while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. [?] stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA doing blastx repeats ERROR: Failed while collecting blastn reports ERROR: Chunk failed at level:1, tier_type:3 FAILED CONTIG:scaffold_01 polishig ESTs ERROR: Chunk failed at level:4, tier_type:0 FAILED CONTIG:scaffold_01 If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. Mikael _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Fri Feb 22 15:21:16 2013 From: mikael.durling at slu.se (=?Windows-1252?Q?Mikael_Brandstr=F6m_Durling?=) Date: Fri, 22 Feb 2013 22:21:16 +0000 Subject: [maker-devel] failing contigs In-Reply-To: References: <35FD181EEB48324AB043FDB803E7D1C6140354B4@exchange2-2>, Message-ID: <16A5EBD9-1CAC-4D4C-9C6E-9BDAF6E79B24@slu.se> You are most likely right. When you mention that I remember that I stumbled across that particular problem in another analysis pipeline based on blast I wrote long ago. Interesting that such an arbitrary limit has not been lifted. Thanks for the hint. I will try mangling the fasta ids to something shorter. Mikael --- Sent from a crippled computer (a.k.a a phone) 22 feb 2013 kl. 22:40 skrev "Daniel Ence" : > Hi Mikael, > > Sometimes really long fasta ID's (like the one in your example) cause blast to fail. I'm not super familiar with how short they need to be, but I know I've had this issue in the past. If you can try shortening the IDs, and then retry maker, I think that will solve your problem. > > Thanks, > Daniel > > Daniel Ence > Graduate Student > Eccles Institute of Human Genetics > University of Utah > 15 North 2030 East, Room 2100 > Salt Lake City, UT 84112-5330 > ________________________________________ > From: maker-devel-bounces at yandell-lab.org [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m Durling [mikael.durling at slu.se] > Sent: Friday, February 22, 2013 6:40 AM > To: maker-devel at yandell-lab.org > Subject: [maker-devel] failing contigs > > Hi, > > while at trying to find why things don't work for me, I have encountered another problem. Sometimes contigs fail with these lines found in the logs: > > WARNING: Cannot find >0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA, trying to re-index the fasta. > [?] > stop here:0_0_CCPB1339.b1_CCOZ_CCPA_CCPB_CCPC_EXTA > doing blastx repeats > ERROR: Failed while collecting blastn reports > ERROR: Chunk failed at level:1, tier_type:3 > FAILED CONTIG:scaffold_01 > > polishig ESTs > ERROR: Chunk failed at level:4, tier_type:0 > FAILED CONTIG:scaffold_01 > > If I then grep for that specific EST in the fasta file, the sequence is there. Sometimes the issue is resolved by deleting the mpi_blastdb and restarting MAKER, sometimes not. > > Mikael > _______________________________________________ > maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From mikael.durling at slu.se Mon Feb 25 03:57:21 2013 From: mikael.durling at slu.se (=?iso-8859-1?Q?Mikael_Brandstr=F6m_Durling?=) Date: Mon, 25 Feb 2013 10:57:21 +0000 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: References: Message-ID: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Hi Carson and Daniel, It seems that this issue might be related to the other issue about maker not finding sequences in the protein and est fasta files due to too long sequence ids. For those annotation setups where I have truncated the ids to short unique names, it seems maker runs fine on all cpu:s allocated. While on the other sets, it seems maker ranks stall once they can't polish an est or protein alignment. (I have no firm evidence of this, as i havn't found a way to separate the logs from the different ranks.) Anyhow, now that I have truncated the names, I have not seen any messages of failed polishing (yet). Mikael 21 feb 2013 kl. 16:58 skrev Carson Holt : > Which version of MAKER are you using? We recently deployed MAKER on the > large NIH TACC computer cluster with some additional modifications for > very large MPI jobs (> 1500 cpus). Some of the modifications focus on > reannotating very large contigs as opposed to the small contig de novo > annotation that MAKER already works well on. The modifications will be > merged back into the MAKER downloadable release soon, but I could give you > access to test with especially if you are running large MPI jobs or on > very large multi-megabase contigs. > > --Carson > > > On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" > wrote: > >> Hi Daniel, >> the genomes I work with are in the order of 30-60 Mb. Other assemblies >> have been quick jobs for maker without any problems. >> >> If I run maker with -debugmpi, I get sets of debug printouts from the >> different ranks now and then: >> >> COMM INITIALIZATION | SEND | who_I_am | 3 >> --> 0 | 46731 >> COMM INITIALIZATION | SEND | what_I_want | 3 >> --> 0 | 46732 >> COMM INITIALIZATION | RECV | what_I_want | 0 >> <-- 3 | 312850 >> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >> --> 3 | 312851 >> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >> <-- 3 | 312852 >> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >> <-- 0 | 46733 >> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >> --> 0 | 46734 >> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >> --> 3 | 312853 >> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >> <-- 0 | 46735 >> COMM INITIALIZATION | RECV | who_I_am | 0 >> <-- ANY | 312854 >> >> and then they seem to stay waiting while a single rank continues to run >> the normal analysis. I have filtered the assembly for contigs shorter >> than then minimum length set in maker_opts.ctl. >> >> I did some strace:ing of the ranks that do nothing, and it seems they >> loop over running a subprocess that basically does a process listing. >> >> I might be completely off in my guesses of what the problem might be. I'm >> sort of afraid that I'm bitten by some NFS related problem as I have been >> quite a few times by know. I will soon try to reannotate a genome >> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make sure >> that make behaves as expected with that genome. >> Mikael >> >> >> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >> >>> Hi Mikael, Depending on the genome size, the assembly you've described >>> shouldn't be too difficult to work with. The process activity that >>> you're describing sounds more like a race condition, where one process >>> is hogging all the work and all the other processes keep trying to find >>> work, but keep getting in each others' way. >>> >>> How much of the genome has maker completed when the processes start >>> doing this? >>> >>> Thanks, >>> Daniel >>> >>> Daniel Ence >>> Graduate Student >>> Eccles Institute of Human Genetics >>> University of Utah >>> 15 North 2030 East, Room 2100 >>> Salt Lake City, UT 84112-5330 >>> ________________________________________ >>> From: maker-devel-bounces at yandell-lab.org >>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>> Durling [mikael.durling at slu.se] >>> Sent: Wednesday, February 20, 2013 6:12 AM >>> To: maker-devel at yandell-lab.org >>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>> >>> Hi, >>> >>> I'm trying to run MAKER on a rather fragmented assembly. I know this is >>> not optimal, as I will most likely miss a substantial part of the gene >>> complement due to the fragmentation. Disregarding this, my question is >>> if there are other problems with running maker on these kinds of genomes >>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, run >>> with MPI (mpich2) behaves rather in a rather strange way, with basically >>> one of the ranks staying at 100% cpu, and the others lingering at about >>> 0%. Now and then I see a burst of activity in the other ranks before >>> they get back to low activity. Could this be a result of the >>> fragmentation level, or should I look for other problems? (Like the all >>> to common problems of running over NFS with locking etc). >>> >>> cheers, >>> Mikael >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> >> _______________________________________________ >> maker-devel mailing list >> maker-devel at box290.bluehost.com >> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > > From carsonhh at gmail.com Mon Feb 25 08:05:34 2013 From: carsonhh at gmail.com (Carson Holt) Date: Mon, 25 Feb 2013 10:05:34 -0500 Subject: [maker-devel] Running MAKER on highly fragmented assembly In-Reply-To: <35FD181EEB48324AB043FDB803E7D1C617CF0FB7@exchange2-2> Message-ID: Good to know it is working so far. I know that long names can also wrap around in output files causing the BioPerl parser to not splice the name pieces back together (so you get truncated names). For splitting the reports, I think there is a way to get MPI itself to tell you who each message originated from (--tag-output in OpenMPI). Alternatively you could try hacking the SUB{__WARN__} and SUB{__DIE__} entries at the top of the maker script to print the ranks out before all warning or failures. --Carson On 13-02-25 5:57 AM, "Mikael Brandstr?m Durling" wrote: >Hi Carson and Daniel, > >It seems that this issue might be related to the other issue about maker >not finding sequences in the protein and est fasta files due to too long >sequence ids. For those annotation setups where I have truncated the ids >to short unique names, it seems maker runs fine on all cpu:s allocated. >While on the other sets, it seems maker ranks stall once they can't >polish an est or protein alignment. (I have no firm evidence of this, as >i havn't found a way to separate the logs from the different ranks.) > >Anyhow, now that I have truncated the names, I have not seen any messages >of failed polishing (yet). > >Mikael > > > >21 feb 2013 kl. 16:58 skrev Carson Holt : > >> Which version of MAKER are you using? We recently deployed MAKER on the >> large NIH TACC computer cluster with some additional modifications for >> very large MPI jobs (> 1500 cpus). Some of the modifications focus on >> reannotating very large contigs as opposed to the small contig de novo >> annotation that MAKER already works well on. The modifications will be >> merged back into the MAKER downloadable release soon, but I could give >>you >> access to test with especially if you are running large MPI jobs or on >> very large multi-megabase contigs. >> >> --Carson >> >> >> On 13-02-21 10:39 AM, "Mikael Brandstr?m Durling" >> >> wrote: >> >>> Hi Daniel, >>> the genomes I work with are in the order of 30-60 Mb. Other assemblies >>> have been quick jobs for maker without any problems. >>> >>> If I run maker with -debugmpi, I get sets of debug printouts from the >>> different ranks now and then: >>> >>> COMM INITIALIZATION | SEND | who_I_am | 3 >>> --> 0 | 46731 >>> COMM INITIALIZATION | SEND | what_I_want | 3 >>> --> 0 | 46732 >>> COMM INITIALIZATION | RECV | what_I_want | 0 >>> <-- 3 | 312850 >>> COMM HAVE C_RESULT | SEND | c_res_status (no_c_res) | 0 >>> --> 3 | 312851 >>> HELPER/RESULT REQUESTED | RECV | work_order (num_helpers_req) | 0 >>> <-- 3 | 312852 >>> COMM HAVE C_RESULT | RECV | c_res_status (is c_res?) | 3 >>> <-- 0 | 46733 >>> HELPER/RESULT REQUESTED | SEND | work_order (num_helpers_req) | 3 >>> --> 0 | 46734 >>> HELPER/RESULT REQUESTED | SEND | req_stat (no_helpers_avail) | 0 >>> --> 3 | 312853 >>> HELPER/RESULT REQUESTED | RECV | req_stat (is helper avail?) | 3 >>> <-- 0 | 46735 >>> COMM INITIALIZATION | RECV | who_I_am | 0 >>> <-- ANY | 312854 >>> >>> and then they seem to stay waiting while a single rank continues to run >>> the normal analysis. I have filtered the assembly for contigs shorter >>> than then minimum length set in maker_opts.ctl. >>> >>> I did some strace:ing of the ranks that do nothing, and it seems they >>> loop over running a subprocess that basically does a process listing. >>> >>> I might be completely off in my guesses of what the problem might be. >>>I'm >>> sort of afraid that I'm bitten by some NFS related problem as I have >>>been >>> quite a few times by know. I will soon try to reannotate a genome >>> sequenced by the JGI where we have 35Mb in 15 scaffolds just to make >>>sure >>> that make behaves as expected with that genome. >>> Mikael >>> >>> >>> 20 feb 2013 kl. 17:29 skrev Daniel Ence : >>> >>>> Hi Mikael, Depending on the genome size, the assembly you've described >>>> shouldn't be too difficult to work with. The process activity that >>>> you're describing sounds more like a race condition, where one process >>>> is hogging all the work and all the other processes keep trying to >>>>find >>>> work, but keep getting in each others' way. >>>> >>>> How much of the genome has maker completed when the processes start >>>> doing this? >>>> >>>> Thanks, >>>> Daniel >>>> >>>> Daniel Ence >>>> Graduate Student >>>> Eccles Institute of Human Genetics >>>> University of Utah >>>> 15 North 2030 East, Room 2100 >>>> Salt Lake City, UT 84112-5330 >>>> ________________________________________ >>>> From: maker-devel-bounces at yandell-lab.org >>>> [maker-devel-bounces at yandell-lab.org] on behalf of Mikael Brandstr?m >>>> Durling [mikael.durling at slu.se] >>>> Sent: Wednesday, February 20, 2013 6:12 AM >>>> To: maker-devel at yandell-lab.org >>>> Subject: [maker-devel] Running MAKER on highly fragmented assembly >>>> >>>> Hi, >>>> >>>> I'm trying to run MAKER on a rather fragmented assembly. I know this >>>>is >>>> not optimal, as I will most likely miss a substantial part of the gene >>>> complement due to the fragmentation. Disregarding this, my question is >>>> if there are other problems with running maker on these kinds of >>>>genomes >>>> with roughly 1500 scaffolds and an N50 of 60 kb? I find that maker, >>>>run >>>> with MPI (mpich2) behaves rather in a rather strange way, with >>>>basically >>>> one of the ranks staying at 100% cpu, and the others lingering at >>>>about >>>> 0%. Now and then I see a burst of activity in the other ranks before >>>> they get back to low activity. Could this be a result of the >>>> fragmentation level, or should I look for other problems? (Like the >>>>all >>>> to common problems of running over NFS with locking etc). >>>> >>>> cheers, >>>> Mikael >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>>> >>>> _______________________________________________ >>>> maker-devel mailing list >>>> maker-devel at box290.bluehost.com >>>> >>>>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >>> >>> >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org >> >> > From jokelley at stanford.edu Tue Feb 26 15:11:26 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 14:11:26 -0800 Subject: [maker-devel] perl Message-ID: Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 18:47:23 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 20:47:23 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: The shebang line (#!) should use whatever perl was used in the "perl Build.PL" step of the configuration. Then "Build install" will overwrite the existing scripts with the ones using a different perl. --Carson From: Joanna Kelley Date: Tuesday, 26 February, 2013 5:11 PM To: Subject: [maker-devel] perl Hello, Is there an easy way to change the hard coding of #!/usr/bin/perl to another perl location in the configuration / installation of maker? Thanks, Joanna -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlee.nakasugi at sydney.edu.au Tue Feb 26 21:09:34 2013 From: kenlee.nakasugi at sydney.edu.au (Kenlee Nakasugi) Date: Wed, 27 Feb 2013 15:09:34 +1100 Subject: [maker-devel] resuming maker Message-ID: <1361938174.2353.28.camel@waterhouse874-8> Hi, When resuming maker on the same dataset, is the 'theVoid' folder and all its contents required to allow it to start out where it stopped? i.e. is this what is used to let maker know that the scaffold has already be calculated? thanks, Ken -- Kenlee Nakasugi | Research Fellow School of Molecular Bioscience Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 T: +61 2 9114 1321 E: kenlee.nakasugi at sydney.edu.au From carsonhh at gmail.com Tue Feb 26 21:10:55 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:10:55 -0500 Subject: [maker-devel] resuming maker In-Reply-To: <1361938174.2353.28.camel@waterhouse874-8> Message-ID: Yes. theVoid directory contains raw analysis reports and log files. --Carson On 13-02-26 11:09 PM, "Kenlee Nakasugi" wrote: >Hi, > >When resuming maker on the same dataset, is the 'theVoid' folder and all >its contents required to allow it to start out where it stopped? >i.e. is this what is used to let maker know that the scaffold has >already be calculated? > >thanks, >Ken > >-- >Kenlee Nakasugi | Research Fellow >School of Molecular Bioscience >Level 8, SMB Building (G08)| The University of Sydney | NSW | 2006 >T: +61 2 9114 1321 >E: kenlee.nakasugi at sydney.edu.au > > >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From ranjani at uga.edu Tue Feb 26 21:22:43 2013 From: ranjani at uga.edu (Sivaranjani Namasivayam) Date: Wed, 27 Feb 2013 04:22:43 +0000 Subject: [maker-devel] MAKER prediction- evidence weightage Message-ID: Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 21:35:28 2013 From: carsonhh at gmail.com (Carson Holt) Date: Tue, 26 Feb 2013 23:35:28 -0500 Subject: [maker-devel] MAKER prediction- evidence weightage In-Reply-To: Message-ID: Not so much "weight" as much as different evidence types help infer different parts of a gene model. ESTs can be used to infer introns, UTR, and splice sites. Proteins help infer exons and CDS. They become "hints" to the gene predictors for the location of these sub-features. Then if ESTs have matching exons that extend beyond the new model produced by SNAP or Augustus using the hints, then they can be added on directly as UTR. That error can happen in the older version 2.10, but should not happen with 2.27. To get rid of it you have to edit several lines of code in maker and BioPerl to remove every mention of SDBM_File. It really doesn't effect anything, as the module in question ends up using DB_File regardless. Thanks, Carson From: Sivaranjani Namasivayam Date: Tuesday, 26 February, 2013 11:22 PM To: "maker-devel at yandell-lab.org" Subject: [maker-devel] MAKER prediction- evidence weightage Hi, Does MAKER give more weightage to certain type of evidence than others, that is, does it base its annotation more on EST, protein or ab-initio gene prediction? If so which one and is there a way to alter this. Also, when I run MAKER I see the following in the error/log file Can't locate package SDBM_File for @AnyDBM_File::ISA at /usr/local/perl/5.14.1/l ib/5.14.1/x86_64-linux-thread-multi/DB_File.pm line 293. Can you please tell me how this impacts the results. Thanks, Ranjani _______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jokelley at stanford.edu Tue Feb 26 22:53:52 2013 From: jokelley at stanford.edu (Joanna Kelley) Date: Tue, 26 Feb 2013 21:53:52 -0800 Subject: [maker-devel] perl In-Reply-To: References: Message-ID: Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl > Build.PL" step of the configuration. Then "Build install" will overwrite > the existing scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.com > http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonhh at gmail.com Tue Feb 26 23:14:48 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 01:14:48 -0500 Subject: [maker-devel] perl In-Reply-To: Message-ID: Did you run './Build install' after running '/opt/perl/bin/perl Build.PL ' to overwrite the old scripts? You may have to delete the /maker/bin directory before running './Build install' to guarantee that they are overwritten. PS. I'm glad the shebang line could brighten your day. Thanks, Carson From: Joanna Kelley Date: Wednesday, 27 February, 2013 12:53 AM To: Carson Holt Cc: Subject: Re: [maker-devel] perl Hi Carson, First, I just learned the term shebang line, that's awesome! Second, what you said isn't the case, because I ran; [jokelley at scg3-ln01 src ]$ /opt/perl/bin/perl Build.PL [jokelley at scg3-ln01 src ]$ grep "/usr/bin/perl" */* ... bin/maker:#!/usr/bin/perl bin/maker2chado:#!/usr/bin/perl bin/maker2eval_gtf:#!/usr/bin/perl bin/maker2jbrowse:#!/usr/bin/perl ... So it's not using the /opt/perl/bin/perl any other ideas for changing that? Because it's not working. On Tue, Feb 26, 2013 at 5:47 PM, Carson Holt wrote: > The shebang line (#!) should use whatever perl was used in the "perl Build.PL" > step of the configuration. Then "Build install" will overwrite the existing > scripts with the ones using a different perl. > > --Carson > > > From: Joanna Kelley > Date: Tuesday, 26 February, 2013 5:11 PM > To: > Subject: [maker-devel] perl > > Hello, > > Is there an easy way to change the > hard coding of #!/usr/bin/perl > to another perl location in the > configuration / installation of maker? > > Thanks, > Joanna > > > > -- > Joanna L. Kelley, PhD > Department of Genetics > Stanford School of Medicine > 300 Pasteur Drive > Lane Building, Room L-333 > Stanford, CA 94305-5120 > jokelley at stanford.edu > > _______________________________________________ maker-devel mailing list > maker-devel at box290.bluehost.comhttp://box290.bluehost.com/mailman/listinfo/mak > er-devel_yandell-lab.org -- Joanna L. Kelley, PhD Department of Genetics Stanford School of Medicine 300 Pasteur Drive Lane Building, Room L-333 Stanford, CA 94305-5120 jokelley at stanford.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjfields at illinois.edu Wed Feb 27 14:05:47 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Wed, 27 Feb 2013 21:05:47 +0000 Subject: [maker-devel] Preformatted BLAST databases? Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> All, We have a number of pre-formatted BLAST databases set up locally (taxonomically separated, for example) that we would like to use for annotation of specific assemblies. We would also really like to save space where possible and not copy auto-generated BLAST databases where possible. These don't currently seem to be supported however (at least, when I attempt to use them via the config they don't work). Is this possible via MAKER currently, or are there any plans to support this in the future? chris From carsonhh at gmail.com Wed Feb 27 14:15:33 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:15:33 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: <118F034CF4C3EF48A96F86CE585B94BF6E705886@CHIMBX5.ad.uillinois.edu> Message-ID: MAKER has to make separate databases (min 10) for each input fasta, so that it can split the work up more efficiently (not just one big database per fasta). For space savinf purposes, you could set a different location for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be shared across multiple MAKER instances and not be regenerated constantly. --Carson On 13-02-27 4:05 PM, "Fields, Christopher J" wrote: >All, > >We have a number of pre-formatted BLAST databases set up locally >(taxonomically separated, for example) that we would like to use for >annotation of specific assemblies. We would also really like to save >space where possible and not copy auto-generated BLAST databases where >possible. > >These don't currently seem to be supported however (at least, when I >attempt to use them via the config they don't work). Is this possible >via MAKER currently, or are there any plans to support this in the >future? > >chris >_______________________________________________ >maker-devel mailing list >maker-devel at box290.bluehost.com >http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From carsonhh at gmail.com Wed Feb 27 14:17:13 2013 From: carsonhh at gmail.com (Carson Holt) Date: Wed, 27 Feb 2013 16:17:13 -0500 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: Message-ID: FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not already there. --Carson On 13-02-27 4:15 PM, "Carson Holt" wrote: >MAKER has to make separate databases (min 10) for each input fasta, so >that it can split the work up more efficiently (not just one big database >per fasta). For space savinf purposes, you could set a different >location >for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >shared across multiple MAKER instances and not be regenerated constantly. > >--Carson > > >On 13-02-27 4:05 PM, "Fields, Christopher J" >wrote: > >>All, >> >>We have a number of pre-formatted BLAST databases set up locally >>(taxonomically separated, for example) that we would like to use for >>annotation of specific assemblies. We would also really like to save >>space where possible and not copy auto-generated BLAST databases where >>possible. >> >>These don't currently seem to be supported however (at least, when I >>attempt to use them via the config they don't work). Is this possible >>via MAKER currently, or are there any plans to support this in the >>future? >> >>chris >>_______________________________________________ >>maker-devel mailing list >>maker-devel at box290.bluehost.com >>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org From cjfields at illinois.edu Wed Feb 27 23:18:20 2013 From: cjfields at illinois.edu (Fields, Christopher J) Date: Thu, 28 Feb 2013 06:18:20 +0000 Subject: [maker-devel] Preformatted BLAST databases? In-Reply-To: References: Message-ID: <118F034CF4C3EF48A96F86CE585B94BF6E7062FF@CHIMBX5.ad.uillinois.edu> Thanks Carson, that should work. chris On Feb 27, 2013, at 3:17 PM, Carson Holt wrote: > FYI. Just add mpi_blastdb=/some/path to the maker_opts.ctl file if not > already there. > > --Carson > > > On 13-02-27 4:15 PM, "Carson Holt" wrote: > >> MAKER has to make separate databases (min 10) for each input fasta, so >> that it can split the work up more efficiently (not just one big database >> per fasta). For space savinf purposes, you could set a different >> location >> for the mpi_blastdb by adding mpi_blastdb=/some/path. Then it would be >> shared across multiple MAKER instances and not be regenerated constantly. >> >> --Carson >> >> >> On 13-02-27 4:05 PM, "Fields, Christopher J" >> wrote: >> >>> All, >>> >>> We have a number of pre-formatted BLAST databases set up locally >>> (taxonomically separated, for example) that we would like to use for >>> annotation of specific assemblies. We would also really like to save >>> space where possible and not copy auto-generated BLAST databases where >>> possible. >>> >>> These don't currently seem to be supported however (at least, when I >>> attempt to use them via the config they don't work). Is this possible >>> via MAKER currently, or are there any plans to support this in the >>> future? >>> >>> chris >>> _______________________________________________ >>> maker-devel mailing list >>> maker-devel at box290.bluehost.com >>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org > >