<HTML>
<HEAD>
<TITLE>Re: [maker-devel] Status check?</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Thanks. I’m downloading the files now. <BR>
<BR>
Carson<BR>
<BR>
<BR>
On 10/22/09 11:05 AM, "Xavier Watkins" <<a href="xavier@flymine.org">xavier@flymine.org</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>You should have got an email with the dropbox url.<BR>
Unfortunately was not able to get wublast before the change to AB-Blast.<BR>
<BR>
Thanks,<BR>
Xavier<BR>
<BR>
On 22 Oct 2009, at 17:53, Carson Holt wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'> I think it would still be useful. Since that one change seemed to make all the difference, I’ll run some tests comparing cross_match and wublast under different configurations on our system.<BR>
<BR>
Thanks,<BR>
Carson<BR>
<BR>
<BR>
<BR>
On 10/22/09 10:50 AM, "Xavier Watkins" <<a href="xavier@flymine.org">xavier@flymine.org</a>> wrote:<BR>
<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Switching to "Drosophila" made all the difference, it's now whizzing through the analysis! Did you still want me to upload the files to my dropbox?<BR>
<BR>
Many thanks,<BR>
Xavier<BR>
<BR>
On 22 Oct 2009, at 17:45, Carson Holt wrote:<BR>
<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'> I’m also surprised you said that RepeatMasker is taking so long. I’m wondering if this is related to cross_match as wublast always seems to scream through this step. The suggestion to switch from ‘all’ to ‘Drosophila’ seems like a good start, but I’d like to check some other things as well. Is there anyway you could provide me with your datasets and maker control files? You could try an online storage site like <a href="http://www.getdropbox.com/">http://www.getdropbox.com/</a> which gives you 2 Gb of free storage online. They also let you share folders. You could then send me a link to the shared folder.<BR>
<BR>
Thanks,<BR>
Carson<BR>
<BR>
<BR>
On 10/22/09 5:38 AM, "Xavier Watkins" <<a href="xavier@flymine.org">xavier@flymine.org</a>> wrote:<BR>
<BR>
<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi,<BR>
Thanks everyone for your help. I'm already running mpi_maker, planning <BR>
on using more CPUs next time...<BR>
<BR>
I'm using the following:<BR>
<BR>
RepeatMasker with cross_match (running cross match seems to be the bit <BR>
that's taking ages)<BR>
<BR>
snap version 2006-07-28<BR>
GeneMarkS<BR>
NCBI blastall 2.2.20 for blastx<BR>
<BR>
Not really sure what happens in RepeatMasker but from what I <BR>
understand it tries to blast the genome against itself to build a <BR>
library of possible repeats? Is there a way of building this library <BR>
of repeats in a more efficient way?<BR>
<BR>
Many thanks,<BR>
Xavier<BR>
<BR>
<BR>
On 21 Oct 2009, at 16:07, Mark Yandell wrote:<BR>
<BR>
><BR>
> Hi Xavier,<BR>
><BR>
> I agree: This seems way to long. I can basically reproduce flybase's <BR>
> annotations and blast data in about 3 days on my laptop-- so 2 weeks <BR>
> on 5 processor's seems way too long.<BR>
><BR>
> Is there some. special, really huge dataset you are running? Are you <BR>
> doing TBLASTX to align hits from a large database of sequences?<BR>
><BR>
> --mark<BR>
><BR>
> Mark Yandell<BR>
> Associate Professor of Human Genetics<BR>
> Eccles Institute of Human Genetics<BR>
> University of Utah<BR>
> 15 North 2030 East, Room 2100<BR>
> Salt Lake City, UT 84112-5330<BR>
> ph:801-587-7707<BR>
> ________________________________________<BR>
> From: <a href="maker-devel-bounces@yandell-lab.org">maker-devel-bounces@yandell-lab.org</a> [<a href="maker-devel-bounces@yandell-lab.org">maker-devel-bounces@yandell-lab.org</a><BR>
> ] On Behalf Of Carson Holt<BR>
> Sent: Wednesday, October 21, 2009 8:57 AM<BR>
> To: Xavier Watkins; <a href="maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><BR>
> Subject: Re: [maker-devel] Status check?<BR>
><BR>
> The time spent depends primarily on the size of the protein, EST, <BR>
> and repeat protein databases provided. BLAST actually makes up <BR>
> about 90% of the run time for MAKER. If your using 5 processors, I <BR>
> suggest using mpi_maker instead of regular maker. It gets better <BR>
> performance on multiprocessor systems. The number of slices is <BR>
> dependant on what you set max_dna_len to be in the maker_opt.ctl <BR>
> file. Increasing the number increases memory usage. Just divide <BR>
> the contig length by that number. Doing a test run on the entire <BR>
> Drosophila genome could take a while especially if you used large <BR>
> protein and EST databases for the analysis. It is 120 Megabases in <BR>
> size, and with the default max_dna_len of 100,000, it would be <BR>
> divided into 1,200 chunks. It could take anywhere from 4 days to 3 <BR>
> weeks depending on the BLAST databases used.<BR>
><BR>
> I guess Barry already answered the question on how to check on run <BR>
> status. Individual contigs also create a file called run.log. <BR>
> These will be under theVoid directory for each individual contig in <BR>
> the MAKER datastore directory. These files also contain entries <BR>
> with labels like STARTED and FINISHED for each individual analysis. <BR>
> The master_datastore_index.log file has status tags for entire <BR>
> contigs as apposed to individual analyses.<BR>
><BR>
> I hope that helps. Let us know how it goes.<BR>
><BR>
> Thanks,<BR>
> Carson<BR>
><BR>
><BR>
> On 10/21/09 3:21 AM, "Xavier Watkins" <<a href="xavier@flymine.org">xavier@flymine.org</a>> wrote:<BR>
><BR>
> Hi,<BR>
> I'm currently doing a test run of Maker on the D. mel genome and I <BR>
> would like to estimate the time it takes to run on our system (it <BR>
> has now been running for 2 weeks on 5 processors).<BR>
> Is there a way to know how many processes are left to run when <BR>
> running MAKER, or to know which contigs (chromosomes in my case) <BR>
> have finished running? From what I see it chops up the contigs into <BR>
> slices when running RepeatMasker (currently on .151) is there a way <BR>
> to know the total number of slices?<BR>
><BR>
> Apologies if I've missed this info in the documentation, I couldn't <BR>
> find it.<BR>
><BR>
> All the best,<BR>
> Xavier<BR>
><BR>
><BR>
><BR>
<BR>
<BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'> <BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'> <BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>