[maker-devel] Fwd: about running MAKER

[Carson Holt]

If the log is growing then it is still working.  The time it takes is a function of the evidence dataset, genome size, and gene density.  Is there any way, you could try starting a second maker process in the same directory with additional CPUs to get it to complete sooner (each process will detect each other and avoid overlapping on their work, so it will still get divided up efficiently).  Also if you combine the length of all finished contigs, you will get a better idea of how much run time is remaining (i.e. See if contigs remaining are short or long compared to most contigs in the assembly).

Thanks,
Carson


From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 23, 2013 1:12 PM
To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Thanks Carson,

gephy.transcripts.fa is 64406072
geph_AS_masked.fa is 296693827

The log is growing. It finished 21700 scaffolds. I think we have nearly 38149 scaffolds. Is that means the program running good?

Sincerely,
Yunxi



2013/7/22 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
Nothing looks out of the ordinary.  How big is this file --> gephy.transcripts.fa
How big is this file --> geph_AS_masked.fa

Also is the STDERR log growing or is it frozen?

--Carson


From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Monday, July 22, 2013 12:24 PM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

Sorry to bother you again. I ran the MAKER for like around two weeks, but it is still running now. I attached my control files. Can you help me to see if I set something wrong? I used the masked genome as my input, so I skipped the repeatmasker in maker. I think it should be faster. I ran this mpiexec -n 8 maker

I'm running another maker using unmasked data in 16 threads, so I ran this in 8. I already got 34G data for this. Because the time cost in running already out of schedule. I'm worry about something wrong in the setting.

Thank you.
Yunxi


2013/7/14 Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Thank you Carson.


2013/7/14 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
I would use the same data to train both.

More information here --> https://groups.google.com/forum/#!msg/maker-devel/Tnkc5RCkgQQ/Qtykuh9hVc8J

--Carson

From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Friday, July 12, 2013 9:04 PM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

One more is when I train the SNAP and Augustus, should I do them in the same time with the est2genome and protein2genome, or do them separately? Thank you.


2013/7/12 Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Hi Carson

The MAKER is running fine right now. I'm confusing about using the CEGMA with MAKER to train the SNAP and Augustus.

I'm running the MAKER from est2genome with protein2genome without using the result from CEGMA. I think the data I already enough for generating the hmm.

And I got the zff from the result of CEGMA, and I followed the link last time you gave me to generate the file for SNAP and Augustus.

I'm not that clear how to combine the hmm model from CEGMA and MAKER.

Thank you for your help.

Sincerely,
Yunxi


2013/7/2 Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Thank you Carson. I think I should better to use the default, in case it occurs an error after several days' running.


2013/7/2 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
Some operations are not NFS safe (NFS behaves different than normal file systems).  So maker performs those operations in the temporary directory.  If you redirect those files to an NFS location, you may get errors (basically MAKER just dies).  It won't necessarily happen, but the warning lets you know that, if it fails then you should set TMP= somewhere else before trying again.

--Carson


From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 2, 2013 3:22 PM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

I got an error message about this:
ERROR: Temporary directory set to an NFS location.
TMP=/home/lin11/tmp
The temporary directory in MAKER is specifically for
operations that are not NFS-safe. You must set TMP
to a locally mounted directory such as /tmp or add
--ignore_nfs_tmp to the maker command line to
override this error message.

Does the MAKER need a big temporary file? Because our system temporary directory may not have that much space. If it does not need that big tmp file, I can set it back to default. Or I just ignore the error message?

Thank you.


2013/7/2 Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Thank you.


2013/7/2 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
Anything from maker under the ~/perl5/ should be deleted.  MAKER should be self contained in its own directory.  Do 'rm ~/perl5/bin/*maker*'.  If you see anything else that is maker related there delete it as well.

Example:
.../maker/bin/
.../maker/lib/
.../maker/exe/

--Carson


From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 2, 2013 1:40 PM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

I installed the new version of MAKER. But right now have a problem of running it. Because I was missing perl dependencies, I installed the perl local library. And then I tried to run maker, it will automatically go to the one in local library called ~/perl5/bin/maker. Do you know how to get rid of this path? I found this path in .bashrc : export PATH="/home/lin11/perl5/bin:$PATH". If that ok to delete it?

Thank you.


2013/7/2 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
It's been stable enough that I could remove the beta tag (no major updates in 2 months).  In fact I'll go ahead and do that.

--Carson



From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 2, 2013 12:21 PM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

Is that the 2.28 only have the beta version?


2013/7/2 Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Thank you Carson. I'll try then let you know.


2013/7/2 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
You could just install it in a new directory (I.e. maker_2.28 rather than maker).  The 2.1 version more than 2 years old (lots of changes and improvements since then).

--Carson


From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 2, 2013 12:11 PM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson
I think I may need to update MAKER too. My version is 2.1. Is the new version of MAKER changed a lot? Because someone running MAKER on the server now, can I just update it instead of delete the old one then reinstall the new version?


2013/7/2 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
If you are using MAKER 2.28, and let it do the install for you it will install mpich2 from this location --> http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.5/mpich2-1.5.tar.gz

The installation will be in the .../maker/exe/mpich2 directory.

Alternatively, if you do the installation yourself, all you have to do set a few flags during install configuration.
--> ./configure --prefix=$path --enable-shared
Set $path to the location where you want MPICH2 to be installed.
Then run 'make install'

Thanks,
Carson



From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 2, 2013 11:59 AM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

I think it installed the 1.3.1 version. Can I update it?


2013/7/2 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
It should be installed under .../maker/exe/mpich2.  Having it under ~/perl5/exe/mpich2 is odd as maker should do that.

Type 'which mpiexec' to see if there is another mpiexec installed somewhere.  Also run 'mpiexec --version'.  What does it give?  I ask because the error message you got before is the same as a bug in an old version of MPICH2 (version 1.3).  MAKER will install version 1.5.

Also run 'mpiexec -n 16 hostname'
What happens?

Thanks,
Carson


From: csusm <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, July 2, 2013 11:17 AM

To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

How's the trip?

I did it followed your instruction. I ran ./build mpich2 first
It installed it into ~/perl5/exe/mpich2 directory.
And then i go back to maker/src ran perl build.pl<http://build.pl> and type Y to accept the MPI.
Then I ran ./build install
After that, I went to ~/.bash_profile to add the ~/perl5/exe/mpich2/bin into the PATH.
I used mpiexec -n 16 maker as the command.
Do I need to set some special variable in that three control file first?

Thank you.

Yunxi Lin

On Jul 2, 2013, at 7:44 AM, Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>> wrote:

Sorry for not responding sooner (it was a long holiday weekend in Canada).  Did you have maker install MPICH2 or did you install it yourself?  Also what is the exact command line you used to launch?

--Carson


From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Sunday, June 30, 2013 2:09 PM
To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

I got the errors like:
[mpiexec at bioinfo1] HYDU_create_process (./utils/launch/launch.c:69): execvp error on file hydra_pmi_proxy (No such file or directory)
[mpiexec at bioinfo1] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:315): assert (!closed) failed
[mpiexec at bioinfo1] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:76): callback returned error status
[mpiexec at bioinfo1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:342): error waiting for event
[mpiexec at bioinfo1] main (./ui/mpich/mpiexec.c:302): process manager error waiting for completion

Is that means our server only has one cpu so I cannot use mpi?

Thank you.



2013/6/25 Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>>
You can get blast to use more than 1 cpu via the cpus= option, but that is still significantly limiting MAKER's performance.

When you let MAKER install MPICH2, it will be local to the MAKER installation (MAKER only).  It will be in …/maker/exe/mpich2.  This was purposely done for people who have limited access and install MAKER themselves, so they can run via MPI without having to get upgraded privileges.  So I don't know if you installed MAKER yourself, but if you did, then this is an option that will let you run.

--Carson


From: csusm <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Tuesday, 25 June, 2013 11:40 AM
To: Carson Holt <carson.holt at oicr.on.ca<mailto:carson.holt at oicr.on.ca>>

Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

Thank you for your suggestion. Do you mean if I dont use MPI, i could only run it on one cpu?  Because my school own the server, I only have the limit authorization.

Yunxi Lin

On Jun 24, 2013, at 5:39 PM, Carson Holt <Carson.Holt at oicr.on.ca<mailto:Carson.Holt at oicr.on.ca>> wrote:

You are most likely only getting 1 cpu of performance.

You should just install MPICH2.  It's easy just to let MAKER do it for you:
Go to the …/maker/src/ directory
Run './Build mpich2'
Once it finishes installing, it will be in the …/maker/exe/mpich2/bin/ directory.

Setup MAKER again to use MPICH2:
Go to the …/maker/src/ directory
Run 'perl Build.PL'
Say yes to the "use MPI": question
Run './Build install'

Now run MAKER via 'mpiexec'.
Example --> …/maker/exe/mpich2/bin/mpiexec -n 16 maker

The –n flag specifies how many CPUS to use. Mpiexec handles process communication either on the same machine or across machines.  You will get much better performance.

Thanks,
Carson



From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Monday, 24 June, 2013 7:11 PM
To: Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Cc: Amelia Ireland <amelia.ireland at gmod.org<mailto:amelia.ireland at gmod.org>>, <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Subject: Re: [maker-devel] Fwd: about running MAKER

Hi Carson

Thank your for your help.

My genome estimated size is 250M base pairs. I ran it in 16cpu, but we don't have the MPI so I cannot use it. I don't think I'm using the alt_est option. I was following the tutorial to do that. I used TopHat and Cufflinks to generate the ESTs from the assembly sequence based on RNA-seq. I used that ESTs to run the MAKER.

I think I already got more than 10Mb data. The information you mentioned is very helpful. I may go to use them to try to train the SNAP and Augustus.

Because this is my first time using the MAKER, I ran already a month, I was wondering maybe the command I used in a wrong way.

Sincerely,
Yunxi



2013/6/24 Carson Holt <carsonhh at gmail.com<mailto:carsonhh at gmail.com>>
Run time is dependent on the size of your evidence dataset, genome size, and number of processors you use.  If you have a large genome (Gb size) and you are running on a single cpu then that could take a long time.  This is especially true if you use the alt_est option for evidence as these are aligned via tblastx  which is 3-4 times slower than protein alignments, and 10-20 time slower than standard EST alignments. 95% of MAKER's runtime is BLAST alignment so your evidence dataset is the major factor.

Also you do not need results from the entire genome to train SNAP.  If you get results from ~10Mb of the genome that is usually sufficient.  Also make sure you are taking advantage of parallelization.  Launch via MPI to get maximum performance.  I commonly launch on 16 and 32 cpu Linux servers which can annotate most fungal genomes in a few hours and larger genomes in a few days.

--Carson


From: Amelia Ireland <amelia.ireland at gmod.org<mailto:amelia.ireland at gmod.org>>
Date: Sunday, 23 June, 2013 10:15 PM
To: <maker-devel at yandell-lab.org<mailto:maker-devel at yandell-lab.org>>
Cc: <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Subject: [maker-devel] Fwd: about running MAKER

>From the GMOD helpdesk; please cc Lin, lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>.

---------- Forwarded message ----------
From: Yunxi Lin <lin11 at cougars.csusm.edu<mailto:lin11 at cougars.csusm.edu>>
Date: Sun, Jun 23, 2013 at 4:14 PM
Subject: about running MAKER
To: "gmod-help at gmod.org<mailto:gmod-help at gmod.org>" <help at gmod.org<mailto:help at gmod.org>>


Hi

I'm running a eukaryote project on our server. Because our server do not have the GUI, is that still work for MAKER? And our command already ran more than one month to try to generate the model use for the training of SNAP and Augustus. Is that normal? I'm running on a 256G memory 64 Linux server.

Thank you.

Sincerely,
Lin



--
Amelia Ireland
GMOD Community Support
http://gmod.org || @gmodproject

_______________________________________________ maker-devel mailing list maker-devel at box290.bluehost.com<mailto:maker-devel at box290.bluehost.com>http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org















-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130723/c2ccf703/attachment-0002.html>