[maker-devel] master_datastore_index.log file shrinks.]

[Eleanor Stanley]

For the Sanger farm I have a wrapper script to run MPI maker so that the same environmental variables are forced to all nodes.

Eleanor


On 19 Mar 2013, at 15:54, Michael Nuhn wrote:

> Hello Carson!
> 
> Thanks for the pointers. I'll give mpi another shot.
> 
> Cheers,
> Michael.
> 
> On 03/19/2013 03:22 PM, Carson Holt wrote:
>> I have MAKER working under OpemnMPI 1.4.3 (intel compiled).
>> 
>> I had to set a couple of environmental variables prior to setup. You would
>> probably need to set these values as well.  If you your OpenMPI path was
>> here for example --> /software/openmpi-1.4.3/, run the following commands
>> (path set accordingly) before even attempting maker setup.
>> 
>> export OMPI_MCA_mpi_warn_on_fork  0
>> export LD_PRELOAD /software/openmpi-1.4.3/lib/libmpi.so:$LD_PRELOAD
>> 
>> These not only need to be set before compilation, but also before any run
>> (so add them to you ~.bashrc or ~/.bash_profile or any module load scripts
>> thanks).  The LD_PRELOAD statement needs to be set for any program using
>> OpenMPI's shared libraries and not just MAKER, so it's normally a good
>> idea to have that set system wide for all users.  The detail can be found
>> in the OpenMPI documentation.  Note sometimes system library updates can
>> break OpenMPI's shared libraries while not breaking OpenMPI itself, so you
>> might also need to recompile OpenMPI if it has broken shared libraries.
>> 
>> Once you have those commands in place, run the perl Buil.PL step. Say yes
>> to install with MPI. Then run ./Build install
>> 
>> Thanks,
>> Carson
>> 
>> 
>> 
>> On 13-03-19 11:02 AM, "Carson Holt" <carsonhh at gmail.com> wrote:
>> 
>>> Try it with the no_locks option then.  Make sure to let one instance
>>> finish populating the mpi_blastdb directory before running other
>>> instances
>>> as that is where most initial locking occurs.
>>> 
>>> I'll send you more details on how to install with OpenMPI, so you can
>>> give
>>> that a shot while your jobs are also running serially (so you don't lose
>>> time).  Also instead of 50 serial instances, you could try 10 with -cpus
>>> set to 5.
>>> 
>>> Thanks,
>>> Carson
>>> 
>>> 
>>> 
>>> On 13-03-19 11:19 AM, "Michael Nuhn" <mnuhn at ebi.ac.uk> wrote:
>>> 
>>>> Hello Carson!
>>>> 
>>>> On 03/19/2013 02:27 PM, Carson Holt wrote:
>>>>> Yes.  If at all possible use MPI.  It removes the overhead of locks
>>>>> which happen per primary instance of MAKER.  So one maker job using
>>>>> 1000
>>>>> cpus via MPI will have one shared set of locks.  1000 serial instances
>>>>> of MAKER on the other hand would have 1000x the locks.
>>>> 
>>>> I don't know a thing about MPI.
>>>> 
>>>> I tried installing maker (2.2.7) with mpich-3.0.2, mpich2-1.4.1 and open
>>>> mpi and none of them worked for me. I also tried the automatic
>>>> installation that comes with maker, but it didn't work for me either.
>>>> 
>>>> If need be, I could spend time getting to the bottom of this, but there
>>>> is no telling how long this would take me so I'd rather not, if there is
>>>> an alternative.
>>>> 
>>>> Would the approach I outlined before work? (Treating the split files as
>>>> separate genomes to annotate and then combine the gffs afterwards)
>>>> 
>>>> I also like this approach, because I would select a few contigs in the
>>>> beginning which I would run on their own. They would complete early and
>>>> this way I would get a preview of the results of the run instead of
>>>> having to wait for everything to complete.
>>>> 
>>>> It might also be more robust, because file locking issues would be
>>>> confined to the instances working on a sequence chunk, but the rest of
>>>> the instances could continue working.
>>>> 
>>>> Cheers,
>>>> Michael.
>>>> 
>>>>> Alternatively if you do need to continue without MPI for some reason, I
>>>>> just finished a devel version of MAKER that has a --no_locks option.
>>>>>   You can never start two instances using the same input fasta when
>>>>> --no_locks is specified, but the splitting to use different input
>>>>> fastas
>>>>> I mentioned before in the example will still work fine.
>>>>> 
>>>>> I also have updated the indexing/reindexing, so if indexing failures
>>>>> happen, MAKER will switch between the current working directory and the
>>>>> TMP= directory from the maker_opts.ctl file so as to try different IO
>>>>> locations (I.e. NFS and non-NFS).  Note you should never set TMP= in
>>>>> the
>>>>> control files to an NFS mounted location (it not only makes things a
>>>>> lot
>>>>> slower, but berkleydb and sqllite will get frequent errors on NFS).
>>>>>   TMP= defaults to /tmp when not specified
>>>>> 
>>>>> I'll send you download information in a separate e-mail.  Try a regular
>>>>> MAKER run to see if the indexing/reindexing changes are sufficient
>>>>> before attempting the ‹no_locks option.
>>>>> 
>>>>> Thanks,
>>>>> Carson
>> 
>> 
> 
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.