[maker-devel] SQLite database locked error, maker MPI using several nodes

[Carson Holt]

Sorry for the slow reply, I'm currently traveling.  Try deleting any *.db
diles in the  maker output directory to force the SQLite database to be
rebuilt.  Also you can try the current version of MAKER at yandell-lab.org.
MAKER is supposed to try and copy the database to the /tmp directory before
it starts work.  That way the actual working copy will be local, and will be
independent for each node.

--Carson


From:  Malcolm Hinsley <mhinsley at ebi.ac.uk>
Date:  Monday, September 16, 2013 3:51 AM
To:  <maker-devel at yandell-lab.org>
Subject:  [maker-devel] SQLite database locked error, maker MPI using
several nodes

    
 Hello
 
 I'm trying to get maker to run on MPI using several nodes.  I have an
installation set up by a colleague which includes maker 2.27 and
openmpi-1.4.3. 
 
 Previously it has only been used (here at EBI) with maker processes running
on one node only, but i find that it can wait a very long time before being
scheduled by LSF.
 
 The command used to submit is like this (as per recommendations from
systems) (uses 8 cpus on each of 8 nodes)
 
 
 
export OMP_NUM_THREADS=64
 
bsub -q mpi -M 40000 -R "rusage[mem=40000] && span[ptile=8]"  -n 64 -o
lsf_log  -a openmpi mpirun.lsf -np 64 -mca btl tcp,self  maker 2>&1
 
 
 and  requires environment be set up in ~/.bashrc for openMPI.
 
 This runs but produces a lot of errors like:
 
      DBD::SQLite::db do failed: database is locked at
/nfs/production/panda/ensemblgenomes/external/maker/2.27_mpi/maker/bin/../li
b/GFFDB.pm line 407.
 
 
 I've looked at 
https://groups.google.com/forum/#!searchin/maker-devel/database$20locked/mak
er-devel/TscBgbQfBX4/pae016DqlIMJ which suggests that "It means that your
GFF3 results will not be integrated" (but i'm not sure what's meant by that,
but the number of genes i'm getting is around 2k, expect more like 15k) and
that the problem is SQLite using NFS (a known issue), and the fix is to use
/tmp.
 
 I have TMP= set as per default in maker_opts.ctl, and there are maker
directories in /tmp on the runtime nodes, but the database (i guess) is in
/nfs/...../maker/<maker_out_dir>/<prefix>.scf.db.
 
 I don't see how i could set the working directory to a non-NFS file systems
and still use more than one node, but this error only seems to appear (so
far) with est2genome, not when running SNAP/ Augustus.
 
 Is there a work around to stop getting the locked error or some way to
recover from it after maker has finished?  Or is it necessary to run the
est2genome step  (or maker generally) on one node?
 
 An obvious option is to split the assembly but i was hoping to avoid that.
 
-- 
malcolm hinsley | EnsEMBL Genomes | +44 (0)1223 49 4669
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Wellcome Trust Genome Campus, Hinxton, Cambridge CB10 1SD
United Kingdom 
 
_______________________________________________ maker-devel mailing list
maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130917/bdc207e0/attachment-0003.html>