[maker-devel] MPI MAKER hanging NFS

[Evan Ernst]

Hi Carson,

For these runs, -TMP is set to the $TMPDIR environment variable via maker
command line argument in the cluster job script to use the local disk on
each node. We can see files being generated in those locations on each
node, so it seems this is working as expected.

In maker_opts.ctl, I commented out the TMP line. I'm not sure if this is
relevant, but I'm also setting mpi_blastdb= to consolidate the databases
onto a different, faster nfs mount than the working dir where the mpi****
files are being written.

Thanks,
Evan



On Tue, May 14, 2013 at 9:01 PM, Carson Holt <Carson.Holt at oicr.on.ca> wrote:

> No it does not use ROMIO.
>
> The locking may be do to how your NFS is implemented.  MAKER does a lot of
> small writes.  Some NFS implementations do not handle that well and only
> like large infrequent writes and frequent reads?
> MAKER also uses a variant of the File:::NFSLock module which uses
> hardlinks to force a flush of the NFS IO cache when asyncrynous IO is
> enabled (described here
> http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html).
> I know that the FhGFS implementation of NFS has broken hard link
> functionality.
>
>
> Also make sure you do not set TMP= in the maker_opt.ctl file to an NFS
> mounted location.  It must be local (/tmp for example).  This is because
> certain types of operations are not always NFS safe and need a local
> location to work with (anything involving berkley DB or SQLite for
> example).  Make sure you are not setting that to an NFS mounted scratch
> location.  The mpi**** files, are examples of some short lived files that
> should not be in NFS.  They hold chunks of data from threads that are
> processing the genome and are very rapidly created and deleted.  They will
> be cleaned up automatically when maker finished or killed by standard
> signals such as when you hit ^C or use kill 15.
>
>
> Thanks,
> Carson
>
>
>
>
> On 13-05-14 4:42 PM, "Heywood, Todd" <heywood at cshl.edu> wrote:
>
> >We have been getting hung NFS mounts on some nodes when running MPI MAKER
> >(version 2.27). Processes go into a "D" state and cannot be killed. We
> >end up having to reboot nodes to recover them. We are running MPICH2
> >version 1.4.1p1
> >with RHEL 6.3. Questions:
> >
> >(1) Does MPI MAKER use MPI-IO (ROMIO)? The state "D" processes are hung
> >on a sync_page system call under NFS. That *might* imply some locking
> >issues.
> >
> >(2) Has anyone else seen this?
> >
> >(3) The root directory (parent of genome.maker.output directory) has lots
> >of mpi***** files, all of which have the first line
> >"pst0Process::MpiChunk". Is this expected?
> >
> >I'm able to reproducibly hang NFS on some nodes when using at least 4
> >32-core nodes and 128 running MPI tasks.
> >
> >Thanks,
> >
> >Todd Heywood
> >CSHL
> >
> >
>
>
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20130515/0d3cdab3/attachment-0003.html>