[maker-devel] MPI MAKER hanging NFS

[Carson Holt]

No it does not use ROMIO.

The locking may be do to how your NFS is implemented.  MAKER does a lot of
small writes.  Some NFS implementations do not handle that well and only
like large infrequent writes and frequent reads?
MAKER also uses a variant of the File:::NFSLock module which uses
hardlinks to force a flush of the NFS IO cache when asyncrynous IO is
enabled (described here
http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html).
I know that the FhGFS implementation of NFS has broken hard link
functionality.


Also make sure you do not set TMP= in the maker_opt.ctl file to an NFS
mounted location.  It must be local (/tmp for example).  This is because
certain types of operations are not always NFS safe and need a local
location to work with (anything involving berkley DB or SQLite for
example).  Make sure you are not setting that to an NFS mounted scratch
location.  The mpi**** files, are examples of some short lived files that
should not be in NFS.  They hold chunks of data from threads that are
processing the genome and are very rapidly created and deleted.  They will
be cleaned up automatically when maker finished or killed by standard
signals such as when you hit ^C or use kill 15.


Thanks,
Carson




On 13-05-14 4:42 PM, "Heywood, Todd" <heywood at cshl.edu> wrote:

>We have been getting hung NFS mounts on some nodes when running MPI MAKER
>(version 2.27). Processes go into a "D" state and cannot be killed. We
>end up having to reboot nodes to recover them. We are running MPICH2
>version 1.4.1p1
>with RHEL 6.3. Questions:
>
>(1) Does MPI MAKER use MPI-IO (ROMIO)? The state "D" processes are hung
>on a sync_page system call under NFS. That *might* imply some locking
>issues.
>
>(2) Has anyone else seen this?
>
>(3) The root directory (parent of genome.maker.output directory) has lots
>of mpi***** files, all of which have the first line
>"pst0Process::MpiChunk". Is this expected?
>
>I'm able to reproducibly hang NFS on some nodes when using at least 4
>32-core nodes and 128 running MPI tasks.
>
>Thanks,
>
>Todd Heywood
>CSHL
>
>