<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>
<div>Could you use the attached file to replace maker/src/bin/maker and maker/bin/maker?</div>
<div><br>
</div>
<div>You will have to rerun 'maker/src/Build install' or just edit the shebang line (#!) if perl is located anywhere other than /usr/bin/perl.</div>
<div><br>
</div>
<div>I explicitly tell it to use the system TMPDIR rather than letting it get set implicitly. See if that stops the mpi***** files in the working directory. It's always possible that this is just a slight difference in behavior for the version of the File::Temp
module that is packaged with your perl.</div>
<div><br>
</div>
<div>--Carson</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px; ">
</div>
</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Evan Ernst <<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a>><br>
<span style="font-weight:bold">Date: </span>Monday, 20 May, 2013 8:20 PM<br>
<span style="font-weight:bold">To: </span>Carson Holt <<a href="mailto:carsonhh@gmail.com">carsonhh@gmail.com</a>><br>
<span style="font-weight:bold">Cc: </span>Carson Holt <<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a>>, "<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>" <<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>,
"Heywood, Todd" <<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [maker-devel] MPI MAKER hanging NFS<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/tmp/uge/1031236.1.primary.q</div>
<div>/opt/uge/default/common/starter_with_limit.sh: line 4: /sonas-hs/it/hpc/data/eernst/maker_carson_debug/"/opt/hpc/lib64/mpich2/bin/hydra_pmi_proxy": No such file or directory</div>
<div>/opt/uge/default/common/starter_with_limit.sh: line 4: exec: /sonas-hs/it/hpc/data/eernst/maker_carson_debug/"/opt/hpc/lib64/mpich2/bin/hydra_pmi_proxy": cannot execute: No such file or directory</div>
<div><br>
</div>
<div><br>
</div>
<div>Todd, are these errors from the starter_with_limit.sh wrapper harmless?<br>
</div>
<div style=""><br>
</div>
<div style="">Thanks,</div>
<div style="">Evan</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, May 20, 2013 at 7:50 PM, Carson Holt <span dir="ltr">
<<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Could you run the following command for me and share the ouptut with me?<br>
<br>
mpiexec -n 8 perl -e 'use File::Spec; print File::Spec->tmpdir()."\n"'<br>
<div class="im"><br>
Thanks,<br>
Carson<br>
<br>
<br>
<br>
From: Evan Ernst <<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a>>><br>
</div>
Date: Monday, 20 May, 2013 4:36 PM<br>
<div class="im">To: Carson Holt <<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a><mailto:<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a>>><br>
</div>
Cc: "<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>" <<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>>,
"Heywood, Todd" <<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>>><br>
<div class="im">Subject: Re: [maker-devel] MPI MAKER hanging NFS<br>
<br>
Hi Carson,<br>
<br>
</div>
<div class="im">The SGE launch script looks like this (sans SGE args):<br>
<br>
mpiexec -n 8 maker -TMP $TMPDIR maker_opts.ctl maker_bopts.ctl maker_exe.ctl >>logs/final.$SGE_TASK_ID.mpi.log 2>&1<br>
<br>
Snooping on the running jobs (see attached image), it looks like $TMPDIR is evaluated to a local directory by the shell of the MPI master node as intended, so the evaluated path, not the env var reference, is being passed to the MPI workers.<br>
<br>
Despite this, the mpi*** files are still being created in the working directory.<br>
<br>
If I understand correctly, these mpi*** files are meant to be written to the directory given by TMP= (maker_opts.ctl) or -TMP (command line arg), which should be equivalent, but this doesn't seem to be the case.<br>
<br>
Thanks,<br>
Evan<br>
<br>
<br>
<br>
<br>
</div>
<div class="im">On Fri, May 17, 2013 at 9:40 AM, Carson Holt <<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a>>> wrote:<br>
I'm glad your getting better results.<br>
<br>
With respect to environmental variables. One common error in MPI<br>
execution is that the environment variables will not always be the same on<br>
the other nodes since only the root node is attached to a terminal, so<br>
variables in launch scripts (.bashrc etc.) may not be available on all<br>
nodes. Many clusters that are part of the XSEDE network and use SGE for<br>
example have scripts that wrap mpiexec to guarantee export of all<br>
environmental variables when using MPI to avoid just this type of common<br>
error. So like anything, you start with the most common cause of errors<br>
and then work to the less common. Kernel bugs usually rank low on the<br>
list :-) But I'm glad it's working for you now.<br>
<br>
Thanks,<br>
Carson<br>
<br>
<br>
<br>
<br>
<br>
</div>
<div class="im">On 13-05-17 9:25 AM, "Heywood, Todd" <<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>>> wrote:<br>
<br>
>It appears that a kernel bug caused the NFS hang, at least for limlted<br>
>scale testing (6 nodes, 192 tasks). I upgraded the kernel from<br>
>2.6.32-279.9.1.el6.x86_64 to 2.6.32-358.6.1.el6.x86_64 on 6 nodes and<br>
>cannot reproduce the hangs.<br>
><br>
>As far a TMPDIR, I'm not really sure I understand. We use SGE, and the<br>
>TMPDIR we are referring to is set by SGE within a job to be<br>
>/tmp/uge/JobID.TaskID.QueueName. Have you run via SGE?<br>
><br>
>Todd<br>
><br>
><br>
><br>
><br>
</div>
>From: Carson Holt <<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a>><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a>>>><br>
<div class="im">>Date: Wednesday, May 15, 2013 1:15 PM<br>
</div>
>To: "Ernst, Evan" <<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a>><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a>>>><br>
>Cc: Todd Heywood <<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>>>>,<br>
>"<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>>"<br>
><<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>>><br>
<div class="im">>Subject: Re: [maker-devel] MPI MAKER hanging NFS<br>
><br>
>The mpi**** files should be generated in the $TMPDIR or TMP= location.<br>
>If they are happening in the working directory, then there is a problem.<br>
>If you are not setting TMP=, perhaps TMPDIR is not being exported when<br>
>'mpiexec' is launched. You may have to manually specify that it needs to<br>
>be exported to the other nodes using the mpiexec command line flags.<br>
>OpenMPI for example does not export all environmental variables by<br>
>default to the other nodes.<br>
><br>
>Thanks,<br>
>Carson<br>
><br>
><br>
><br>
</div>
>From: Evan Ernst <<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a>><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a><mailto:<a href="mailto:eernst@cshl.edu">eernst@cshl.edu</a>>>><br>
<div class="im">>Date: Wednesday, 15 May, 2013 1:08 PM<br>
</div>
>To: Carson Holt <<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a><mailto:<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a>><mailto:<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a><mailto:<a href="mailto:carson.holt@oicr.on.ca">carson.holt@oicr.on.ca</a>>>><br>
>Cc: "Heywood, Todd" <<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>>>>,<br>
>"<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>>"<br>
><<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a><mailto:<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>>>><br>
<div class="im">>Subject: Re: [maker-devel] MPI MAKER hanging NFS<br>
><br>
>Hi Carson,<br>
><br>
>For these runs, -TMP is set to the $TMPDIR environment variable via maker<br>
>command line argument in the cluster job script to use the local disk on<br>
>each node. We can see files being generated in those locations on each<br>
>node, so it seems this is working as expected.<br>
><br>
>In maker_opts.ctl, I commented out the TMP line. I'm not sure if this is<br>
>relevant, but I'm also setting mpi_blastdb= to consolidate the databases<br>
>onto a different, faster nfs mount than the working dir where the mpi****<br>
>files are being written.<br>
><br>
>Thanks,<br>
>Evan<br>
><br>
><br>
><br>
>On Tue, May 14, 2013 at 9:01 PM, Carson Holt<br>
</div>
<div>
<div class="h5">><<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a>><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a><mailto:<a href="mailto:Carson.Holt@oicr.on.ca">Carson.Holt@oicr.on.ca</a>>>>
wrote:<br>
>No it does not use ROMIO.<br>
><br>
>The locking may be do to how your NFS is implemented. MAKER does a lot of<br>
>small writes. Some NFS implementations do not handle that well and only<br>
>like large infrequent writes and frequent reads?<br>
>MAKER also uses a variant of the File:::NFSLock module which uses<br>
>hardlinks to force a flush of the NFS IO cache when asyncrynous IO is<br>
>enabled (described here<br>
><a href="http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html" target="_blank">http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html</a>).<br>
>I know that the FhGFS implementation of NFS has broken hard link<br>
>functionality.<br>
><br>
><br>
>Also make sure you do not set TMP= in the maker_opt.ctl file to an NFS<br>
>mounted location. It must be local (/tmp for example). This is because<br>
>certain types of operations are not always NFS safe and need a local<br>
>location to work with (anything involving berkley DB or SQLite for<br>
>example). Make sure you are not setting that to an NFS mounted scratch<br>
>location. The mpi**** files, are examples of some short lived files that<br>
>should not be in NFS. They hold chunks of data from threads that are<br>
>processing the genome and are very rapidly created and deleted. They will<br>
>be cleaned up automatically when maker finished or killed by standard<br>
>signals such as when you hit ^C or use kill 15.<br>
><br>
><br>
>Thanks,<br>
>Carson<br>
><br>
><br>
><br>
><br>
>On 13-05-14 4:42 PM, "Heywood, Todd"<br>
</div>
</div>
<div class="im">><<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a><mailto:<a href="mailto:heywood@cshl.edu">heywood@cshl.edu</a>>>>
wrote:<br>
><br>
>>We have been getting hung NFS mounts on some nodes when running MPI MAKER<br>
>>(version 2.27). Processes go into a "D" state and cannot be killed. We<br>
>>end up having to reboot nodes to recover them. We are running MPICH2<br>
>>version 1.4.1p1<br>
>>with RHEL 6.3. Questions:<br>
>><br>
>>(1) Does MPI MAKER use MPI-IO (ROMIO)? The state "D" processes are hung<br>
>>on a sync_page system call under NFS. That *might* imply some locking<br>
>>issues.<br>
>><br>
>>(2) Has anyone else seen this?<br>
>><br>
>>(3) The root directory (parent of genome.maker.output directory) has lots<br>
>>of mpi***** files, all of which have the first line<br>
>>"pst0Process::MpiChunk". Is this expected?<br>
>><br>
>>I'm able to reproducibly hang NFS on some nodes when using at least 4<br>
>>32-core nodes and 128 running MPI tasks.<br>
>><br>
>>Thanks,<br>
>><br>
>>Todd Heywood<br>
>>CSHL<br>
>><br>
>><br>
><br>
><br>
>_______________________________________________<br>
>maker-devel mailing list<br>
</div>
><a href="mailto:maker-devel@box290.bluehost.com">maker-devel@box290.bluehost.com</a><mailto:<a href="mailto:maker-devel@box290.bluehost.com">maker-devel@box290.bluehost.com</a>><mailto:<a href="mailto:maker-devel@box290.bluehost.com">maker-devel@box290.bluehost.com</a><mailto:<a href="mailto:maker-devel@box290.bluehost.com">maker-devel@box290.bluehost.com</a>>><br>
<div class="HOEnZb">
<div class="h5">><a href="http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org" target="_blank">http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org</a><br>
><br>
<br>
<br>
_______________________________________________ maker-devel mailing list <a href="mailto:maker-devel@box290.bluehost.com">
maker-devel@box290.bluehost.com</a><mailto:<a href="mailto:maker-devel@box290.bluehost.com">maker-devel@box290.bluehost.com</a>>
<a href="http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org" target="_blank">
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span>
</body>
</html>