<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
It’s a little more complicated than that. MAKER is written in Perl, and Perl doesn’t give me the low level access that a language like C would for controlling memory access (I don’t control that). All I get is Perl’s standard implementation of forks. So
it’s not really a matter of MAKER changing, it would be a matter of changing Perl itself (which I have no power over, and I don’t think will be changing anytime soon).</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
For now you just have to add this flag to OpenMPI when running MAKER with mpiexec —> -mca btl ^openib</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Example :</div>
<blockquote style="margin:0 0 0 40px; border:none; padding:0px;">
<div><font face="Calibri,sans-serif">mpiexec -mca btl ^openib -n 20 maker</font></div>
</blockquote>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Thanks,</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Carson</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>UMD Bioinformatics <<a href="mailto:bioinformatics.umd@gmail.com">bioinformatics.umd@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, February 27, 2014 at 9:46 AM<br>
<span style="font-weight:bold">To: </span><<a href="mailto:maker-devel@yandell-lab.org">maker-devel@yandell-lab.org</a>><br>
<span style="font-weight:bold">Subject: </span>Problem with OpenFabrics and infiniband<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hello,
<div><br>
</div>
<div>I’ve had my IT folks install maker on our cluster at UMD. I’m having a SEGFAULT error when running maker on inifiniband nodes vs gigE nodes. According to the logs this appears to be an issue with forks but I’m not sure how to fix this. I would simply use
the gigE nodes but we are in the process of updating everything to inifiniband so I’ll need to address this issue as some point. I’ve attached the error log from the MPI run as well as commentary from my HPCC team. </div>
<div><br>
</div>
<div>IT suggestions</div>
<div><br>
</div>
<div><span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">If you look at the top of the error log for the problematic job, it clearly</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">warns of an issue with doing 'fork's within openmpi/openfabrics framework.</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">In particular, the use of the fork system call is only partially supported</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">in the OpenFabrics software (this is the drivers, etc for the infiniband</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">connections). See e.g. </span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<a href="http://www.open-mpi.org/faq/?category=openfabrics#ofa-fork" style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">http://www.open-mpi.org/faq/?category=openfabrics#ofa-fork</a><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">for more information. In particular the paragraphs starting with the</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">sentence with the red highlighted "it does not mean that your fork()-calling </span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">application is safe". (The kernel, openMPI version, and OFED version are </span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">sufficiently recent to mean that there is _some_ fork support).</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">The fact that the job runs over gigE but not IB, in conjunction with the</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">warning from openmpi, strongly suggests that this is the issue that you are </span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">encountering. I suspect that maker touches registered memory before the fork,</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">which would result in a segfault (matching what was observed).</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">You can try adding the arguments</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">--mca mpi_warn_on_fork 0 </span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">to the mpirun command, just in case the crash was somehow caused by openmpi's</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">warning, but I would not hold out much hope for that.</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<br>
</div>
<div>###UPDATE### This does not fix the problem.<br>
<br>
<br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">Basically, it looks like maker uses some system calls like fork in a manner</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">which is incompatible with the current OpenFabrics software, and thus will</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">not work with infiniband. This situation is likely to remain until either</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">maker changes to be compatible with OFED, or OFED's support for the fork</span><br style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;">system call is broadened.</span></div>
<div><span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;"><br>
</span></div>
<div><span style="font-family: Arial, Helvetica, sans-serif; font-size: 14px;"><br>
</span></div>
<div></div>
</div>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div></div>
</div>
</div>
</span>
</body>
</html>