[maker-devel] maker with MPI and perl using threads

[John Damm Sørensen]

Thanks Carson,

One thing bothers me. That's this from Perl forks documentation:

module load order: forks first

Since forks overrides core Perl functions, you are *strongly* encouraged 
to load the forks module before any other Perl modules. This will insure 
the most consistent and stable system behavior. This can be easily done 
without affecting existing code, like:

     perl -Mforks  script.pl

But in the maker perlscript the module FindBin that in sturn loads a 
bunch of other modules is loaded before forks.

Is that intentionally?

Best

John




Den 29-06-2017 kl. 22:56 skrev Carson Holt:
> Also when you see calls like threads->new() in the maker script. It is creating a fork and not a thread. It’s a convenience feature activated by the ‘use forks’ line at the top of the script. It allows you to use thread like syntax when working with forks.
>
> —Carson
>
>
>
>> On Jun 29, 2017, at 2:43 PM, Carson Holt <carsonhh at gmail.com> wrote:
>>
>> MAKER doesn’t use threads. It uses forks. There are several reasons for this, including that Perl already requires you to use hidden fork operations every time you call system() or open(). So trying to get forks, threads, and MPI together is just a mess. So we stuck with just forks and MPI. With MPI flavors with direct infiniband support you can get weird errors because of how forks affect registered memory when the MPI flavor uses OpenIB libraries. There is also another issue with how perl wraps malloc calls that affects registered memory. The best way around both these issues is too disable direct infiniband support, and then set the MPI flavor to use -tcp over the ip-over-infiniband virtual adaptor (usually ib0) or use the ethernet adapter. Sometime last year after a system update we also got an OpenMPI error (only on CentOS6) that referred to a file used for Perl threads (which should not even be in use since we are using forks). We worked around that issue by compile Perl without thread support so that the library couldn't be called.
>>
>> If you were using OpenMPI and were seeing a reference to Perl threads, then the error you saw may have been related to the latter one I mentioned.
>>
>> I have tried the MPI_Init_thread option before and had run into issues because of it without it helping any of the previously mentioned fork related issues. But that was some time ago, so I could try it as a solution to the last issue I mentioned rather than installing a no-thread version of perl (if I can ever replicate the error because it went away when we updated from CentOS kernel 6 to kernel 7).
>>
>> Thanks,
>> Carson
>>
>>
>>> On Jun 28, 2017, at 2:54 AM, John Damm Sørensen <john at hovedpuden.dk> wrote:
>>>
>>> Hello,
>>>
>>> Recently I assisted one of my customers with problems solving maker using MPI.
>>>
>>> It seems that the main reason for the trouble was maker not initializing the MPI environment for thread save execution.
>>>
>>> In the MPI.pm module you call MPI_Init whereas for a threaded environment you should call MPI_Init_thread.
>>>
>>> I think it would be a good idea to detect whether the users Perl is with thread support and init MPI accordingly or clearly state that maker is for unthreaded Perl only.
>>>
>>> During the debugging we also found that it was beneficial to have the latest mxm.c installed:
>>>
>>> https://community.mellanox.com/thread/3439
>>>
>>>
>>> Best Regards
>>>
>>> John Damm Sørensen
>>>
>>> IT consultant
>>>
>>>
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org