<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Look for hidden files with .NFSLock in the name, delete them, and see if they come back.<div class=""><br class=""></div><div class="">find <search_folder> | grep .NFSLock | xargs rm</div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">find <search_folder> | grep .NFSLock</span></div><div class=""><br class=""></div><div class="">If the files come back after deleting them, it can mean another MAKER job is still running and updating the lock. Can happen in weird situations like when process managers like slurm OOM kill a job but only wipe out some of the processes and not all.</div><div class=""><br class=""></div><div class="">—Carson</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jan 21, 2021, at 6:30 AM, Michele Vidotto <<a href="mailto:michele.vidotto@gmail.com" class="">michele.vidotto@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><span style="font-family: Roboto, RobotoDraft, Helvetica, Arial, sans-serif; font-size: 14px;" class="">Dear all,</span><br class=""></div><div class=""><span style="font-family: Roboto, RobotoDraft, Helvetica, Arial, sans-serif; font-size: 14px;" class=""><br class=""></span></div><div class=""><span style="font-family: Roboto, RobotoDraft, Helvetica, Arial, sans-serif; font-size: 14px;" class="">as reported in the subject I'm having issues with locking mechanism of MAKER when it is runs in parallel-mode through mpi.</span></div><div class=""><span style="font-family: Roboto, RobotoDraft, Helvetica, Arial, sans-serif; font-size: 14px;" class="">I'm using maker version </span><font color="rgba(0, 0, 0, 0.870588235294118)" face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif" class=""><span style="font-size:14px" class="">3.01.03 but the same happens in my system when I build and install version </span></font><font color="#a00029" face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif" class=""><span style="font-size:14px" class="">2.31.11.</span></font></div><div class=""><font color="#a00029" face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif" class=""><span style="font-size:14px" class="">All prerequisites were installed in a conda environment. Perl was installed from anaconda channel in version 5.26.2. Hard-coded paths to the compilers were fixed. Necessary perl modules were installed via cpanm:</span><br class=""><br class=""><div class=""><div class=""><span style="font-size:14px" class="">"DBD::SQLite",</span></div><div class=""><span style="font-size:14px" class="">"DBI",</span></div><div class=""><span style="font-size:14px" class="">"Error",</span></div><div class=""><span style="font-size:14px" class="">"Error::Simple",</span></div><div class=""><span style="font-size:14px" class="">"File::NFSLock",</span></div><div class=""><span style="font-size:14px" class="">"File::Which",</span></div><div class=""><span style="font-size:14px" class="">"forks",</span></div><div class=""><span style="font-size:14px" class="">"forks::shared",</span></div><div class=""><span style="font-size:14px" class="">"Inline",</span></div><div class=""><span style="font-size:14px" class="">"Inline::C",</span></div><div class=""><span style="font-size:14px" class="">"IO::All",</span></div><div class=""><span style="font-size:14px" class="">"IO::Prompt",</span></div><div class=""><span style="font-size:14px" class="">"LWP::Simple"</span></div><div class=""><span style="font-size:14px" class="">"Perl::Unsafe::Signals",</span></div><div class=""><span style="font-size:14px" class="">"PerlIO::gzip",</span></div><div class=""><span style="font-size:14px" class="">"Proc::Simple",</span></div><div class=""><span style="font-size:14px" class="">"URI::Escape",</span></div><div class=""><span style="font-size:14px" class="">"DBD::Pg"</span></div></div></font><font color="#a00029" face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif" class=""><div style="font-size:14px" class=""><br class=""></div><div style="font-size:14px" class="">additional libraries and components were installed via conda</div><div style="font-size:14px" class=""><br class=""></div><div class=""><div style="font-size:14px" class=""> - gcc_linux-64=7.3.0</div><div style="font-size:14px" class=""> - gxx_linux-64=7.3.0</div><div style="font-size:14px" class=""> - openmpi=4.1.0</div><div style="font-size:14px" class=""> - zlib=1.2.11</div><div style="font-size:14px" class=""> - libdb=6.1.26</div><div style="font-size:14px" class=""> - expat=2.2.9</div><div style="font-size:14px" class=""> - libxml2=2.9.10</div><div style="font-size:14px" class=""> - exonerate=2.4.0</div><div style="font-size:14px" class=""> - snoscan=1.0</div><div style="font-size:14px" class=""> - rapsearch=2.24</div><div style="font-size:14px" class=""><br class=""></div><div style="font-size:14px" class="">other components were installed manually. MAKER compile and install with no errors, but when I execute the program via MPI with:</div><div style="font-size:14px" class=""><br class=""></div><div style="font-size:14px" class=""># to devoid <span style="font-weight:bold;font-size:small" class="">OPEN MPI segmentation fault</span></div><div class=""><div class=""><span style="font-size:14px" class="">export THREADS_DAEMON_MODEL=1</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">mpiexec -mca btl ^openib -n 1 \</span></div><div class=""><span style="font-size:14px" class="">maker \</span></div><div class=""><span style="font-size:14px" class="">-force \</span></div><div class=""><span style="font-size:14px" class="">-cpus 8 \</span></div><div class=""><span style="font-size:14px" class="">--fix_nucleotides \</span></div><div class=""><span style="font-size:14px" class="">maker_opts.ctl \</span></div><div class=""><span style="font-size:14px" class="">maker_bopts.ctl \</span></div><div class=""><span style="font-size:14px" class="">maker_exe.ctl</span></div></div><div style="font-size:14px" class=""><br class=""></div><div style="font-size:14px" class="">It always ends up with following error:</div><div style="font-size:14px" class=""><br class=""></div><div style="font-size:14px" class=""><br class=""></div><div class=""><div class=""><span style="font-size:14px" class="">STATUS: Parsing control files...</span></div><div class=""><span style="font-size:14px" class="">ERROR: The directory is locked. Perhaps by an instance of MAKER.</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">--> rank=NA, hostname=<a href="http://april.corp.igatechnology.com/" class="">april.corp.igatechnology.com</a></span></div><div class=""><span style="font-size:14px" class="">--------------------------------------------------------------------------</span></div><div class=""><span style="font-size:14px" class="">Primary job terminated normally, but 1 process returned</span></div><div class=""><span style="font-size:14px" class="">a non-zero exit code. Per user-direction, the job has been aborted.</span></div><div class=""><span style="font-size:14px" class="">--------------------------------------------------------------------------</span></div><div class=""><span style="font-size:14px" class="">--------------------------------------------------------------------------</span></div><div class=""><span style="font-size:14px" class="">mpiexec detected that one or more processes exited with non-zero status, thus causing</span></div><div class=""><span style="font-size:14px" class="">the job to be terminated. The first process to do so was:</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class=""> Process name: [[19321,1],0]</span></div><div class=""><span style="font-size:14px" class=""> Exit code: 10</span></div><div class=""><span style="font-size:14px" class="">--------------------------------------------------------------------------</span></div></div><div style="font-size:14px" class=""><br class=""></div></div><div style="font-size:14px" class="">if I look inside *.maker.output directory a lock file remains: </div><div style="font-size:14px" class=""><br class=""></div><div class=""><span style="font-size:14px" class="">.NFSLock.gi_lock.NFSLock</span><br class=""></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">If instead I run maker with the -nolock flag. MAKER runs with no problems at all.</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">My filesystem is oneFS from ISILON, exported to a virtual server through nfs4 protocol.</span></div><div class=""><span style="font-size:14px" class="">By looking at the code MAKER uses File::NFSLock Perl module for locking. This module fails some tests when installed on my system with cipanm:</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><div class=""><span style="font-size:14px" class=""># Failed test at t/300_bl_sh.t line 115.</span></div><div class=""><span style="font-size:14px" class="">Shared locks not running simultaneously at t/300_bl_sh.t line 116, <$rd3> line 18.</span></div><div class=""><span style="font-size:14px" class=""># Looks like your test exited with 4 just after 27.</span></div><div class=""><span style="font-size:14px" class="">t/300_bl_sh.t ..... Dubious, test returned 4 (wstat 1024, 0x400)</span></div><div class=""><span style="font-size:14px" class="">Failed 47/73 subtests</span></div><div class=""><span style="font-size:14px" class="">t/400_kill.t ...... ok</span></div><div class=""><span style="font-size:14px" class="">t/410_die.t ....... ok</span></div><div class=""><span style="font-size:14px" class="">t/420_crash.t ..... ok</span></div><div class=""><span style="font-size:14px" class="">t/430_taint.t ..... ok</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">Test Summary Report</span></div><div class=""><span style="font-size:14px" class="">-------------------</span></div><div class=""><span style="font-size:14px" class="">t/300_bl_sh.t (Wstat: 1024 Tests: 27 Failed: 1)</span></div><div class=""><span style="font-size:14px" class=""> Failed test: 27</span></div><div class=""><span style="font-size:14px" class=""> Non-zero exit status: 4</span></div><div class=""><span style="font-size:14px" class=""> Parse errors: Bad plan. You planned 73 tests but ran 27.</span></div></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">But anyway I was able to install it with --notest flag.</span><br class=""></div><div class=""><span style="font-size:14px" class="">Do you have any idea on how I can overcome my problem and have MAKER run in parallel with MPI?</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class="">Thanks in advance,</span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div class=""><span style="font-size:14px" class=""><br class=""></span></div><div style="font-size:14px" class=""><br class=""></div></font></div><div class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class="">---</div><div class="">Michele Vidotto<br class="">mailto: <a href="mailto:michele.vidotto@studenti.unipd.it" target="_blank" class="">michele.vidotto@gmail.com</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br class="">maker-devel mailing list<br class=""><a href="mailto:maker-devel@yandell-lab.org" class="">maker-devel@yandell-lab.org</a><br class="">http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org<br class=""></div></blockquote></div><br class=""></div></body></html>