[maker-devel] Maker-Error when started with IMPI
Rainer Rutka
rainer.rutka at uni-konstanz.de
Fri Feb 24 01:43:21 MST 2017
Hi Carson.
First of all THANK YOU FOR YOUR HELP.
MUCH APPRECIATED. :-)
Am 23.02.2017 um 21:22 schrieb Carson Holt:
> It means one of two things.
> 1. The mpiexec you called is not from Intel MPI (try 'which mpiexec' to verify the location and that it is not some other random mpiexec executable).
No, I am starting the right version of mpiexec.
This is a list of all our current available MPI-Versions, corresponding
to their compiler:
UC:[kn at uc1n997 ~]$ module avail mpi
------------------------------------- /opt/bwhpc/common/modulefiles
--------------------------------------
mpi/impi/4.1.3-gnu-4.4 mpi/openmpi/1.10-intel-15.0
mpi/impi/4.1.3-gnu-4.7 mpi/openmpi/1.10-intel-16.0
mpi/impi/4.1.3-intel-14.0 mpi/openmpi/1.8-gnu-4.5
mpi/impi/5.0.3-gnu-4.4 mpi/openmpi/1.8-gnu-4.7
mpi/impi/5.0.3-gnu-4.7 mpi/openmpi/1.8-gnu-4.7-m32
mpi/impi/5.0.3-intel-15.0 mpi/openmpi/1.8-gnu-4.8
mpi/impi/5.1.3-gnu-4.7 mpi/openmpi/1.8-gnu-4.9
mpi/impi/5.1.3-gnu-system mpi/openmpi/1.8-intel-14.0(default)
mpi/impi/5.1.3-intel-16.0(default)
mpi/openmpi/1.8-intel-15.0
mpi/openmpi/1.10-gnu-4.5 mpi/openmpi/2.0-gnu-4.7
mpi/openmpi/1.10-gnu-4.7 mpi/openmpi/2.0-gnu-4.8
mpi/openmpi/1.10-gnu-4.8 mpi/openmpi/2.0-gnu-5.2
mpi/openmpi/1.10-gnu-4.9 mpi/openmpi/2.0-intel-15.0
mpi/openmpi/1.10-gnu-5.2 mpi/openmpi/2.0-intel-16.0
mpi/openmpi/1.10-intel-14.0
Here I load the MPI-Module including all it's dependencies:
UC:[kn at uc1n997 ~]$ module load mpi/impi/5.1.3-intel-16.0
Loading module dependency 'compiler/intel/16.0'.
Result:
UC:[kn at c1n997 ~]$ which mpiexec
/opt/bwhpc/common/compiler/intel/compxe.2016.4.258/impi/5.1.3.223/intel64/bin/mpiexec
UC:[kn at uc1n997 ~]$
> 2. It is an extremely old version of Intel MPI (in which case the instructions I gave you would not apply)
Extremely old?
[...]
Intel(R) MPI Library for Linux* OS, Version 5.1.3 Build 20160601 (build
id: 15562)
> MPD is an old launcher used in MPICH1 and early versions of MPICH2. It’s been abandoned since about 2008 when MPICH2 switched to the hydra launcher which is still used in MPICH3.
Yes. And we don't even use MPD at our clusters :-)
> So either you are pointing to an old version of MPICH or you have a very old version of Intel MPI based off of MPICH.
I do not have any MPICH available on our cluster(s) now. All of the old
versions had
been removed some years ago.
At a glance
---------------
Maker is available as a so-called module on our cluster system. It's
been build
on a developers' node I can access. But, the MPI-modules are built by other
fellows (e.g. at the KIT in Karlsruhe/Germany) on other nodes.
Please check the module-file (included in this mail)
bio-maker-2.31.8_impi
to see how Maker was build including the envoronments set by this
module.
e.g.:
./Build status
verify dependencies:
===================================================
STATUS MAKER v2.31.8
===================================================
PERL Dependencies: VERIFIED
External Programs: VERIFIED
External C Libraries: VERIFIED
MPI SUPPORT: ENABLED
MWAS Web Interface: DISABLED
MAKER PACKAGE: CONFIGURATION OK
At least, please(!) have a look at our m.moab file (included in this mail).
This is the way how we submit a Maker job to our cluster. Maybe something
is wrong here?
Sorry again for wasting your time. But we imperatively need the Maker
software running in parallel mode.
:-)
--
Rainer Rutka
University of Konstanz
Communication, Information, Media Centre (KIM)
* High-Performance-Computing (HPC)
* KIM-Support and -Base-Services
Room: V511
78457 Konstanz, Germany
+49 7531 88-5413
-------------- next part --------------
#%Module1.0
#
# Cluster: bwunicluster
# Module: bio/maker/2.31.8
# Revision: knbw01
# TargetSystem: Red-Hat-Enterprise
# MainLocation: /opt/bwhpc/common
# Status: optional
# License: GPL, Artistic License
# URL: http://www.yandell-lab.org/software/maker.html
# 2ndLevelSupport: bwhpc [at] uni-konstanz.de
#
##### (A) Revision history:
#
# knbw01 20160630 r.rutka Initial revision knbw01 of module version 2.31.8
# knbw02 20161121 r.rutka Initial revision knbw02 of module version 2.31.8
# knbw03 20160222 r.rutka Initial revision knbw02 of module version 2.31.8 with impi
#
##### (B) Dependencies:
#
# conflict: any other maker version
# module load compiler/intel/16.0
# module load mpi/impi/5.1.3-intel-16.0
#
# # Main environments for build
#
# VERSION="2.31.8_impi" && echo "Version: ${VERSION}"
# MAIN_DIR="/opt/bwhpc/common" && echo ${MAIN_DIR}
# SOURCE_DIR=/home/kn/kn_kn/kn_pop235844/src/bio/maker/2.31.8
# TARGET_DIR="${MAIN_DIR}/bio/maker/${VERSION}" && echo ${TARGET_DIR}
#
##### (C) How to obtain software?
# # You have to register at first before you can download the SW.
# # http://yandell.topaz.genetics.utah.edu/cgi-bin/maker_license.cgi
#
# # Download and unzip the binary distributions
# cd ${SOURCE_DIR} && pwd
# [ ! -f maker-${VERSION/_impi}.tgz ] && wget http://yandell.topaz.genetics.utah.edu/maker_downloads/CE63/461D/BD41/4510A183DC4571D5EA619E459572/maker-2.31.8.tgz
# [[ -d ${TARGET_DIR} ]] && mv ${TARGET_DIR} ${TARGET_DIR}_$(date +%Y.%m.%d:%H.%M)
#
##### (D) How to build and install software?
#
# mkdir -vp ${TARGET_DIR/$VERSION}
# cd ${TARGET_DIR/$VERSION} && pwd
# tar -xvzf ${SOURCE_DIR}/maker-${VERSION/_impi}.tgz
# mv maker ${VERSION}
# cd ${VERSION} && pwd
# export LD_PRELOAD=${MPI_LIB_DIR}/libmpi.so
# cd src
# # Build required ad-ons. Lot's of required packages will be installed.
#
# perl Build.PL
# | Would you like to configure MAKER for MPI.. ? [N ] Y
# | # accept the next default paths and press ENTER
# # ignore the error-messages - just continue with the next step
#
# ./Build installdeps # installs missing PERL dependencies
# | ... MAKER .. local installataion: Y
# | # Accept the defaults of _all_ further questions by pressing ENTER.
# | # ** This is an annoying and long procedure :-( **
#
# ./Build installexes # installs missing external programs
# | # If are a registered user of RepBase, then MAKER can
# | # download and install RepBase for RepeatMasker for you.
# | # Register at: http://www.girinst.org/
# | ... download and install RepBase ... : N # We install RepBase later!
# # If you type Y, this will not work
# # See Install RepBase later in this file
#
# | # ERROR: Exonerate can't be found. URL Error 404. :-(
# | # Locate exonerate entry in locations file and modify path to (line 33):
# | # http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.2.0-x86_64.tar.gz
# | # restart ./Build installexes
# ./Build installexes
#
# # Install RepBase to prevent "RepBase is not insalled for RepeatMasker" error-message
# # http://www.girinst.org/server/RepBase/index.php
# p=$(pwd)
# cd ${TARGET_DIR}/exe/RepeatMasker && pwd
# # http://www.girinst.org/server/RepBase/index.php
# # FASTA-Format
# wget --user RainerRutka --password 13m5c3 http://www.girinst.org/server/RepBase/protected/RepBase22.01.fasta.tar.gz
# tar xvzf RepBase22.01.fasta.tar.gz
# #
# # NEU!
# wget --user RainerRutka --password 13m5c3 http://www.girinst.org/server/RepBase/protected/repeatmaskerlibraries/RepBaseRepeatMaskerEdition-20170127.tar.gz
# # # wget --user RainerRutka --password 13m5c3 http://www.girinst.org/server/RepBase/protected/repeatmaskerlibraries/repeatmaskerlibraries-20160829.tar.gz
# tar xvzf RepBaseRepeatMaskerEdition-20170127.tar.gz
# # # tar xvzf repeatmaskerlibraries-20160829.tar.gz
#
# perl ./configure
# | Enter path perl ... : ENTER
# | Enter path ... RepeatMasker: ENTER
# | Enter path TRF programm: /opt/bwhpc/common/bio/maker/2.31.8_impi/exe/RepeatMasker/trf
# | Add a search engine: 2 RMBlast
# | Enter path: /opt/bwhpc/common/bio/maker/2.31.8_impi/exe/RepeatMasker/rmblast/bin
# | search engine ... default: Y or ENTER
# | Enter Selection: 5 (Done)
# #
# rm *.tar.gz
# cd $p && pwd
#
# # Now check if anything's fine until now...
# ./Build status
# | # veryfy dependencies:
# | # ===================================================
# | # STATUS MAKER v2.31.8
# | # ===================================================
# | # PERL Dependencies: VERIFIED
# | # External Programs: VERIFIED
# | # External C Libraries: VERIFIED
# | # MPI SUPPORT: ENABLED
# | # MWAS Web Interface: DISABLED
# | # MAKER PACKAGE: CONFIGURATION OK
#
# ./Build install
#
# # Test
# cd ${TARGET_DIR}
# bin/maker -version # 2.31.8
#
# # Create module file (this one):
# # ${SOURCE_DIR}/modulefiles/bio-maker-2.31.8
#
# # Create Moab example files (and read the instructions inside of it):
# # ${SOURCE_DIR}/bwhpc-examples/bwunicluster-maker-example.moab
#
# # Copy Moab/bwHPC, special-scripts, examples and more...
# mkdir -vp ${TARGET_DIR}/bwhpc-examples
# cp ${SOURCE_DIR}/bwhpc-examples/* ${TARGET_DIR}/bwhpc-examples/.
#
# # Copy module file (this one):
# mkdir -vp ${TARGET_DIR}/modulefiles
# cp -v ${SOURCE_DIR}/modulefiles/bio-maker-${VERSION} ${TARGET_DIR}/modulefiles/
# chmod -c 644 ${TARGET_DIR}/modulefiles/*
#
# # Create module link:
# mkdir -vp ${MAIN_DIR}/modulefiles/bio/maker
# ln -fs ${TARGET_DIR}/modulefiles/bio-maker-${VERSION} ${MAIN_DIR}/modulefiles/bio/maker/${VERSION}
# module avail bio/maker
#
# # Cleanup build and installation target dir:
# # chgrp -R -h -c uc1-adm-sw ${TARGET_DIR}
# chmod 777 ${TARGET_DIR}/bwhpc-examples
# chmod 666 ${TARGET_DIR}/bwhpc-examples/*
# rm -rf ${TARGET_DIR}/src
##### END COMMENTS
### Define procedure "set_envVAR" to work around a module-unload-load bug (optional):
# Exported environment variables, which do not change their values,
# disappear after an automatic unload and load sequence within a module file.
# This is most likely a bug in the modules environment. Unchanged
# variables are not re-exported upon load in an automatic unload-load
# sequence. The workaround is to "unset" the variable in the right moment
# (so the load of the unload-load-chain can set the variable again)
# and, in addition, to set the environment variable explicitly.
proc set_envVAR {envVAR VARcontent} {
# Use global function env:
global env
# Unset envVAR explicitly:
if { [info exists env($envVAR)] } {
catch {
unset env($envVAR)
}
}
# Call overloaded module command setenv:
setenv $envVAR $VARcontent
# Set envVAR explicitly:
set env([set envVAR]) $VARcontent
}
### Define fallback values and source global functions and variables from modulerc file:
set first_level_support_email "bwhpc (at) uni-konstanz.de"
set global_modulerc_file "/opt/bwhpc/common/etc/modules.conf"
if { [ file readable "${global_modulerc_file}" ] } {
source "${global_modulerc_file}"
}
# Override global first_level_support_email email:
set first_level_support_email "bwhpc (at) uni-konstanz.de"
### Define version, base_dir and whatis entry:
set version "2.31.8_impi"
set base_dir "/opt/bwhpc/common/bio/maker/$version"
module-whatis "maker $version is a portable and configurable genome annotation pipeline"
### Define convenience environment variables:
set maker_version "${version}"
set maker_home "${base_dir}"
set maker_exa_dir "${base_dir}/bwhpc-examples"
set maker_bin_dir "${base_dir}/bin"
set maker_bpr_url "http://www.bwhpc-c5.de/wiki/index.php/Maker"
set maker_blast_bin "${base_dir}/exe/blast/bin"
set maker_exonerate_bin "${base_dir}/exe/exonerate/bin"
set maker_snap_bin "${base_dir}/exe/snap"
set maker_repeatmasker_bin "${base_dir}/exe/RepeatMasker"
set_envVAR MAKER_VERSION "${maker_version}"
set_envVAR MAKER_HOME "${maker_home}"
set_envVAR MAKER_EXA_DIR "${maker_exa_dir}"
set_envVAR MAKER_BIN_DIR "${maker_bin_dir}"
set_envVAR MAKER_BPR_URL "http://www.bwhpc-c5.de/wiki/index.php/Maker"
set_envVAR MAKER_BLAST_BIN "${maker_blast_bin}"
set_envVAR MAKER_EXONERATE_BIN "${maker_exonerate_bin}"
set_envVAR MAKER_SNAP_BIN "${maker_snap_bin}"
set_envVAR MAKER_REPEATMASKER_BIN "${maker_repeatmasker_bin}"
### Update path environments and define application specific environment vars:
prepend-path PATH "${maker_bin_dir}"
prepend-path PATH "${maker_blast_bin}"
prepend-path PATH "${maker_exonerate_bin}"
prepend-path PATH "${maker_snap_bin}"
prepend-path PATH "${maker_repeatmasker_bin}"
### Define help text:
proc ModulesHelp { } {
global maker_home maker_exa_dir first_level_support_email maker_bpr_url maker_blast_bin maker_exonerate_bin maker_snap_bin maker_repeatmasker_bin
puts stderr "
DESCRIPTION
MAKER is a portable and easily configurable genome annotation pipeline.
Its purpose is to allow smaller eukaryotic and prokaryotic genome projects
to independently annotate their genomes and to create genome databases.
MAKER identifies repeats, aligns ESTs and proteins to a genome, produces
ab-initio gene predictions and automatically synthesizes these data into
gene annotations having evidence-based quality values.
EXTERNAL PROGRAMS
BLAST Command Line Applications (2.2.28)
$maker_blast_bin
EXONERATE Generic Sequence Comparison Tool (2.2.0)
$maker_exonerate_bin
SNAP Semi-HMM-based Nucleic Acid Parser (version 2006-07-28)
$maker_snap_bin
REPEATMASKER Mask repetitive DNA (open-4.0.5)
$maker_repeatmasker_bin
DOCUMENTATION
* Main MAKER site
http://www.yandell-lab.org/software/maker.html
* MAKER User Guide, Documents, Wiki-Article
http://weatherby.genetics.utah.edu/MAKER/wiki/index.php
$maker_home/README
* MAKER Download
http://yandell.topaz.genetics.utah.edu/maker_downloads/CE63/461D/BD41/4510A183DC4571D5EA619E459572/maker-2.31.8.tgz
* bwHPC examples and a moab example script
$maker_exa_dir
* bwHPC Best Practices Repository
$maker_bpr_url
CITATION
n./a.
+-------------------------------------------+
| THIS IS THE IMPI-VERSION BUILT WITH INTEL |
+-------------------------------------------+
In case of problems, please contact: $first_level_support_email
This module is available for all users.
"
}
module load compiler/intel/16.0
module load mpi/impi/5.1.3-intel-16.0
conflict bio/maker
-------------- next part --------------
#!/bin/bash
#MSUB -N maker_impi-job
#MSUB -j oe
#MSUB -o $(JOBNAME).$(JOBID)
#MSUB -m ae
#MSUB -l nodes=1:ppn=1
#MSUB -l mem=20gb
#MSUB -l walltime=10:00:00
#
start=$(date +%s)
echo " "
echo "### Setting up shell environment ..."
echo " "
# if test -e "/etc/profile"; then source "/etc/profile"; fi;
if test -e "$HOME/.bash_profile"; then source "$HOME/.bash_profile"; fi;
unset LANG; export LC_ALL="C"; export MKL_NUM_THREADS=1; export OMP_NUM_THREADS=1
export USER=${USER:=`logname`}
export MOAB_JOBID=${MOAB_JOBID:=`date +%s`}
export MOAB_SUBMITDIR=${MOAB_SUBMITDIR:=`pwd`}
export MOAB_JOBNAME=${MOAB_JOBNAME:=`basename "$0"`}
export MOAB_JOBNAME=$(echo "${MOAB_JOBNAME}" | sed 's/[^a-zA-Z0-9._-]/_/g')
export MOAB_NODECOUNT=${MOAB_NODECOUNT:=1}
export MOAB_PROCCOUNT=${MOAB_PROCCOUNT:=1}
ulimit -s 200000
echo " "
echo "### Printing basic job infos to stdout ..."
echo " "
echo "START_TIME = `date +'%y-%m-%d %H:%M:%S %s'`"
echo "HOSTNAME = ${HOSTNAME}"
echo "USER = ${USER}"
echo "MOAB_JOBNAME = ${MOAB_JOBNAME}"
echo "MOAB_JOBID = ${MOAB_JOBID}"
echo "MOAB_SUBMITDIR = ${MOAB_SUBMITDIR}"
echo "MOAB_NODECOUNT = ${MOAB_NODECOUNT}"
echo "MOAB_PROCCOUNT = ${MOAB_PROCCOUNT}"
echo "SLURM_NODELIST = ${SLURM_NODELIST}"
echo "PBS_NODEFILE = ${PBS_NODEFILE}"
if test -f "${PBS_NODEFILE}"; then
echo "PBS_NODEFILE (begin) ---------------------------------"
NO_NODES=$(wc -l < ${PBS_NODEFILE})
cat "${PBS_NODEFILE}"
sort -u $PBS_NODEFILE > mpd.hosts
echo "PBS_NODEFILE (end) -----------------------------------"
else
NO_NODES=1
fi
echo " "
echo "### Creating TMP_WORK_DIR directory and changing to it ..."
echo " "
# Using "$TMPDIR" is strongly recommended for Maker jobs
# since these jobs can create a lot of disk IO.
# NEVER EVER calculate in your home directory.
TMP_BASE_DIR="$TMPDIR"
JOB_WORK_DIR="${MOAB_JOBNAME}.${MOAB_JOBID##*.}.$(date +%y%m%d_%H%M%S)"
TMP_WORK_DIR="${TMP_BASE_DIR}/${JOB_WORK_DIR}"
echo "JOB_WORK_DIR = ${JOB_WORK_DIR}"
echo "TMP_BASE_DIR = ${TMP_BASE_DIR}"
echo "TMP_WORK_DIR = ${TMP_WORK_DIR}"
mkdir -vp "${TMP_WORK_DIR}"
cd "${TMP_WORK_DIR}"
echo " "
echo "### Loading MAKER module:"
echo " "
module load bio/maker/2.31.8_impi
[ "$MAKER_VERSION" ] || { echo "ERROR: Failed to load module 'bio/maker/2.31.8_impi'."; exit 1; }
echo "MAKER_VERSION = $MAKER_VERSION"
module list
echo " "
echo "### Copying input examples files for job:"
echo " "
cp -v ${MAKER_EXA_DIR}/*.{fasta,ctl} .
cp -Rv ${MAKER_EXA_DIR}/_Inline .
sleep 2
echo " "
echo "### Display internal Maker/bwHPC environments..."
echo " "
echo "MAKER_BIN_DIR = ${MAKER_BIN_DIR}"
echo "MAKER_EXA_DIR = ${MAKER_EXA_DIR}"
echo ""
echo " "
echo "### Runing Maker example"
echo " "
#
# Environmental variables to set:
#
export LD_PRELOAD=${MPI_LIB_DIR}/libmpi.so
echo "LD_PRELOAD=${MPI_LIB_DIR}/libmpi.so"
export OMPI_MCA_mpi_warn_on_fork=0
echo "OMPI_MCA_mpi_warn_on_fork=0"
export I_MPI_CPUINFO="proc"
echo "I_MPI_CPUINFO=proc"
export I_MPI_PMI_LIBRARY=${MPI_LIB_DIR}/libpmi.so
echo "I_MPI_PMI_LIBRARY=${MPI_LIB_DIR}/libpmi.so"
export I_MPI_PIN_DOMAIN=node
echo "I_MPI_PIN_DOMAIN=node"
# otherwise MAKER calls to BLAST and other programs which are parallelized independent
# of MPI may not work
export I_MPI_FABRICS='shm:tcp'
echo "I_MPI_FABRICS=shm:tcp"
# avoid potential complication with OpenFabrics libraries (they block system calls because of
# how they use registered memory, i.e. MAKER calling BLAST would fail)
# set to eth0 if you don’t have an infiniband over
# ip inerface (required because of the above I_MPI_FABRICS change)
export I_MPI_HYDRA_IFACE=ib0
echo "I_MPI_HYDRA_IFACE=ib0"
# The –c nobrs. option will try and run the BLAST jobs on
# nobrs. cpus on the same machine, but will not run via MPI.
# -nolocal # wtf?
# mpiexec -n $SLURM_NPROCS -env I_MPI_DEBUG 5 ./hello
echo "starting mpiexec..."
mpiexec -nc 1 maker
echo "### Cleaning up files ... removing unnecessary scratch files ..."
echo " "
# rm -fv
sleep 3 # Sleep some time so potential stale nfs handles can disappear.
echo " "
echo "### Compressing results and copying back result archive ..."
echo " "
cd "${TMP_BASE_DIR}"
mkdir -vp "${MOAB_SUBMITDIR}" # if user has deleted or moved the submit dir
echo "Creating result tgz-file '${MOAB_SUBMITDIR}/${JOB_WORK_DIR}.tgz' ..."
tar -zcvf "${MOAB_SUBMITDIR}/${JOB_WORK_DIR}.tgz" "${JOB_WORK_DIR}" \
|| { echo "ERROR: Failed to create tgz-file. Please cleanup TMP_WORK_DIR '$TMP_WORK_DIR' on host '$HOSTNAME' manually (if not done automatically by queueing system)."; exit 102; }
# Remarks:
# * The resulting tgz file is copied back to the submit directory.
# The name of the tgz file looks similar too
# "bwunicluster-maker-example.moab.275.110528_101755.tgz"
echo " "
echo "### Final cleanup: Remove TMP_WORK_DIR ..."
echo " "
rm -rvf "${TMP_WORK_DIR}"
echo "END_TIME = `date +'%y-%m-%d %H:%M:%S %s'`"
end=$(date +%s)
echo " "
echo "### Calculate duration ..."
echo " "
diff=$[end-start]
if [ $diff -lt 60 ]; then
echo "Runtime (approx.): '$diff' secs"
elif [ $diff -ge 60 ]; then
echo 'Runtime (approx.): '$[$diff / 60] 'min(s) '$[$diff % 60] 'secs'
fi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5055 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20170224/cdae8f4f/attachment-0003.p7s>
More information about the maker-devel
mailing list