[maker-devel] Maker-run with no clean finish on openMPI-cluster
Florian
fdolze at students.uni-mainz.de
Thu Feb 11 03:43:51 MST 2016
Hi all,
I am no expert for MPI so maybe this is something very trivial or maybe
not caused by MAKER at all but I'd be glad to have your thoughts on
this. I installed MAKER 2.31.8 with MPI support (openMPI 1.8.1) on our
cluster.
I ran maker with the options attached and the command in bsub_maker, and
I _think_ it worked fine. Here is the last output of maker:
running exonerate search.
#--------- command -------------#
Widget::exonerate::protein2genome:
/gpfs/fs1/cluster/Apps/bioinf/maker/2.31.8/exe/exonerate/bin/exonerate
-q /project/molgen/workbench_Florian/riparius_MAKER_v2/tmp_fil
es/maker_yZhQlA/49/sp%7CQ4JHE0%7CXB36_ORYSJ.for.114901-115619.49.fasta
-t /project/molgen/workbench_Florian/riparius_MAKER_v2/tmp_fil
es/maker_yZhQlA/49/scaffold299_size115619.114901-115619.49.fasta -Q
protein -T dna -m protein2genome --softmasktarget --percent 20 -
-showcigar >
/project/molgen/workbench_Florian/riparius_MAKER_v2/tmp_files/maker_yZhQlA/49/scaffold299_size115619.114901-115619.sp%7
CQ4JHE0%7CXB36_ORYSJ.p.exonerate
#-------------------------------#
cleaning blastx...
in cluster::shadow_cluster...
...finished clustering.
in cluster::shadow_cluster...
...finished clustering.
cleaning clusters....
total clusters:37 now processing 0
...processing 0 of 11
...processing 1 of 11
...processing 2 of 11
...processing 3 of 11
...
...processing 174 of 177
...processing 175 of 177
...processing 176 of 177
flattening protein clusters
prepare section files
Maker is now finished!!!
Start_time: 1454700985
End_time: 1455023070
Elapsed: 322085
but my cluster job didnt finish here, instead I got the following errors
until my runtime limit of 5 days was reached:
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
SIGTERM received
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
SIGTERM received
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
SIGTERM received
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
SIGTERM received
SIGTERM received
SIGTERM received
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
SIGTERM received
SIGTERM received
Perl exited with active threads:
1 running and unjoined
0 finished and unjoined
0 running and detached
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
[a0238:09542] *** Process received signal ***
[a0238:09542] Signal: Segmentation fault (11)
[a0238:09542] Signal code: Address not mapped (1)
[a0238:09542] Failing at address: 0xa80
[a0238:09542] [ 0] /lib64/libpthread.so.0(+0xf710)[0x2ba955727710]
[a0238:09542] [ 1]
/usr/lib64/perl5/CORE/libperl.so(Perl_csighandler+0x22)[0x2ba954715002]
[a0238:09542] [ 2] /lib64/libpthread.so.0(+0xf710)[0x2ba955727710]
[a0238:09542] [ 3] /lib64/libc.so.6(__poll+0x53)[0x2ba955a170d3]
[a0238:09542] [ 4]
/cluster/mpi/gcc_4.4.7/OpenMPI-1.8.1/lib/libopen-pal.so.6(+0x6cfca)[0x2ba955fb4fca]
[a0238:09542] [ 5]
/cluster/mpi/gcc_4.4.7/OpenMPI-1.8.1/lib/libopen-pal.so.6(opal_libevent2021_event_base_loop+0x221)[0x2ba955fabf11]
[a0238:09542] [ 6]
/cluster/mpi/gcc_4.4.7/OpenMPI-1.8.1/lib/libopen-rte.so.7(+0x376ae)[0x2ba955d076ae]
[a0238:09542] [ 7] /lib64/libpthread.so.0(+0x79d1)[0x2ba95571f9d1]
[a0238:09542] [ 8] /lib64/libc.so.6(clone+0x6d)[0x2ba955a208fd]
[a0238:09542] *** End of error message ***
Argument "ALRM" isn't numeric in exit at
/home/fdolze/perl5/lib/perl5/x86_64-linux-thread-multi/forks.pm line
2184.
SIGTERM received
SIGTERM received
...
maybe someone experienced something similar before or can give me some
hint if this is caused by my setup or by maker.
kind regards,
Florian Dolze
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20160211/78c171eb/attachment-0002.html>
-------------- next part --------------
#-----BLAST and Exonerate Statistics Thresholds
blast_type=ncbi+ #set to 'ncbi+', 'ncbi' or 'wublast'
pcov_blastn=0.8 #Blastn Percent Coverage Threhold EST-Genome Alignments
pid_blastn=0.85 #Blastn Percent Identity Threshold EST-Genome Aligments
eval_blastn=1e-10 #Blastn eval cutoff
bit_blastn=40 #Blastn bit cutoff
depth_blastn=0 #Blastn depth cutoff (0 to disable cutoff)
pcov_blastx=0.5 #Blastx Percent Coverage Threhold Protein-Genome Alignments
pid_blastx=0.4 #Blastx Percent Identity Threshold Protein-Genome Aligments
eval_blastx=1e-06 #Blastx eval cutoff
bit_blastx=30 #Blastx bit cutoff
depth_blastx=0 #Blastx depth cutoff (0 to disable cutoff)
pcov_tblastx=0.8 #tBlastx Percent Coverage Threhold alt-EST-Genome Alignments
pid_tblastx=0.85 #tBlastx Percent Identity Threshold alt-EST-Genome Aligments
eval_tblastx=1e-10 #tBlastx eval cutoff
bit_tblastx=40 #tBlastx bit cutoff
depth_tblastx=0 #tBlastx depth cutoff (0 to disable cutoff)
pcov_rm_blastx=0.5 #Blastx Percent Coverage Threhold For Transposable Element Masking
pid_rm_blastx=0.4 #Blastx Percent Identity Threshold For Transposbale Element Masking
eval_rm_blastx=1e-06 #Blastx eval cutoff for transposable element masking
bit_rm_blastx=30 #Blastx bit cutoff for transposable element masking
ep_score_limit=20 #Exonerate protein percent of maximal score threshold
en_score_limit=20 #Exonerate nucleotide percent of maximal score threshold
-------------- next part --------------
#-----Location of Executables Used by MAKER/EVALUATOR
makeblastdb=/cluster/Apps/bioinf/BLAST/2.2.28/bin/makeblastdb #location of NCBI+ makeblastdb executable
blastn=/cluster/Apps/bioinf/BLAST/2.2.28/bin/blastn #location of NCBI+ blastn executable
blastx=/cluster/Apps/bioinf/BLAST/2.2.28/bin/blastx #location of NCBI+ blastx executable
tblastx=/cluster/Apps/bioinf/BLAST/2.2.28/bin/tblastx #location of NCBI+ tblastx executable
formatdb= #location of NCBI formatdb executable
blastall= #location of NCBI blastall executable
xdformat= #location of WUBLAST xdformat executable
blasta= #location of WUBLAST blasta executable
RepeatMasker=/gpfs/fs1/cluster/Apps/bioinf/maker/2.31.8/bin/../exe/RepeatMasker/RepeatMasker #location of RepeatMasker executable
exonerate=/gpfs/fs1/cluster/Apps/bioinf/maker/2.31.8/bin/../exe/exonerate/bin/exonerate #location of exonerate executable
#-----Ab-initio Gene Prediction Algorithms
snap=/gpfs/fs1/cluster/Apps/bioinf/maker/2.31.8/bin/../exe/snap/snap #location of snap executable
gmhmme3=/project/molgen/Maker_additional_tools/genemark-4.32/gmhmme3 #location of eukaryotic genemark executable
gmhmmp= #location of prokaryotic genemark executable
augustus=/project/molgen/Maker_additional_tools/augustus-3.2.1/bin/augustus #location of augustus executable
fgenesh= #location of fgenesh executable
tRNAscan-SE=/project/molgen/Maker_additional_tools/tRNAscan/bin/tRNAscan-SE #location of trnascan executable
snoscan=/project/molgen/Maker_additional_tools/snoscan/bin/snoscan #location of snoscan executable
#-----Other Algorithms
probuild=/project/molgen/Maker_additional_tools/genemark-4.32/probuild #location of probuild executable (required for genemark)
-------------- next part --------------
#-----Genome (these are always required)
genome= /project/molgen/workbench_Florian/riparius_MAKER_v2/Crip_genome_v20_newHead.fa
organism_type=eukaryotic #eukaryotic or prokaryotic. Default is eukaryotic
#-----Re-annotation Using MAKER Derived GFF3
maker_gff= #MAKER derived GFF3 file
est_pass=0 #use ESTs in maker_gff: 1 = yes, 0 = no
altest_pass=0 #use alternate organism ESTs in maker_gff: 1 = yes, 0 = no
protein_pass=0 #use protein alignments in maker_gff: 1 = yes, 0 = no
rm_pass=0 #use repeats in maker_gff: 1 = yes, 0 = no
model_pass=0 #use gene models in maker_gff: 1 = yes, 0 = no
pred_pass=0 #use ab-initio predictions in maker_gff: 1 = yes, 0 = no
other_pass=0 #passthrough anyything else in maker_gff: 1 = yes, 0 = no
#-----EST Evidence (for best results provide a file for at least one)
est=/project/molgen/workbench_Florian/riparius_MAKER_v2/riparius_cDNA_formatedHeader.fa #set of ESTs or assembled mRNA-seq in fasta format
altest= #EST/cDNA sequence file in fasta format from an alternate organism
est_gff= #aligned ESTs or mRNA-seq from an external GFF3 file
altest_gff= #aligned ESTs from a closly relate species in GFF3 format
#-----Protein Homology Evidence (for best results provide a file for at least one)
protein=/project/molgen/workbench_Florian/riparius_MAKER_v2/uniprot_sprot.fasta #protein sequence file in fasta format (i.e. from mutiple oransisms)
protein_gff= #aligned protein homology evidence from an external GFF3 file
#-----Repeat Masking (leave values blank to skip repeat masking)
model_org=all #select a model organism for RepBase masking in RepeatMasker
rmlib=/project/molgen/workbench_Florian/riparius_MAKER_v2/20151208_Custom_Crip_repeat_library_final.fas #provide an organism specific repeat library in fasta format for RepeatMasker
repeat_protein=/gpfs/fs1/cluster/Apps/bioinf/maker/2.31.8/data/te_proteins.fasta #provide a fasta file of transposable element proteins for RepeatRunner
rm_gff= #pre-identified repeat elements from an external GFF3 file
prok_rm=0 #forces MAKER to repeatmask prokaryotes (no reason to change this), 1 = yes, 0 = no
softmask=1 #use soft-masking rather than hard-masking in BLAST (i.e. seg and dust filtering)
#-----Gene Prediction
snaphmm=/project/molgen/workbench_Florian/riparius_MAKER_v2/cegmasnap.hmm #SNAP HMM file
gmhmm=/project/molgen/workbench_Florian/riparius_MAKER_v2/gmhmm.mod #GeneMark HMM file
augustus_species=Riparius_Neu #Augustus gene prediction species model
fgenesh_par_file= #FGENESH parameter file
pred_gff= #ab-initio predictions from an external GFF3 file
model_gff= #annotated gene models from an external GFF3 file (annotation pass-through)
est2genome=1 #infer gene predictions directly from ESTs, 1 = yes, 0 = no
protein2genome=1 #infer predictions from protein homology, 1 = yes, 0 = no
trna=1 #find tRNAs with tRNAscan, 1 = yes, 0 = no
snoscan_rrna=/project/molgen/workbench_Florian/riparius_MAKER_v2/C.thummi_28S_rDNA_gene.fasta #rRNA file to have Snoscan find snoRNAs
unmask=0 #also run ab-initio prediction programs on unmasked sequence, 1 = yes, 0 = no
#-----Other Annotation Feature Types (features MAKER doesn't recognize)
other_gff= #extra features to pass-through to final MAKER generated GFF3 file
#-----External Application Behavior Options
alt_peptide=C #amino acid used to replace non-standard amino acids in BLAST databases
cpus=1 #max number of cpus to use in BLAST and RepeatMasker (not for MPI, leave 1 when using MPI)
#-----MAKER Behavior Options
max_dna_len=2100000 #length for dividing up contigs into chunks (increases/decreases memory usage)
min_contig=1 #skip genome contigs below this length (under 10kb are often useless)
pred_flank=200 #flank for extending evidence clusters sent to gene predictors
pred_stats=0 #report AED and QI statistics for all predictions as well as models
AED_threshold=1 #Maximum Annotation Edit Distance allowed (bound by 0 and 1)
min_protein=0 #require at least this many amino acids in predicted proteins
alt_splice=0 #Take extra steps to try and find alternative splicing, 1 = yes, 0 = no
always_complete=0 #extra steps to force start and stop codons, 1 = yes, 0 = no
map_forward=0 #map names and attributes forward from old GFF3 genes, 1 = yes, 0 = no
keep_preds=1 #Concordance threshold to add unsupported gene prediction (bound by 0 and 1)
split_hit=10000 #length for the splitting of hits (expected max intron size for evidence alignments)
single_exon=1 #consider single exon EST evidence when generating annotations, 1 = yes, 0 = no
single_length=250 #min length required for single exon ESTs if 'single_exon is enabled'
correct_est_fusion=0 #limits use of ESTs in annotation to avoid fusion genes
tries=2 #number of times to try a contig if there is a failure for some reason
clean_try=0 #remove all data from previous run before retrying, 1 = yes, 0 = no
clean_up=0 #removes theVoid directory with individual analysis files, 1 = yes, 0 = no
TMP=/project/molgen/workbench_Florian/riparius_MAKER_v2/tmp_files #specify a directory other than the system default temporary directory for temporary files
-------------- next part --------------
#!/bin/bash
#BSUB -n 128
#BSUB -q long
#BSUB -W 7200
#BSUB -o mogon_maker_MPIrun_5_feb.log
#BSUB -J riparius_makerMPI
#BSUB -app Reserve1G
mpiexec -mca btl ^openib -n 128 /project/molgen/Bio/maker-2.31.8_MPI-1.8.1/bin/maker -base maker_MPIrun3 -fix_nucleotides
More information about the maker-devel
mailing list