<div dir="ltr">Hi Carson,<div><br></div><div>Unfortunately, it looks like I am running MPICH version 3.0.4, so that doesn't explain the problem. I regenerated the error for you:</div><div><br></div><div>





<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">STATUS: Parsing control files...</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">STATUS: Processing and indexing input FASTA files...</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">STATUS: Setting up database for any GFF3 input...</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">A data structure will be created for you at:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/home/ubuntu/PGScratch/scaff1/scaff0.maker.output/scaff0_datastore</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">To access files for individual sequences use the datastore index:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/home/ubuntu/PGScratch/scaff1/scaff0.maker.output/scaff0_master_datastore_index.log</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">STATUS: Now running MAKER...</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">examining contents of the fasta file and run log</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">--Next Contig--</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Processing run.log file...</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MAKER WARNING: The file scaff0.maker.output/scaff0_datastore/0B/6F/PGA_scaffold0__8_contigs__length_34808196//theVoid.PGA_scaffold0__8_contigs__length_34808196/PGA_scaffold0__8_contigs__length_34808196.abinit_masked.1.BUSCO_hemistictus_3028031324.augustus</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">did not finish on the last run and must be erased</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MAKER WARNING: The file scaff0.maker.output/scaff0_datastore/0B/6F/PGA_scaffold0__8_contigs__length_34808196//theVoid.PGA_scaffold0__8_contigs__length_34808196/PGA_scaffold0__8_contigs__length_34808196.abinit_masked.0.BUSCO_hemistictus_3028031324.augustus</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">did not finish on the last run and must be erased</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MAKER WARNING: The file scaff0.maker.output/scaff0_datastore/0B/6F/PGA_scaffold0__8_contigs__length_34808196//theVoid.PGA_scaffold0__8_contigs__length_34808196/PGA_scaffold0__8_contigs__length_34808196.abinit_masked.2.BUSCO_hemistictus_3028031324.augustus</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">did not finish on the last run and must be erased</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#---------------------------------------------------------------------</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Now retrying the contig!!</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SeqID: PGA_scaffold0__8_contigs__length_34808196</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Length: 34808196</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Tries: 2!!</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#---------------------------------------------------------------------</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">setting up GFF3 output and fasta chunks</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">preparing ab-inits</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">running<span class="gmail-Apple-converted-space">  </span>augustus.</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#--------- command -------------#</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Widget::augustus:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/home/ubuntu/PGTools/Augustus/bin/augustus --species=BUSCO_hemistictus_3028031324 --UTR=off /tmp/maker_pzUjBq/PGA_scaffold0__8_contigs__length_34808196.abinit_masked.0 > /tmp/maker_pzUjBq/PGA_scaffold0__8_contigs__length_34808196.abinit_masked.0.BUSCO_hemistictus_3028031324.augustus</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#-------------------------------#</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/home/ubuntu/PGTools/Augustus/bin/augustus: ERROR</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Invalid nucleotide '' encountered.</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/home/ubuntu/PGTools/Augustus/bin/augustus: ERROR</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Invalid nucleotide '?' encountered.</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ERROR: Augustus failed</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">--> rank=NA, hostname=ip-172-31-14-148</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ERROR: Failed while preparing ab-inits</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ERROR: Chunk failed at level:0, tier_type:2</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">FAILED CONTIG:PGA_scaffold0__8_contigs__length_34808196</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ERROR: Chunk failed at level:4, tier_type:0</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">FAILED CONTIG:PGA_scaffold0__8_contigs__length_34808196</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">examining contents of the fasta file and run log</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">--Next Contig--</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Processing run.log file...</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MAKER WARNING: The file scaff0.maker.output/scaff0_datastore/0B/6F/PGA_scaffold0__8_contigs__length_34808196//theVoid.PGA_scaffold0__8_contigs__length_34808196/PGA_scaffold0__8_contigs__length_34808196.abinit_masked.0.BUSCO_hemistictus_3028031324.augustus</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">did not finish on the last run and must be erased</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Maker is now finished!!!</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Start_time: 1615317386</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">End_time: <span class="gmail-Apple-converted-space">  </span>1615317388</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Elapsed:<span class="gmail-Apple-converted-space">    </span>2</span></p></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 7:41 PM Carson Holt <<a href="mailto:carsonhh@gmail.com" target="_blank">carsonhh@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Hi Hayley,</div><div><br></div><div>Which flavor of MPI are you using? OpenMPI, MVAPICH2, MPICH, or Intel MPI?  There is a known issue with infiniband network cards that causes them to freeze on certain system calls, and I bet that is what you are seeing.  It is not a MAKER issue, but a limitation in order to allow performance benefits elsewhere with MPI.  You need to add extra command line flags to tell MPI to communicate a bit differently to get around it. You can find multiple threads with more info in the maker-devel archives here —> <a href="https://groups.google.com/g/maker-devel" target="_blank">https://groups.google.com/g/maker-devel</a></div><div><br></div><div><br></div><div>For OpenMPI you may need to add these flags to the mpiexec command line —> --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 --mca btl_openib_want_fork_support 1 --mca mpi_warn_on_fork 0</div><div><br></div><div>For intel MPI set the environmental variable I_MPI_FABRICS=shm:tcp before running mpiexec</div><div><br></div><div>For MPICH, no modifications should be needed (it doesn’t experience the issue).</div><div><br></div><div>For <font color="#000000"><span>MVAPICH2, there is no work around and you just need to use one of the other MPI flavors.</span></font></div><div><br></div><div><br></div><div>For the second failure, if you can regenerate it, send me the STDERR and I will take a look.</div><div><br></div><div>Thanks,</div><div>Carson</div><div><br></div><div><br></div><div><br></div><div><blockquote type="cite"><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>From:</b><span> </span>Hayley Mangelson <<a href="mailto:hayley@phasegenomics.com" style="color:rgb(149,79,114);text-decoration:underline" target="_blank">hayley@phasegenomics.com</a>><span> </span><br><b>Sent:</b><span> </span>Thursday, March 4, 2021 12:03 PM<br><b>To:</b><span> </span>AARON A DUFFY <<a href="mailto:aaron.duffy@utah.edu" style="color:rgb(149,79,114);text-decoration:underline" target="_blank">aaron.duffy@utah.edu</a>><br><b>Cc:</b><span> </span>Mary Wood <<a href="mailto:mary@phasegenomics.com" style="color:rgb(149,79,114);text-decoration:underline" target="_blank">mary@phasegenomics.com</a>><br><b>Subject:</b><span> </span>MAKER Commercial User -- Question<u></u><u></u></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Hi Aaron, <u></u><u></u></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Mary and I are bioinformaticians at Phase Genomics, and we are using the MAKER software (for which Phase purchased a licence) to annotate a couple of genomes. We are using the same workflow that I used for a previous project, but are having the same issues while working on separate projects. I am hoping you could help us out or direct us to someone who can help!<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Here is some background. I am trying to get it to run using<span> </span><i>mpiexec</i><span> </span>for parallelization. I let it run for about a week and all processes remained “active”, but no new output seemed to be generated. It was hung up on “prepare section files” for nearly the whole 1-week period. I canceled the run, then tried running again without parallelization (in the same folder as the failed run). I got an error about each contig failing, and something about Augustus. Unfortunately, I didn’t save the error message. I tried to regenerate the error, but it looks like I would probably need to completely rerun… this is the error message now (repeated for each contig), if I run in the same folder again.<br><br><i>The contig failed 2 times!!<br>Maker will not try again!!<br>The contig will be stored in a fasta file that you can use for debugging.<br>SeqID: PGA_scaffold4__8_contigs__length_31799528<br>Length: 31799528<br>FASTA: /home/ubuntu/PGScratch/chunk1/chunk1.maker.output/chunk1_datastore/10/91/PGA_scaffold4__8_contigs__length_31799528//PGA_scaffold4__8_contigs__length_31799528.died.fasta</i><br><br>Any thoughts about why<span> </span><i>mpiexec<span> </span></i>seems unable to finish? I tried a run on the test data using parallelization, and it completed in under a minute.<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Thanks!<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Hayley<br>--<span> </span><u></u><u></u></div><div><div><div><div><div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(102,102,102)">Hayley Mangelson</span><u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(102,102,102)">Bioinformatics Scientist</span><u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(102,102,102)">Phase Genomics, Inc</span></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><font color="#666666">Hayley Mangelson<br></font></div><div><font color="#666666">Bioinformatics Scientist</font></div><div><font color="#666666">Phase Genomics, Inc</font></div></div></div></div></div></div>