<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<span id="mailbox-conversation"><div>Hi Everyone,</div>
<div><br></div>
<div>I’m attempting to run Maker on Amazon EC2 using MIT’s starcluster— I’ve started a 200 node cluster, and enabled MPICH2 (Starcluster by default uses OpenMPI). I plan on documenting this setup once I’ve figured out how to run things reliably.</div>
<div><br></div>
<div>I’m having a persistent issue where something fails on one of the nodes, and std error is flooded with:</div>
<div><br></div>
<div>
<div>examining contents of the fasta file and run log</div>
<div>[67] ERROR: could not make datastore directory</div>
<div>[67] --> rank=67, hostname=node067</div>
<div>[67] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[67] ERROR: Chunk failed at level:0, tier_type:0</div>
<div id="mb-reply">[67] FAILED CONTIG:Scaffold261</div>
<div id="mb-reply"><br></div>
<div id="mb-reply">This error repeats for each “next” scaffold for some time. When I go back to find the “source” of the error in the log, the following is the first error message on that node:</div>
<div id="mb-reply"><br></div>
<div id="mb-reply">
<div>67] #-------------------------------#</div>
<div>[67] deleted:-60 hits</div>
<div>[67] collecting blastx reports</div>
<div>[67] ERROR: Could not colapse BLAST reports</div>
<div>[67] at /root/maker/bin/../lib/GI.pm line 2524 thread 1.</div>
<div>[67] GI::combine_blast_report(FastaChunk=HASH(0x108e1a90), ARRAY(0x1b874938), ARRAY(0xf127ad8), runlog=HASH(0x4d54ed8)) called at /root/maker/bin/../lib/Process/MpiChunk.pm line 2760 thread 1</div>
<div>[67] Process::MpiChunk::__ANON__() called at /root/maker/bin/../lib/Error.pm line 415 thread 1</div>
<div>[67] eval {...} called at /root/maker/bin/../lib/Error.pm line 407 thread 1</div>
<div>[67] Error::subs::try(CODE(0x1514eb00), HASH(0x9cbeb568)) called at /root/maker/bin/../lib/Process/MpiChunk.pm line 4215 thread 1</div>
<div>[67] Process::MpiChunk::_go(Process::MpiChunk=HASH(0x13976308), "run", HASH(0x12e04268), 9, 3) called at /root/maker/bin/../lib/Process/MpiChunk.pm line 341 thread 1</div>
<div>[67] Process::MpiChunk::run(Process::MpiChunk=HASH(0x13976308), 67) called at /root/maker/bin/maker line 1457 thread 1</div>
<div>[67] main::node_thread("/mnt/data/paramormyrops_new_annotation/supercontigs.maker.out"...) called at /usr/local/lib/perl/5.14.2/forks.pm line 799 thread 1</div>
<div>[67] eval {...} called at /usr/local/lib/perl/5.14.2/forks.pm line 799 thread 1</div>
<div>[67] threads::new("threads", CODE(0x3dc5b38), "/mnt/data/paramormyrops_new_annotation/supercontigs.maker.out"...) called at /root/maker/bin/maker line 917 thread 1</div>
<div>[67] --> rank=67, hostname=node067</div>
<div>[67] ERROR: Failed while collecting blastx reports</div>
<div>[67] ERROR: Chunk failed at level:9, tier_type:3</div>
<div>[67] FAILED CONTIG:Scaffold66</div>
<div>[67] </div>
<div>[67] ERROR: Chunk failed at level:4, tier_type:0</div>
<div id="mb-reply">[67] FAILED CONTIG:Scaffold66</div>
<div id="mb-reply"><br></div>
<div id="mb-reply"><br></div>
<div id="mb-reply">I’ve attempted to ignore the error to see if things will proceed on the other 199 processors. When I returned to the “master” node after the evening, Maker keeps repeating the same error code over and over (same scaffold):</div>
<div id="mb-reply">
<div>] examining contents of the fasta file and run log</div>
<div>[67] ERROR: could not make datastore directory</div>
<div>[67] --> rank=67, hostname=node067</div>
<div>[67] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[67] ERROR: Chunk failed at level:0, tier_type:0</div>
<div id="mb-reply">[67] FAILED CONTIG:Scaffold1589</div>
<div id="mb-reply"><br></div>
<div id="mb-reply">I stop the job, and restart, and after only a few minutes of running, the same error is reported, this time on a new scaffold. Strangely here, the error is reported in the MPI tag of node001, but the error originates at node137:</div>
<div id="mb-reply"><br></div>
<div id="mb-reply">
<div>ERROR: Could not colapse BLAST reports</div>
<div>[1] at /root/maker/bin/../lib/GI.pm line 2524.</div>
<div>[1] GI::combine_blast_report(FastaChunk=HASH(0xf4aa9b8), ARRAY(0xf628f90), ARRAY(0x325fea78), runlog=HASH(0x133cc8e8)) called at /root/maker/bin/../lib/Process/MpiChunk.pm line 2760</div>
<div>[1] Process::MpiChunk::__ANON__() called at /root/maker/bin/../lib/Error.pm line 415</div>
<div>[1] eval {...} called at /root/maker/bin/../lib/Error.pm line 407</div>
<div>[1] Error::subs::try(CODE(0x352c9b8), HASH(0xdab3b690)) called at /root/maker/bin/../lib/Process/MpiChunk.pm line 4215</div>
<div>[1] Process::MpiChunk::_go(Process::MpiChunk=HASH(0x3545d90), "run", HASH(0x30aa710), 9, 3) called at /root/maker/bin/../lib/Process/MpiChunk.pm line 341</div>
<div>[1] Process::MpiChunk::run(Process::MpiChunk=HASH(0x3545d90), 137) called at /root/maker/bin/maker line 979</div>
<div>[1] --> rank=137, hostname=node137</div>
<div>[1] ERROR: Failed while collecting blastx reports</div>
<div>[1] ERROR: Chunk failed at level:9, tier_type:3</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] ERROR: Chunk failed at level:4, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div>[1] FAILED CONTIG:Scaffold249</div>
<div>[1]</div>
<div>[1] examining contents of the fasta file and run log</div>
<div>[1] ERROR: could not make datastore directory</div>
<div>[1] --> rank=1, hostname=node001</div>
<div>[1] ERROR: Failed while examining contents of the fasta file and run log</div>
<div>[1] ERROR: Chunk failed at level:0, tier_type:0</div>
<div id="mb-reply">[1] FAILED CONTIG:Scaffold249</div>
<div id="mb-reply"><br></div>
<div id="mb-reply">I’d appreciate any guidance as how best to diagnose this error!</div>
<div id="mb-reply"><br></div>
<div id="mb-reply">Many thanks,</div>
<div id="mb-reply">Jason Gallant</div>
</div>
<div id="mb-reply"><br></div>
<div id="mb-reply"><br></div>
</div>
<div id="mb-reply"><br></div>
</div>
</div></span><div class="mailbox_signature">
<br>—<br>Dr. Jason R. Gallant<div>Assistant Professor</div>
<div>Room 38 Natural Sciences<br><div>Department of Zoology</div>
<div>Michigan State University</div>
<div>East Lansing, MI 48824</div>
<div>jgallant@msu.edu</div>
</div>
<div>office: 517-884-7756</div>
</div>
</body></html>