<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><span style="background-color: rgb(254, 254, 254);" class=""><br class=""></span><div><br class=""><blockquote type="cite" class=""><div class="">On Feb 4, 2019, at 11:39 AM, DECKER, KEITH F [AG/1005] <<a href="mailto:keith.decker@bayer.com" class="">keith.decker@bayer.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks,<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Do you have metrics on how MAKER performs on annotating a single chromosome on a single machine? For example, will I see anything close to 16X speed-up using a 16 core machine, and does performance improvement saturate at a certain number of cores?<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">-Keith<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Carson Holt <<a href="mailto:carsonhh@gmail.com" style="color: purple; text-decoration: underline;" class="">carsonhh@gmail.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, February 4, 2019 at 12:33 PM<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"DECKER, KEITH F [AG/1005]" <<a href="mailto:keith.decker@bayer.com" style="color: purple; text-decoration: underline;" class="">keith.decker@bayer.com</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:maker-devel@yandell-lab.org" style="color: purple; text-decoration: underline;" class="">maker-devel@yandell-lab.org</a>" <<a href="mailto:maker-devel@yandell-lab.org" style="color: purple; text-decoration: underline;" class="">maker-devel@yandell-lab.org</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [maker-devel] MAKER on AWS<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">You can try and stand up a cluster inside AWS, or like you said just start independent instances each with their own piece of the total dataset. There is a tools called fasta_tool inside of maker that makes it easy to split up the dataset into equal sized chunks.<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Alternatively, CyVerse has set up an interesting MAKER wrapper (WQ-MAKER) that launches multiple cloud instances for MAKER and handles data chunking for you (they’ve been using XSEDE cloud resources through the NSF) —><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://ccl.cse.nd.edu/research/papers/maker-service-ic2e2018.pdf" style="color: purple; text-decoration: underline;" class="">http://ccl.cse.nd.edu/research/papers/maker-service-ic2e2018.pdf</a><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Here is an example of an external project using their setup —> <a href="http://onsnetwork.org/kubu4/2018/08/07/genome-annotation-olympia-oyster-genome-using-wq-maker-instance-on-jetstream/" style="color: purple; text-decoration: underline;" class="">http://onsnetwork.org/kubu4/2018/08/07/genome-annotation-olympia-oyster-genome-using-wq-maker-instance-on-jetstream/</a><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">—Carson<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Feb 4, 2019, at 11:09 AM, DECKER, KEITH F [AG/1005] <<a href="mailto:keith.decker@bayer.com" style="color: purple; text-decoration: underline;" class="">keith.decker@bayer.com</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I would like to evaluate the use of MAKER on AWS, but I am unsure what the best approach to parallelization would be.<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I found this old post on STARCLUSTER,<span class="apple-converted-space"> </span><a href="http://efish.integrativebiology.msu.edu/2015/02/10/annotate.html" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">http://efish.integrativebiology.msu.edu/2015/02/10/annotate.html</span></a><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">but my understanding is that STARCLUSTER and its successors (cfncluster and parallel cluster) can be challenging to set up and use. <span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">So my questions are<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">1. Has anyone had recent success running MAKER on cfncluster or parallel cluster in AWS?<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2. Would it be reasonable to just split up N chromosomes across N ECS instances and collect the results at the end? If so, does it make sense to run each chromosome level annotation on for example an m4.16xlarge instance with 64 cores and 256 GB of RAM? Or is there a maximum number of cores at which the benefits from parallelization saturate?<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks and sorry for the long question<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Keith<span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Helvetica;" class=""><br style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><br class=""></span><o:p class=""></o:p></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 9pt; color: blue;" class=""><o:p class=""> </o:p></span></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 9pt; color: blue;" class="">This system contains confidential and copyrighted information. Access to the system is limited to users only and only for approved business purposes.<o:p class=""></o:p></span></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 9pt; color: blue;" class="">Anyone obtaining access to and using this system acknowledges that all information on this system including but not limited to electronic mail, word processing, directories and files, constitutes private property belonging to the Company.<o:p class=""></o:p></span></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 9pt; color: blue;" class="">Anyone using of viewing this system is further advised that the use of this system may be recorded and the information contained herein may be monitored, retrieved and reviewed if, in the Company’s sole discretion there is a business reason to do so.<o:p class=""></o:p></span></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 9pt; color: blue;" class="">If improper activity or use is suspected, all available information may be used by the Company for possible disciplinary action, prosecution, civil claim or any remedy or lawful purpose.<o:p class=""></o:p></span></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 9pt; color: blue;" class=""><o:p class=""> </o:p></span></pre><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Helvetica;" class=""><br class="">_______________________________________________<br class="">maker-devel mailing list<br class=""></span><a href="mailto:maker-devel@box290.bluehost.com" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica; color: rgb(149, 79, 114);" class="">maker-devel@box290.bluehost.com</span></a><span style="font-size: 9pt; font-family: Helvetica;" class=""><br class=""></span><a href="http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica; color: rgb(149, 79, 114);" class="">http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org</span></a><o:p class=""></o:p></div></div></blockquote></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div></div><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New"; caret-color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font color="blue" class="">
This system contains confidential and copyrighted information. Access to the system is limited to users only and only for approved business purposes.
Anyone obtaining access to and using this system acknowledges that all information on this system including but not limited to electronic mail, word processing, directories and files, constitutes private property belonging to the Company.
Anyone using of viewing this system is further advised that the use of this system may be recorded and the information contained herein may be monitored, retrieved and reviewed if, in the Company’s sole discretion there is a business reason to do so.
If improper activity or use is suspected, all available information may be used by the Company for possible disciplinary action, prosecution, civil claim or any remedy or lawful purpose.
</font></pre><br class="Apple-interchange-newline"></div></blockquote></div><br class=""></div></body></html>