[maker-devel] Hyperthreading

[Carson Holt]

One last thing to check if using CentOS or RedHat. I’ve seen it happen on a handful of clusters where transparent hugepages can create odd load issues and very high sys CPU usage under top (not just with maker but with BWA, GATK, and other programs that can have larger memory footprints). If using CentOS or RedHat, you may want to disable defrag for hugepages.

You do this on CentOS 6 to disable it (the process is similar on CentOS 7 and RedHat but you may have to google it) —> 

echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

—Carson


> On May 23, 2017, at 2:31 PM, admin at genome.arizona.edu wrote:
> 
> Carson Holt wrote on 05/23/2017 01:19 PM:
>> You can pull back to 128 would if you find that you are running low on RAM or have a high IO burden (both of which will double if you go from 128 to 256 even though CPU isn’t really doubling). Also MAKER per job performance plateaus at around 200 processes due to communication overhead. Above that threshold it is often useful to divide datasets into multiple separate jobs that can run simultaneously.
> 
> Yes, with '-n 192' we found the load on the cluster will initially go up to 360-380 but then continually decreases until maker is finished. Memory usage was very low during the processing (under 20%).
> 
> _______________________________________________
> maker-devel mailing list
> maker-devel at box290.bluehost.com
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org