[maker-devel] Possible to redirect maker output?

[Carson Holt]

We run on a standard cluster.  We have traditional NFS as well as more advanced Lustre options for shared storage. Each node has both locally mounted disk and in memory storage available (I never use the in memory storage though because MAKER requires a lot of temporary storage).

I run using OpenMPI (it scales better than MPICH2 - also MAKER is incompatible with MVAPICH2 because of a known registered memory defect in that MPI flavor). We use the SLURM scheduler although previously we had PBS. I usually run job sizes of between 100 and 200 CPU cores (10 to 20 nodes). We have mixed node types of 12, 16, 20. and 24 core nodes.

I always set TMP= to a locally mounted disk (never NFS or RAM disk). The working directory is always NFS or Lustre.

I've also run under a similar configuration on the TACC and XSEDE clusters (https://www.xsede.org <https://www.xsede.org/>). They use SLURM and previously SGE for their scheduler. I’ve been able to run on 600 plus CPU cores per job there, but I get better efficiency with multiple jobs at ~200 CPU cores (communication overhead gets too high for a single root process to handle effectively above 200 cores).

MAKER will need ~2 Gb of RAM for every core you give it with MPI.

—Carson



> On Mar 3, 2016, at 4:01 AM, Florian <fdolze at students.uni-mainz.de> wrote:
> 
> Hello Carson,
> 
> May I ask on what kind of hardware setup you guys are running MAKER?
> 
> I cant seem to get this running performantly on our cluster. There are usually only 2-3 cores running on 100% and the rest is idle waiting (I THINK due to I/O blockage but I'm not sure). Any ideas how I could find the cause for this problem?
> 
> I attached a screenshot of the node status for the first hour of the last MAKER run if this is any help.
> 
> On 29.02.2016 20:09, Carson Holt wrote:
>> You can try setting TMP= in the control files to a RAM disk location (You will need a lot of RAM though, perhaps 500Gb). Even then some components used by MAKER may not function properly with tmpfs, but you can try. If it doesn’t work you’ll get an error. The main output directory on the other hand must be globally accessible to all nodes if working with MPI, and a RAM disk will only exist and be accessible on a single node (even though a directory with the same name may exists on multiple nodes, they will actually be separate and distinct locations, i.e. /dev/shm).
>> 
>> —Carson
>> 
>> 
>>> On Feb 26, 2016, at 7:16 AM, Florian<fdolze at students.uni-mainz.de>  wrote:
>>> 
>>> Hi all,
>>> 
>>> I am trying to run maker on a cluster (2 nodes with 64 cores each), to speed things up I copied all input files to a ramdisk to reduce I/O time, but all subsequent results are still written to hdd.
>>> 
>>> Is there a way I can tell maker to write the maker.results files to ramdisk (or generally any other directory than the current working dir) too? (are they actually used for the current run or are only files in the temp files location used?)
>>> 
>>> Is anybody experienced with running maker on a similar setup and could tell me how you are handling this?
>>> 
>>> 
>>> thanks,
>>> Florian
>>> 
>>> _______________________________________________
>>> maker-devel mailing list
>>> maker-devel at box290.bluehost.com
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
> 
> <Screenshot from 2016-03-03 11:35:41.png>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20160303/260b1232/attachment-0003.html>