[maker-devel] Maker regularly fails and just lost all of the previous work!

[Mark Ebbert]

Thanks Carson,

I’ve been restarting it using the same commands several times in a row. Unless that ‘find’ command has the potential to modify any important files, then I don’t think I modified anything. All I ran was:

“find . -name *.NFSLock* -exec rm {} \;”

“sbatch maker.slurm”

I’m inclined to nuke it all and start over. Is it possible to salvage previous work, or is it all gone?

Mark T. W. Ebbert

Please note my new email address: mark.ebbert at gmail.com

On Mon, Oct 17, 2016 at 12:35 PM Carson Holt

<
mailto:Carson Holt <carsonhh at gmail.com>
> wrote:

a, pre, code, a:link, body { word-wrap: break-word !important; }

If you made a change that affects downstream steps, MAKER erases affected intermediate files, and recalculates. It’s possible that you erased required checkpoiunt files, so MAKER thinks a change has been made that requires some things to be rerun.

Also if the STDERR is too big. Set -quiet or -qq (really quiet) on the command line. 

In general the error you see at the end is not the cause. The real error is further back in the log. MAKER tries to recover/retry, so the final failure you see is basically MAKER saying, I give up. But the original cause is further back in the log often behind the output of other MAKER threads that are writing to the log simultaneously. Iif you have 100 CPUs writing to the same output log, you may bury the real error behind the output of other threads (the log is not truly linear), so you have to look further back.

If you use the beta, you can also specify -nolock, but be warned that the locks themselves are important to avoid file corruption (i.e. you accidentally launch MAKER twice).

—Carson

On Oct 13, 2016, at 3:57 PM, Mark Ebbert <
mailto:mark.ebbert at gmail.com
> wrote:

Hi,

I’ve been working with maker for several months off and on with varying success. It worked great the first time I ran it, but ever since, it fails every run without any specific errors. Just says that one of the processes failed. I’ve been limping along by just running the following command to remove any locks and re-starting: “find . -name *.NFSLock* -exec rm {} \;”

This has been working, but for some reason maker started over from the beginning and lost all of the previous work! I don’t even know where to start interrogating. Should I nuke the whole maker directory structure and start from scratch? Maybe something got corrupted??

I already deleted the log files before I realized maker started over because the log files get way too big.

I really appreciate your help!

Mark T. W. Ebbert

_______________________________________________

maker-devel mailing list
mailto:maker-devel at box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20161017/5cb1ef08/attachment-0001.html>