[maker-devel] Maker regularly fails and just lost all of the previous work!

[Carson Holt]

MAKER should automatically try and salvage things on restart (that is the purpose of the checkpoint files). You can set clean_try=1 if you want. It will then delete failed contigs before retrying on any failure.

—Carson



> On Oct 17, 2016, at 2:09 PM, Mark Ebbert <mark.ebbert at gmail.com> wrote:
> 
> 
> Thanks Carson,
> 
> I’ve been restarting it using the same commands several times in a row. Unless that ‘find’ command has the potential to modify any important files, then I don’t think I modified anything. All I ran was:
> 
> “find . -name *.NFSLock* -exec rm {} \;”
> “sbatch maker.slurm”
> 
> I’m inclined to nuke it all and start over. Is it possible to salvage previous work, or is it all gone?
> 
> Mark T. W. Ebbert
> Please note my new email address: mark.ebbert at gmail.com <mailto:mark.ebbert at gmail.com>
> 
> On Mon, Oct 17, 2016 at 12:35 PM Carson Holt <Carson Holt  <mailto:Carson Holt <carsonhh at gmail.com>>> wrote:
> If you made a change that affects downstream steps, MAKER erases affected intermediate files, and recalculates. It’s possible that you erased required checkpoiunt files, so MAKER thinks a change has been made that requires some things to be rerun.
> 
> Also if the STDERR is too big. Set -quiet or -qq (really quiet) on the command line. 
> 
> In general the error you see at the end is not the cause. The real error is further back in the log. MAKER tries to recover/retry, so the final failure you see is basically MAKER saying, I give up. But the original cause is further back in the log often behind the output of other MAKER threads that are writing to the log simultaneously. Iif you have 100 CPUs writing to the same output log, you may bury the real error behind the output of other threads (the log is not truly linear), so you have to look further back.
> 
> If you use the beta, you can also specify -nolock, but be warned that the locks themselves are important to avoid file corruption (i.e. you accidentally launch MAKER twice).
> 
> —Carson
> 
> 
>> On Oct 13, 2016, at 3:57 PM, Mark Ebbert <mark.ebbert at gmail.com <mailto:mark.ebbert at gmail.com>> wrote:
>> 
>> 
>> Hi,
>> 
>> I’ve been working with maker for several months off and on with varying success. It worked great the first time I ran it, but ever since, it fails every run without any specific errors. Just says that one of the processes failed. I’ve been limping along by just running the following command to remove any locks and re-starting: “find . -name *.NFSLock* -exec rm {} \;”
>> 
>> This has been working, but for some reason maker started over from the beginning and lost all of the previous work! I don’t even know where to start interrogating. Should I nuke the whole maker directory structure and start from scratch? Maybe something got corrupted??
>> 
>> I already deleted the log files before I realized maker started over because the log files get way too big.
>> 
>> I really appreciate your help!
>> 
>> Mark T. W. Ebbert
>> _______________________________________________
>> maker-devel mailing list
>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20161017/43cdcd52/attachment-0002.html>