[maker-devel] questions on master_datastore_index.log file
Carson Holt
carsonhh at gmail.com
Wed Jan 10 12:05:03 MST 2018
The error is saying exactly that the file MAKER just created does not exist. The only time we ever see this is when using network mounted locations under heavy IO load. Most network storage options use asynchronous IO, which means the system returns success on file operation before they actually complete. So they can say they finished writing a file before it actually exist. So if you try and open it right away, it doesn’t really exist and everything fails. But that only happens if there is heavy IO (lots of things going on in that mount location). So if you are getting persitent failures you may want to try a different work directory, or get your IT to troubleshoot IO load in the directory you are using.
—Carson
> On Jan 9, 2018, at 11:10 AM, Qihua Liang <qlian003 at ucr.edu> wrote:
>
> Hi Carson,
>
> I just check with the system administrator and we think the disk space should be working fine. And actually I also ran another attempt with much fewer processors days ago and I am having the same issues.
>
> Maybe I will try renaming the contig names to see how the new attempt works? Or any other suggestions?
>
> Thank you!
> Qihua
>
>> On Jan 9, 2018, at 9:14 AM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>
>> Your contig names may create issues. Specifically the ‘;’ character, but you should also remove the ‘=‘ character. However, I believe your problem may be IO. If you are running under MPI or are running multiple jobs, the disk one of the machines may have that location unmounted, it may be full, you may have hit a system file quota limit, or the IO load is slowing it is not actually finished writing the file when MAKER tries to read it. If IO load, is the issue, then you just need to run fewer processes. The other possibilities would mean you need to make space, fix the mount, or raise any quotas on your systems.
>>
>> Thanks,
>> Carson
>>
>> On Jan 6, 2018, at 4:09 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>
>>> Hi Carson,
>>>
>>> I am pasting more lines of error messages. I notice an error of "ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq”, the seq name of “ScsGwly” is ">ScsGwly_6124;HRSCAF=6247”, is it because of the seq naming that makes the temp file name weird?
>>>
>>> Thanks
>>> Qihua
>>>
>>> #--------- command -------------#
>>> Widget::blastx:
>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_nJDkCL/te_proteins%2Efasta.mpi.10.9 -query /tmp/maker_nJDkCL/0/ScsG
>>> wly_5932%3BHRSCAF=6050.0 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes
>>> -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/D7/4
>>> A/ScsGwly_5932%3BHRSCAF=6050//theVoid.ScsGwly_5932%3BHRSCAF=6050/0/ScsGwly_5932%3BHRSCAF=6050.0.te_proteins%2Efasta.repeatrunner.temp_dir/te_
>>> proteins%2Efasta.mpi.10.9.repeatrunner
>>> #-------------------------------#
>>> deleted:0 hits
>>> collecting blastx repeatmasking
>>> processing all repeats
>>> in cluster::shadow_cluster...
>>> ...finished clustering.
>>> preparing masked sequence
>>> preparing ab-inits
>>> running snap.
>>> #--------- command -------------#
>>> Widget::snap:
>>> /24-2/home/qliang/0.soft/maker/exe/snap/snap /home/qliang/cowpea/annotation/09.tingting/4.Abintio/2.CEGMA/3.maker/maker1.hmm/maker1.snap.hmm
>>> /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.maker1%2Esnap%2Eh
>>> mm.snap
>>> #-------------------------------#
>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>> scoring....decoding.10.20.30.40.50.60.70.80.90.100 done
>>> running augustus.
>>> #--------- command -------------#
>>> Widget::augustus:
>>> /usr/local/augustus.2.7/bin/augustus --species=cowpea_new --UTR=off /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0 > /tmp/maker
>>> _nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_masked.0.cowpea_new.augustus
>>> #-------------------------------#
>>> deleted:0 hits
>>> collecting blastx repeatmasking
>>> processing all repeats
>>> in cluster::shadow_cluster...
>>> ...finished clustering.
>>> deleted:0 hits
>>> doing blastx repeats
>>> running blast search.
>>> #--------- command -------------#
>>> Widget::blastx:
>>> /24-2/home/qliang/0.soft/maker/bin/../exe/blast/bin/blastx -db /tmp/maker_mvdRkd/te_proteins%2Efasta.mpi.10.6 -query /tmp/maker_mvdRkd/0/chr10.75 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/09/chr10//theVoid.chr10/7/chr10.75.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
>>> #-------------------------------#
>>> doing blastx repeats
>>> re reading blast report.
>>> /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/0/ScsGwly_6124%3BHRSCAF=6247.0.te_proteins%2Efasta.repeatrunner
>>> deleted:0 hits
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> doing blastx repeats
>>> collecting blastx repeatmasking
>>> processing all repeats
>>> in cluster::shadow_cluster...
>>> ...finished clustering.
>>> ERROR: Can't open seq file: /24-2/home/qliang/cowpea/annotation/22.dovetail.assembly/map.maker.output/map_datastore/ED/F1/ScsGwly_6124%3BHRSCAF=6247//theVoid.ScsGwly_6124%3BHRSCAF=6247/query.masked.gff.seq
>>> No such file or directory
>>>
>>> at /24-2/home/qliang/0.soft/maker/bin/../lib/Dumper/GFF/GFFV3.pm line 199.
>>> Dumper::GFF::GFFV3::finalize(Dumper::GFF::GFFV3=HASH(0x5000ab8)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 700
>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>> eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>> Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>> Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>> Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>> Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>> eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>> Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>> Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>> Process::MpiTiers::next_chunk(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 286
>>> Process::MpiTiers::run_all(Process::MpiTiers=HASH(0x4fb3350), 0) called at /home/qliang/0.soft/maker/bin/maker line 695
>>> --> rank=NA, hostname=H4
>>> ERROR: Failed while builing masking tiers
>>> --> rank=NA, hostname=H4
>>> --> rank=NA, hostname=H4
>>> ERROR: Can not get next level
>>> running genemark.
>>> #--------- command -------------#
>>> Widget::genemark:
>>> /24-2/home/qliang/0.soft/PerlPackages/ActivePerl-5.22/bin/perl-static /24-2/home/qliang/0.soft/maker/bin/../lib/Widget/genemark/gmhmm_wrap -m /home/qliang/cowpea/annotation/05.CEGMA/2.genemask/output/gmhmm.mod -g /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/gmhmme3 -p /24-2/home/qliang/0.soft/makerPackages/gm_et_linux_64/gmes_petap/probuild -o /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0.gmhmm%2Emod.genemark /tmp/maker_nJDkCL/ScsGwly_5932%3BHRSCAF=6050.abinit_nomask.0
>>> #-------------------------------#
>>> FAILED CONTIG:ScsGwly_6124;HRSCAF=6247
>>>
>>> examining contents of the fasta file and run log
>>>
>>>
>>>
>>> --Next Contig--
>>>
>>> #---------------------------------------------------------------------
>>> Now starting the contig!!
>>> SeqID: ScsGwly_6140;HRSCAF=6263
>>> Length: 1247
>>> #---------------------------------------------------------------------
>>>
>>>
>>>
>>>
>>>> On Jan 5, 2018, at 7:22 PM, Carson Holt <carsonhh at gmail.com <mailto:carsonhh at gmail.com>> wrote:
>>>>
>>>> That’s the stack trace. The error is going to be a few lines further back. It would be best to get a few hundred lines right around the area you are showing.
>>>>
>>>> —Carson
>>>>
>>>>> On Jan 4, 2018, at 2:36 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>
>>>>> Hi Ence,
>>>>>
>>>>> When I searched for “E/error” in the output file, here is what first showed up:
>>>>> Process::MpiChunk::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>> eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>> Error::subs::try(CODE(0x502bbb0), HASH(0x5007788)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 4269
>>>>> Process::MpiChunk::_go(Process::MpiChunk=HASH(0x50a1a18), "flow", HASH(0x50ad0f0), 2, 0) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiChunk.pm line 378
>>>>> Process::MpiChunk::_flow(Process::MpiChunk=HASH(0x50a1a18), HASH(0x50ad0f0), 2, 0, Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 318
>>>>> Process::MpiTiers::__ANON__() called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 415
>>>>> eval {...} called at /24-2/home/qliang/0.soft/maker/bin/../lib/Error.pm line 407
>>>>> Error::subs::try(CODE(0x50a9348), HASH(0x4ff0ec0)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 338
>>>>> Process::MpiTiers::_next_level(Process::MpiTiers=HASH(0x4fb3350)) called at /24-2/home/qliang/0.soft/maker/bin/../lib/Process/MpiTiers.pm line 179
>>>>>
>>>>> Is this what you may need?
>>>>>
>>>>> Qihua
>>>>>
>>>>>> On Jan 4, 2018, at 6:16 AM, Ence,daniel <d.ence at ufl.edu <mailto:d.ence at ufl.edu>> wrote:
>>>>>>
>>>>>> Hi, Before we can give any help to debug it, we need the error messages. These should be in the same file that the “maker is finished” message is in. Look for the first error message (the one closest to the top of the file) and send that to the mailing list.
>>>>>>
>>>>>> Thanks,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>>> On Jan 3, 2018, at 8:52 PM, Qihua Liang <qlian003 at ucr.edu <mailto:qlian003 at ucr.edu>> wrote:
>>>>>>>
>>>>>>> Dear Maker Develop Team,
>>>>>>>
>>>>>>> I have successfully run Maker for several times before. But I came across a strange thing days ago when I ran Maker again on a different assembly with the same input files and settings.
>>>>>>>
>>>>>>> I saw the message of "Maker is now finished!!!” but got empty GFF3 and no fasta files. And then I checked the master_datastore_index.log and realized that there are a lot of “failed”s and “retry”s and “failed” again. What does this mean? Since I used same inputs as previous successful runs, could you provide some instructions on how to debug and solve it?
>>>>>>>
>>>>>>> Thank you so much
>>>>>>> Qihua
>>>>>>> _______________________________________________
>>>>>>> maker-devel mailing list
>>>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__box290.bluehost.com_mailman_listinfo_maker-2Ddevel-5Fyandell-2Dlab.org&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=12jzlNvGVD0AlPJ4E7cTlw1Dvu6n9cb4kMCobJ28XPs&m=nUDCP_0kFOhDYlTHgOpWtf_zdL77aQFeQwYOGIQwP8c&s=9Z4T1hdtxOyIjpn6f70qhrQRuGsZxXdV-oLSJF1zGkY&e=>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> maker-devel mailing list
>>>>> maker-devel at box290.bluehost.com <mailto:maker-devel at box290.bluehost.com>
>>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org <http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20180110/ce8ad4cd/attachment-0002.html>
More information about the maker-devel
mailing list