Bad Hammer zone statistics
Antony T Curtis
atcurtis at gmail.com
Fri Jun 3 14:50:12 PDT 2016
Would it be fixable by using mirror-copy to a different volume and then
mirror-copy back after newfs?
On Fri, Jun 3, 2016, 14:24 Tomohiro Kusumi <kusumi.tomohiro at gmail.com>
wrote:
> Using a 5TB disk, I filled up the first layer1 (layer#0 for the first
> 4TB of the filesystem) to see if the statistics gets broken by a bug
> of statistics itself.
> As a result, it's not broken, so I guess your raid issue did break
> your filesystem metadata.
>
>
> [root@]~# df -T /HAMMER
> Filesystem Type 1K-blocks Used Avail Capacity Mounted on
> TEST hammer 4881580032 4310634400 570945632 88% /HAMMER
> [root@]~# umount /HAMMER
> [root@]~# hammer -vvf /dev/da4 blockmap > out
> [root@]~# grep "layer1 " out -A 5
> layer1 4000000000000000 @2000000000800000 blocks-free 0
> 4000000000000000 zone=4 vol=0 L1#=0 L2#=0 L1=0
> L2=0 app=8388608 free=0 fill=100.0
> crc=9d65da9d-6663e9c8
> 4000000000800000 zone=4 vol=0 L1#=0 L2#=1 L1=0
> L2=16 app=8388608 free=0 fill=100.0
> crc=9d65da9d-6663e9c8
> 4000000001000000 zone=4 vol=0 L1#=0 L2#=2 L1=0
> L2=32 app=8388608 free=0 fill=100.0
> crc=9d65da9d-6663e9c8
> 4000000001800000 zone=3 vol=0 L1#=0 L2#=3 L1=0
> L2=48 app=8388608 free=0 fill=100.0
> crc=9d65da9d-12fb0047
> 4000000002000000 zone=3 vol=0 L1#=0 L2#=4 L1=0
> L2=64 app=8388608 free=0 fill=100.0
> crc=9d65da9d-12fb0047
> --
> layer1 4000040000000000 @2000000001000000 blocks-free 69741
> 4000040000000000 zone=10 vol=0 L1#=1 L2#=0 L1=32
> L2=0 app=8388608 free=0 fill=100.0
> crc=5cccfffc-8f523ad6
> 4000040000800000 zone=10 vol=0 L1#=1 L2#=1 L1=32
> L2=16 app=8388608 free=0 fill=100.0
> crc=5cccfffc-8f523ad6
> 4000040001000000 zone=10 vol=0 L1#=1 L2#=2 L1=32
> L2=32 app=8388608 free=0 fill=100.0
> crc=5cccfffc-8f523ad6
> 4000040001800000 zone=10 vol=0 L1#=1 L2#=3 L1=32
> L2=48 app=8388608 free=0 fill=100.0
> crc=5cccfffc-8f523ad6
> 4000040002000000 zone=10 vol=0 L1#=1 L2#=4 L1=32
> L2=64 app=8388608 free=0 fill=100.0
> crc=5cccfffc-8f523ad6
> [root@]~# tail -22 ./out
> HAMMER zone statistics
> zone # blocks items used[B] used[%]
> zone 0 69741 0 0 0
> zone 1 0 0 0 0
> zone 2 0 0 0 0
> zone 3 128 0 1073741824 100
> zone 4 3 0 25165824 100
> zone 5 0 0 0 0
> zone 6 0 0 0 0
> zone 7 0 0 0 0
> zone 8 1075 0 8795586560 97.5363
> zone 9 2 0 5610928 33.4437
> zone 10 525078 0 4404626128896 99.9989
> zone 11 0 0 0 0
> zone 12 0 0 0 0
> zone 13 0 0 0 0
> zone 14 0 0 0 0
> zone 15 452549 0 3796256161792 100
>
> ----------------------------------------------------------------------
> total 1048576 0 8210782395824 93.3458
> 0 bad layer1
> 0 bad layer2
>
>
> 2016-06-03 3:27 GMT+09:00 Tomohiro Kusumi <kusumi.tomohiro at gmail.com>:
> > Your total block count matches your disk size (12TB),
> > so it's either layer2's bytes_free field is broken or this statistics
> has a bug.
> >
> >
> >> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496
> >>
> >> BM block=200005e72e000000 zone=9 calc 5999696 free, got 5999824
> >>
> >> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360
> >>
> >> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536
> >>
> >> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464
> >>
> >> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928
> >>
> >> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536
> >>
> >> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576
> >
> > These errors come from the reason I mentioned above.
> >
> > Technically speaking, these offsets says the bad data are stored in
> > the second layer1 entry, where 1 layer1 entry is 4TB.
> > In other words these are beyond 4TB of the filesystem.
> > I'll check if hammer blockmap/checkmap have a bug in this case
> > tomorrow, before I say your metadata is broken.
> >
> >
> > 2016-06-03 2:54 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
> >> Overall size of the disk is around 12 Tb.
> >>
> >> slightly different result from checkmap, likely because it is a mounted
> >> volume...
> >>
> >>> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496
> >>>
> >>> BM block=200005e72e000000 zone=9 calc 5999696 free, got 5999824
> >>>
> >>> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360
> >>>
> >>> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536
> >>>
> >>> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464
> >>>
> >>> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928
> >>>
> >>> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536
> >>>
> >>> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576
> >>>
> >>> HAMMER zone statistics
> >>>
> >>> zone # blocks items used[B] used[%]
> >>>
> >>> zone 0 0 0 0 0
> >>>
> >>> zone 1 0 0 0 0
> >>>
> >>> zone 2 0 0 0 0
> >>>
> >>> zone 3 128 0 1073741824 100
> >>>
> >>> zone 4 4 0 33554432 100
> >>>
> >>> zone 5 0 0 0 0
> >>>
> >>> zone 6 0 0 0 0
> >>>
> >>> zone 7 0 0 0 0
> >>>
> >>> zone 8 1029 0 7988363264 92.5449
> >>>
> >>> zone 9 1443 0 2183889824 18.0416
> >>>
> >>> zone 10 330736 0 2911421278384 104.938
> >>>
> >>> zone 11 1154 0 14288724672 147.604
> >>>
> >>> zone 12 0 0 0 0
> >>>
> >>> zone 13 0 0 0 0
> >>>
> >>> zone 14 0 0 0 0
> >>>
> >>> zone 15 0 0 0 0
> >>>
> >>> ----------------------------------------------------------------------
> >>>
> >>> total 334494 0 2936989552400 104.67
> >>>
> >>> 0 bad nodes
> >>>
> >>> 8 errors
> >>
> >>
> >>
> >> On 2 June 2016 at 10:42, Tomohiro Kusumi <kusumi.tomohiro at gmail.com>
> wrote:
> >>>
> >>> 1. What's your approximate disk size (or sum of disks) ?
> >>>
> >>> 2. If your total block counts from blockmap output matches your disk
> size,
> >>> > zone 0 1095221 0 18446744073709486080
> 2.00784e+06
> >>> this line from blockmap output shows your layer2 metadata probably had
> >>> bad (broken) free_bytes.
> >>> Can't tell how it was broken from this output though.
> >>>
> >>> 3. What are the 10 errors for checkmap ?
> >>> You have output before statistics that indicates those errors.
> >>>
> >>>
> >>> 2016-06-03 2:32 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
> >>> > Hammer checkmap shows this instead:
> >>> >>
> >>> >> HAMMER zone statistics
> >>> >>
> >>> >> zone # blocks items used[B] used[%]
> >>> >>
> >>> >> zone 0 0 0 0 0
> >>> >>
> >>> >> zone 1 0 0 0 0
> >>> >>
> >>> >> zone 2 0 0 0 0
> >>> >>
> >>> >> zone 3 128 0 1073741824 100
> >>> >>
> >>> >> zone 4 4 0 33554432 100
> >>> >>
> >>> >> zone 5 0 0 0 0
> >>> >>
> >>> >> zone 6 0 0 0 0
> >>> >>
> >>> >> zone 7 0 0 0 0
> >>> >>
> >>> >> zone 8 1029 0 7988092928 92.5418
> >>> >>
> >>> >> zone 9 1443 0 2183767648 18.0406
> >>> >>
> >>> >> zone 10 330734 0 2911406172336 104.938
> >>> >>
> >>> >> zone 11 1154 0 14287486768 147.591
> >>> >>
> >>> >> zone 12 0 0 0 0
> >>> >>
> >>> >> zone 13 0 0 0 0
> >>> >>
> >>> >> zone 14 0 0 0 0
> >>> >>
> >>> >> zone 15 0 0 0 0
> >>> >>
> >>> >>
> ----------------------------------------------------------------------
> >>> >>
> >>> >> total 334492 0 2936972815936 104.67
> >>> >>
> >>> >> 0 bad nodes
> >>> >>
> >>> >> 10 errors
> >>> >
> >>> >
> >>> >
> >>> > On 2 June 2016 at 10:14, Antony T Curtis <atcurtis at gmail.com> wrote:
> >>> >>
> >>> >> This is from a hammer blockmap command.
> >>> >>
> >>> >> On 2 June 2016 at 10:13, Tomohiro Kusumi <kusumi.tomohiro at gmail.com
> >
> >>> >> wrote:
> >>> >>>
> >>> >>> From which command is this from ?
> >>> >>> It's not hammer show (because hammer show has non 0 items), so I
> >>> >>> assume either blockmap or checkmap.
> >>> >>>
> >>> >>> 2016-06-03 2:05 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
> >>> >>> > Is there any info as to how to repair a Hammer volume or should I
> >>> >>> > simply try
> >>> >>> > to backup and restore?
> >>> >>> >
> >>> >>> >> HAMMER zone statistics
> >>> >>> >>
> >>> >>> >> zone # blocks items used[B]
> used[%]
> >>> >>> >>
> >>> >>> >> zone 0 1095221 0 18446744073709486080
> >>> >>> >> 2.00784e+06
> >>> >>> >>
> >>> >>> >> zone 1 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 2 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 3 128 0 1073741824 100
> >>> >>> >>
> >>> >>> >> zone 4 4 0 33554432 100
> >>> >>> >>
> >>> >>> >> zone 5 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 6 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 7 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 8 1029 0 7986941952
> 92.5284
> >>> >>> >>
> >>> >>> >> zone 9 1443 0 2183756816
> 18.0405
> >>> >>> >>
> >>> >>> >> zone 10 330680 0 2910941826784
> 104.939
> >>> >>> >>
> >>> >>> >> zone 11 1154 0 14287415824
> 147.59
> >>> >>> >>
> >>> >>> >> zone 12 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 13 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 14 0 0 0 0
> >>> >>> >>
> >>> >>> >> zone 15 143205 0 1201290608640 100
> >>> >>> >>
> >>> >>> >>
> >>> >>> >>
> ----------------------------------------------------------------------
> >>> >>> >>
> >>> >>> >> total 1572864 0 4137797780736
> 31.3609
> >>> >>> >>
> >>> >>> >> 0 bad layer1
> >>> >>> >>
> >>> >>> >> 0 bad layer2
> >>> >>> >
> >>> >>> >
> >>> >>> >
> >>> >>> >
> >>> >>> > --
> >>> >>> > Antony T Curtis
> >>> >>> >
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Antony T Curtis
> >>> >> 0523 C487 9187 6972 6894
> >>> >> AEC7 3087 F819 B477 B687
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Antony T Curtis
> >>> > 0523 C487 9187 6972 6894
> >>> > AEC7 3087 F819 B477 B687
> >>
> >>
> >>
> >>
> >> --
> >> Antony T Curtis
> >> 0523 C487 9187 6972 6894
> >> AEC7 3087 F819 B477 B687
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dragonflybsd.org/pipermail/users/attachments/20160603/005e9cc0/attachment-0003.htm>
More information about the Users
mailing list