Bad Hammer zone statistics

Tomohiro Kusumi kusumi.tomohiro at gmail.com
Fri Jun 3 22:58:44 PDT 2016


Probably safer to just copy (cp/scp) files without using hammer's mirroring,
provided that your file contents are still safe.

2016-06-04 6:50 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
> Would it be fixable by using mirror-copy to a different volume and then
> mirror-copy back after newfs?
>
>
> On Fri, Jun 3, 2016, 14:24 Tomohiro Kusumi <kusumi.tomohiro at gmail.com>
> wrote:
>>
>> Using a 5TB disk, I filled up the first layer1 (layer#0 for the first
>> 4TB of the filesystem) to see if the statistics gets broken by a bug
>> of statistics itself.
>> As a result, it's not broken, so I guess your raid issue did break
>> your filesystem metadata.
>>
>>
>> [root@]~# df -T /HAMMER
>> Filesystem  Type   1K-blocks        Used     Avail Capacity  Mounted on
>> TEST        hammer 4881580032 4310634400 570945632    88%    /HAMMER
>> [root@]~# umount /HAMMER
>> [root@]~# hammer -vvf /dev/da4 blockmap > out
>> [root@]~# grep "layer1 " out -A 5
>>   layer1 4000000000000000 @2000000000800000 blocks-free 0
>>         4000000000000000 zone=4  vol=0   L1#=0      L2#=0      L1=0
>>    L2=0       app=8388608 free=0       fill=100.0
>> crc=9d65da9d-6663e9c8
>>         4000000000800000 zone=4  vol=0   L1#=0      L2#=1      L1=0
>>    L2=16      app=8388608 free=0       fill=100.0
>> crc=9d65da9d-6663e9c8
>>         4000000001000000 zone=4  vol=0   L1#=0      L2#=2      L1=0
>>    L2=32      app=8388608 free=0       fill=100.0
>> crc=9d65da9d-6663e9c8
>>         4000000001800000 zone=3  vol=0   L1#=0      L2#=3      L1=0
>>    L2=48      app=8388608 free=0       fill=100.0
>> crc=9d65da9d-12fb0047
>>         4000000002000000 zone=3  vol=0   L1#=0      L2#=4      L1=0
>>    L2=64      app=8388608 free=0       fill=100.0
>> crc=9d65da9d-12fb0047
>> --
>>   layer1 4000040000000000 @2000000001000000 blocks-free 69741
>>         4000040000000000 zone=10 vol=0   L1#=1      L2#=0      L1=32
>>    L2=0       app=8388608 free=0       fill=100.0
>> crc=5cccfffc-8f523ad6
>>         4000040000800000 zone=10 vol=0   L1#=1      L2#=1      L1=32
>>    L2=16      app=8388608 free=0       fill=100.0
>> crc=5cccfffc-8f523ad6
>>         4000040001000000 zone=10 vol=0   L1#=1      L2#=2      L1=32
>>    L2=32      app=8388608 free=0       fill=100.0
>> crc=5cccfffc-8f523ad6
>>         4000040001800000 zone=10 vol=0   L1#=1      L2#=3      L1=32
>>    L2=48      app=8388608 free=0       fill=100.0
>> crc=5cccfffc-8f523ad6
>>         4000040002000000 zone=10 vol=0   L1#=1      L2#=4      L1=32
>>    L2=64      app=8388608 free=0       fill=100.0
>> crc=5cccfffc-8f523ad6
>> [root@]~# tail -22 ./out
>> HAMMER zone statistics
>>         zone #  blocks       items              used[B]
>> used[%]
>>         zone 0  69741        0                  0                   0
>>         zone 1  0            0                  0                   0
>>         zone 2  0            0                  0                   0
>>         zone 3  128          0                  1073741824          100
>>         zone 4  3            0                  25165824            100
>>         zone 5  0            0                  0                   0
>>         zone 6  0            0                  0                   0
>>         zone 7  0            0                  0                   0
>>         zone 8  1075         0                  8795586560
>> 97.5363
>>         zone 9  2            0                  5610928
>> 33.4437
>>         zone 10 525078       0                  4404626128896
>> 99.9989
>>         zone 11 0            0                  0                   0
>>         zone 12 0            0                  0                   0
>>         zone 13 0            0                  0                   0
>>         zone 14 0            0                  0                   0
>>         zone 15 452549       0                  3796256161792       100
>>
>> ----------------------------------------------------------------------
>>         total   1048576      0                  8210782395824
>> 93.3458
>> 0 bad layer1
>> 0 bad layer2
>>
>>
>> 2016-06-03 3:27 GMT+09:00 Tomohiro Kusumi <kusumi.tomohiro at gmail.com>:
>> > Your total block count matches your disk size (12TB),
>> > so it's either layer2's bytes_free field is broken or this statistics
>> > has a bug.
>> >
>> >
>> >> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496
>> >>
>> >> BM block=200005e72e000000 zone=9  calc 5999696 free, got 5999824
>> >>
>> >> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360
>> >>
>> >> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536
>> >>
>> >> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464
>> >>
>> >> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928
>> >>
>> >> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536
>> >>
>> >> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576
>> >
>> > These errors come from the reason I mentioned above.
>> >
>> > Technically speaking, these offsets says the bad data are stored in
>> > the second layer1 entry, where 1 layer1 entry is 4TB.
>> > In other words these are beyond 4TB of the filesystem.
>> > I'll check if hammer blockmap/checkmap have a bug in this case
>> > tomorrow, before I say your metadata is broken.
>> >
>> >
>> > 2016-06-03 2:54 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
>> >> Overall size of the disk is around 12 Tb.
>> >>
>> >> slightly different result from checkmap, likely because it is a mounted
>> >> volume...
>> >>
>> >>> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496
>> >>>
>> >>> BM block=200005e72e000000 zone=9  calc 5999696 free, got 5999824
>> >>>
>> >>> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360
>> >>>
>> >>> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536
>> >>>
>> >>> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464
>> >>>
>> >>> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928
>> >>>
>> >>> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536
>> >>>
>> >>> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576
>> >>>
>> >>> HAMMER zone statistics
>> >>>
>> >>> zone #  blocks       items              used[B]             used[%]
>> >>>
>> >>> zone 0  0            0                  0                   0
>> >>>
>> >>> zone 1  0            0                  0                   0
>> >>>
>> >>> zone 2  0            0                  0                   0
>> >>>
>> >>> zone 3  128          0                  1073741824          100
>> >>>
>> >>> zone 4  4            0                  33554432            100
>> >>>
>> >>> zone 5  0            0                  0                   0
>> >>>
>> >>> zone 6  0            0                  0                   0
>> >>>
>> >>> zone 7  0            0                  0                   0
>> >>>
>> >>> zone 8  1029         0                  7988363264          92.5449
>> >>>
>> >>> zone 9  1443         0                  2183889824          18.0416
>> >>>
>> >>> zone 10 330736       0                  2911421278384       104.938
>> >>>
>> >>> zone 11 1154         0                  14288724672         147.604
>> >>>
>> >>> zone 12 0            0                  0                   0
>> >>>
>> >>> zone 13 0            0                  0                   0
>> >>>
>> >>> zone 14 0            0                  0                   0
>> >>>
>> >>> zone 15 0            0                  0                   0
>> >>>
>> >>> ----------------------------------------------------------------------
>> >>>
>> >>> total   334494       0                  2936989552400       104.67
>> >>>
>> >>> 0 bad nodes
>> >>>
>> >>> 8 errors
>> >>
>> >>
>> >>
>> >> On 2 June 2016 at 10:42, Tomohiro Kusumi <kusumi.tomohiro at gmail.com>
>> >> wrote:
>> >>>
>> >>> 1. What's your approximate disk size (or sum of disks) ?
>> >>>
>> >>> 2. If your total block counts from blockmap output matches your disk
>> >>> size,
>> >>> > zone 0  1095221      0                  18446744073709486080
>> >>> > 2.00784e+06
>> >>> this line from blockmap output shows your layer2 metadata probably had
>> >>> bad (broken) free_bytes.
>> >>> Can't tell how it was broken from this output though.
>> >>>
>> >>> 3. What are the 10 errors for checkmap ?
>> >>> You have output before statistics that indicates those errors.
>> >>>
>> >>>
>> >>> 2016-06-03 2:32 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
>> >>> > Hammer checkmap shows this instead:
>> >>> >>
>> >>> >> HAMMER zone statistics
>> >>> >>
>> >>> >> zone #  blocks       items              used[B]             used[%]
>> >>> >>
>> >>> >> zone 0  0            0                  0                   0
>> >>> >>
>> >>> >> zone 1  0            0                  0                   0
>> >>> >>
>> >>> >> zone 2  0            0                  0                   0
>> >>> >>
>> >>> >> zone 3  128          0                  1073741824          100
>> >>> >>
>> >>> >> zone 4  4            0                  33554432            100
>> >>> >>
>> >>> >> zone 5  0            0                  0                   0
>> >>> >>
>> >>> >> zone 6  0            0                  0                   0
>> >>> >>
>> >>> >> zone 7  0            0                  0                   0
>> >>> >>
>> >>> >> zone 8  1029         0                  7988092928          92.5418
>> >>> >>
>> >>> >> zone 9  1443         0                  2183767648          18.0406
>> >>> >>
>> >>> >> zone 10 330734       0                  2911406172336       104.938
>> >>> >>
>> >>> >> zone 11 1154         0                  14287486768         147.591
>> >>> >>
>> >>> >> zone 12 0            0                  0                   0
>> >>> >>
>> >>> >> zone 13 0            0                  0                   0
>> >>> >>
>> >>> >> zone 14 0            0                  0                   0
>> >>> >>
>> >>> >> zone 15 0            0                  0                   0
>> >>> >>
>> >>> >>
>> >>> >> ----------------------------------------------------------------------
>> >>> >>
>> >>> >> total   334492       0                  2936972815936       104.67
>> >>> >>
>> >>> >> 0 bad nodes
>> >>> >>
>> >>> >> 10 errors
>> >>> >
>> >>> >
>> >>> >
>> >>> > On 2 June 2016 at 10:14, Antony T Curtis <atcurtis at gmail.com> wrote:
>> >>> >>
>> >>> >> This is from a hammer blockmap command.
>> >>> >>
>> >>> >> On 2 June 2016 at 10:13, Tomohiro Kusumi
>> >>> >> <kusumi.tomohiro at gmail.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> From which command is this from ?
>> >>> >>> It's not hammer show (because hammer show has non 0 items), so I
>> >>> >>> assume either blockmap or checkmap.
>> >>> >>>
>> >>> >>> 2016-06-03 2:05 GMT+09:00 Antony T Curtis <atcurtis at gmail.com>:
>> >>> >>> > Is there any info as to how to repair a Hammer volume or should
>> >>> >>> > I
>> >>> >>> > simply try
>> >>> >>> > to backup and restore?
>> >>> >>> >
>> >>> >>> >> HAMMER zone statistics
>> >>> >>> >>
>> >>> >>> >> zone #  blocks       items              used[B]
>> >>> >>> >> used[%]
>> >>> >>> >>
>> >>> >>> >> zone 0  1095221      0                  18446744073709486080
>> >>> >>> >> 2.00784e+06
>> >>> >>> >>
>> >>> >>> >> zone 1  0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 2  0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 3  128          0                  1073741824          100
>> >>> >>> >>
>> >>> >>> >> zone 4  4            0                  33554432            100
>> >>> >>> >>
>> >>> >>> >> zone 5  0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 6  0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 7  0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 8  1029         0                  7986941952
>> >>> >>> >> 92.5284
>> >>> >>> >>
>> >>> >>> >> zone 9  1443         0                  2183756816
>> >>> >>> >> 18.0405
>> >>> >>> >>
>> >>> >>> >> zone 10 330680       0                  2910941826784
>> >>> >>> >> 104.939
>> >>> >>> >>
>> >>> >>> >> zone 11 1154         0                  14287415824
>> >>> >>> >> 147.59
>> >>> >>> >>
>> >>> >>> >> zone 12 0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 13 0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 14 0            0                  0                   0
>> >>> >>> >>
>> >>> >>> >> zone 15 143205       0                  1201290608640       100
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> ----------------------------------------------------------------------
>> >>> >>> >>
>> >>> >>> >> total   1572864      0                  4137797780736
>> >>> >>> >> 31.3609
>> >>> >>> >>
>> >>> >>> >> 0 bad layer1
>> >>> >>> >>
>> >>> >>> >> 0 bad layer2
>> >>> >>> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>> > --
>> >>> >>> > Antony T Curtis
>> >>> >>> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Antony T Curtis
>> >>> >> 0523 C487 9187 6972 6894
>> >>> >> AEC7 3087 F819 B477 B687
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Antony T Curtis
>> >>> > 0523 C487 9187 6972 6894
>> >>> > AEC7 3087 F819 B477 B687
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Antony T Curtis
>> >> 0523 C487 9187 6972 6894
>> >> AEC7 3087 F819 B477 B687



More information about the Users mailing list