<p dir="ltr">Would it be fixable by using mirror-copy to a different volume and then mirror-copy back after newfs?</p>
<br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 3, 2016, 14:24 Tomohiro Kusumi <<a href="mailto:kusumi.tomohiro@gmail.com">kusumi.tomohiro@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Using a 5TB disk, I filled up the first layer1 (layer#0 for the first<br>
4TB of the filesystem) to see if the statistics gets broken by a bug<br>
of statistics itself.<br>
As a result, it's not broken, so I guess your raid issue did break<br>
your filesystem metadata.<br>
<br>
<br>
[root@]~# df -T /HAMMER<br>
Filesystem Type 1K-blocks Used Avail Capacity Mounted on<br>
TEST hammer 4881580032 4310634400 570945632 88% /HAMMER<br>
[root@]~# umount /HAMMER<br>
[root@]~# hammer -vvf /dev/da4 blockmap > out<br>
[root@]~# grep "layer1 " out -A 5<br>
layer1 4000000000000000 @2000000000800000 blocks-free 0<br>
4000000000000000 zone=4 vol=0 L1#=0 L2#=0 L1=0<br>
L2=0 app=8388608 free=0 fill=100.0<br>
crc=9d65da9d-6663e9c8<br>
4000000000800000 zone=4 vol=0 L1#=0 L2#=1 L1=0<br>
L2=16 app=8388608 free=0 fill=100.0<br>
crc=9d65da9d-6663e9c8<br>
4000000001000000 zone=4 vol=0 L1#=0 L2#=2 L1=0<br>
L2=32 app=8388608 free=0 fill=100.0<br>
crc=9d65da9d-6663e9c8<br>
4000000001800000 zone=3 vol=0 L1#=0 L2#=3 L1=0<br>
L2=48 app=8388608 free=0 fill=100.0<br>
crc=9d65da9d-12fb0047<br>
4000000002000000 zone=3 vol=0 L1#=0 L2#=4 L1=0<br>
L2=64 app=8388608 free=0 fill=100.0<br>
crc=9d65da9d-12fb0047<br>
--<br>
layer1 4000040000000000 @2000000001000000 blocks-free 69741<br>
4000040000000000 zone=10 vol=0 L1#=1 L2#=0 L1=32<br>
L2=0 app=8388608 free=0 fill=100.0<br>
crc=5cccfffc-8f523ad6<br>
4000040000800000 zone=10 vol=0 L1#=1 L2#=1 L1=32<br>
L2=16 app=8388608 free=0 fill=100.0<br>
crc=5cccfffc-8f523ad6<br>
4000040001000000 zone=10 vol=0 L1#=1 L2#=2 L1=32<br>
L2=32 app=8388608 free=0 fill=100.0<br>
crc=5cccfffc-8f523ad6<br>
4000040001800000 zone=10 vol=0 L1#=1 L2#=3 L1=32<br>
L2=48 app=8388608 free=0 fill=100.0<br>
crc=5cccfffc-8f523ad6<br>
4000040002000000 zone=10 vol=0 L1#=1 L2#=4 L1=32<br>
L2=64 app=8388608 free=0 fill=100.0<br>
crc=5cccfffc-8f523ad6<br>
[root@]~# tail -22 ./out<br>
HAMMER zone statistics<br>
zone # blocks items used[B] used[%]<br>
zone 0 69741 0 0 0<br>
zone 1 0 0 0 0<br>
zone 2 0 0 0 0<br>
zone 3 128 0 1073741824 100<br>
zone 4 3 0 25165824 100<br>
zone 5 0 0 0 0<br>
zone 6 0 0 0 0<br>
zone 7 0 0 0 0<br>
zone 8 1075 0 8795586560 97.5363<br>
zone 9 2 0 5610928 33.4437<br>
zone 10 525078 0 4404626128896 99.9989<br>
zone 11 0 0 0 0<br>
zone 12 0 0 0 0<br>
zone 13 0 0 0 0<br>
zone 14 0 0 0 0<br>
zone 15 452549 0 3796256161792 100<br>
----------------------------------------------------------------------<br>
total 1048576 0 8210782395824 93.3458<br>
0 bad layer1<br>
0 bad layer2<br>
<br>
<br>
2016-06-03 3:27 GMT+09:00 Tomohiro Kusumi <<a href="mailto:kusumi.tomohiro@gmail.com" target="_blank">kusumi.tomohiro@gmail.com</a>>:<br>
> Your total block count matches your disk size (12TB),<br>
> so it's either layer2's bytes_free field is broken or this statistics has a bug.<br>
><br>
><br>
>> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496<br>
>><br>
>> BM block=200005e72e000000 zone=9 calc 5999696 free, got 5999824<br>
>><br>
>> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360<br>
>><br>
>> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536<br>
>><br>
>> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464<br>
>><br>
>> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928<br>
>><br>
>> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536<br>
>><br>
>> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576<br>
><br>
> These errors come from the reason I mentioned above.<br>
><br>
> Technically speaking, these offsets says the bad data are stored in<br>
> the second layer1 entry, where 1 layer1 entry is 4TB.<br>
> In other words these are beyond 4TB of the filesystem.<br>
> I'll check if hammer blockmap/checkmap have a bug in this case<br>
> tomorrow, before I say your metadata is broken.<br>
><br>
><br>
> 2016-06-03 2:54 GMT+09:00 Antony T Curtis <<a href="mailto:atcurtis@gmail.com" target="_blank">atcurtis@gmail.com</a>>:<br>
>> Overall size of the disk is around 12 Tb.<br>
>><br>
>> slightly different result from checkmap, likely because it is a mounted<br>
>> volume...<br>
>><br>
>>> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496<br>
>>><br>
>>> BM block=200005e72e000000 zone=9 calc 5999696 free, got 5999824<br>
>>><br>
>>> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360<br>
>>><br>
>>> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536<br>
>>><br>
>>> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464<br>
>>><br>
>>> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928<br>
>>><br>
>>> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536<br>
>>><br>
>>> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576<br>
>>><br>
>>> HAMMER zone statistics<br>
>>><br>
>>> zone # blocks items used[B] used[%]<br>
>>><br>
>>> zone 0 0 0 0 0<br>
>>><br>
>>> zone 1 0 0 0 0<br>
>>><br>
>>> zone 2 0 0 0 0<br>
>>><br>
>>> zone 3 128 0 1073741824 100<br>
>>><br>
>>> zone 4 4 0 33554432 100<br>
>>><br>
>>> zone 5 0 0 0 0<br>
>>><br>
>>> zone 6 0 0 0 0<br>
>>><br>
>>> zone 7 0 0 0 0<br>
>>><br>
>>> zone 8 1029 0 7988363264 92.5449<br>
>>><br>
>>> zone 9 1443 0 2183889824 18.0416<br>
>>><br>
>>> zone 10 330736 0 2911421278384 104.938<br>
>>><br>
>>> zone 11 1154 0 14288724672 147.604<br>
>>><br>
>>> zone 12 0 0 0 0<br>
>>><br>
>>> zone 13 0 0 0 0<br>
>>><br>
>>> zone 14 0 0 0 0<br>
>>><br>
>>> zone 15 0 0 0 0<br>
>>><br>
>>> ----------------------------------------------------------------------<br>
>>><br>
>>> total 334494 0 2936989552400 104.67<br>
>>><br>
>>> 0 bad nodes<br>
>>><br>
>>> 8 errors<br>
>><br>
>><br>
>><br>
>> On 2 June 2016 at 10:42, Tomohiro Kusumi <<a href="mailto:kusumi.tomohiro@gmail.com" target="_blank">kusumi.tomohiro@gmail.com</a>> wrote:<br>
>>><br>
>>> 1. What's your approximate disk size (or sum of disks) ?<br>
>>><br>
>>> 2. If your total block counts from blockmap output matches your disk size,<br>
>>> > zone 0 1095221 0 18446744073709486080 2.00784e+06<br>
>>> this line from blockmap output shows your layer2 metadata probably had<br>
>>> bad (broken) free_bytes.<br>
>>> Can't tell how it was broken from this output though.<br>
>>><br>
>>> 3. What are the 10 errors for checkmap ?<br>
>>> You have output before statistics that indicates those errors.<br>
>>><br>
>>><br>
>>> 2016-06-03 2:32 GMT+09:00 Antony T Curtis <<a href="mailto:atcurtis@gmail.com" target="_blank">atcurtis@gmail.com</a>>:<br>
>>> > Hammer checkmap shows this instead:<br>
>>> >><br>
>>> >> HAMMER zone statistics<br>
>>> >><br>
>>> >> zone # blocks items used[B] used[%]<br>
>>> >><br>
>>> >> zone 0 0 0 0 0<br>
>>> >><br>
>>> >> zone 1 0 0 0 0<br>
>>> >><br>
>>> >> zone 2 0 0 0 0<br>
>>> >><br>
>>> >> zone 3 128 0 1073741824 100<br>
>>> >><br>
>>> >> zone 4 4 0 33554432 100<br>
>>> >><br>
>>> >> zone 5 0 0 0 0<br>
>>> >><br>
>>> >> zone 6 0 0 0 0<br>
>>> >><br>
>>> >> zone 7 0 0 0 0<br>
>>> >><br>
>>> >> zone 8 1029 0 7988092928 92.5418<br>
>>> >><br>
>>> >> zone 9 1443 0 2183767648 18.0406<br>
>>> >><br>
>>> >> zone 10 330734 0 2911406172336 104.938<br>
>>> >><br>
>>> >> zone 11 1154 0 14287486768 147.591<br>
>>> >><br>
>>> >> zone 12 0 0 0 0<br>
>>> >><br>
>>> >> zone 13 0 0 0 0<br>
>>> >><br>
>>> >> zone 14 0 0 0 0<br>
>>> >><br>
>>> >> zone 15 0 0 0 0<br>
>>> >><br>
>>> >> ----------------------------------------------------------------------<br>
>>> >><br>
>>> >> total 334492 0 2936972815936 104.67<br>
>>> >><br>
>>> >> 0 bad nodes<br>
>>> >><br>
>>> >> 10 errors<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > On 2 June 2016 at 10:14, Antony T Curtis <<a href="mailto:atcurtis@gmail.com" target="_blank">atcurtis@gmail.com</a>> wrote:<br>
>>> >><br>
>>> >> This is from a hammer blockmap command.<br>
>>> >><br>
>>> >> On 2 June 2016 at 10:13, Tomohiro Kusumi <<a href="mailto:kusumi.tomohiro@gmail.com" target="_blank">kusumi.tomohiro@gmail.com</a>><br>
>>> >> wrote:<br>
>>> >>><br>
>>> >>> From which command is this from ?<br>
>>> >>> It's not hammer show (because hammer show has non 0 items), so I<br>
>>> >>> assume either blockmap or checkmap.<br>
>>> >>><br>
>>> >>> 2016-06-03 2:05 GMT+09:00 Antony T Curtis <<a href="mailto:atcurtis@gmail.com" target="_blank">atcurtis@gmail.com</a>>:<br>
>>> >>> > Is there any info as to how to repair a Hammer volume or should I<br>
>>> >>> > simply try<br>
>>> >>> > to backup and restore?<br>
>>> >>> ><br>
>>> >>> >> HAMMER zone statistics<br>
>>> >>> >><br>
>>> >>> >> zone # blocks items used[B] used[%]<br>
>>> >>> >><br>
>>> >>> >> zone 0 1095221 0 18446744073709486080<br>
>>> >>> >> 2.00784e+06<br>
>>> >>> >><br>
>>> >>> >> zone 1 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 2 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 3 128 0 1073741824 100<br>
>>> >>> >><br>
>>> >>> >> zone 4 4 0 33554432 100<br>
>>> >>> >><br>
>>> >>> >> zone 5 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 6 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 7 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 8 1029 0 7986941952 92.5284<br>
>>> >>> >><br>
>>> >>> >> zone 9 1443 0 2183756816 18.0405<br>
>>> >>> >><br>
>>> >>> >> zone 10 330680 0 2910941826784 104.939<br>
>>> >>> >><br>
>>> >>> >> zone 11 1154 0 14287415824 147.59<br>
>>> >>> >><br>
>>> >>> >> zone 12 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 13 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 14 0 0 0 0<br>
>>> >>> >><br>
>>> >>> >> zone 15 143205 0 1201290608640 100<br>
>>> >>> >><br>
>>> >>> >><br>
>>> >>> >> ----------------------------------------------------------------------<br>
>>> >>> >><br>
>>> >>> >> total 1572864 0 4137797780736 31.3609<br>
>>> >>> >><br>
>>> >>> >> 0 bad layer1<br>
>>> >>> >><br>
>>> >>> >> 0 bad layer2<br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> > --<br>
>>> >>> > Antony T Curtis<br>
>>> >>> ><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >> --<br>
>>> >> Antony T Curtis<br>
>>> >> 0523 C487 9187 6972 6894<br>
>>> >> AEC7 3087 F819 B477 B687<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > --<br>
>>> > Antony T Curtis<br>
>>> > 0523 C487 9187 6972 6894<br>
>>> > AEC7 3087 F819 B477 B687<br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Antony T Curtis<br>
>> 0523 C487 9187 6972 6894<br>
>> AEC7 3087 F819 B477 B687<br>
</blockquote></div>