panic: assertion: layer2->zone == zone in hammer_blockmap_free

[Matthew Dillon]

:>     I am going to need some additional information from you.  Please sync
:>     your filesystem and then do the following:
:> 
:>     hammer -f <device> blockmap > textfile
:>     hammer -f <device> -vvv show >> textfile
:> 
:>     And gzip and upload that to leaf.
:
:Done as ~y0netan1/crash/blockmap.gz.
:
:Cheers.

    Excellent!  This is great!  Ok, here is the big-block blockmap entry:

        4000000174800000 zone=9 app=8388608 free=8388480

    Each big-block can hold 8MB (8388608 bytes).  This one has 128 bytes
    allocated out of it.  From the B-Tree dump four inodes are allocated
    out of that block, though, so it should have read 512 bytes
    allocated (free=8388096).

    I already see a pattern, and I'm hoping it will lead to the bug.
    The LO field is 0002xxxx == PFS#2.  The tids are create:delete... 
    notice that one has a delete_tid of 0, which means it is live.
    The others have a non-zero delete_tid, those have been deleted.

    We get a nice ctime too... the ctime is in microseconds.  First, here
    are the inode records:

G------ ELM 14 R obj=0000000000000001 key=0000000000000000 
		 lo=00020001 rt=01 ot=01		    <<<< top 16 bits 
								 of LO is the
								 PFS id.
                 tids 000000010848ffbb:0000000000000000   <<<<<  create:delete
                 dataoff=9000000174dda260/128 crc=149d91f4  <<<< dataoff
                 fills=z9:745=1%
                 size=0 nlinks=1 mode=01777 uflags=00000000
                 ctime=0004525a5174d38a pobjid=0000000000000000 obj_type=1
                 mtime=0004537db932151f
			^^^^^^^^^^^^
			time stamps

G------ ELM 15 R obj=0000000000000001 key=0000000000000000
		 lo=00020001 rt=01 ot=01
               d tids 0000000108490171:0000000108490179   <<<< create:delete
                 dataoff=9000000174dda440/128 crc=761cd3d0
                 fills=z9:745=1%
                 size=0 nlinks=1 mode=00755 uflags=00000000
                 ctime=0004525b5cd8b43f pobjid=0000000000000000 obj_type=1
                 mtime=0004525b5cd8b43f

G------ ELM 22 R obj=0000000107e44a04 key=0000000000000000
		 lo=00020001 rt=01 ot=02
               d tids 0000000108490169:0000000108490175 
                 dataoff=9000000174dda2e0/128 crc=1861f98c
                 fills=z9:745=1%
                 size=0 nlinks=2 mode=00644 uflags=00000000
                 ctime=0004525b4031d815 pobjid=0000000000000001 obj_type=2
                 mtime=0004525b4031d815

G------ ELM 23 R obj=0000000107e44a05 key=0000000000000000
		 lo=00020001 rt=01 ot=07
               d tids 000000010849016d:0000000108490175 
                 dataoff=9000000174dda360/128 crc=c57654ef
                 fills=z9:745=1%
                 size=3 nlinks=1 mode=00755 uflags=00000000
                 ctime=0004525b40e6eaf6 pobjid=0000000000000001 obj_type=7
                 mtime=0004525b40e6eaf6

    The converted ctimes are:

    Fri Jul 18 23:09:33 2008
    Sat Jul 19 00:24:20 2008
    Sat Jul 19 00:16:19 2008
    Sat Jul 19 00:16:31 2008

    So the inodes related to this broken freemap block were all created
    on Jul 18th and 19th.  They are all root inodes for PFS #2.

    This is very good news.  It's either related to the cross-device link
    bug that we fixed, or its related to the PFS root inode creation or
    deletion code.

    I will try to cross reference the data offsets against the rest of
    the dump to see if there are any more freemap discrepancies.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>