File system panic on recent HEAD
dillon at apollo.backplane.com
Wed May 21 10:54:02 PDT 2008
:* Matthew Dillon wrote:
:> Matthias, did that partition (/build) at any point in the past fill up
:> to 100% ?
:No. The partition has enough space left:
:/dev/ad10s2 256G 33G 203G 14% /build
:> In anycase, what I recommend is that you umount the partition and do
:> another manual fsck of it, then be careful not to fill it up and see
:> if the problem reoccurs.
:Manual fsck did not help. I did it several times before and now again.
:Sadly the machine crashed again last night. And again no coredump:
:dev = #ad/0x1e130057, block = 30408, fs = /build
:panic: ffs_blkfree: freeing free block
:mp_lock = 00000000; cpuid = 0
:boot() called on cpu#0
:syncing disks... Warning: vfsync_bp skipping dirty buffer 0xc4491550
:640 Warning: vfsync_bp skipping dirty buffer 0xc4491550
:619 Warning: vfsync_bp skipping dirty buffer 0xc4491550
Another possibility is that fsck is not able to repair the problem.
Try completely wiping the filesystem by re-newfs'ing it maybe?
If THAT doesn't work then we've wiped out any chance of it being the
only currently known UFS bug and something in the package build you
are running is tweaking another bug.
I am going to go ahead and bring in the fix from FreeBSD related to
full UFS filesystems, but I don't think it fixes the issue you are
Which leads us to a second option: Get the absolute latest HEAD running
on that box and run a HAMMER filesystem for that partition instead of
UFS. With the proviseo that you absolutely cannot fill it up or HAMMER
will blow up (that being the last bit of the filesystem that I am
working on this week). If there is any corruption occuring in the
driver/DMA path HAMMER should be able to detect it. Also make sure
the latest version of the hammer utility and newfs_hammer is installed
Something like this:
newfs_hammer -L BUILD /dev/ad10s2
And in fstab:
/dev/ad10s2 /build hammer rw,nohistory 2 2
I'm presuming you wouldn't want it to retain historical information.
However, even if you mount with the nohistory option it is still a
good idea to reblock the filesystem every so often to clean up loose
ends. Something like this in a once-a-day cron job would do the
hammer -t 600 -c /var/db/reblock-build.cycle reblock-inodes /build 100
hammer -t 600 -c /var/db/reblock-build.cycle reblock-btree /build 100
hammer -t 600 -c /var/db/reblock-build.cycle reblock-data /build 95
In anycase, I am not suggesting that you start using HAMMER for
production stuff yet, but it might give us another data point on whether
the corruption is due to a bug in UFS, or due to a bug in the rest of
the kernel, or due to driver/DMA issues. And there's a risk you might
hit a HAMMER bug too and start tearing your hair out for real.
I am going to also note that this particular bug has been reported
sporatically on FreeBSD and DragonFly, NetBSD, and OpenBSD.
<dillon at backplane.com>
More information about the Bugs