No space left on device

[Predrag Punosevac]

Hi DragonFly users,

My DragonFly file server which worked like a swiss watch for number of
years

dfly# uname -a
DragonFly dfly.int.bagdala2.net 6.2-RELEASE DragonFly
v6.2.2.3.gca806c-RELEASE #33: Sun Jul 10 23:48:15 EDT 2022
root at dfly.int.bagdala2.net:/usr/obj/usr/src/sys/X86_64_GENERIC  x86_64

appears to be broken. The machine was recently upgraded to DragonFly
6.2.2 per

https://marc.info/?l=dragonfly-users&m=165490764112773&w=2

This morning I tried to upgrade packages (no particular reason) when I
got this message

dfly# pkg upgrade
pkg: Loading of plugin 'provides' failed: Shared object "libpcre.so.1"
not found, required by "provides.so"
pkg: Plugins cannot be loaded

I would swear that I checked for the package updates after upgrading OS
but I thought that pkg could be temporary broken as it happened once
last year due to some Lua libraries. It appears that is not the case so
I started suspecting that I didn't make install-all or make upgrade due
to my advanced age :-)

Anyhow this gave me the clue

dfly# git pull
remote: Enumerating objects: 80, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 30 (delta 26), reused 0 (delta 0), pack-reused 0
error: unable to create temporary file: No space left on device
fatal: failed to write object
fatal: unpack-objects failed


dfly# df -h
Filesystem                  Size   Used  Avail Capacity  Mounted on
ROOT                       21.7G  21.7G     0B   100%    /
devfs                      1024B  1024B     0B   100%    /dev
/dev/serno/B620550018.s1a  1008M   734M   194M    79%    /boot
/pfs/@@-1:00001            21.7G  21.7G     0B   100%    /var
/pfs/@@-1:00002            21.7G  21.7G     0B   100%    /tmp
/pfs/@@-1:00003            21.7G  21.7G     0B   100%    /home
/pfs/@@-1:00004            21.7G  21.7G     0B   100%    /usr/obj
/pfs/@@-1:00005            21.7G  21.7G     0B   100%    /var/crash
/pfs/@@-1:00006            21.7G  21.7G     0B   100%    /var/tmp
procfs                     4096B  4096B     0B   100%    /proc
BACKUP                     1862G   419G  1443G    23%    /data
MIRROR                      465G   163G   303G    35%    /mirror
/data/pfs/@@-1:00001       1862G   419G  1443G    23%    /data/backups
/data/pfs/@@-1:00002       1862G   419G  1443G    23%    /data/nfs
tmpfs                      1944M     0B  1944M     0%    /var/run/shm


That should not happen as the cronjob should take care of system
snapshots. I am using HAMMER1 on this machine as it was originally
provisioned long before HAMMER2 was production ready. I did have a
power-outage last night so that is why I started tinkering with this to
begin with. After manually running 

hammer cleanup

the system looks better

dfly# df -h
Filesystem                  Size   Used  Avail Capacity  Mounted on
ROOT                       21.7G  12.0G  9997M    55%    /
devfs                      1024B  1024B     0B   100%    /dev
/dev/serno/B620550018.s1a  1008M   734M   194M    79%    /boot
/pfs/@@-1:00001            21.7G  12.0G  9997M    55%    /var
/pfs/@@-1:00002            21.7G  12.0G  9997M    55%    /tmp
/pfs/@@-1:00003            21.7G  12.0G  9997M    55%    /home
/pfs/@@-1:00004            21.7G  12.0G  9997M    55%    /usr/obj
/pfs/@@-1:00005            21.7G  12.0G  9997M    55%    /var/crash
/pfs/@@-1:00006            21.7G  12.0G  9997M    55%    /var/tmp
procfs                     4096B  4096B     0B   100%    /proc
BACKUP                     1862G   419G  1443G    23%    /data
MIRROR                      465G   162G   303G    35%    /mirror
/data/pfs/@@-1:00001       1862G   419G  1443G    23%    /data/backups
/data/pfs/@@-1:00002       1862G   419G  1443G    23%    /data/nfs
tmpfs                      1944M     0B  1944M     0%    /var/run/shm


pkg upgrade works as expected

dfly# pkg upgrade
Updating Avalon repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.pkg: 100%    6 MiB   1.1MB/s    00:06    
Processing entries: 100%
Fetching provides database: 100%   12 MiB   2.6MB/s    00:05    
Extracting database....success
Avalon repository update completed. 30176 packages processed.
All repositories are up to date.
Checking for upgrades (1 candidates): 100%
Processing candidates (1 candidates): 100%
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        curl: 7.83.1 -> 7.84.0 [Avalon]

Number of packages to be upgraded: 1

1 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching curl-7.84.0.pkg: 100%    1 MiB 371.4kB/s    00:04    
Checking integrity... done (0 conflicting)
[1/1] Upgrading curl from 7.83.1 to 7.84.0...
[1/1] Extracting curl-7.84.0: 100%


I can upgrade the sytem

dfly# git pull
remote: Enumerating objects: 80, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 30 (delta 26), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (30/30), 7.70 KiB | 63.00 KiB/s, done.
From git://git.dragonflybsd.org/dragonfly
   353c2689d5..c0211a14ce  master     -> origin/master
Already up to date.



Where do I go from here? Any hints? Did anybody notice any regressions
after the last upgrade? The file system on my machine still seems a bit
too large 12GB but that is due to the existance of /usr/src as well 
/usr/obj/usr directory used for build the sytem. The last directory is
3.2GB

Best,
Predrag