File changes propagating in older snapshots: what am I doing wrong?

[Andrea]

Hi, with help form this list I've set-up my main main workstation  and 
almost ported all my work to DragonFly BSD (thanks!)

Now I'm moving all my data and but I'm left with one problem I'm not 
able to sort out: some of the changes to files are propagated in older 
snapshots! It's obviously impossible so or I've done something horribly 
wrong during disk set-up or I don't understand how snapshots works.

The "short" version:

1) /home/andrea/ is my home directory in a master PFS
2) /home/attic/ is another directory in another master PFS
3) Both PFSs have a snapshot done yesterday.
4) If I add or remove files sometimes changes are not visible in 
yesterday's snapshots, but sometimes they are.

As an example, today I've added (added, not modified) this file:
% ls -l /home/attic/disk_stats.txt
-rw-r--r--  1 andrea  wheel  167 Apr  7 11:38 /home/attic/disk_stats.txt

And I found it in yesterday's snapshot:
% ls -l /var/hammer/home/attic/snap-20170406-0957/disk_stats.txt
-rw-r--r--  1 andrea  wheel  167 Apr  7 11:37 
/var/hammer/home/attic/snap-20170406-0957/disk_stats.txt

File date in the snapshot is 1 minute earlier than original but the file 
shouldn't be there at all.

5) Another example: today I moved a whole directory from /home/attic/ to 
/home/andrea/ and:
- It disappeared from /home/attic/ yesterday's snapshot
- It appeared in /home/andrea/ yesterday's snapshot
- Files in /home/andrea/ kept the last modify dates while files in the 
snapshot have the date of the moment I moved the directory

6) I'm 100% sure I didn't move file directly inside the snapshot directory


Some details:

1) OS version is DragonFly v4.8.0-RELEASE #4
2) Both directories /home/andrea/ and /home/attic/ are mirrored on a 
second disk.
3) This is the output of snapls and pfs-status for /home/andrea/:

# hammer snapls /home/andrea/
Snapshots on /home/andrea/	PFS#1
Transaction ID		Timestamp		Note
0x0000000000000000	2017-04-06 09:57:34 CEST	-

# hammer pfs-status /home/andrea/
/home/andrea/	PFS#1 {
     sync-beg-tid=0x0000000000000001
     sync-end-tid=0x00000001183f03c0
     shared-uuid=f030302c-1895-11e7-b1dd-714d7b6e28cc
     unique-uuid=f0303031-1895-11e7-b1dd-714d7b6e28cc
     label=""
     prune-min=00:00:00
     operating as a MASTER
     snapshots directory defaults to /var/hammer/<pfs>
}

4) Here I find my snapshot for /home/andrea/:

# ls -l /var/hammer/home/andrea/snap-20170406-0957
lrwxr-xr-x  1 root  wheel  33 Apr  6 09:57 
/var/hammer/home/andrea/snap-20170406-0957 -> 
/home/andrea/@@0x0000000000000000

5) The output of snapls and pfs-status for /home/attic/:

% hammer snapls /home/attic/
Snapshots on /home/attic/	PFS#2
Transaction ID		Timestamp		Note
0x0000000000000000	2017-04-06 09:57:34 CEST	-

% hammer pfs-status /home/attic/
/home/attic/	PFS#2 {
     sync-beg-tid=0x0000000000000001
     sync-end-tid=0x000000011840a4b0
     shared-uuid=7bb3b3bc-18ab-11e7-9d0e-714d7b6e28cc
     unique-uuid=7bb3b3e1-18ab-11e7-9d0e-714d7b6e28cc
     label=""
     prune-min=00:00:00
     operating as a MASTER
     snapshots directory defaults to /var/hammer/<pfs>
}

6) Here I find my snapshot for /home/attic:

% ls -l /var/hammer/home/attic/snap-20170406-0957
lrwxr-xr-x  1 root  wheel  32 Apr  6 09:57 
/var/hammer/home/attic/snap-20170406-0957 -> 
/home/attic/@@0x0000000000000000

7) I try to reproduce the problem but everything seems ok:

I delete a file in my home directory and I still find it in the snapshot:

$ rm /home/andrea/0
$ ls /home/andrea/0 /var/hammer/home/andrea/snap-20170406-0957/0
ls: /home/andrea/0: No such file or directory
/var/hammer/home/andrea/snap-20170406-0957/0

Now I add a file in my home directory and I don't find it in yesterday's 
snapshot:

$ touch xxx
$ ls -1 /home/andrea/xxx  /var/hammer/home/andrea/snap-20170406-0957/xxx
ls: /var/hammer/home/andrea/snap-20170406-0957/xxx: No such file or 
directory
/home/andrea/xxx

8) But, I moved a directory in my home directory *today* and I find it 
in *yesterday's* snapshot:

% ls -l /home/andrea/bin/ | grep launcher
-rwxr-xr-x  1 andrea  wheel        57 Jun 11  2013 ll.urxvt_launcher

% ls -l /var/hammer/home/andrea/snap-20170406-0957/bin/ | grep launcher
-rwxr-xr-x  1 andrea  wheel        57 Apr  7 11:42 ll.urxvt_launcher

In other words: today I added the whole /bin directory inside my home 
and I find it in yesterday's snapshot. Files in my home directory have 
the original dates (2013 in this case) while version in the snapshot 
have the date of the moment I added the files. As you can see from the 
output above there's a file dated 2017-04-07 inside the snapshot dated 
2017-04-06 which should be impossible

9) In the source directory the directory disappeared from yesterday's 
snapshots.

10) I didn't mess with the snapshot directories directly (I double 
checked my command history to be sure).


I'm completely lost here, any suggestion about what I'm doing wrong or 
what I can do to diagnose further?


Regards,
Andrea