cvsup

[Vincent Stemen]

On 2008-01-15, Simon 'corecode' Schubert <corecode at fs.ei.tum.de> wrote:
>
> Yes.  There are a couple of mirrors which serve the cvs repo via rsync. 
> For instance use chlamydia.fs.ei.tum.de.  There are more, I guess.

Thanks.  

I just checked the list, but most of the sites say they only mirror
_Daily snapshots and official ISOs_.  The rsync link to the one you
mentioned, *chlamydia.fs.ei.tum.de*, does take you to a web page that
says they have the cvsup collections, but they do not provide the full
path to access them via rsync.

_TheShell.com_ is the only other one that lists *Code* under _Mirrored
Data_.  Their rsync link takes you to
rsync://rsync.theshell.com/pub/DragonFly but there are no instructions
on the full path to access the repository.  I can only guess
    rsync://rsync.theshell.com/pub/DragonFly/dragonfly-cvs-src
I will experiment with it.


>
>> Also, has anybody considered writing some front end scripts to rsync to emulate
>> the necessary features of cvsup?  I have found rsync to be very fast since it
>> efficiently only transfers what has changed.  I also do not see any reason for
>> the mirroring tool to need any knowledge of a specific repository type (e.g
>> cvs, svn, etc).  You are just mirroring files.
>
> Rsync is very slow when you tag a repo, because all the files change, but 
> only one line.  It's not really optimized for this task.  But this doesn't 
> happen very often, though.

That is interesting.  The rsync docs claim to be very fast and say this

    The rsync remote-update protocol allows rsync to transfer just the dif-
    ferences between two sets of files across the network connection, using an
    efficient  checksum-search  algorithm ...

I ran a test transfering 8 text files over the internet that are each about
450K and the performance seems pretty good when I changed just one line in each
file.  In my test I told rsync to use ssh, so the transfer was also encrypted.

Here are the results.

*Initial download*

$ time rsync --stats  --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test .
receiving file list ... done
test/
test/testfile1
test/testfile2
test/testfile3
test/testfile4
test/testfile5
test/testfile6
test/testfile7
test/testfile8

Number of files: 9
Number of files transferred: 8
Total file size: 3607368 bytes
Total transferred file size: 3607368 bytes
Literal data: 3607368 bytes
Matched data: 0 bytes
File list size: 168
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 202
Total bytes received: 3608346

sent 202 bytes  received 3608346 bytes  379847.16 bytes/sec
total size is 3607368  speedup is 1.00
rsync --stats --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test .  0.19s user 0.04s system 2% cpu 9.059 total
============

*After modifying each file*

$ time rsync --stats  --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test .
receiving file list ... done
test/
test/testfile1
test/testfile2
test/testfile3
test/testfile4
test/testfile5
test/testfile6
test/testfile7
test/testfile8

Number of files: 9
Number of files transferred: 8
Total file size: 3607376 bytes
Total transferred file size: 3607376 bytes
Literal data: 5608 bytes
Matched data: 3601768 bytes
File list size: 168
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 31162
Total bytes received: 26842

sent 31162 bytes  received 26842 bytes  23201.60 bytes/sec
total size is 3607376  speedup is 62.19
rsync --stats --del --rsh="ssh $wh_ssh_args" -auv $whr:/var/tmp/test .  0.08s user 0.02s system 6% cpu 1.469 total
============

As you can see, it took 9 seconds for a full download, but only less than 1.5
seconds to update them.  That seems reasonably fast to me.  Is cvsup really
faster than that?  I am sceptical that it could be much faster.