distcc port override is finally here!

[Devon H. O'Dell]

Hey everyone,

As many people have noticed, our world does not build with distcc. This 
is no longer the case!

In discussing the issue with Simon and Jörg, it became apparent that 
distcc was not equipt with the functionality that our build system 
needs. Namely, when we bootstrap our GCC, we rely on the OBJFORMAT_PATH 
variable in several places to tell objformat which CC to use.

The fix? Allow distcc to export environment variables.

The quirks? You're required to share /usr/obj via NFS when you want to 
use distcc for building the world.

The result? I'm somewhat disappointed, actually. I remember doing a 
buildworld on Leaf with no -j flags (it's not allowed) and it finished 
in just under fifteen minutes. On DFCluster (my 7-blade P3 800 cluster) 
it takes 1 hour and 10 minutes. There are a couple of possible 
bottlenecks that I faced here:

1) I was running this under script buildworld.out, so that I could 
determine where things were going wrong when / if they went wrong,

2) DFCluster uses 4200RPM hard drives (please, if you want to donate an 
old SCSI fibre channel RAID array, I'd welcome it!)

3) I'm not utilizing any of the VFS improvements that have been made, 
this was on a 1.0A system.

4) 2 blades are not functioning due to a botched upgrade on them.

5) I know nothing about NFS tuning, I started up portmap, mountd and 
nfsd with no special options, no kernel hacks and a fairly standard 
exports line:

usr -alldirs -mapall=root -network 192.168.1 -mask 255.255.255.0

which then had /usr/obj mounted on all of the blades.

Distcc was run as a handler for cc and c++. My buildworld line was:

make -j 20 CC=/usr/local/bin/distcc C++=/usr/local/bin/distcc buildworld

The world I built was yesterdays DragonFly_Stable tagged world.

So, overall, with 5 active blades (4000MHz of processor power and 4.5GB 
RAM overall shared between them), I suppose I should be happy. I'll be 
doing more testing in the future with it, now that it works. Namely, I'd 
like to see how long it takes to build the world on _one_ blade, so that 
we can get an idea what kind of improvement it provides.

Where do you get it? I've got a tarball for inclusion in dfports at: 
http://www.the-bofh.org/projects/distcc-env/distcc.tar.gz

Have fun!

Devon H. O'Dell