buildworld fails with distcc, buildkernel works fine though

[YONETANI Tomokazu]

Hi.
On Mon, Oct 11, 2004 at 10:14:04PM -0700, Daimao wrote:
> I'm posting again with an update, hopefully I can get a response out
> of someone this time.  I need a bit of help here.
> 
> First off, the good news.  I've had two successfull kernel builds
> using distcc on both of my dfly systems.  So that should work for
> anyone else too, I think.  Didn't require any voodoo whatsoever, just
> install the package through "pkg_add -r" and follow the docs on the
> distcc site.  Then do:
> cd /usr/src
> setenv DISTCC_HOSTS "localhost 123.123.123.123"
> make -j5 CC=/usr/local/bin/distcc buildkernel
> According to the distcc site, you should do two times the number of
> cpus plus one, as ball park figure.  So that's what I did.
> 
> Now, the bad news.  Buildworld fails, and it appears to fail in a
> different spot each time.  Now, I'm not much of a coder so I don't
> really know what's going on, but each time there was a lot of include
> files with warnings about "unrecognized data at end of #line".  Or
> some other such thing.  The other thing is the actual file failing to
> build always seems to be on the remote distcc system.  I'll have to
> figure out how to get distcc to tell me what the actual compile time
> error was though, it only tells me that a file failed.

- showing the exact place where your buildworld failed and the actual
  error messages will give us better information than just saying
  "some such things" or stuff like that :)

- the system clock on each node must be in sync or the make program
  gets screwed.

- if you keep your source tree on other place than /usr/src, you need to
  have exactly the same realpath for your source tree on every node,
  because build tree is created using realpath under $MAKEOBJDIRPREFIX
  where you probably going to share with NFS.

- you probably have to install exact same version of OS on each
  node, same make.conf file, and the same environment variables.
  diffent configurations produce diffent outputs, so mixing them up
  will result in unexpected errors.

- you should not use CPUTYPE higher than your build nodes accept,
  because buildworld uses compiler and other tools that get built
  in the early stages.

- I think you'll get better result by distributing sub-makes to remote
  machines rather than compiler processes, because buildworld involves
  many other tasks than just compiling C codes, like formatting manpages
  and documents.