link: "Recursive Make Considered Harmful"

[Andrew Hacking]

On Fri, 2005-01-14 at 02:13, Joerg Sonnenberger wrote:
> On Thu, Jan 13, 2005 at 02:35:49PM +1000, Andrew Hacking wrote:
> > It would seem that systems which require *all dependency knowledge* up
> > front are inappropriate where it is desirable to _not_ have a
> > pre-patched universe checked out on your machine, unless of course you
> > recursively used the build tool..and now we have come full circle to the
> > typical recursive make approach.
> 
> Well, it certainly is possible to allow dependency tracking tools to
> work increment to a certain degree. The problem with this comes from
> the fact that certain a-priory assumptions become void. All build tools
> I'm aware of depend on the "build-at-most-once" assumption, if only to
> avoid recursions. Another problem is variable modification in the
> dependency + build phase can lead to a lot of nasty problems.

Yes, have been bitten in the past by 'evaluate on use' variables.

> 
> For me, it is easier to do a recursion on demand, than an overdesigned
> build tool. Jam would allow us to do a recursion-free buildworld without
> any nasty tricks we need a recursion-free make buildworld, by proper
> exploiting the target local variable name space. The issue about jam
> not allowing running commands to read variables can be solved either by
> adding this or avoiding it completely. The problem of filename as target
> is avoided by the same way we already solve it for make: keep the
> subdirectory structure for the object file space as well.
> The problem of addiontal dependencies from external tools, which will come
> up again once we start to split out certain parts of contrib even more,
> can be solved by doing a full recursion there, they should by definition
> be self-contained and just have a fixed list of dependencies on the
> outer-world.
> 

Agree with your points. Thats why I _don't_ consider recursive builds
harmful in many circumstances. There is real world justification for it,
eg building a port, or other similar situations where the source is not
immediately available or in its final state at dependency calculation
time. ie cases such as checkout from a VCS, or fetch via ftp/http or the
source requires patching (which can effect the resultant dependency
calculation on the source).

However, as you have demonstrated, good old make may not necessarily be
the best tool. Your results show that jam can evaluate most dependencies
up front in typical usage (at least for the DFly sources) which allows
the build to be parallelised more than a typical recursive decent make
approach can.

It is certainly possible that extending jam to allow assignment of
variables as the output of a shell command (at the parsing phase only)
would not necessarily corrupt the dependency calculation logic.  The
strongest guarantee comes from using only commands that have no
side-effects. Such a policy however, wouldn't allow  fetching or
patching the source, unless this sort of thing could be
deferred/implemented in the guise of a dependency generator.

-Andrew Hacking