Compatability with FreeBSD Ports

[Chris Pressey]

On Tue, 16 Aug 2005 15:23:17 +0200
Joerg Sonnenberger <joerg at xxxxxxxxxxxxxxxxx> wrote:

> On Tue, Aug 16, 2005 at 12:11:01AM -0700, Chris Pressey wrote:
> > On Tue, 16 Aug 2005 01:19:44 +0200
> > Joerg Sonnenberger <joerg at xxxxxxxxxxxxxxxxx> wrote:
> > 
> > > On Mon, Aug 15, 2005 at 03:55:07PM -0700, Chris Pressey wrote:
> > > > That's not part of package installation, in my view.  That's
> > > > part of package configuration.  All pkg_add should be doing is
> > > > putting the software on my hard drive.  It should not be setting
> > > > it up for me.
> > > 
> > > Package installation can consist of things like restarting
> > > daemons.
> > 
> > Then have a "daemon" class for this purpose, and have it's "upgrade"
> > method issue "kill -HUP ${pid}".  It still doesn't require executing
> > arbitrary commands.
> 
> So you are starting exactly what I said before -- a white list.

No, I am talking about a system of available classes of objects, and
their behaviour - how each of them is allowed to execute commands.  It
is "white" only in the sense that it describes what behaviour is
allowed, and prohibits everything else.  But it is not the same as
merely a list of allowed commands that can be arbitrarily executed,
which is what I think of when you say "white list".

> > > Limiting their use by anything but policies can easily
> > > hurt for those cases where you really need the advanced
> > > possibilities.
> > 
> > Joerg, I will be happy to reconsider my position if presented with
> > evidence to the contrary - such is the reason for debate, after all.
> > 
> > But first I would have to see such evidence.  Can you give examples
> > of these "advanced capabilities" and name some packages that require
> > them?
> 
> packages are used to setup machines, they are a valid mean to automate
> most of the configuration. This is not done for the typical packages,

Because it is a *different problem*.  That's no longer package
installation.  That's system configuration.  I don't think they should
be conflated in the same tool.

> but again Debian shows what can be done. Think about updating the boot
> loader via package, you want or want not it to automatically reinstall
> itself. Under Linux the problem of rerunning lilo after a kernel
> update is a good example of why it should not be forgotten.

But do you have any *existing* examples?

Because, even if, in the future, updates to (say) the NetBSD kernel are
under the domain of pkgsrc, I don't think it would have the same problem
with its bootloader as Debian does with lilo.  (Correct me if I'm
wrong.)  So this example is highly hypothetical, and unpersuasive to
me... but regardless, I don't see why it couldn't be handled (and
handled fairly well) by introducing a "kernel" class to the package
system, if it came down to it.

> > This analogy is specious because all userland programs do
> > *different* things, while all packages have the *same* things done
> > to them - "be installed", "be removed", "be upgraded".
> 
> *most* packages are as simple as install - remove - upgrade. Some are
> more advanced.

You're not telling me which ones - or *why* they have to be treated as
"more advanced" - so I'm getting suspicious.  :)

> > To be clearer on some (not necessarily all) points, I find:
> > 
> > - pkg_*(1)'s methodology (specifically, directives in plists) to be
> >   messy and insecure;
> 
> While some parts can be improved (e.g. reference counting for
> directories, automatic handling for certain file types), I don't
> see what is messy and insecure here. 

@cwd, @exec, @unexec.

(But you will probably say that these aren't messy because they're
needed because the scope of package installation also includes playing
Hunt the Wumpus and making toast, or at least it might someday for a
powerful enough package??)

> > > Building is exactly nothing but scripting a bunch of commands.
> > 
> > Building is nigh-always already taken care of by the 3rd party
> > package.  Issuing "./configure && gmake" is trivial, and if that's
> > all there was to bsd.pkg.mk it wouldn't be ~5000 LoC.
> 
> Sorry, but lol. Just thinking that running "./configure && gmake" is
> enough is funny.

It is quite enough *if everything else is set up correctly*.

Setting everything else up correctly *is not best viewed as just a
matter of scripting a bunch of commands*.

> As Hiten asked for reasons to choose pkgsrc over
> ports, I'll repeat one. pkgsrc ensures that configure doesn't see
> anything it should not see.

This is in fact an example of what I was talking about below - the bulk
of the complexity of the package-build system is about system
organization, for example how the filesystem is organized.  It is not
just about "scripting commands" as you suggest - unless you insist on
looking at it at that level, which I suggest is decidedly suboptimal.

> > Importantly, the building part itself is not where the
> > complexity is.  The complexity is "out there", in the filesystem
> > ("where shall we put the include files and libraries?") and the 3rd
> > party programs themselves ("where is this expecting the include
> > files and libraries to be?")  This part of it is clearly NOT "just
> > scripting a bunch of commands".
>
> Actually, this is just scripting a bunch of commands.

OK, apparently you do insist on looking at it on that level.

> It's setting a
> few variables, sometimes a bit of patching around idiosyncraties. The
> policy which decides the values is something completely different.

The policy decides those commands, at a higher level of abstraction. 
The commands are merely a consequence - an artefact, even - of the
policy.  The policy is what the programmer should be working with - not
the commands themselves, any more than a C programmer should normally be
working with assembly.

> >  No programming language in the world can address
> > this complexity - only policy can do that ("include files and
> > libraries are always located here and always named like this.")
> > There may, however, be some programming languages more suited to
> > expressing these sorts of policies than others.
> 
> Well, *which* one is?

Although I'm no fan of the buzzword, the ideal one would be
"domain-specific" and almost certainly not invented yet.  However, that
doesn't mean that you couldn't code up the basic abstractions that it
would need in a general-purpose, high-level language (or even just
Python! ;)

As a simple example, it is possible to implement the policy "all header
files for package X version Y go under .../include/X-Y", as a method
on the package object, like foo.get_include_dir().  This method can then
be used consistently everywhere this information is needed: for
determining where to put foo's headers when installing foo, for
configuring packages which need foo, and anything else.

Now, I'm sure pkgsrc does try to do things like that.  But I'm also sure
that make does not help them do it, because make was never designed for
building abstractions.  And that was admittedly only a simple example.

> The typical candidates either make the scripting
> part necessary much more complicated

So you keep asserting, but on what reasoning and evidence?  That it is a
few more keystrokes to type sys("foo") than it is to type just foo?

> or have similiar issues to make. 

Perhaps, insofar as they support the same programming style; but it
would inevitably have *fewer* issues than make, insofar as it could
support *better* abstractions.  Such as local variables, to name but one
simple one.

> If you have the alternative between a complex tool and no tool, which
> do you choose?

The existing tool, obviously.

Note that I have not been saying "DragonFly shouldn't use ports/pkgsrc".
What I've been saying is "ports/pkgsrc sucks".  I'm not saying "don't
use things that suck."  I use Windows and Unix and C, after all...

> > [five needed qualities]
> 
> How is that different from any other program? It's says nothing about
> the complexity at all. The details of the normal C RTL are not very
> different from the POSIX toolbox, so I don't see your point here.

It's not much different from C (because, guess what?  C is a pretty
lousy language too.)  My point was simply that it _is_ complex, despite
it being painted by you as being a simple matter of following steps.

Anything that can be done to _actually_ make it simpler - for example,
reducing the number of interactions between symbols, reducing the number
of arbitrary and inconsistent syntax rules - or even to stop modelling
problems in terms of steps and start modelling them in terms of rules -
is a good thing for tractability.  Better tractability means more users
that are capable of acting as developers, which means more maintenance,
which means fewer bugs, which means higher quality in the long term.

> > > The big problems here are the various edge cases,
> > > which are obstacles to understand. But exactly those edge cases
> > > create the power of the system and allow the simple descriptive
> > > Makefiles in most ports/packages.
> > 
> > I don't follow you here.  Could you please give an example of one of
> > these 'edge cases'?
> 
> See above.

You didn't mention edge cases anywhere above.

-Chris