Compatability with FreeBSD Ports

[Joerg Sonnenberger]

On Tue, Aug 16, 2005 at 12:11:01AM -0700, Chris Pressey wrote:
> On Tue, 16 Aug 2005 01:19:44 +0200
> Joerg Sonnenberger <joerg at xxxxxxxxxxxxxxxxx> wrote:
> 
> > On Mon, Aug 15, 2005 at 03:55:07PM -0700, Chris Pressey wrote:
> > > That's not part of package installation, in my view.  That's part of
> > > package configuration.  All pkg_add should be doing is putting the
> > > software on my hard drive.  It should not be setting it up for me.
> > 
> > Package installation can consist of things like restarting daemons.
> 
> Then have a "daemon" class for this purpose, and have it's "upgrade"
> method issue "kill -HUP ${pid}".  It still doesn't require executing
> arbitrary commands.

So you are starting exactly what I said before -- a white list.

> > Look at Debian, you might or might not agree with it, but it is
> > difficult to clearly cut lines. Also keep in mind that the
> > INSTALL/DEINSTALL scripts are certainly not the most dangerous parts
> > of a package.
> 
> But that's not a reason for them to _stay_ dangerous.
> 
> > Limiting their use by anything but policies can easily
> > hurt for those cases where you really need the advanced possibilities.
> 
> Joerg, I will be happy to reconsider my position if presented with
> evidence to the contrary - such is the reason for debate, after all. 
> But first I would have to see such evidence.  Can you give examples of
> these "advanced capabilities" and name some packages that require them?

packages are used to setup machines, they are a valid mean to automate
most of the configuration. This is not done for the typical packages,
but again Debian shows what can be done. Think about updating the boot
loader via package, you want or want not it to automatically reinstall
itself. Under Linux the problem of rerunning lilo after a kernel update
is a good example of why it should not be forgotten.

> This analogy is specious because all userland programs do *different*
> things, while all packages have the *same* things done to them -
> "be installed", "be removed", "be upgraded".

*most* packages are as simple as install - remove - upgrade. Some are
more advanced. Nevertheless to come back to the original point, asking
before arbitrary commands are to be executed would be a nice thing. But
it must be able to turn it off. -I is not enough for that.

> To be clearer on some (not necessarily all) points, I find:
> 
> - pkg_*(1)'s methodology (specifically, directives in plists) to be
>   messy and insecure;

While some parts can be improved (e.g. reference counting for
directories, automatic handling for certain file types), I don't
see what is messy and insecure here. 

> - that the two exhibit an undesirable tight coupling.

There are three parts where bsd.pkg.mk and pkg_* interact: ensuring that
all dependencies are available, registration of an installed package and
building a package. I don't see how this could be more light. Removing
the need for the second is on the TODO list, but it is not a simple
thing either.

> > Building is exactly nothing but scripting a bunch of commands.
> 
> Building is nigh-always already taken care of by the 3rd party package. 
> Issuing "./configure && gmake" is trivial, and if that's all there was
> to bsd.pkg.mk it wouldn't be ~5000 LoC.

Sorry, but lol. Just thinking that running "./configure && gmake" is
enough is funny. As Hiten asked for reasons to choose pkgsrc over ports,
I'll repeat one. pkgsrc ensures that configure doesn't see anything it
should not see. For example, if you don't want mplayer to build with
vorbis support, you have to ensure that configure can't find the
headers. That's what the buildlink framework ensures. It's complicated,
but very useful.

Also those packages for which ./configure && gmake && gmake install is
enough is the nice, easy to build fraction. Now add the magic for Perl
modules, Python modules, Java engines, source patching, dealing with
incompatible base utilities (BSD make vs. GNU make vs. imake, bash vs.
sh, etc.), ensuring that all requirement are meet etc. Sure, for one
individual package you might not need all of it -- heck, 5% would often
be enough. But if you only want to have that 5%, you end up duplicating
a lot of scripts like the infamous debian rules in many tarballs do. I
don't consider that useful either.

> So to say "bsd.pkg.mk" is "about building packages" is at best slightly
> misleading.  Importantly, the building part itself is not where the
> complexity is.  The complexity is "out there", in the filesystem ("where
> shall we put the include files and libraries?") and the 3rd party
> programs themselves ("where is this expecting the include files and
> libraries to be?")  This part of it is clearly NOT "just scripting a
> bunch of commands".

Actually, this is just scripting a bunch of commands. It's setting a few
variables, sometimes a bit of patching around idiosyncraties. The policy
which decides the values is something completely different.

>  No programming language in the world can address
> this complexity - only policy can do that ("include files and libraries
> are always located here and always named like this.") There may,
> however, be some programming languages more suited to expressing these
> sorts of policies than others.

Well, *which* one is? The typical candidates either make the scripting
part necessary much more complicated or have similiar issues to make. 

> First you said "make is about as good or bad as many other solutions",
> now you say "You can improve make e.g. when looking at ant or jam [...]
> A lot of things might be simpler for pkgsrc when moving e.g. to jam".
> 
> I'm not certain what to take from this.  Which do you believe?

They can be simplified, but they can't remove the complexity. It would a
huge effort to port the existing rules to a new system, the question is
whether the gain is worth it. I've postponed by afforts for our base
system and that is by an order of magnitude simpler than what pkgsrc has
to deal with.

To come back to your question what I believe, I do consider make as good
or as bad as jam, since the latter might be more expressive, but doesn't
reduce the magnitude of complexity.

> > Just an example from the subject I study: the integrate
> > command of Computer Algreba Systems is very easy to use. You give it a
> > term, the independent variable and maybe also the limits, it computes
> > the closed-form of the integral if possible. The actual algorithm has
> > the size of the Bible, even Mathematica and Maple needed *years* to
> > fix most bugs.  Does that make the tool of symbolic integrate less
> > useful?
> 
> Less useful?  No.  Easy to use?  Yes.
> More expensive?  Yes.  More bug-prone?  Yes.

If you have the alternative between a complex tool and no tool, which do
you choose? At least it is easy to verify that the solution is correct,
something which isn't possible in all situations.

> - intimate familiarity with the nature of each step (in our case, the
>   individual commands;)

Like the basic POSIX tools? Yes, that's requirement.

> - intimate familiarity with the flow of execution between steps (in our
>   case, how execution happens in make and sh;)

sh is just used execute (groups of) commands,it is entirely controlled
by make. You have to understand the sh and make syntax, sure. But that's
like requiring you to know the full C syntax.

> - intimate familiarity with what each symbol being worked upon
>   represents (in our case, the syntax, and variable and target naming;)

The same.

> - the mental capacity to visualize the cumulative effects of each step
>   on the global state (in our case, the filesystem and environment;)
> - the patience to trace out however many steps might be involved
>   (in our case, maybe in the dozens to hundreds range per package.)

How is that different from any other program? It's says nothing about
the complexity at all. The details of the normal C RTL are not very
different from the POSIX toolbox, so I don't see your point here.

> > The big problems here are the various edge cases,
> > which are obstacles to understand. But exactly those edge cases create
> > the power of the system and allow the simple descriptive Makefiles in
> > most ports/packages.
> 
> I don't follow you here.  Could you please give an example of one of
> these 'edge cases'?

See above.

Joerg