Compatability with FreeBSD Ports

[Chris Pressey]

On Mon, 15 Aug 2005 22:04:25 +0200
Joerg Sonnenberger <joerg at xxxxxxxxxxxxxxxxx> wrote:

> On Mon, Aug 15, 2005 at 12:54:12PM -0700, Chris Pressey wrote:
> > On Mon, 15 Aug 2005 21:03:02 +0200
> > Joerg Sonnenberger <joerg at xxxxxxxxxxxxxxxxx> wrote:
> > 
> > > On Mon, Aug 15, 2005 at 10:59:35AM -0700, Chris Pressey wrote:
> > > > - package install/deinstall can execute arbitrary commands
> > > 
> > > This issue exists for every packaging system out there, simply
> > > because it is necessary for proper operation.
> > 
> > I disagree.  There is absolutely no reason that any package should
> > be able to execute 'fdisk', for example.
> 
> Are you sure about that?

Yes.

> What about an emulator for example, which sets up the disk image to
> use?

That's not part of package installation, in my view.  That's part of
package configuration.  All pkg_add should be doing is putting the
software on my hard drive.  It should not be setting it up for me.

(Writers of package systems always seem to want to endow them with a lot
of "power".  I don't know why.  I don't want a "powerful" package system
any more than I want a "powerful" pet.  Which is why I own a cat, and
not a lion.)

> How do you want to determine which programs it is allowed to execute?

What does it need to get onto my system?  Files?  Then the "install"
command.  Users?  Then "pw useradd" or its equivalent.  Some other
object?  Then the commands which add/remove that object from the system.

Nothing more.

> White lists are difficult to maintain and grow
> very large and black lists have the same issue.

White/blacklists are an awfully ad hoc way of controlling execution.
That the idea is considered at all, probably only comes from the
associated preoccupation with looking at every problem on the level of
individual commands (see below.)  Organization of the resources to be
installed into classes, and determination of what tools are needed for
each class (instead of each individual package, or (holy moley!) each
individual command,) would be much less complex and error-prone.

> If you do bother about which commands will be executed, check the
> scripts first.

I shouldn't have to audit 'N' individual package-scripts when all that
is needed is to make the *one* package system running them secure.

> > > > - bsd.port.mk and friends are almost unreadable/unmaintainable
> > > 
> > > Well, anything comparable to bsd.port.mk or bsd.pkg.mk is a
> > > complicated beast.
> > 
> > Make(1) is basically the wrong tool for organizing something of this
> > level of complexity - especially considering that 80% or more of
> > what ports/pkgsrc does, has *nothing* to do with what make(1) was
> > designed to do (dependency ordering & elimination of redundant
> > actions.)
> 
> There aren't many proper tools around. E.g. plain shell is much better
> either, Python not good for command scripting etc.

One of the reasons our package tools (and many other tools) are not very
good is because package management is not essentially a "command
scripting" problem (nor are many other problems.)

Thinking of everything in terms of "commands" - incontestable, linear,
irreversible, piecemeal updates to a chunk of shared state - is one of
the reasons modern software in general is such a mess.  Package systems
are no exception.  We need more effective abstractions in our tools.

> I've looked at this before and make is about as good or bad as many
> other solutions.

How many of the tools that you have looked at are not essentially
imperative?  Anything functional, deductive, declarative, dataflow,
multi-paradigmal even?

(Python is at least object-oriented.  There should be no real barrier to
making it good at command scripting (assuming that was even what was
called for here) since it's definately possible to create abstractions
suited for such things in an object-oriented language -- in fact, that's
exactly what I did in the installer.)

make(1) doesn't really count as multiparadigmal since what little
logical deduction it does is not being fully utilized, while its crude
imperative abilities (command execution, .if, .for) are being overused
(and, I would argue, abused.)  Not to mention the conceptual impedance
mismatch that results from how the two styles have been thrown
together...  make(1)'s oddball syntax rules, every-variable-is-a-
lazily-expanded-string approach to data, and its hybrid sometimes-lazy-
sometimes-eager flow-control semantics should be enough marks against it
to prevent its use for anything more ambitious than a small project
Makefile.  I dare suggest that people who think otherwise have probably
been holding the make-hammer so long that everything has started to look
like a nail-target.

The only reason that individual ports/pkgsrc Makefiles are as tolerable
as they currently are, is that they're *pretending to be something
they're not* - namely, declarative configuration files.

> > > I agree neither with the unreadable nor with the
> > > unmaintainable, at least for pkgsrc.
> > 
> > How long do you think it would it take the average developer to read
> > and understand the following code excerpt from bsd.pkg.mk?
> 
> Why does he have to?

He/she may or may not have to - that was not the point.  The point was
to demonstrate that the code is not simple, and that being written in
"The Make Programming Language" does not help.  (In regards to that, you
did not answer the question:  How long DO you think it would take the
average developer to read and understand that code?)

However, if you want me to make it the point, I'll tell you:

Because otherwise, the control of the tools remains only in the hands of
those most expert and most familiar with a particular system (in this
case pkgsrc, but this is only one system of thousands.)

In essence, the complexity sets up a dependency between the users and
casual developers who do not understand the system, and a class of "core
developers" who do understand it.  Insofar as that complexity is
unnecessary, that dependency is unnecessary too.

If the tools were made simpler, they would be understood by more
developers.  As such, their evolution would be subject to wider, more
democratic forces.

If the tools were made very simple, their evolution might even be in the
hands of the users.  Imagine that!

Of course, it might not be possible to make them that simple.  But
dismissing the possibility before even trying is (at least to me) akin
to giving up on democracy and settling for an elitist oligarchy.

Is it acceptable to you that the average developer (never mind the
average user!) is not expected to be able to understand the tools that
they use?  What is the point of the source being "open" in this
situation, when it's still effectively "closed" to most people?

If it is acceptable to you, then I'd like to know: how do you reconcile
this with your position of "if you're worried that a call to pkg_add
might trojan your machine, you should audit the install script yourself
before doing it"?  Are users expected to be skilled enough to audit
package install scripts, but not skilled enough to understand how the
package system software itself works?  Or are users simply expected to
put their absolute trust in the package developers?

-Chris