first stab at simple mailer
dillon at apollo.backplane.com
Wed Mar 21 11:01:01 PDT 2007
:The queue works like this:
:- create a temp file
:- this queue file has a "header" prepended
:- read the mail from stdin and write to the temp file
:- each recipient gets assigned a queue id, which is the filename at the s=
:- the temp file gets linked to the queue ids
:- when a recipient was processed, the link gets removed
:this way I only have one instance of the mail on disk and I don't need ad=
:ditional files to track for one mail.
It's a fair idea, but very very filesystem intensive. Using softlinks
might be less invasive (at least then the inodes don't have to be
updated every time you make a delivery).
An even better solution would be to generate a worklist in one
or more files... maybe chunk it up, 20 recipients per file or something
like that. Doing it that way then gives you the option of pre-sorting
the worklist when generating it (in the future) without having to also
change the backend code that processes it.
You need to do this anyway to avoid making a separate connection to
any external entity for each recipient in a list. It's not optional.
:> * Remote delivery, which you have as 'XXX implement me'... you nee=
:> to be able to sort by the target domain's MX lookup and you need =
:> do MX backoff & forwarding for any remote delivery. It isn't too=
:> bad but DNS lookups can be difficult to implement efficiently.
:I don't really care about efficiency. This mailer is supposed to be a lo=
:w-end thing, just able to enable a base system to deliver common day mail=
: loads (by periodic, cron and users). This is *NOT* supposed to be a hig=
:h-performance mail server. Anybody who needs any sort of sophisticated m=
:ail processing can use the mailer they want.
Yes, but I think it still needs to be designed in such a way that
the efficiency is reasonable, and to leave the door open for
future work to make it even more efficient. Continue reading on
for more on this.
:> * Configuration file. Yup. you need one :-).
:Not sure what I want to configure. Maybe a smart/relayhost, but nothing =
The various hardwired constants you have at the moment at the very
least. Since it will be interfacing with the user you must also be
able to configure maximum mail header and body sizes.
:> How serious are you about this? There are a number of issues that
:> would have to be addressed, even for a simple mailer. Handoffs for=
:> .forward files (including processing exit codes properly),
:What do you mean with handoff? I want to support .forward, but only for =
:file and address forwards, not for command forwards.
You have to support command forwards, or it isn't useful. Many, many
people use things like procmail and it must work in the base system.
It actually is not all that hard to do, you basically just create a
pipe and fork/exec (with appropriate permissions and ownership changes)
the program, pipe the data to it, process the response (if any), and
handle the exit code.
:> /etc/mail/aliases and /etc/mail/virtusertable,
:yes, aliases is in work. I've never heard about virtusertable before, an=
:d I don't see where this might be necessary in a simple mail setup. This=
: mailer accepts only local mails. No listening on port 25. As soon as y=
:ou need mail rewriting, a "big" mailer should be installed.
You need it even for simple mail setups. Hosting multiple domains is
something that is very common these days and it has to be supported
out of the box. But, more importantly, the local mailer either needs
to be told that a recipient is a local recipient, or it needs to be
able to figure out in absolute terms whether the recipient is local
or not, or you risk creating severe mail loops.
With regards to these files... don't go the DBM route that sendmail
took. Just index them internally (in memory) and do a stat() check
to determine when they have changed (so the daemon doesn't have to be
:> and handling a more
:> sophisticated local delivery check (using a DNS lookup on the targe=
:> domain to see if its a CNAME to the actual host, or something simil=
:I thought about this, but I decided to go down the easy road for now. Th=
:is shouldn't be much of an issue. Again, the mailer is not designed to a=
:ct as an MX. How do I find out if an IP address is local? This check wo=
:uld basically be the only important check.
I don't think it is optional. The mailer really needs to be able to
determine whether a recipient is local or not. You could list the
local domains in a file (in the case of sendmail, the file is called
/etc/mail/local-host-names) and not bother with the DNS lookup.
Doing a DNS lookup is far more convenient, though (in addition to having
the file), because then the user doesn't have to maintain a list of
:> There are many other issues as well, including mail size limitation=
:> smarter queueing mechanisms (sorting by target recipient to prevent=
:> stalled target host from stalling the entire queue), handling
:> out-of-disk-space issues gracefully, etc.
:There is no global queue manager. Each mail gets processed on its own. =
:Yes, I am aware of the fact that this might open two connections to a des=
:tination mailer for two mails. However, this is a simple mailer and for =
:simple mail loads, this won't be a problem.
You could make it an MX-forward-only mailer, sure. That's reasonable.
But you still need to efficiently queue the mail for forwarding. You
can't just make separate connections to the target for each recipient.
That makes it impossible for the target to optimize its own mail storage.
In a MX-forward-only setup you still basically want to make just a
single connection to the forwarding host and pass all the non-local
recipients to it for any given piece of mail.
In a direct-delivery setup you still need to optimize multiple recipients
going to the same target. Again, its an absolute requirement for even
a simple mail system. It's unfair to burden target systems with separate
connections that make it impossible for those systems to efficiently
store the mail for further processing (just as you want to efficiently
store multiple recipients in your own code, so must other targets be
able to do the same thing with the mail you send to them).
:Out of disk space is being handled: if the mail can be queued, everythin=
:g is fine. If not, the mailer errors out (nothing else it can do).
:I don't think I want to care about mail size limitations. That's the job=
: of a "real" mail server.
It isn't something that has to be implemented right off the bat, but
generally speaking you need to be able to set limits for anything
that interfaces to userland. You don't want root mail from cron jobs
to break because some userland program filled up the mail queue
(intentionally or not).
:> the back-end over a localhost socket and pass creds with sendmsg() =
:> and SCM_CREDS (with regards to a mail.local equivallent).
:I don't really want to split this into backend daemon and frontend tool. =
: Besides, where is the difference? One of them always needs to run with =
:elevated privileges. I do *not* want to make it setgid wheel. Just setg=
:id mail, where mail may write to /var/spool/mqueue and to /var/mail/*. I=
:t's really supposed to be simple. No running daemons required.
The difference is that the user can manipulate the frontend tool far
more then the user can manipulate the backend tool. For exmaple, ^C,
or ^Z, and so on and so forth. It's very dangerous. Having SUID or
SGID programs directly manipulate subsystems is a bad idea in general.
<dillon at backplane.com>
More information about the Kernel