Dual-instance sendmail with amavisd-new
---------------------------------------
Mark Martinec, 2003-05-06
(based on initial research by Ricardo Stella)
updated on: 2005-09-22 (added a reference to 'milter-ahead');
updated on: 2005-09-29 (added custom rules to reject unknown users outright,
provided by Matej Vela, thanks to Simone Marx)
updated on: 2006-09-15 (placement of DKIM/DK milters, mention the feature
FEATURE(`nocanonify',`canonify_hosts'), the absence
of which can make header procesing by sm *very* long)
The most recent version of this document can be found at:
http://www.ijs.si/software/amavisd/README.sendmail-dual
==========================================================================
The setup is very similar to the one described in README.sendmail
(by Rainer Link) in section 'Scanning incoming/outgoing and relayed mail',
except that it uses SMTP protocol over inet socket (instead of pipes
to commands) to transfer files between MTA and amavisd-new and back,
and that it uses a permanently running second sendmail instance
in 'queue only' delivery mode, instead of bringing it up every time
a new checked mail comes from amavisd.
==========================================================================
Comparing the setup described in this document with the sendmail milter
setup, as described in README.milter:
milter - reasons in favour:
- can REJECT on the original SMTP session, instead of generating a bounce
(sending a non-delivery notification _after_ the mail has been enqueued);
- only one sendmail daemon need be running, only one config file needed,
no additional queue area needed (although starting with sendmail 8.12
more than one queue area is already a norm: clientmqueue, queue groups;
and MSP already uses a different .cf file);
dual-MTA - reasons in favour:
- Full amavisd-new functionality is available, including adding spam and
virus information header fields, adding address extensions and removing
certain recipients from delivery while delivering the same message to
the rest (*_lovers). Also a message can be split if different recipients
need different header edits. All this is not available when using
amavis-milter helper program.
- Content scanning need not be performed at the time of mail reception.
This allows better control on CPU-intensive content filtering: mail
checking can be streamlined and performed at optimum throughput setting
(number of content checker processes) so as not to overwhelm host resources,
instead of leaving it at the mercy of the current number of incoming
SMTP sessions where available crude controls are mostly based on system
load. Typically the number of incoming SMTP sessions (tiny processes)
is desired to be many times above the number of content filtering
processes (heavy resource consumers).
- No helper programs needed, MTA communicates with amavisd-new directly
via SMTP, saves on creating one directory and one file for each message,
and deleting it (at the cost of one additional transfer);
- Receiving sendmail daemon (MTA-RX) need not run as root (using option
RunAsUser) since it does not need to run any local delivery agents (LDA)
or to access user .forward files. This avoids external SMTP clients
talking directly to a process running as root.
==========================================================================
The following setup is described in this document:
............................ ............................
: sendmail instance MTA-RX : : sendmail instance MTA-TX :
: : : :
25 -----> \ (mqueue-rx) : : (mqueue) / -------> forward
587 -----> > -queue- : : -queue- ->-----+ :
^ : / | MAIL_HUB, : : | \ -------> local
| : v SMART_HOST : : ^ : delivery
msp ...........|................ ....|.......................
| ^ loopback interface
v | port 10025
loopback interf.| port 10024 |
.....|.......................|.............
: $inet_socket_port=10024 | :
: | :
: $forward_method='smtp:127.0.0.1:10025' :
: $notify_method ='smtp:127.0.0.1:10025' :
: :
: amavisd-new :
...........................................
The setup is based on the recent sendmail (8.12.9 or later) with its set of
m4 configuration macros. Because of several security problems with earlier
versions of sendmail it is advised to stick to the most recent version,
although the functionality needed for this setup has long been available.
If a particular macro or feature is not available with some older version,
it is usually possible to achieve the same or similar by manually writing
a new 'mailer' specification and/or tweaking the .cf file.
We'll prepare two sendmail daemon instances (processes), let's call them
MTA-RX (receiving, accepting) and MTA-TX (transmitting, delivering).
For convenience we keep the name of the configuration file and the
queue (spool) area at default names for one mailer instance, and choose
non-default names for the other. Let's choose the MTA-TX to keep default
names, and supply non-default names to MTA-RX explicitly. This will make
admin utilities like mailq, newaliases, hoststat and purgestat operate
on the outgoing mailer instance unless explicitly told otherwise.
It could just as well be the other way around.
MTA-RX (receiving mailer) will be responsible for accepting mail from
the Internet or from internal hosts on port 25, optionally accepting local
message submissions on tcp port 587 (rfc2476), and for message submissions
via sendmail program. It will forward all mail (both for local and for
nonlocal recipients) via SMTP protocol (or LMTP) to 127.0.0.1 (a loopback
interface) on tcp port 10024, where amavisd daemon will be listening.
- its queue: /var/spool/mqueue-rx
- its config file: /etc/mail/sendmail-rx.cf, /etc/mail/submit.cf
- the source (.mc) of the configuration file: thishost-rx.mc
(where 'thishost' is often by convention the name of the system (uname))
MTA-TX (transmitting mailer) will be responsible for accepting checked
mail and notifications from amavisd-new via SMTP on the loopback interface
(127.0.0.1) at tcp port 10025, and will forward all mail to its final
destinations, either for local delivery, or delivering outgoing mail
to the Internet or to other internal mailers.
- its queue: /var/spool/mqueue
- its config file: /etc/mail/sendmail.cf
- the source (.mc) of the configuration file: thishost-tx.mc
In-between the two MTAs an amavisd daemon will accept mail via SMTP (or LMTP)
protocol on tcp port 10024, check it, and forward checked mail and
notifications via SMTP to MTA-TX.
If you already have an existing sendmail installation, you already
have a queue directory /var/spool/mqueue and the configuration file(s)
(.mc source and the compiled .cf file). Most of the existing settings
in your .mc file can be reused, and are to be moved to the new files
thishost-rx.mc or thishost-tx.mc, or (some of them) to both.
The settings pertaining to receiving mail, including recource limits,
should go to thishost-rx.mc; settings pertaining to delivering mail
(locally or to other mailers) should go to thishost-tx.mc, and general
settings should go to both.
The MTA-TX should have none or hardly any resource limits, or at least
have them larger than MTA-RX. Large messages, common errors in mail, and
mail rush-ins should be stopped or limited at their entry to the system.
Accepting them first, but choking later can lead to trouble or at least
to wasted resources.
The file names thishost-rx.mc and thishost-tx.mc are arbitrary, they only
serve as source (to the m4 macro processor) for producing .cf files,
which control sendmail's behaviour. Sendmail never uses .mc files directly.
MTA-TX already got its queue directory during sendmail installation.
For MTA-RX a new queue directory needs to be created where incoming
mail can be collected. Use the same ownership and protection as used
for /var/spool/mqueue, e.g:
# mkdir /var/spool/mqueue-rx
# chown root:wheel /var/spool/mqueue-rx
# chmod 700 /var/spool/mqueue-rx
SECURITY NOTE:
starting with sendmail 8.12 it is possible to start sendmail daemon
as root and let it drop privileges (become user specified by RunAsUser)
after binding to port 25. This is normally used by MSP, and it can
just as well be used by MTA-RX, since it has no need to access user
mailboxes and .forward files. To use this feature, specify user and
group in the macro confRUN_AS_USER (file thishost-rx.mc), and set the
ownership of mqueue-rx to this user and group:
# chown smmsp:smmsp /var/spool/mqueue-rx
# chmod 770 /var/spool/mqueue-rx
More complex queue setup is possible if needed, like separating
sendmail work area and core dump area from actual queues.
For details about queue groups see sendmail documentation.
Create file thishost-rx.mc:
---cut-here------------------------------
dnl To be used for MTA-RX, the first MTA instance (receiving mail)
dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.
dnl Specify here also access controls, relayable domains, anti-spam measures
dnl including milter settings if needed, mail submission settings, client
dnl authentication, resource controls, maximum mail size and header size,
dnl confMIN_FREE_BLOCKS, and other settings needed for receiving mail.
dnl
dnl NOTE:
dnl confMIN_FREE_BLOCKS at MTA-RX should be kept higher than the same
dnl setting at MTA-TX to quench down clients when disk space is low,
dnl and not to stop processing the already received mail.
dnl
dnl In particular, here are some settings to be considered:
dnl ( see also http://www.sendmail.org/m4/anti_spam.html )
dnl
dnl FEATURE(`access_db')
dnl VIRTUSER_DOMAIN(`sub1.example.com')dnl list valid users here
dnl VIRTUSER_DOMAIN(`sub2.example.com')dnl list valid users here
dnl FEATURE(`virtusertable')
dnl define(`confUSERDB_SPEC', `/etc/mail/userdb.db')
dnl FEATURE(`blacklist_recipients')
dnl FEATURE(`use_cw_file')
dnl FEATURE(`use_ct_file')
dnl FEATURE(`nocanonify', `canonify_hosts')dnl
dnl INPUT_MAIL_FILTER(...)
dnl define(`confPRIVACY_FLAGS', `noexpn,novrfy,authwarnings') nobodyreturn ?
dnl define(`confDONT_PROBE_INTERFACES')
dnl MASQUERADE_AS(...) FEATURE(`allmasquerade') FEATURE(`masquerade_envelope')
dnl define(`confTO_IDENT', `0')dnl Disable IDENT
dnl define(`confMAX_MESSAGE_SIZE',`10485760')
dnl define(`confMAX_MIME_HEADER_LENGTH', `256/128')
dnl define(`confNO_RCPT_ACTION', `add-to-undisclosed')
dnl define(`confBIND_OPTS', ...)
dnl define(`confTO_RESOLVER_*... )
dnl define(`confDELAY_LA, 8)
dnl define(`confREFUSE_LA', 12)
dnl define(`confMAX_DAEMON_CHILDREN',20)
dnl define(`confMIN_FREE_BLOCKS', `10000')
dnl define(`confDEF_USER_ID', ...)
define(`confRUN_AS_USER',`smmsp:smmsp')dnl Drop privileges (see SECURITY NOTE)
define(`confPID_FILE', `/var/run/sendmail-rx.pid')dnl Non-default pid file
define(`STATUS_FILE', `/etc/mail/stat-rx')dnl Non-default stat file
define(`QUEUE_DIR', `/var/spool/mqueue-rx')dnl Non-default queue area
define(`confQUEUE_SORT_ORDER',`Modification')dnl Modif or Random are reasonable
dnl Match the number of queue runners (R=) to the number of amavisd-new child
dnl processes ($max_servers). 2 to 7 OK, 10 is plenty, 20 is too many
QUEUE_GROUP(`mqueue', `P=/var/spool/mqueue-rx, R=2, F=f')dnl
dnl Direct all mail to be forwarded to amavisd-new at 127.0.0.1:10024
FEATURE(stickyhost)dnl Keep envelope addr "u@local.host" when fwd to MAIL_HUB
define(`MAIL_HUB', `esmtp:[127.0.0.1]')dnl Forward all local mail to amavisd
define(`SMART_HOST', `esmtp:[127.0.0.1]')dnl Forward all other mail to amavisd
define(`LOCAL_RELAY',`esmtp:[127.0.0.1]')dnl
define(`confDELIVERY_MODE',`q')dnl Delivery mode: queue only (a must,
dnl ... otherwise the advantage of this setup of being able to specify
dnl ... the number of queue runners is lost)
define(`ESMTP_MAILER_ARGS',`TCP $h 10024')dnl To tcp port 10024 instead of 25
MODIFY_MAILER_FLAGS(`ESMTP', `+z')dnl Speak LMTP (this is optional)
define(`SMTP_MAILER_MAXMSGS',`10')dnl Max no. of msgs in a single connection
define(`confTO_DATAFINAL',`20m')dnl 20 minute timeout for content checking
DAEMON_OPTIONS(`Name=MTA-RX')dnl Daemon name used in logged messages
dnl Disable local delivery, as all local mail will go to MAIL_HUB
undefine(`ALIAS_FILE')dnl No aliases file, all local mail goes to MAIL_HUB
define(`confFORWARD_PATH')dnl Empty search path for .forward files
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
undefine(`DECNET_RELAY')dnl
MAILER(smtp)
dnl The following solution to reject unknown recipients outright
dnl is provided by Matej Vela <m...@irb.hr>, see:
dnl http://groups.google.com/group/comp.mail.sendmail/
dnl browse_thread/thread/88cc72d7c4d3a6e/ee2a9474b3a4558d
dnl The FEATURE(stickyhost) short-circuits FEATURE(luser_relay) so that a:
dnl define(`LUSER_RELAY',`error:5.1.1:"550 User unknown"') can't be used.
dnl A simple solution is to disable FEATURE(stickyhost). If this is not
dnl possible, the alternative is to replace FEATURE(luser_relay) with custom
dnl rules below. The latter has the advantage of properly handling special
dnl aliases like ("|program", "/mailbox", and ":include:/list"). If choosing
dnl this route, one should NOT use `undefine(`ALIAS_FILE')dnl', and use the
dnl following custom rules:
dnl
LOCAL_CONFIG
Kaliasp hash -m /etc/mail/aliases
Kuserp user -m
LOCAL_RULESETS
SLocal_check_rcpt
R$* $: <?> $&{rcpt_addr}
R<?> $+ @ $=w $: <@> $1 mark local address
R<?> $* @ $* $@ OK ignore remote address
R<?> $+ $: <@> $1 mark unqualified user
R<@> $+ + $* $: < $(aliasp $1+$2 $: @ $) > $1 + * plussed alias?
R<@> $+ + $* $: < $(aliasp $1+$2 $: @ $) > $1 +* alias?
R<@> $+ $: < $(aliasp $1 $: @ $) > $1 normal alias?
R<@> $+ $: < $(userp $1 $: @ $) > $1 system user?
R<@> $+ $#error $@ 5.1.1 $: "550 User unknown" nope, go away
---end-----------------------------------
Create file thishost-tx.mc:
---cut-here------------------------------
dnl To be used for MTA-TX, the second MTA instance
dnl (delivering outgoing and local mail)
dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.
dnl Specify here also the required outgoing mail processing and
dnl local delivery settings such as mailertables, needed mailers, aliases,
dnl local delivery mailer settings, smrsh, delivery mode, queue groups, ...
dnl Don't use milters here - for all common filtering purposes they belong
dnl to MTA-RX; an exception to this rule would be DKIM or DomainKeys mail
dnl signing milters (signature _verification_ milters still belong to MTA-RX).
define(`confREFUSE_LA',999)dnl Disable the feature, limiting belongs to MTA-RX
define(`confMAX_DAEMON_CHILDREN',0)dnl Disable, limiting belongs to MTA-RX
FEATURE(`no_default_msa')dnl No need for another MSA, MTA-RX already has one
FEATURE(`nocanonify')dnl Host/domain names are considered canonical
DAEMON_OPTIONS(`Addr=127.0.0.1, Port=10025, Name=MTA-TX')dnl Listen on lo:10025
define(`confSMTP_LOGIN_MSG', `$w.tx.$m Sendmail $v/$Z; $b')dnl
define(`confTO_IDENT', `0')dnl Disable IDENT
MAILER(smtp)
MAILER(local)
---end-----------------------------------
Now macro-expand .mc files into .cf files:
(adjust the path if needed to where your cf/m4/cf.m4 file resides)
# m4 /usr/share/sendmail/cf/m4/cf.m4 thishost-rx.mc >/etc/mail/sendmail-rx.cf
# m4 /usr/share/sendmail/cf/m4/cf.m4 thishost-tx.mc >/etc/mail/sendmail.cf
Start MTA-RX and MTA-TX daemons:
# /usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf -L sm-mta-rx -bd -qp
# /usr/sbin/sendmail -L sm-mta-tx -bd -q15m
Start queue runner for the MSP client queue as usual, if using it:
# /usr/sbin/sendmail -Ac -L sm-msp-queue -q10m
Start amavisd-new:
# amavisd
Test if MTA-RX is listening:
# telnet localhost 25
QUIT
Test if MTA-RX is listening on MSA port 587 (a newer sendmail invention)
# telnet localhost 587
QUIT
Test if MTA-TX is listening:
# telnet localhost 10025
QUIT
Test if amavisd is listening:
# telnet localhost 10024
QUIT
For convenience some shell aliases may be defined:
alias mailq-rx='mailq -C/etc/mail/sendmail-rx.cf'
alias mailq-tx='mailq'
alias sendmail-rx='/usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf'
alias sendmail-tx='/usr/sbin/sendmail'
All done!
NOTES
- In amavisd.conf file follow the 'POSTFIX or EXIM V4 or dual MTA setup',
which is also the default.
- The $final_*_destiny should not specify D_REJECT.
The D_BOUNCE (or D_PASS or D_DISCARD) is preferred.
- To make MTA-RX content-check only some mail but not all, one may use
mailertables instead of MAIL_HUB and SMART_HOST. For example setting
some recipient domains to be passed to MTA-TX at 127.0.0.1:10025
directly (e.g. via mailer 'esmtp'), while sending all the rest to
amavisd at 127.0.0.1:10024. To be able to specify the port number,
a new 'mailer' needs to be defined, let's call it 'amavis', with similar
settings as the already defined 'esmtp', except with port number 10024.
- depending on how local addresses are translated by MTA-RX, the
%local_domains (or @local_domains_maps) in amavisd.conf needs to be
adjusted accordingly to be able to recognize local domains. Check the
amavisd-new log what recipient addresses it sees for local recipients.
The '[127.0.0.1]' may need to be added to the @local_domains.
- To make MTA-RX reject mail for nonexistent local users by itself
(instead of generating a bounce later on), one may use the 'virtusertable'
in thishost-rx.mc, listing all known recipients, and rejecting the rest,
e.g.:
VIRTUSER_DOMAIN(`example.com')dnl
FEATURE(`virtusertable', `hash /etc/mail/virtusertable')dnl
jim@example.com %1%3
joe@example.com %1%3
postmaster@example.com %1
@example.com error:5.7.0:550 No such user here
You may use the righthand side of the map to specify local user
(e.g. %1%3, or just jim, without domain name) in which case MAIL_HUB will
be used for forwarding, or specify an explicit domain name that is not
in the {w} class, in which case the SMART_HOST will get consulted.
Perhaps what Stephane Lentz writes is even better:
Dictionary attacks and messages to retired accounts can be bounced with
sendmail: just replicate your aliases or write some SLocal_check_rcpt
rule-set that checks addresses of your domain against a map of valid users
(valid_addresses.db). I hope some standard FEATURE will be provided
with sendmail - something like FEATURE(checkdomainaddresses) and
CHECKDOMAINADDRESSES(mydomain.com).
An alternative solution is to use a milter to do address verification
against the second MTA in chain. See the milter-ahead project:
http://www.milter.info/sendmail/milter-ahead/
PERFORMANCE NOTES
- Mail handling is I/O-intensive. For better performance one may place
the two mail queue areas (/var/spool/mqueue and /var/spool/mqueue-rx),
and the /var/amavis work directory ($TEMPBASE) on three separate disks.
The /var/amavis/tmp may be a tmpfs or a RAMdisk or delayed-sync fs.
- One of the important arguments for choosing the dual-MTA setup is to be
able to keep the number of content filtering processes under control,
and not at the mercy of current mail inflow. Don't blow this advantage
by setting the number of amavisd-new processes and MTA-RX queue runners
too high!
Throughput optimum is somewhere between 3 and 10 with fast daemonized
av-scanner (or no av scanner) (with or without SpamAssassin),
and between 2 and 3 with many command line scanners (regardless of SA).
If the host is low on memory and when spam checking (SpamAssassin)
is used, even 2 may be a lot for an elderly host.
Start conservatively, e.g. at 2 or 3, and if everything works normally
and higher throughput is needed, try a bit more. Anything above the point
where throughput function levels off is just a waste of memory and
gains nothing!
The optimum may be higher if high-latency external SpamAssassin tests
are enabled (e.g. Razor, RBL), Still, never go beyond available memory.
For example with SpamAssassin enabled, the 20..25 processes on a 256 MB
machine is where throughput begins to drop rapidly on a way to a swapping
tar pit.
