AMAVIS POLICY DELEGATION PROTOCOL (AM.PDP)
==========================================
Author: Mark Martinec <Mark.Martinec@ijs.si>
Revisions:
2003-11-10 created
2005-03-18
2005-06-22
2006-08-18 (added attributes: version_server, insheader and quarantine),
2007-05-17 (allow attribute value: request=requeue)
2008-06-21 (new attribute partition_tag=xx on release requests)
2009-11-16
2010-01-22 (new attribute log_id on response)
NOTE: at the end of this document there is a description
(by Stephane Lentz) of the old protocol, now called AM.CL.
Amavis policy delegation protocol (AM.PDP) is intended to replace the
old amavis client protocol (AM.CL) as spoken between amavisd-new helper
programs (amavis.c or amavis-milter.c) and the amavisd daemon. The server
side is implemented by amavisd-new daemon. A sample AM.PDP client in Perl
is helper-progs/amavis.pl, and a rewrite by Petr Rehor of the helper
program amavis-milter.c to use the new AM.PDP protocol is available
as a separate project, see:
http://sourceforge.net/projects/amavisd-milter/
The new amavisd client/server protocol is based on the
'Postfix policy delegation protocol', described in the Postfix
document file README_FILES/SMTPD_POLICY_README, which can be used to
delegate policy decisions to an external server that runs outside Postfix.
It is conceptually similar to sendmail/milter mechanism, but based
on a documented text protocol which is easier to test and debug,
instead of using an undocumented binary protocol such as sendmail/milter.
There are a few departures from the original Postfix policy delegation
protocol:
- Postfix policy delegation protocol terminates lines with LF,
AM.PDP protocol terminates lines with CR LF;
( This was changed to facilitate debugging via telnet, and to make it
more similar to related protocols, e.g. SMTP, HTTP, FTP, ... )
- Postfix policy delegation protocol restricts the reply to one field only,
AM.PDP allows arbitrary number of fields passed from server back to client;
( This allows additional functionality like modifying or removing
recipient addresses, adding or editing mail header fields, etc. )
- Postfix policy delegation protocol expects the same attribute name
to be used only once; AM.PDP allows repeated appearances of the attribute,
which makes passing of lists easier (such as multiple recipient addresses);
- attribute value may consist of more than one field. Fields are delimited
by exactly one non-encoded space. Spaces within a field must be encoded
like any other restricted character (see below).
The protocol may be spoken over a Unix STREAM socket, or over an
inet tcp socket.
The client request is a sequence of name=value attributes, each terminated
by CR LF. The sequence is terminated by an empty line (only CR LF).
The server reply has the same structure. After client receives server
response sequence terminated by an empty line, it may close the session,
or issue another request on the same session. During normal operation
the server should not close the socket until it has been closed by the client.
Only under special circumstances is the server allowed to close the session,
e.g. in response to a timeout or fatal error condition.
The order of attributes does not matter, except for the 'request'
attribute which must appear first in the client request.
The policy client as well as the policy server should ignore
any attributes that it does not care about.
An attribute name must not contain non-encoded characters: "=", "%",
space, null, CR or LF. An attribute value must not contain
non-encoded characters: "%", space, null, CR, or LF.
All restricted characters must be hex-encoded to a sequence of three
characters: a percent sign, followed by two hex digits, e.g. %2A.
Hexadecimal digits 0..9,a..f or 0..9,A..F are allowed.
Any other character MAY be encoded as well. Although not mandated,
it is prudent to encode all non-printable characters.
The line length is not limited by this protocol. Both the server and the
client must be prepared to handle arbitrary line lengths without breaking
the protocol or rising security issues. Both are allowed to internally
truncate unreasonably long lines to a sensible length, and issue a warning.
Neither the client nor the server must make any assumptions that certain
characters will not be used in the attribute name or values. E.g. a presence
of encoded null or newline or other special character in the attribute name
or value must be safely and appropriately handled. If such a character
does not comply with the expected syntax, the case should be handled
to the best of client or server understanding and capability,
e.g. character ignored, attribute ignored, and/or a warning logged.
The following example Perl expression may be used for encoding/decoding:
to decode attribute name and attribute values:
s/%([0-9a-fA-F]{2})/pack("C",hex($1))/eg
to encode attribute name:
s/([^0-9a-zA-Z_-])/sprintf("%%%02x",ord($1))/eg;
to encode attribute values (each space-separated field individually):
s/([^\041-\044\046-\176])/sprintf("%%%02x",ord($1))/eg;
Attributes in the client request are:
-------------------------------------
request=AM.PDP
is a required first attribute, its value must be AM.PDP
sender=<foo@example.com>
specifies the envelope sender address (reverse-path).
The attribute should appear exactly once. The attribute value syntax
is specified in rfc5321 as 'Reverse-path' (i.e. smtp-quoted form,
enclosed in <>); a null reverse path is specified as <>.
recipient=<user1@example.net>
specifies the envelope recipient address. The attribute appears once
for each recipient address, the order of addresses must be preserved
and might be significant for some setups or functions.
The attribute value syntax is specified in rfc5321 as 'Forward-path'.
tempdir=/var/amavis/amavis-milter-MWZmu9Di
Specifies a temporary work directory to be used for mail unpacking,
typically also containing the original mail file - see attribute
'mail_file' below. This attribute should be present exactly once.
The server is allowed to use the specified directory to create additional
temporary files if it chooses so. As a security precaution, currently
amavisd restricts the temporary directory path, which must be a
subdirectory under $TEMPBASE or $MYHOME.
tempdir_removed_by=client
Specifies the client will be responsible for removing the temporary
directory. The server must not remove the file email.txt nor the directory,
but it may remove temporary files and subdirectories it has created.
tempdir_removed_by=server
Specifies the server is responsible to remove the temporary directory
if/when it deems appropriate. This is a default in the absence of this
attribute (for compatibility with traditional amavis clients).
mail_file=/var/amavis/amavis-milter-MWZmu9Di/email.txt
Specifies a file name (full file path) of a file containing the original
mail with header and body. This attribute should be present at most once.
In its absence the file name defaults to <tempdir>/email.txt.
delivery_care_of=client
Specifies that server should NOT actively forward the mail to recipients,
but should only report its opinion in its reply, and let the client
act on it. This is a default in the absence of this attribute.
delivery_care_of=server
Specifies that server is responsible to actively forward the mail
to recipients if it deems the message appropriate for forwarding.
This attribute value indicates that the client has no capability
or intention to forward mail by itself.
queue_id=qid
optional informational attribute: MTA queue id;
protocol_name=ESMTP
optional informational attribute: the name of the protocol used by the
original SMTP client to deliver the mail to the client (or to its
associated MTA). Common values are ESMTP, SMTP, LMTP.
helo_name=b.example.com
optional informational attribute: the value of the HELO or EHLO or LHLO
command specified to our MTA by the original SMTP client;
client_address=10.2.3.4
optional informational attribute: the IP address of the original SMTP
client;
client_name=mail.example.com
optional informational attribute: the DNS name of the original SMTP client
as obtained by DNS reverse mapping of the original SMTP client;
policy_bank=TLS,ORIGINATING,MYNETS
value is a comma-separated list of policy bank names. Names of
nonexistent banks are silently ignored, so are leading and trailing spaces
and TABs around each name. The order of policy bank loading generally
follows the order in which information about a message were obtained:
- interface or socket -based policy banks (when MTA connects to amavisd);
- MYNETS (when client's IP address becomes known);
- the list as specified in the policy_bank attribute of AM.PDP;
- MYUSERS (when sender e-mail address becomes known);
Attributes in the server response are:
--------------------------------------
The current set of attributes maps almost exactly to the capabilities
of libmilter, which should facilitate initial implementation.
See sendmail libmilter documentation as well.
version_server=2
Since amavisd-new-2.4.3 the 'version_server=2' is inserted, older
versions did not provide this attribute (assumed version was 1).
With version 2 the following changes to the protocol were made:
- version_server=2 is provided by the server as the first attribute;
- delheader and chgheader now stand in a response before insheader
and addheader, assuming that milter MTA will execute these in
the same order;
- new attribute insheader, see below;
- new attribute quarantine, see below.
log_id=xxx
Provided since amavisd-new-2.7.0, tells a log_id under which the
amavisd daemon logged the request and related debugging events;
delrcpt=<user1@example.net>
The specified recipient should be removed from the list of
recipients of this mail. The specified address must exactly match
the recipient addresses as specified in the client request,
i.e. a smtp-quoted form enclosed in <> should be specified;
addrcpt=<user1@example.net>
The specified recipient should be added to the list of recipients
of this mail. Paired with 'delrcpt' the pair indicates an existing
recipient address to be replaced by a modified address, e.g. to
add an address extension.
delheader=index hdr_head
Specifies an existing mail header field should be removed.
Index is a decimal integer indicating which of the header fields
with the same head should be affected, count starts with 1.
'hdr_head' does not include a colon!
chgheader=index hdr_head hdr_body
Specifies an existing mail header field should have its body replaced by
a new hdr_body. Index is a decimal integer indicating which of the header
fields with the same head should be affected, count starts with 1.
'hdr_head' does not include a colon!
insheader=index hdr_head hdr_body
Similar to addheader, but specifies a mail header field to be inserted to
the mail header at a given position, index 0 implies top of the header.
Amavisd-new passes a value of $prepend_header_fields_hdridx as the index
argument, which is configurable and defaults to 1 for compatibility with
dkim-milter and dk-milter signing milters. Alternative useful value is 0,
which may be used in absence of other milters inserting their header fields
at index 1. Header fields to be prepended are listed in reverse order,
the last one listed in AM.PDP is to appear at the top of a resulting mail
header. Inserting a header field at an arbitrary position is a later
addition to sendmail milter protocol
(introduced with sendmail 8.13.0 2004-06-20)
addheader=hdr_head hdr_body
Specifies a mail header field to be appended to the mail header.
Note the use of exactly two value fields, separated by exactly one space.
As described above, spaces in each field must be hex-encoded.
'hdr_head' does not include the colon!
replacebody=new_body
Not implemented - for future consideration.
quarantine=reason
place message on hold or to a quarantine maintained by MTA, and supply
a reason text (e.g. client may call smfi_quarantine milter routine;
btw, smfi_quarantine was introduced with sendmail 8.13);
For future use - it is currently (2.4.3 or earlier) never used.
return_value=val
where val can be any of: continue, accept, reject, discard, tempfail
This attribute should be present exactly once, and indicates to the
client what should be done with the mail and how the original SMTP
session should be treated.
setreply=rcode xcode message
SMTP response code and text suggested by server to the client
to be used in response to the original SMTP client;
There are exactly three value fields, separated by a single space.
exit_code=n
Similar in semantics to return_value attribute, but uses
sysexits.h -based exit values for the purpose. Useful with old
amavis clients, but should be ignored by new clients,
which should base its decision on return_value instead.
An example
==========
Indentation and text in parenthesis in the following example
is not part of the actual session.
$ telnet 127.0.0.1 9998
Trying 127.0.0.1...
Connected to localhost.example.com.
Escape character is '^]'.
(client->server request)
request=AM.PDP
sender=me@example.com
recipient=user1@example.net
recipient=user2@example.net
protocol_name=ESMTP
client_address=10.2.3.4
tempdir=/var/amavis/amavis-milter-MWZmu9Di
(server->client response)
setreply=250 2.5.0 Ok,%20id=MWZmu9Di,%20continue%20delivery
return_value=continue
exit_code=0
(or:)
setreply=250 2.7.1 Ok,%20discarded,%20UBE,%20id=mYOljdn2
return_value=discard
exit_code=99
(or:)
setreply=550 5.7.1 Message%20content%20rejected,%20UBE,%20id=S7uS4qvA
return_value=reject
exit_code=69
(or:)
setreply=451 4.5.0 Error%20in%20processing,%20id=...
return_value=tempfail
exit_code=75
===============================================================================
Releasing a message from a quarantine:
request=release (or request=requeue, available since 2.5.1)
mail_id=xxxxxxxxxxxx
secret_id=xxxxxxxxxxxx (authorizes a release)
partition_tag=xx (optional, but recommended if info is available)
quar_type=x F/Z/B/Q/M (file/zipfile/bsmtp/sql/mailbox)
mail_file=... (optional: overrides automatics; $QUARANTINEDIR prepended)
requested_by=<releaser@example.com> (optional: lands in Resent-From:)
sender=<foo@example.com> (optional: replaces envelope sender)
recipient=<bar1@example.net> (optional: replaces envelope recips)
recipient=<bar2@example.net>
recipient=...
In reply, for each recipient a SMTP status response is returned
(similar to LMTP)
quar_type defaults to Q if spam_quarantine_method is sql:, otherwise to F.
===============================================================================
AMAVIS SIMPLE CLIENT/SERVER PROTOCOL (traditional)
description by Stephane Lentz
2003-11-06
amavisd is the daemon part of AMAVIS in charge of scanning SMTP messages.
It receives messages from other applications (clients) using either
SMTP or a simple protocol which is detailed in this document.
The protocol being used depends on the MTA and architecture chosen.
The "simple protocol" is most often used with sendmail in a MILTER
set-up (the client program in such a case is amavis-client).
AMAVISD receives messages from clients through a UNIX socket :
The UNIX socket used is by default /var/amavis/amavisd.sock .
It is defined :
- at the amavisd server level in amavisd.conf as $unix_socketname
- at the client level when using MILTER (amavis-milter available in
helper-progs) as a configure option : --with-sockname swith
The protocol used between the client and server is simple (basic & limited).
There is no possibility for the server to ask the client to remove/add/change
headers. The server can only say if the message was detected as CLEAN, as
UNSAFE and to be rejected/discarded) or not analysed successfully due to
some errors.
PROTOCOL IN DETAILS :
The client connects to the AMAVISD server's socket.
IF successful then for each incoming message :
- the client computes and create a new temporary directory ($tempdir)
to store the new incoming message.
- the incoming message is stored as $tempdir/email.txt
- the client sends the directory name to the SERVER
- the server sends \1 to the client if the directory is ok
- the client sends the envelope sender recipient address
- the server sends \1 to the client if ok
- the client sends the envelope recipient addresses one by one to the server:
the client trims the address if it is longer than the
maximum length possible
the client sends this address to the server
the server sends \1 to the client if ok
- the client sends some request to analyze the message to the SERVER.
The character used is EOT (end of transmission) :\3
- the server processes the mail stored in the directory ($tempdir/email.txt)
- the server sends a STATUS number to the CLIENT. This number returned is
either :
EX_OK (2) : message CLEAN
EX_UNAVAILABLE (69) : message UNSAFE to be rejected at the SMTP LEVEL
(550 reject)
99 : message UNSAFE to be silently (250 code) discarded
at the SMTP LEVEL
EX_TEMPFAIL (75) : message not processed successfully (error in
communication, or server error, ...)
