Cyrus IMAP Server: Overview and Concepts
This document gives an overview of the Cyrus IMAP server. The Cyrus
IMAP (Internet Message Access Protocol) server provides access to
personal mail and system-wide bulletin boards through the IMAP
protocol. The Cyrus IMAP server is a scalable enterprise mail system
designed for use from small to large enterprise environments using
standards-based technologies.
A full Cyrus IMAP implementation allows a seamless mail and bulletin
board environment to be set up across multiple servers. It differs from
other IMAP server implementations in that it is run on "sealed"
servers, where users are not normally permitted to log in. The mailbox
database is stored in parts of the filesystem that are private to the
Cyrus IMAP system. All user access to mail is through software using
the IMAP, IMAPS, POP3, POP3S, or KPOP protocols.
The private mailbox database design gives the server large advantages
in efficiency, scalability, and administratability. Multiple concurrent
read/write connections to the same mailbox are permitted. The server
supports access control lists on mailboxes and storage quotas on
mailbox hierarchies.
This document is organized into the following areas:
* Mailbox Namespace
* Standard (Internal) Namespace
* Alternate Namespace
* Access Control Lists
* Access Rights
* Identifiers
* "anonymous" and "anyone"
* Kerberos vs. Unix Authorization
* Negative Rights
* Calculating Users' Rights
* Implicit Rights for Administrators on Personal Mailboxes
* Initial ACLs for Newly Created Mailboxes
* Login Authentication
* Anonymous Logins
* Kerberos Logins
* Unix Logins
* Quotas
* Supports Quotas on Storage
* Quota Roots
* Mail Delivery Behavior
* Quota Warnings Upon Select When User Has "d" Rights
* Quotas and Partitions
* New Mail Notification
* Partitions
* Specifying Partitions with "create"
* Changing Partitions with "rename"
* News
* "news" Partition
* Netnews
* collectnews
* rmnews
* syncnews
* POP3 Server
* The syslog facility
* Mail Directory Recovery
* Reconstructing Mailbox Directories
* Reconstructing the Mailboxes File
* Reconstructing Quota Roots
* Removing Quota Roots
* Subscriptions
* Configuration Directory
* "log" Directory
* "proc" Directory
* Message Delivery
* Local Mail Transfer Protocol
* Single Instance Store
* Duplicate Delivery Suppression
* Sieve, a Mail Filtering Language
Mailbox Namespace
By default, the Cyrus IMAP server presents mailboxes using the netnews
namespace convention. Mailbox names are case-sensitive. A mailbox name
may not start or end with a "." character, nor may it contain two "."
characters in a row.
In this implementation, non-ASCII characters and shell metacharacters
are not permitted in mailbox names.
Optionally, the server can present mailboxes using the UNIX hierarchy
convention.
Standard (Internal) Namespace
All personal mailboxes for user "bovik" begin with the string
"user.bovik.". For example, if user "bovik" had a personal "work"
mailbox, it would be called "user.bovik.work". To user "bovik",
however, the prefix "user.bovik." normally appears as "INBOX.". The
mailbox "user.bovik.work" would therefore appear as "INBOX.work". If
the access control list of the mailbox permitted other users to see
that mailbox, it would appear to them as "user.bovik.work".
The mailbox "user.bovik" is where the user "bovik" normally receives
new mail, and normally appears to user "bovik" as "INBOX". The mailbox
"user.bovik" is referred to in this document as user "bovik"'s INBOX.
Administrators create and delete users by creating and deleting the
users' INBOX. If a user has an INBOX, then they are allowed to
subscribe to mailboxes. Only users without dots in their userid are
permitted to have an INBOX. (A user with a dot in their userid would be
able to login but would not be able to receive mail. Note that when
using the unix hierarchy seperator, this is not the case, and any user
may have a dot in their userid.)
When an administrator deletes a user's INBOX, all of the user's
personal mailboxes are deleted as well.
With the one notable exception of INBOX, all mailbox names are
system-wide--they refer to the same mailbox regardless of the user.
Access control lists determine which users can access or see which
mailboxes. Using
In contexts which permit relative mailbox names, the mailbox namespace
works as follows:
* Names that do not start with "." are fully qualified.
* Names that start with "." are relative to the current context.
Thus, if you are working with folder names and the top of the hierarchy
is named "cmu.", the name "comp.infosystems.www" resolves to
"comp.infosystems.www" and the name ".comp.infosystems.www" resolves to
"cmu.comp.infosystems.www".
Alternate Namespace
The Cyrus IMAP server can also use an alternate namespace which allows
a user's personal mailboxes to appear as if they reside at the same
level as that user's INBOX as opposed to children of it. With this
feature, it may appear that there are non-unique names for mailboxes
between users (2 different users may each have a top level "work"
mailbox), but the internal representation is still user.name.work.
Access Control Lists
Access to each mailbox is controlled by each mailbox's access control
list. Access Control Lists (ACLs) provide a powerful mechanism for
specifying the users or groups of users who have permission to access
the mailboxes. An ACL is a list of zero or more entries. Each entry has
an identifier and a set of rights. The identifier specifies the user or
group of users for which the entry applies. The set of rights is one or
more letters or digits, each letter or digit conferring a particular
privilege.
Access Rights
The defined rights are:
l lookup - The user may see that the mailbox exists.
r read - The user may read the mailbox. The user may select the
mailbox, fetch data, perform searches, and copy messages from
the mailbox.
s seen - Keep per-user seen state. The "Seen" and "Recent" flags
are preserved for the user.
w write - The user may modify flags and keywords other than
"Seen" and "Deleted" (which are controlled by other sets of
rights).
i insert - The user may insert new messages into the mailbox.
p post - The user may send mail to the submission address for the
mailbox. This right differs from the "i" right in that the
delivery system inserts trace information into submitted
messages.
c create - The user may create new sub-mailboxes of the mailbox,
or delete or rename the current mailbox.
d delete - The user may store the "Deleted" flag, and perform
expunges.
a administer - The user may change the ACL on the mailbox.
You can combine access rights in different ways. For example:
lrs The user can read the mailbox.
lrsp The user can read the mailbox, and can post to it through the
delivery system. Most delivery systems do not provide
authentication, so the "p" right usually has meaning only for
the "anonymous" user.
lr The user can see the mailbox and can read it, but the server
does not preserve the "Seen" and "Recent" flags. This set of
rights is primarily useful for "anonymous IMAP."
rs The user can read the mailbox and the server preserves the
"Seen" and "Recent" flags, but the mailbox is not visible to
the user through the various mailbox listing commands. The user
must know the name of the mailbox to be able to access it.
lrsip The user can read and append to the mailbox, either through
IMAP, or through the delivery system.
Identifiers
The identifier part of an ACL entry specifies the user or group for
which the entry applies. The meaning of an identifier usually depends
on the authorization mechanism being used.
"anonymous" and "anyone"
With any authorization mechanism, two special identifiers are defined.
The identifier "anonymous" refers to the anonymous, or unauthenticated
user. The identifier "anyone" refers to all users, including the
anonymous user.
Kerberos vs. Unix Authorization
The Cyrus IMAP server comes with two authorization mechanisms, one for
use with Unix "/etc/passwd" files, one for use with Kerberos.
In the Unix authorization mechanism, identifiers are either a userid
listed in "/etc/passwd" or the string "group": followed by a group
listed in "/etc/group". Thus:
root Refers to the user root
group:staff Refers to the group staff
Using the Kerberos authorization mechanism, identifiers are of the
form:
principal.instance@realm
If ".instance" is omitted, it defaults to the null string. If "@realm"
is omitted, it defaults to the local realm.
"Principal", "instance", or "realm" may be replaced by the "*"
wildcard.
* A "*" in an identifier represents all principals with the null
instance at the local realm.
* A "*.*@FOO.COM" in an identifier represents all principals with all
instances at FOO.COM.
* A "*.*@*" in an identifier represents all Kerberos-authenticated
identities.
None of the above refer to the anonymous user.
The file "/etc/krb.equiv" contains mappings between Kerberos
principals. The file contains zero or more lines, each containing two
fields. Any identity matching the first field of a line is changed to
the second identity during canonicalization. For example, a line in
"/etc/krb.equiv" of:
bovik@REMOTE.COM bovik
will cause the identity "bovik@REMOTE.COM" to be treated as if it were
the local identity "bovik".
A site may wish to write their own authorization mechanism, perhaps to
implement a local group mechanism.
Negative Rights
Any of the above defined identifiers may be prefixed with a "-"
character. The associated rights are then removed from that identifier.
These are referred to as "negative rights".
Calculating Users' Rights
To calculate the set of rights granted to a user, the server first
calculates the union of all of the rights granted to the user and to
all groups the user is a member of. The server then calculates and
removes the union of all the negative rights granted to the user and to
all groups the user is a member of.
For example, given the ACL:
anyone lrsp
fred lwi
-anonymous s
The user "fred" will be granted the rights "lrswip" and the anonymous
user will be granted the rights "lrp".
Implicit Rights for Administrators on Personal Mailboxes
Regardless of the ACL on a mailbox, users who are listed in the
"admins" configuration option in "/etc/imapd.conf" implicitly have the
"l" and "a" rights on all mailboxes. Users also implicitly have the "l"
and "a" rights on their INBOX and all of their personal mailboxes.
Initial ACLs for Newly Created Mailboxes
When a mailbox is created, its ACL starts off with a copy of the ACL of
its closest parent mailbox. When a user is created, the ACL on the
user's INBOX starts off with a single entry granting all rights to the
user. When a non-user mailbox is created and does not have a parent,
its ACL is initialized to the value of the "defaultacl" option in
"/etc/imapd.conf"
Login Authentication
This section discusses different types of logins that can be used with
IMAP.
The Cyrus IMAP server uses the Cyrus SASL library for authentication.
This section describes how to configure SASL with use with Cyrus imapd.
Please consult the Cyrus SASL System Administrator's Guide for more
detailed, up-to-date information.
Anonymous Logins
Regardless of the SASL mechanism used by an individual connection, the
server can support anonymous login. If the "allowanonymouslogin" option
in "/etc/imapd.conf" is turned on, then the server will permit
plaintext password logins using the user "anonymous" and any password.
Additionally, the server will enable any SASL mechanisms that allow
anonymous logins.
Plaintext Authentication
The SASL library has several ways of verifying plaintext passwords
Plaintext passwords are passed either by the IMAP LOGIN command or by
the SASL PLAIN mechanism.
* PAM
* Kerberos v4 Plaintext passwords are verified by obtaining a ticket
for the server's Kerberos identity, to protect against Kerberos
server spoofing attacks.
* /etc/passwd
* /etc/shadow
* etc.
sasl_auto_transition automatically creates secrets for shared secret
authentication when given a password
Kerberos Logins
The Kerberos SASL mechanism supports the KERBEROS_V4 authentication
mechanism. The mechanism requires that a srvtab file exist in the
location given in the "srvtab" configuration option. The srvtab file
must be readable by the Cyrus server and must contain a
"imap.<host>@<realm>" service key, where <host> is the first component
of the server's host name and <realm> is the server's Kerberos realm.
The server will permit logins by identities in the local realm and
identities in the realms listed in the "loginrealms" option in
"/etc/imapd.conf".
The file "/etc/krb.equiv" contains mappings between Kerberos
principals. The file contains zero or more lines, each containing two
fields. Any identity matching the first field of a line is permitted to
log in as the identity in the second field.
If the "loginuseacl" configuration option is turned on, than any
Kerberos identity that is granted the "a" right on the user's INBOX is
permitted to log in as that user.
Shared Secrets Logins
Some mechanisms require the user and the server to share a secret
(generally a password) that can be used for comparison without actually
passing the password in the clear across the network. For these
mechanism (such as CRAM-MD5 and DIGEST-MD5), you will need to supply a
source of passwords, such as the sasldb (which is described more fully
in the Cyrus SASL distribution)
Quotas
Quotas allow server administrators to limit resources used by
hierarchies of mailboxes on the server.
Supports Quotas on Storage
The Cyrus IMAP server supports quotas on storage, which is defined as
the number of bytes of the relevant RFC-822 messages, in kilobytes.
Each copy of a message is counted independently, even when the server
can conserve disk space use by making hard links to message files. The
additional disk space overhead used by mailbox index and cache files is
not charged against a quota.
Quota Roots
Quotas are applied to quota roots, which can be at any level of the
mailbox hierarchy. Quota roots need not also be mailboxes.
Quotas on a quota root apply to the sum of the usage of any mailbox at
that level and any sub-mailboxes of that level that are not underneath
a quota root on a sub-hierarchy. This means that each mailbox is
limited by at most one quota root.
For example, if the mailboxes
user.bovik
user.bovik.list.imap
user.bovik.list.info-cyrus
user.bovik.saved
user.bovik.todo
exist and the quota roots
user.bovik
user.bovik.list
user.bovik.saved
exist, then the quota root "user.bovik" applies to the mailboxes
"user.bovik" and "user.bovik.todo"; the quota root "user.bovik.list"
applies to the mailboxes "user.bovik.list.imap" and
"user.bovik.list.info-cyrus"; and the quota root "user.bovik.saved"
applies to the mailbox "user.bovik.saved".
Quota roots are created automatically when they are mentioned in the
"setquota" command. Quota roots may not be deleted through the
protocol, see Removing Quota Roots for instructions on how to delete
them.
Mail Delivery Behavior
Normally, in order for a message to be inserted into a mailbox, the
quota root for the mailbox must have enough unused storage so that
inserting the message will not cause the block quota to go over the
limit.
Mail delivery is a special case. In order for a message to be delivered
to a mailbox, the quota root for the mailbox must not have usage that
is over the limit. If the usage is not over the limit, then one message
may be delivered regardless of its size. This puts the mailbox's usage
over the quota, causing a user to be informed of the problem and
permitting them to correct it. If delivery were not permitted in this
case, the user would have no practical way of knowing that there was
mail that could not be delivered.
If the usage is over the limit, then the mail delivery will fail with a
temporary error. This will cause the delivery system to re-attempt
delivery for a couple of days (permitting the user time to notice and
correct the problem) and then return the mail to the sender.
Quota Warnings Upon Select When User Has "d" Rights
When a user selects a mailbox whose quota root has usage that is close
to or over the limit and the user has "d" rights on the mailbox, the
server will issue an alert notifying the user that usage is close to or
over the limit. The threshold of usage at which the server will issue
quota warnings is set by the "quotawarn" configuration option.
The server only issues warnings when the user has "d" rights because
only users with "d" rights are capable of correcting the problem.
Quotas and Partitions
Quota roots are independent of partitions. A single quota root can
apply to mailboxes in different partitions.
New Mail Notification
The Cyrus IMAP server comes with a notification daemon which supports
multiple mechanisms for notifying users of new mail. Notifications can
be configured to be sent upon normal delivery ("MAIL" class) and/or
sent as requested by a Sieve script ("SIEVE" class).
By default, both types of notifications are disabled. Notifications are
enabled by using one or both of the following configuration options:
* the "mailnotifier" option selects the notifyd method to use for
"MAIL" class notifications
* the "sievenotifier" option selects the notifyd method to use for
"SIEVE" class notifications (when no method is specified by the
Sieve action)
Partitions
Partitions allow administrators to store different mailboxes in
different parts of the Unix filesystem. This is intended to be used to
allow hierarchies of mailboxes to be spread across multiple disks.
Specifying Partitions with "create"
When an administrator creates a new mailbox, the name of the partition
for the mailbox may be specified using an optional second argument to
the "create" command. Non-administrators are not permitted to specify
the partition of a mailbox. If the partition is not specified, then the
mailbox inherits the partition of its most immediate parent mailbox. If
the mailbox has no parent, it gets the partition specified in the
"defaultpartition" configuration option.
The optional second argument to the "create" command can usually be
given only when using a specialized Cyrus-aware administrative client
such as cyradm.
Changing Partitions with "rename"
An administrator may change the partition of a mailbox by using the
rename command with an optional third argument. When a third argument
to rename is given, the first and second arguments can be the
same--this changes the partition of a mailbox without changing its
name. If a third argument to rename is not given and the first argument
is not "INBOX", the partition of a mailbox does not change. If a third
argument to rename is not given and the first argument is "INBOX", the
newly created mailbox gets the same partition it would get from the
"create" command.
News
This section contains an overview of concepts related to exporting news
groups through the IMAP server.
"news" Partition
The partition named "news" is reserved for netnews and cannot be used
for any other purpose.
Netnews
The IMAP server can export netnews newsgroups as IMAP mailboxes. This
is done by creating a partition named "news" which points to the news
spool directory and arranging for the IMAP server's auxiliary databases
to be maintained. The server software includes programs for integrating
with the INN netnews server. The following information assumes working
knowledge of the INN netnews server.
collectnews
The "collectnews" program takes as input a list of article file path
names relative to the news spool directory. It then adds those messages
to the auxiliary databases. If collectnews encounters a newsgroup that
does not have a corresponding IMAP mailbox listed in the mailboxes file
(described later), it adds it there.
The "collectnews" program is normally run from cron as the cyrus user.
It is given as input a news overview feed. Since collectnews needs to
write the auxiliary databases in the news spool area, the cyrus user
must be in the news group and the news group must be able to write to
the news spool area.
rmnews
The "rmnews" program takes a list of article pathnames relative to the
news spool directory. It removes (expunges) any mention of those
messages from the auxiliary databases and unlinks the article files.
The "rmnews" program is used to remove both expired and canceled
articles.
"Rmnews" takes the same input as the "fastrm" program. It is normally
run to clean up after expire by invoking news.daily with the "delayrm"
argument and having RMPROC in expirerm changed from "fastrm" to
"rmnews".
Rmnews is also normally run from cron as the cyrus user to remove
canceled and superseded articles. It is given as input a special cancel
feed.
As "rmnews" is invoked at expire time by the news user and must be able
to write to the auxiliary databases, it must be installed with set-uid
to the cyrus user and set-gid to the news group.
syncnews
The "syncnews" program takes as an argument the name of the news active
file. It compares the list of IMAP news mailboxes with the news active
file. News mailboxes which are not listed in the active file are
removed. Newsgroups listed in the active file but not in the IMAP
mailboxes file have IMAP mailboxes created.
A newsgroup must have a status of ``y'', ``m'', or ``n'' to be
considered listed in the active file. The "syncnews" program should be
invoked by the cyrus user at least daily from cron.
POP3 Server
The Cyrus IMAP server software comes with a compatibility POP3 server.
Due to limitations in the POP3 protocol, the server can only access a
user's INBOX and only one instance of a POP3 server may exist for any
one user at any time. While a POP3 server has a user's INBOX open,
expunge operations from any concurrent IMAP session will fail.
When Kerberos login authentication is being used, the POP3 server uses
the server identity "pop.host@realm" instead of "imap.host@realm",
where "host" is the first component of the server's host name and
"realm" is the server's Kerberos realm. When the POP3 server is invoked
with the "-k" switch, the server exports MIT's KPOP protocol instead of
generic POP3.
The syslog facility
The Cyrus IMAP server software sends log messages to the "local6"
syslog facility. The severity levels used are:
* CRIT - Critical errors which probably require prompt administrator
action
* ERR - I/O errors, including failure to update quota usage. The
syslog message includes the specific file and Unix error.
* WARNING - Protection mechanism failures, client inactivity timeouts
* NOTICE - Authentications, both successful and unsuccessful
* INFO - Mailbox openings, duplicate delivery suppression
Mail Directory Recovery
This section describes the various databases used by the Cyrus IMAP
server software and what can be done to recover from various
inconsistencies in these databases.
Reconstructing Mailbox Directories
The largest database is the mailbox directories. Each mailbox directory
contains the following files:
message files
There is one file per message, containing the message in RFC
822 format. Lines in the message are separated by CRLF, not
just LF. The file name of each message is the message's UID
followed by a dot (.).
In netnews newsgroups, the message files instead follow the
format and naming conventions imposed by the netnews software.
cyrus.header
This file contains a magic number and variable-length
information about the mailbox itself.
cyrus.index
This file contains fixed-length information about the mailbox
itself and each message in the mailbox.
cyrus.cache
This file contans variable-length information about each
message in the mailbox.
cyrus.seen
This file contains variable-length state information about each
reader of the mailbox who has "s" permissions.
The "reconstruct" program can be used to recover from corruption in
mailbox directories. If "reconstruct" can find existing header and
index files, it attempts to preserve any data in them that is not
derivable from the message files themselves. The state "reconstruct"
attempts to preserve includes the flag names, flag state, and internal
date. "Reconstruct" derives all other information from the message
files.
An administrator may recover from a damaged disk by restoring message
files from a backup and then running reconstruct to regenerate what it
can of the other files.
The "reconstruct" program does not adjust the quota usage recorded in
any quota root files. After running reconstruct, it is advisable to run
"quota -f" (described below) in order to fix the quota root files.
Reconstructing the Mailboxes File
NOTE: CURRENTLY UNAVAILABLE
The mailboxes file in the configuration directory is the most critical
file in the entire Cyrus IMAP system. It contains a sorted list of each
mailbox on the server, along with the mailboxes quota root and ACL. To
reconstruct a corrupted mailboxes file, run the "reconstruct -m"
command. The "reconstruct" program, when invoked with the "-m" switch,
scavenges and corrects whatever data it can find in the existing
mailboxes file. It then scans all partitions listed in the imapd.conf
file for additional mailbox directories to put in the mailboxes file.
The cyrus.header file in each mailbox directory stores a redundant copy
of the mailbox ACL, to be used as a backup when rebuilding the
mailboxes file.
Reconstructing Quota Roots
The subdirectory "quota" of the configuration directory (specified in
the "configdirectory" configuration option) contains one file per quota
root, with the file name being the name of the quota root. These files
store the quota usage and limits of each of the quota roots.
The "quota" program, when invoked with the "-f" switch, recalculates
the quota root of each mailbox and the quota usage of each quota root.
Removing Quota Roots
To remove a quota root, remove the quota root's file. Then run "quota
-f" to make the quota files consistent again.
Subscriptions
The subdirectory "user" of the configuration directory contains user
subscriptions. There is one file per user, with a filename of the
userid followed by ".sub". Each file contains a sorted list of
subscribed mailboxes.
There is no program to recover from damaged subscription files. A site
may recover from lost subscription files by restoring from backups.
Configuration Directory
Many objects in the configuration directory are discussed in the
Database Recovery section. This section documents two other directories
that reside in the configuration directory.
"log" Directory
The subdirectory "log" under the configuration directory permits
administrators to keep protocol telemetry logs on a per-user basis.
If a subdirectory of "log" exists with the same name as a user, the
IMAP and POP3 servers will keep a telemetry log of protocol sessions
authenticating as that user. The telemetry log is stored in the
subdirectory with a filename of the server process-id and starts with
the first command following authentication.
"proc" Directory
The subdirectory "proc" under the configuration directory contains one
file per active server process. The file name is the ASCII
representation of the process id and the file contains the following
tab-separated fields:
* hostname of the client
* login name of the user, if logged in
* selected mailbox, if a mailbox is selected
The file may contain arbitrary characters after the first newline
character.
The "proc" subdirectory is normally be cleaned out on server reboot.
Message Delivery
Local Mail Transfer Protocol
LMTP, the Local Mail Transfer Protocol, is a variant of SMTP design for
transferring mail to the final message store. The Cyrus server
implements LMTP via the lmtpd daemon. LMTP can either be used over a
network via TCP or local via a UNIX domain socket.
Many MTAs (such as the latest versions of Sendmail or Postfix) can
deliver "local" mail over a network. This is an easy optimization so
that the IMAP server doesn't need to maintain a queue of messages or
run an MTA.
Authenticated final delivery is also supported in the form of LMTP
AUTH. If the final delivery isn't authenticated, it is treated (for the
purposes of checking ACLs) as if the "anonymous" user is attempting
delivery.
Single Instance Store
If a delivery attempt mentions several recipients (only possible if the
MTA is speaking LMTP to lmtpd), the server attempts to store as few
copies of a message as possible. It will store one copy of the message
per partition, and create hard links for all other recipients of the
message.
Single instance store can be turned off by using the
"singleinstancestore" flag in the configuration file.
Duplicate Delivery Suppression
A message is considered a duplicate if two copies of a message with the
same message-id and the same envelope receipient are received. Cyrus
uses the duplicate delivery database to hold this information, and it
looks approximately 3 days back in the default install.
Duplicate delivery suppression can be turned off by using the
"duplicatesuppression" flag in the configuration file.
Sieve, a Mail Filtering Language
Sieve is a mail filtering language that can filter mail into an
appropriate IMAP mailbox as it is delivered via lmtp. For more
information, look here.
Cyrus Murder, the IMAP Aggregator
Cyrus now supports the distribution of mailboxes across a number of
IMAP servers to allow for horizontal scalability. For information on
setting up this configuration, see here.
