maildropfilter (7) Linux Manual Page

maildropfilter – maildrop’s filtering language

Synopsis

/etc/maildroprc, $HOME/.mailfilter, $HOME/.mailfilters/*, and friends…

Description

This manual page describes the language used by maildrop to filter E-mail messages. The mail filtering instructions are read from a file. The language is loosely structured, it is based on pattern matching. The language has a distinct lexical and syntactical structure, very similar to Perl’s, but it is important to note that it is not Perl, and is very different from Perl, in certain cases. If the filtering instructions do not exist, maildrop delivers the message to the default mailbox without doing any additional processing, making it indistinguishable from the usual mail delivery agent. It is important to note that maildrop reads and parses the filter file before doing anything. If there are any errors maildrop prints an error message, and terminates with the exit code set to EX_TEMPFAIL. A compliant mail transport agent should re-queue the message for a later delivery attempt. Hopefully, most simple syntax errors will not cause mail to be bounced back if the error is caught and fixed quickly.

Environment

maildrop uses variables to access and manipulate messages. Variables are arbitrary text accessed by referring to the name of the variable, such as HOME, or DEFAULT. Text is placed into a variable by using an assignment statement, such as:

FILE="IN.junk"

This statement puts the text "IN.junk" (without the quotes) into a variable whose name is FILE. Later, the contents of a variable are accessed by using the $ symbol and the name for the variable. For example:

to $FILE

This will deliver the current message to the mailbox file (or a maildir directory) named "IN.junk". maildrop initially creates variables from the environment variables of the operating system, UNLESS maildrop runs in delivery mode. Each operating system environment variable becomes a maildrop variable. When running in delivery mode, maildrop does not import the environment for security reasons, except for the environment variables that define the process locale (LANG, LANGUAGE, and LC_*), which are still imported. In all cases maildrop resets the following variables to their default values: HOME, DEFAULT, SHELL, PATH, LOCKEXT, LOCKREFRESH, LOCKSLEEP, LOCKTIMEOUT, MAILDIRQUOTA, SENDMAIL and LOGNAME. There’s one exception to this rule which applies to the version of maildrop that comes with the m[blue]Courier mail serverm[][1]. The following does not apply to the standalone version of maildrop: when running in delivery mode, if the -d flag was not used, or if it specifies the same userid as the one that’s running maildrop: the following variables are automatically imported from the environment: HOME, SHELL, LOGNAME and MAILDIRQUOTA. These environment variables are initialized by the Courier mail server prior to running maildrop. Additionally, the initial value for the DEFAULT maildrop variable is imported from the MAILDROPDEFAULT environment variable. This is because the Courier mail server overloads the DEFAULT environment variable to store the defaulted portion of the local mailbox address. See the m[blue]dot-courier(5)m[][2] man page in the Courier mail server distribution. You can get the Courier mail server’s DEFAULT value by using the import command. Note, however, that this will clobber the old contents of DEFAULT, which is probably not what you want. The right way to do this would be something like this:

SAVEDEFAULT=$DEFAULT
import DEFAULT
LOCALDEFAULT=$DEFAULT
DEFAULT=$SAVEDEFAULT

All internal variables are exported back as environment variables when maildrop runs an external command. Changes to internal variables, made by the filter file, are reflected in the exported environment.

Lexical structure

Most whitespace is generally ignored. The # character introduces a comment running to the end of the line, which is also ignored. Unlike other mail filters, maildrop parses the filter file before taking any action with the message. If there are syntax errors in the file, maildrop displays an error message, and returns EX_TEMPFAIL. That should cause the mail message to remain in the queue, and, hopefully allow the problem to be corrected, without bouncing any mail.

Note
In maildrop, the end of line is a lexical token. In order to continue a long statement on the next line, terminate the line with a backslash character.

Literal text

Literal text in the maildrop filtering language is surrounded by either single or double quotes. In order to enter a single quote into a text literal surrounded by single quotes, or a double quote into a literal surrounded by double quotes, prefix it with a backslash character. Use two backslash characters characters to enter one backslash character in the text literal.

Note
A backslash followed by either a backslash, or a matching quote, is the only situation where the backslash character is actually removed, leaving only the following character in the actual text literal. If a backslash character is followed by any other character, the backslash is NOT removed. Multiple text literals in a row are automatically concatenated, even if they use different quotes. For example:

FOOBAR="Foo"'bar'
SAVEDEFAULT=$DEFAULT
import DEFAULT
LOCALDEFAULT=$DEFAULT
DEFAULT=$SAVEDEFAULT

This sets the variable FOOBAR to the text "Foobar".

Variable substitution

Variable substitution is performed on text literals that’s surrounded by double quotation marks. The "$" character, followed by a variable name, is replaced by that variable’s contents. MAILBOX="$HOME/Mailbox" This sets the variable MAILBOX to the contents of the variable HOME followed by "/Mailbox". Variable names must begin with an uppercase letter, a lowercase letter, or an underscore. Following that, all letters, digits, and underscores are taken as a variable name, and its contents replace the $ sign, and the variable name. It is possible to access variables whose name includes other characters, by using braces as follows: MAILBOX="${HOME-WORD}/Mailbox" Inserts the contents of the HOME-WORD variable. If the variable does not exist, the empty text literal is used to replace the variable name. It is not possible to access variables whose names include the } character. If the $ character is not followed by a left brace, letter, or an underscore, the $ character remains unmolested in the text literal. A backslash followed by the $ character results in a $ character in the text literal, without doing any variable substitution. Variable substitution is not done in text literals which are surrounded by single quotes (apostrophes).

Command line arguments

maildrop initializes special variables: $1, $2, and so on, with additional parameters specified on the maildrop command line. A filter file may use those variables just like any other variables.

Predefined variables

The following variables are automatically defined by maildrop. The default values for the following variables may be changed by the system administrator. For security reasons, the values of the following variables are always reset to their default values, and are never imported from the environment: DEFAULT

The default mailbox to deliver the message to. If the filter file does not indicate a mailbox to deliver this message to, the message is delivered to this mailbox. The default mailbox is defined by the system administrator.

FROM

Message envelope sender. This is usually the same address as what appears in the From: header, but may not be. This information may or may not be available to maildrop on your system. The message envelope sender is usually specified with the -f option to maildrop. If the -f option is not given, maildrop looks for the Return-Path: header in the message. As the last resort, FROM defaults to “MAILER-DAEMON”. Note that FROM may be empty – the message envelope sender is empty for bounce messages.

HOME

Home directory of the user running maildrop.

HOSTNAME

Network name of the machine running maildrop. Obtained from gethostname(3).

LOCKEXT

Extension for dot-lock files (default: .lock).

LOCKREFRESH

Refresh interval, in seconds, for dot-locks (default: 15). When maildrop dot-locks a mailbox, maildrop tries to refresh the lock periodically in order to keep other programs from removing a stale dot-lock. This is only required if a dot-lock exists for a prolonged period of time, which should be discouraged anyway.

LOCKSLEEP

Number of seconds to wait to try again to create a dot-lock file, if one already exists (default: 5).

LOCKTIMEOUT

Number of seconds to wait before removing a stale dot-lock file (default: 60). If a dot-lock file still exists after LOCKTIMEOUT seconds, maildrop assumes that the process holding the lock no longer exists, and the dot-lock file can be safely removed. After removing the dot-lock file, maildrop waits LOCKSLEEP seconds before trying to create its own dot-lock file, in order to avoid a race condition with another process which is also trying to remove the same stale dot-lock, at the same time.

LOGNAME

Name of the user to who the message is being delivered.

MAILDROP_OLD_REGEXP

Revert to using the old legacy pattern matching engine. Versions of maildrop prior to version 2.0 (included in the Courier mail server 0.51, and earlier), used a built-in pattern matching engine, instead of using the PCRE library (see the “Patterns” section). maildrop 1.x used a different syntax for patterns, which is no longer described in this manual page. The old pattern matching engine is still available, by setting MAILDROP_OLD_REGEXP to “1”. Setting this variable will use the legacy pattern matching engine for the rest of the maildrop recipe file. The pattern matching engine will be removed completely in a future version of maildrop. This setting provides for a transitional period of converting old recipes. MAILDROP_OLD_REGEXP can be set to “1” in the global maildroprc file, then reset to “0” in each individual maildrop recipe file, after it gets converted to the new syntax.

MAILFILTER

This is the name of the original filter file that was given to maildrop on the command line. This is mostly useful to -defaultfilter files, it allows them to obtain the m[blue]value of the -M optionm[][3] specified on the command line.

PATH

Command execution path. maildrop resets PATH to the system default (usually /bin:/usr/bin:/usr/local/bin).

SENDMAIL

The mail delivery agent. When maildrop is instructed to deliver the message to a mailbox whose name begins with the ! character, this is interpreted as a request to forward the message. The SENDMAIL command is executed to forward the message.

SHELL

The login shell. The shell is used to execute all commands invoked by maildrop.

VERBOSE

Current Debug level (default: 0). Setting VERBOSE to progressive higher values, between 1 and 9, produces debugging output on standard error. maildrop ignores the VERBOSE variable in delivery mode (in order not to confuse the mail transport agent).

UMASK

The file creation mode mask, in octal. The default setting of 077 creates mailboxes that are readable and writable by the owner only. Use 007 to create mailboxes that are readable/writable by both owner and the group. Use 037 to create mailboxes that are readable by both owner and group, but writable by owner only. Permissions on existing mailboxes are not changed, this setting affects only new mailboxes. When delivering to maildirs this setting sets the permissions on new messages only. Access permissions on messages in maildirs are also affected by the permissions on the maildir directories.

Other special variables

The following variables are automatically used by maildrop when the filter file is being processed: EXITCODE

Return code for maildrop. When maildrop successfully delivers a message, it terminates with this exit code, which defaults to 0. When the to or the cc command is used to deliver the message to an external process, via a pipe, maildrop will set this variable to the exit code of the external process. Since maildrop immediately terminates after completing the to command this means that maildrop‘s exit code will be the exit code of the external process. If the to command does not deliver the message to a process you must set EXITCODE before the to command, since maildrop terminates immediately after finishing the delivery.

FLAGS

The FLAGS variable is used only when delivering a message to a maildir, and may contain only the following letters: “D”, “F”, “R”, and “S”. They may appear in any order. When the message gets delivered to the maildir, the message will be marked with a draft, flag, replied, or seen, attribute, correspondingly. FLAGS must be set before the message is delivered to a maildir. The contents of FLAGS are ignored, when delivering on an mbox folder.

KEYWORDS

The KEYWORDS variable is used only when delivering a message to a maildir, and implements the optional IMAP keyword extension as implemented in the m[blue]Courier IMAP serverm[][1]. It may be optionally initialized to contain a comma-separate list of keywords. The to, or the cc command, delivers the message to the maildir normally, but also associated the list of keywords in KEYWORDS with the newly delivered message. KEYWORDS must be set before the message is delivered to a maildir. The contents of KEYWORDS are ignored, when delivering on an mbox folder.

LINES

Number of lines in the current message. Note that this may be an approximation. It may or may not take into account the -A option. Use this as criteria for filtering, nothing more.

MAILDIRQUOTA

Set this variable in order to manually enforce a maximum size on ANY maildir where the message is delivered. This is an optional feature that must be enabled by the system administrator, see m[blue]maildirquota(8)m[][4] for more information.

RETURNCODE

This variable is set when maildrop runs the m[blue]xfilterm[][5] command, or a command that’s specified within a pair of backtick characters ( command substitution ). The RETURNCODE variable will be set to the exit code of the command, after it completes.

SIZE

Number of bytes in the message. This may or may not include the -A option. Use this as a criteria for filtering, nothing more.

Unquoted text

All text strings in filter files should be in single, or double quotes. However, for convenience sake, quotes can be omitted under certain circumstances. Text that includes ONLY letters, digits, and the following characters: _-.:/${}@ may appear without quotes. Note that this does not allow spaces, or backslashes to be entered, however the text is still variable-substituted, and the substituted text may contain other characters. Also, note that patterns (see below) begin with the slash character. Normally, anything that begins with the slash is interpreted as a pattern. However, text immediately after “VARIABLE=” is interpreted as a string even if it begins with a slash. This is why something like:

MAILDIR=/var/mail

works as expected. Using quotes, though, is highly recommended. You must use quotes to set a variable to a lone slash, because an unquoted slash is interpreted as a division sign. Long double or singly-quoted text can be broken across multiple lines by ending the line with a lone backslash character, like this: TEXT="This is a long \
text string" The backslash, the newline, and all leading whitespace on the next line is removed, resulting in "This is a long text string".

Command substitution

Text enclosed in back-tick characters is interpreted as a shell command. The shell command is executed as a child process by maildrop. Its output is used in place of the command. For example:

DIR=`ls`

places the names of the files in the current directory into the DIR variable. The output of the command will have all newline characters replaced by spaces, and leading and trailing spaces will be stripped (multiple spaces are not removed, though). Also, the contents of the message being delivered is made available to the command on standard input.

Patterns

The pattern syntax in maildrop is similar to the grep command’s syntax, with some minor differences. A pattern takes the following form in the filter file:

/pattern/:options

pattern specifies the text to look for in the message, in the UTF-8 codeset. pattern must not begin with a space, otherwise the leading slash will then be interpreted as a division sign. If you must search for text that starts with a space, use something like "/[ ] … /". The general syntax of maildrop‘s patterns is described in the pcrepattern(3) manual page, with certain exceptions noted below. maildrop uses the m[blue]PCREm[][6] library to implement pattern matching. Not all features in PCRE are available in maildrop, and the “options” part, which follows the pattern specification, changes the pattern matching further. Consult the pcrepattern(3) manual page for more information, but note the following exceptions:

• Internal options settings are not supported (but see the “D” maildrop option, below). Do not include option settings in the pattern, doing so will lead to undefined results.

• Named subpatterns are not implemented. Numbered subpatterns are implemented, see “Pattern Match Results”, below.

• The search pattern gets executed not against the raw message text, but the message transcoded into a canonical UTF-8-based format. This process involves transcoding any non-UTF-8 message content into UTF-8. Additionally, message headers get converted into a canonical format before the search pattern gets executed. For structured headers with email addresses, the process involves removing extraneous punctuation, or adding missing ones (in situations where a missing punctuation character can be deduced). Additionally certain pre-RFC822 obsolete header formats get converted to canonical form. This means that header search patterns that include punctuation character may appear not to work against obviously-matching message text. Use “reformime -u <message.txt”, with message.txt containing the sample message, to see exactly the actual text that gets searched by patterns.

Pattern options

Following /pattern/, there may be an optional colon, followed by one. or more options. The following options may be specified in any order: h

Match this pattern against the message header.

b

Match this pattern against the message body.

D

This is a case sensitive match. Normally the patterns match either uppercase or lowercase text. /john/ will match "John", "john", or "JOHN". Specify the D option for a case-sensitive search: lowercase letters in the pattern must match lowercase letters in the message; ditto for uppercase.

If neither ‘h’ or ‘b’ is specified, the pattern is matched against the header only. Specifying the ‘b’ option causes the pattern to be matched against the message body. Specifying both causes the pattern to be matched against the entire message. Normally, each line in the message gets matched against the pattern individually. When applying patterns to a header, multi-line headers (headers split on several lines by beginning each continuation line with whitespace) are silently combined into a single line, before the pattern is applied.

MIME encoding

The pattern must be a valid text string in the UTF-8 codeset, and maildrop should handle messages that use MIME encodings in other known character sets. Options that specify a message header search result in maildrop searching the initial message headers, and any headers of additional MIME sections, in a multipart MIME message. Options that specify a message body search will search through all "text" MIME content. For a MIME search to succeed, the message must be a well-formed MIME message (with a Mime-Version: 1.0 header).

Weighted scoring

Patterns are evaluated by maildrop as any other numerical expression. If a pattern is found, maildrop‘s filter interprets the results of the pattern match as number 1, or true, for filtering purposes. If a pattern is not found the results of the pattern search is zero. Once a pattern is found, the search stops. Second, and subsequent occurrences of the same pattern are NOT searched for. maildrop can also do weighted scoring. In weighted scoring, multiple occurrences of the same pattern are used to calculate a numerical score. To use a weighted search, specify the pattern as follows:

/pattern/:options,xxx,yyy

where xxx and yyy are two numbers. yyy is optional — it will default to 1, if missing. The first occurrence of the pattern is evaluated as xxx. The second occurrence of the pattern is evaluated as xxx*yyy, the third as xxx*yyy*yyy, etc… All occurrences of the pattern are added up to calculate the final score.

Note
maildrop does not recognize multiple occurrences of the same pattern in the same line. Multiple occurences of the same pattern in one line count as one occurence.

Pattern Match Results

After a pattern is successfully matched, the actual text that is matched is placed in the MATCH variable. For example:

/^From:.*/

matches a line of the form:

From: postmaster [at] localhost

Here the variable MATCH will be set to "From: postmaster [at] localhost", which can be used in subsequent statements. If the pattern contains subpatterns, the portions of the text that match the first subpattern is placed in the MATCH1 variable. The second subpattern, if any, is placed in MATCH2, and so on:

/^From:\s+(.*)@(.*)/

matched against the same line will set MATCH to “From: postmaster [at] localhost”, MATCH1 to “postmaster”, and MATCH2 to “localhost”. Of course, in real world the “From:” header is usually much more complicated, and can’t be handled that easily. This is just an illustrative example.

Note
Subpatterns are not processed in the foreach statement.

Conversion of maildrop 1.x patterns to 2.0

Although the new PCRE-based pattern matching code in maildrop is completely different from the built-in pattern matching code in maildrop 1.x, very few changes will be required to convert recipes to the new syntax. The only major differences are:

• The subexpression format has changed. Any pattern that uses subexpression needs to be converted. Additionally, references to MATCH2 must be replaced with MATCH1, MATCH3 to MATCH2, and so on. References to plain old MATCH will remain the same.

• The “w” pattern option is no longer possible, with PCRE. The very few recipes that use this option, if any actually exist, will have to be rewritten in some other fashion.

Expressions

Although maildrop evaluates expressions numerically, results of expressions are stored as text literals. When necessary, text literals are converted to numbers, then the results of a mathematical operation is converted back into a text literal.
Operators

The following operators carry their usual meaning, and are listed in order from lowest precedence, to the highest:

||
&&
< <= > >= == != lt le gt ge eq ne
|
&
+ -
* /
=~ /pattern/
/pattern/ ! ~ function()

Variable assignment

VARIABLE=expression

Assigns the result of the expression to VARIABLE (note no leading $ in front of variable).

Note
If VARIABLE is NOT surrounded by quotes, then it may contain only letters, numbers, underscores, dashes, and a selected few other characters. In order to initialize a variable whose name contains non-standard punctuation marks, surround the name of the variable with quotes.
cc – deliver a copy of the message

cc expression

The cc statement is very similar to the to statement, except that after delivering the message maildrop continues to process the filter file, unlike the to statement which immediately terminates maildrop after the delivery is complete. Essentially, the message is carbon copied to the given mailbox, and may be delivered again to another mailbox by another cc or to statement. m[blue]See the to statementm[][7] for more details. When cc is used to deliver a message to a process maildrop will set the EXITCODE variable to the process’s exit code.

dotlock – create a manual dot-lock

dotlock expression {
 ...
}

maildrop automatically creates a lock when a message is delivered to a mailbox. Depending upon your system configuration, maildrop will use either dot-locks, or the flock() system call. The dotlock statement creates an explicit dot-lock file. Use the m[blue]flock statementm[][8] to create an explicit flock() lock. The expression is a filename that should be used as a lock file. maildrop creates the indicated dot-lock, executes the filtering instructions contained within the { … } block, and removes the lock. The expression must be the name of the dot-lock file itself, NOT the name of the mailbox file you want to lock.

Note
With manual locking, it is possible to deadlock multiple maildrop processes (or any other processes that try to claim the same locks). No deadlock detection is possible with dot-locks, and since maildrop automatically refreshes all of its dot-locks regularly, they will never go stale. You’ll have maildrop processes hanging in limbo, until their watchdog timers go off, aborting the mail delivery.
echo – output diagnostic information

echo expression

maildrop will print the given text. This is usually used when maildrop runs in embedded mode, but can be used for debugging purposes. Normally, a newline is printed after the text. If text is terminated with a