Spamd

Работа с rspamd

Сигналы

Сигналы можно посылать master процессу (pid которого находится в указанном в конфигурации pidfile):

SIGTERM — остановить работу rspamd
SIGHUP — перечитать конфиг и создать новые процессы-обработчики, старые при этом перестают принимать соединения и через некоторое время завершают работу (примерно 1 минута)
SIGUSR2 — переоткрыть лог-файл.

Если вы используете стартовый скрипт, то он посылает эти сигналы сам, обеспечивая более удобный интерфейс, например, `rspamd reload`.

Передача сообщений rspamd

Для передачи писем rspamd можно использовать две возможности: передача сообщения обычному обработчику, используя расширенный протокол rspamc или же протокол совместимости с SpamAssassin’ом (оба эти протокола похожи на HTTP, но rspamc поддерживает больший набор заголовков, а также метрики, протокол совместимости очень ограничен в заголовках и отдает ответ только по метрике *default*), а также работа rspamd в режиме балансирующего smtp прокси, выполняющего проксирование smtp соединений и проверку их «на лету». Второй способ, безусловно, более удобен, однако он не тестировался в production, и я не рекомендую его для ответственных систем.

spamd {
  servers = r:spam1.test.ru:11333, r:spam2.test.ru:11333;
  connect_timeout = 1s;
  results_timeout = 20s;
  error_time = 10;
  dead_time = 300;
  maxerrors = 10;
  reject_message = "Spam message rejected; If this is not spam contact abuse at test.ru";
};

Также rspamd поставляется с утилитой rspamc, предоставляющей доступ к основным командам rspamd.
Работа с утилитой довольно проста:

rspamc symbols ] — для проверки писем
rspamc -P q1 -с bayes learn_spam ] — для обучения письмами статистику спама
rspamc -P q1 -c bayes learn_ham ] — для обучения письмами статистику хама
rspamc stat — для получения статистики

Обратите внимание, что для обучения ключом -P указывается дополнительный параметр — пароль для привилегированных команд, который настраивается в секции worker процесса-контроллера. Также при обучении rspamd необходимо учитывать, что количество спама и хама должно быть примерно одинаковым, иначе статистика будет работать неверно.

Также при обучении rspamd необходимо учитывать, что количество спама и хама должно быть примерно одинаковым, иначе статистика будет работать неверно.

Roundcube: Virtual Users, etc.

Documentation

/etc/roundcube/config.inc.php

// The default locale setting (leave empty for auto-detection)
$config'language' = 'it_IT';

// Set to false if only registered users can use this service
$config'auto_create_user' = false;
// Enables possibility to log in using email address from user identities
$config'user_aliases' = true;

The only drawback is that you have to populate the users and identities database tables before the user can login into Roundcube:

INSERT INTO users (username, mail_host, language)
    VALUES ('niccolo', '127.0.0.1', 'it_IT');
SELECT user_id FROM users WHERE username = 'niccolo';
INSERT INTO identities (user_id, standard, name, email)
    VALUES (1234, 1, 'Niccolo Rigacci', 'niccolo@domain.org');

The web interface

Part 2: install dovecot imap server on ubuntu & enable tls encryption

rspamd’s web interface

First you need to enable Apache’s module for HTTP proxying:

You can either create a new virtual host configuration or just edit the /etc/apache2/sites-available/webmail.example.org-https.conf file. Anywhere within the VirtualHost tags add:

This piece of configuration will forward any requests to https://webmail.example.org/rspamd to localhost:11334 and thus give you access to the rspamd web interface.

The interface is password protected. Let’s generate a new access password:

This gives you a password like “eiLi1lueTh9mia4”. You could put that password in an rspamd configuration file. But cleartext passwords in configuration files are not quite elegant. Let’s create a hash of the password:

Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is also documented on the .

Create a new configuration file /etc/rspamd/local.d/worker-controller.inc and put your hashed password in there:

That’s it for the configuration. Finally restart both rspamd and Apache to load your changed configuration:

If everything went as expected you should now be able to access the rspamd web interface at https://webmail.example.org/rspamd

Варианты написания текста

Настройка пересылки почты в postfix

Самое трудное при фильтрации контента — учесть разные варианты написания букв. Необходимо, чтобы фильтр смог распознать буквы, которые можно написать как разными способами, так и в разных кодировках.

Буквы

Для примера, берем букву «о» — ее можно заменить цифрой «0» или английской буквой «о», которая имеет другой код и будет по другому восприниматься системой, а читаться будет одинаково.

Таким образом, при составлении привил в файле 99_filter.cf, мы должны вести учет данных вариантов. Вот пример слова «Сова»:

((c|C|с|С)(o|O|о|О|0)(B|в|В)(a|A|а|А))

* в данном примере каждый вариант написания буквы идет в своих скобках. Сами варианты разделены вертикальной линией.

Кодировки

Различные почтовые клиенты будут отправлять русский текст в различных кодировках. Таким образом, наши правила не будут работать — нужно учитывать варианты использования разных кодов для символов. Например, тоже слово «Сова» в кодировке KOI8-R будет написано:

((\xd3|\xf3)(\xcf|\xef)(\xd7|\xf7)(\xc1|\xe1))

* в каждой скобке свой вариант написания буквы — строчной и заглавной.

Все вместе

Теперь нужно объединить два вышеописанных подхода. Получаем, что фильтр по слову «Сова» для KOI8-R должен выглядеть так:

body FILTER_CONTROL /((\xd3|\xf3|c|C|с|С)(\xcf|\xef|o|O|о|О|0)(\xd7|\xf7|B|в|В)(\xc1|\xe1|a|A|а|А))/

Sieve filtering

We will use Sieve for:

Filtering messages with Sanitizer/ClamAV.
Filtering messages with SpamAssassin.
Saving SPAM into the Spam folder (or removing it).
Creating Vacation auto-responder.

Enable Sieve Filtering in Dovecot LDA

To enable Sieve filtering in Dovecot local delivery agent we have to edit /etc/dovecot/conf.d/15-lda.conf:

protocol lda {
  mail_plugins = $mail_plugins sieve
}

Enable sieve_extprograms plugin

By default a Sieve filter cannot execute an external program or filter, we have to enable the sieve_extprograms plugin, which is however installed by the dovecot-sieve Debian package.

To enable the plugin, just edit /etc/dovecot/conf.d/90-sieve.conf and set into the existing plugin section:

plugin {
  ...
  sieve_extensions = +vnd.dovecot.filter
  ...
  sieve_plugins = sieve_extprograms
  ...
}

and configure the plugin itself, editing the /etc/dovecot/conf.d/90-sieve-extprograms.conf file at the plugin section:

plugin {
  ...
  sieve_filter_socket_dir = sieve-filter
  ...
  sieve_filter_bin_dir = /usr/local/lib/dovecot/sieve-filter
  ...
}

NOTICE: we enabled only the filter capability of the sieve_extprograms plugin, there are also the pipe and execute capabilities.

The actual filters are scripts saved into /usr/local/lib/dovecot/sieve-filter/ directory. We make just two examples: one using Sanitizer/ClamAV and one using SpamAssassin (named clamav-filter.sh and spamc-filter.sh respectively):

binsh
usrbinsanitizer etcsanitizer.cfg

#!/bin/sh
usrbinspamc

How to Test an User’s Sieve Filter File

The dovecot-lde searches the user’s active filter into $HOME/.dovecot.sieve (configured in ). We can write a simple filter file:

require ;
if header :contains "subject"  {
    fileinto "Commercial";
}

When a message is received and handled by the dovecot-lda, it can leave some useful log in /var/log/mail.log:

Jan 19 09:51:58 dovecot: lda(username): sieve: Execution of script /home/username/.dovecot.sieve
    failed, but implicit keep was successful (user logfile /home/username/.dovecot.sieve.log may
    reveal additional details)

As reported in log, an error message is recorded into users’ $HOME/.dovecot.sieve.log. In our case the subfolder named Commercial was missing, we have to create it using (notice the leading dot in subfolder name, which does not appear in Sieve filter statements).

A more complete Sieve filter example is the following. Each message is filtered for viruses and spam, then the messages marked as spam are saved into a specific folder:

require ;

# Filter with /usr/local/lib/dovecot/sieve-filter/spamc-filter.sh.
filter "spamc-filter.sh";

# Filter with /usr/local/lib/dovecot/sieve-filter/clamav-filter.sh.
filter "clamav-filter.sh";

if header :contains "X-Spam-Flag" "YES" {
    fileinto "Spam";
}

Every Sieve script will be compiled by Dovecot at the first execution; you will find a file .svbin for each .sieve file. Beware that global Sieve files are usually stored in non-user-writable directories, to the sysadmin must compile them with sievec.

Multiple Sieve Scripts

In the simplest case there is only one Sieve script per each user, which is $HOME/.dovecot.sieve. The user can create several scripts into $HOME/sieve/ directory, and choose which one to activate by making the proper symlink. The managing of these scripts can be done using the Managesieve daemon on the server and a Managesieve client (available e.g. as a Roundcube plugin).

In some cases it is desiderable to have several Sieve scripts and apply them in sequence, beside the one activated by the user and Managesieve. The most flexible way is to use a sieve_before and/or a sieve_after directory.

Scripts sieve_before and sieve_after

In /etc/dovecot/conf.d/90-sieve.conf we can configure some files or directories to contain Sieve scripts (with the proper .sieve extension), to be executed before or after the user’s personal script. If the path starts with it is relative to the user’s home directory (scripts are executed in alphabetical order):

plugin {
    ...
    sieve_before = ~/sieve_before.d/
    sieve_before2 = /var/lib/dovecot/sieve.d/
    #sieve_after =
    ...
}

The Include plugin

Another useful mechanism is to provide some system or user Sieve scripts, and let the users to use them via the include directive. The location for user’s scripts is $HOME/sieve by default, for global scripts you have to define sieve_global, e.g.:

plugin {
    sieve = file:~/sieve;active=~/.dovecot.sieve
    ...
    sieve_global = /etc/dovecot/sieve
    ...
}

once defined the location, the user can create a Sieve script like this:

require ;
include :personal "my_script";
include :global "spam_tests";

The files (with the .sieve extension) will be searched into the proper directory.

Configuration

, defines what Postfix services are enabled and how clients connect to them, see
, the main configuration file, see

Configuration changes need a reload in order to take effect.

Aliases

See .

You can specify aliases (also known as forwarders) in .

You need to map all mail addressed to root to another account since it is not a good idea to read mail as root.

Uncomment the following line, and change to a real account.

root: you

Once you have finished editing you must run the postalias command:

postalias /etc/postfix/aliases

For later changes you can use:

newaliases

Tip: Alternatively you can create the file , e.g. for root. Specify the user to whom root mail should be forwarded, e.g. user@localhost.

/root/.forward

user@localhost

Local mail

To only deliver mail to local system users (that are in ) update to reflect the following configuration. Uncomment, change, or add the following lines:

myhostname = localhost
mydomain = localdomain
mydestination = $myhostname, localhost.$mydomain, localhost
inet_interfaces = $myhostname, localhost
mynetworks_style = host
default_transport = error: outside mail is not deliverable

All other settings may remain unchanged. After setting up the above configuration file, you may wish to set up some and then .

Virtual mail

Virtual mail is mail that does not map to a user account ().

See Virtual user mail system with Postfix, Dovecot and Roundcube for a comprehensive guide how to set it up.

Check configuration

Run the command. It should output anything that you might have done wrong in a config file.

To see all of your configs, type . To see how you differ from the defaults, try .

Checking SSL Certificates

We should periodically check every SSL-enabled service for the certificate validity. The monitoring-plugins-basic Debian package contains some useful scripts that can be used. Here are an example script:

#!/bin/sh
echo "Checking SSL certificate fot SMTP on port 25/TCP..."
usrlibnagiospluginscheck_smtp -H localhost --port=25 --starttls -D 28
echo
echo "Checking SSL certificate fot POP3D on port 995/TCP..."
usrlibnagiospluginscheck_pop -H localhost --ssl --port=995 -D 28
echo
echo "Checking SSL certificate fot IMAP on port 993/TCP..."
usrlibnagiospluginscheck_imap -H localhost --ssl --port=993 -D 28

DEFAULT TAGGING FOR ALL MAILS

These headers are added to all messages, both spam and ham (non-spam).

X-Spam-Checker-Version: header: The version and subversion of SpamAssassin and the host where
SpamAssassin was run.
X-Spam-Level: header: A series of «*» charactes where each one represents a full score point.
X-Spam-Status: header: A string, is set in this header to
reflect the filter status. For the first word, «Yes» means spam and
«No» means ham (non-spam).

INSTALLATION

Note that it is not possible to use the environment variable
to affect where SpamAssassin finds its perl modules, due to limitations
imposed by perl’s «taint» security checks.

For further details on how to install, please read the file
from the SpamAssassin distribution.

DEVELOPER DOCUMENTATION

    Mail::SpamAssassin
        Spam detector and markup engine

    Mail::SpamAssassin::ArchiveIterator
        find and process messages one at a time

    Mail::SpamAssassin::AutoWhitelist
        auto-whitelist handler for SpamAssassin

    Mail::SpamAssassin::Bayes
        determine spammishness using a Bayesian classifier

    Mail::SpamAssassin::BayesStore
        Bayesian Storage Module

    Mail::SpamAssassin::BayesStore::SQL
        SQL Bayesian Storage Module Implementation

    Mail::SpamAssassin::Conf::LDAP
        load SpamAssassin scores from LDAP database

    Mail::SpamAssassin::Conf::Parser
        parse SpamAssassin configuration

    Mail::SpamAssassin::Conf::SQL
        load SpamAssassin scores from SQL database

    Mail::SpamAssassin::Message
        decode, render, and hold an RFC-2822 message

    Mail::SpamAssassin::Message::Metadata
        extract metadata from a message

    Mail::SpamAssassin::Message::Node
        decode, render, and make available MIME message parts

    Mail::SpamAssassin::PerMsgLearner
        per-message status (spam or not-spam)

    Mail::SpamAssassin::PerMsgStatus
        per-message status (spam or not-spam)

    Mail::SpamAssassin::PersistentAddrList
        persistent address list base class

    Mail::SpamAssassin::Plugin
        SpamAssassin plugin base class

    Mail::SpamAssassin::Plugin::Hashcash
        perform hashcash verification tests

    Mail::SpamAssassin::Plugin::RelayCountry
        add message metadata indicating the country code of each relay

    Mail::SpamAssassin::Plugin::SPF
        perform SPF verification tests

    Mail::SpamAssassin::Plugin::URIDNSBL
        look up URLs against DNS blocklists

    Mail::SpamAssassin::SQLBasedAddrList
        SpamAssassin SQL Based Auto Whitelist

COPYRIGHT

SpamAssassin is distributed under the Apache License, Version 2.0, as
described in the file included with the distribution.

Postfix SASL over Dovecot Auth

See Postfix And Dovecot SASL.

First of all we need to activate the socket used by Posfix for authentication. In /etc/dovecot/conf.d/10-master.conf ensure that into the service auth section there is:

service auth {
  ...
  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
  ...
}

To be compatible with more clients, enable both PLAIN and LOGIN methods. In /etc/dovecot/conf.d/10-auth.conf:

auth_mechanisms = plain login

Once reloaded the dovecot.service, you should see the /var/spool/postfix/private/auth socket, with the requested owner and permission.

For the Postfix part, you must declare in /etc/postfix/main.cf:

# Uses Dovecot Auth socket.
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth

After reloading the postfix.service, it should repsond to the EHLO command with:

telnet localhost 25
220 smtp.domain.org ESMTP Postfix (Debian/GNU)
EHLO test
...
250-STARTTLS
250-AUTH PLAIN LOGIN
...

WARNING: The AUTH capability is not announced over an unencrypted connection, if it is enabled only over TLS connections; see the smtpd_tls_auth_only Postfix option. Modern Postfix installations does not offer AUTH over smtp (port 25/tcp), but only over submission (port 587/tcp). Check if the option smtpd_sasl_auth_enable=yes is declared into main.cf or into the submission section of master.cf.

DEFAULT PLUGINS

    Mail::SpamAssassin::Plugin::Hashcash
    Mail::SpamAssassin::Plugin::SPF
    Mail::SpamAssassin::Plugin::URIDNSBL

WEB SITES

    SpamAssassin web site:     http://spamassassin.apache.org/
    Wiki-based documentation:  http://wiki.apache.org/spamassassin/

USER MAILING LIST

A users mailing list exists where other experienced users are often able
to help and provide tips and advice. Subscription instructions are
located on the SpamAssassin web site.

CONFIGURATION FILES

The SpamAssassin rule base, text templates, and rule description text
are loaded from configuration files.

Default configuration data is loaded from the first existing directory
in:

/home/jm/sabuildtools/perl587/var/spamassassin/3.002006
/home/jm/sabuildtools/perl587/share/spamassassin
/home/jm/sabuildtools/perl587/share/spamassassin
/usr/local/share/spamassassin
/usr/share/spamassassin

Site-specific configuration data is used to override any values which had
already been set. This is loaded from the first existing directory in:

/home/jm/sabuildtools/perl587/etc/mail/spamassassin
/home/jm/sabuildtools/perl587/etc/mail/spamassassin
/home/jm/sabuildtools/perl587/etc/spamassassin
/usr/local/etc/spamassassin
/usr/pkg/etc/spamassassin
/usr/etc/spamassassin
/etc/mail/spamassassin
/etc/spamassassin

From those three directories, SpamAssassin will first read files ending in
«.pre» in lexical order and then it will read files ending in «.cf» in
lexical order (most files begin with two numbers to make the sorting
order obvious).

In other words, it will read init.pre first, then 10_default_prefs.cf before
50_scores.cf and 20_body_tests.cf before 20_head_tests.cf.
Options in later files will override earlier files.

Individual user preferences are loaded from the location specified on
the , , or command line (see respective
manual page for details). If the location is not specified,
~/.spamassassin/user_prefs is used if it exists. SpamAssassin will
create that file if it does not already exist, using
user_prefs.template as a template. That file will be looked for in:

/home/jm/sabuildtools/perl587/etc/mail/spamassassin
/home/jm/sabuildtools/perl587/etc/mail/spamassassin
/home/jm/sabuildtools/perl587/share/spamassassin
/etc/spamassassin
/etc/mail/spamassassin
/usr/local/share/spamassassin
/usr/share/spamassassin

TAGGING

The following two sections detail the default tagging and markup that
takes place for messages when running or with
in the default configuration.

Note: before header modification and addition, all headers beginning
with are removed to prevent spammer mischief and also to
avoid potential problems caused by prior invocations of SpamAssassin.

TAGGING FOR SPAM MAILS

By default, all messages with a calculated score of 5.0 or higher are
tagged as spam.

If an incoming message is tagged as spam, instead of modifying the
original message, SpamAssassin will create a new report message and
attach the original message as a message/rfc822 MIME part (ensuring the
original message is completely preserved and easier to recover).

The new report message inherits the following headers (if they are
present) from the original spam message:

From: header
To: header
Cc: header
Subject: header
Date: header
Message-ID: header

By default these message headers are added to spam:

X-Spam-Flag: header: Set to .

spam mail body text: The SpamAssassin report is added to top of the mail message body,
if the message is marked as spam.

DEFAULT PLUGINS

USER MAILING LIST

A users mailing list exists where other experienced users are often able to help and provide tips and advice. Subscription instructions are located on the SpamAssassin web site.

CONFIGURATION FILES

The SpamAssassin rule base, text templates, and rule description text are loaded from configuration files.

Default configuration data is loaded from the first existing directory in:

/var/lib/spamassassin/3.004004
/usr/local/share/spamassassin
/usr/local/share/spamassassin
/usr/local/share/spamassassin
/usr/share/spamassassin

Site-specific configuration data is used to override any values which had already been set. This is loaded from the first existing directory in:

/etc/mail/spamassassin
/usr/local/etc/mail/spamassassin
/usr/local/etc/spamassassin
/usr/local/etc/spamassassin
/usr/pkg/etc/spamassassin
/usr/etc/spamassassin
/etc/mail/spamassassin
/etc/spamassassin

From those directories, SpamAssassin will first read files ending in «.pre» in lexical order and then it will read files ending in «.cf» in lexical order (most files begin with two numbers to make the sorting order obvious).

In other words, it will read init.pre first, then 10_default_prefs.cf before 50_scores.cf and 20_body_tests.cf before 20_head_tests.cf. Options in later files will override earlier files.

Individual user preferences are loaded from the location specified on the , , or command line (see respective manual page for details). If the location is not specified, ~/.spamassassin/user_prefs is used if it exists. SpamAssassin will create that file if it does not already exist, using user_prefs.template as a template. That file will be looked for in:

/etc/mail/spamassassin
/usr/local/etc/mail/spamassassin
/usr/local/share/spamassassin
/etc/spamassassin
/etc/mail/spamassassin
/usr/local/share/spamassassin
/usr/share/spamassassin

TAGGING

The following two sections detail the default tagging and markup that takes place for messages when running or with in the default configuration.

Note: before header modification and addition, all headers beginning with are removed to prevent spammer mischief and also to avoid potential problems caused by prior invocations of SpamAssassin.

Dovecot Authentication

user:password:uid:gid:(gecos):home:(shell):extra_fields

This file will be used for authentication against the password (i.e. as a Password Database), and it will be used for user’s data lookup, like the home directory (i.e. as an User Database). So we need to configure two sections in /etc/dovecot/conf.d/auth-passwdfile.conf.ext

passdb {
  driver = passwd-file
  args = scheme=CRYPT username_format=%u /etc/dovecot/users
}

userdb {
  driver = passwd-file
  args = username_format=%u /etc/dovecot/users
}

The scheme=CRYPT means that libc’s crypt() is used for password encryption, so the same password stored in /etc/shadow can be used.

To activate both system PAM and passwd-like authentication, we set into /etc/dovecot/conf.d/10-auth.conf (needed after the change):

!include auth-system.conf.ext
!include auth-passwdfile.conf.ext

Now we can test both authentication and user’s lookup using the doveadm tool:

doveadm auth test username@domain.org
Password: 
passdb: username@domain.org auth succeeded

doveadm user username@domain.org
field   value
uid     1000
gid     1000
home    /home/username
mail    maildir:~/Maildir

The file /etc/dovecot/users should be built e.g. by a cron-job, joining Postfix and passwords. It is re-read at each lookup. We need also to protect it:

chmod 0640 /etc/dovecot/users
chown root:dovecot /etc/dovecot/users

If you need to generate the hash for a password, you can use the following command line:

openssl passwd -1

Sending spam to the Junk folder

Note: SpamAssassin as used in previous versions of this guide used a slightly different header. “X-Spam-Flag: YES”. Make sure to change your Sieve filters accordingly.

Dovecot supports global Sieve filters that apply to all users. Edit the file /etc/dovecot/conf.d/90-sieve.conf. Look for the “sieve_after” lines. They are commented out. So add a new line there:

The “sieve after” filters are executed after the user’s filters. John can define his own filter rules. And after that Dovecot will run any filter rules it finds in files in /etc/dovecot/sieve-after. Create that directory:

And create a new file /etc/dovecot/sieve-after/spam-to-folder.sieve reading:

Dovecot cannot deal with such human-readable files though. So we need to compile it:

That generated a machine-readable file /etc/dovecot/sieve-after/spam-to-folder.svbin.

Restart Dovecot: