pages tagged muttFeeding the Cloudhttps://feeding.cloud.geek.nz/tags/mutt/Feeding the Cloudikiwiki2024-01-14T00:26:40ZFiltering your own spam using SpamAssassinhttps://feeding.cloud.geek.nz/posts/filtering-own-spam-using-spamassassin/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2024-01-14T00:26:40Z2023-12-19T06:20:00Z
<p>I know that people rave about GMail's spam filtering, but it didn't work for
me: I was seeing too many false positives. I personally prefer to see some
false negatives (i.e. letting some spam through), but to reduce false
positives as much as possible (and ideally have a way to tune this).</p>
<p>Here's the local <a href="https://spamassassin.apache.org/">SpamAssassin</a> setup I
have put together over many years. In addition to the parts I describe here,
I also turn off
<a href="https://en.wikipedia.org/wiki/Greylisting_%28email%29">greylisting</a> on my email
provider (<a href="https://kolabnow.com/">KolabNow</a>) because I don't want to have to
wait for up to 10 minutes for a "2FA" email to go through.</p>
<p>This setup assumes that you download all of your emails to your local
machine. I use <a href="https://www.fetchmail.info/">fetchmail</a> for this, though
similar tools should work too.</p>
<h2 id="Three_tiers_of_emails">Three tiers of emails</h2>
<p>The main reason my setup works for me, despite my receiving hundreds of spam
messages every day, is that I split incoming emails into three tiers via
<a href="https://github.com/BuGlessRB/procmail">procmail</a>:</p>
<ol>
<li><em>not spam</em>: delivered to inbox</li>
<li><em>likely spam</em>: quarantined in a <code>soft_spam/</code> folder</li>
<li><em>definitely spam</em>: silently deleted</li>
</ol>
<p>I only ever have to review the <em>likely spam</em> tier for false positives, which
is on the order of 10-30 spam emails a day. I never even see the the
hundreds that are silently deleted due to a very high score.</p>
<p>This is implemented based on a threshold in my <code>.procmailrc</code>:</p>
<pre><code># Use spamassassin to check for spam
:0fw: .spamassassin.lock
| /usr/bin/spamassassin
# Throw away messages with a score of > 12.0
:0
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*
/dev/null
:0:
* ^X-Spam-Status: Yes
$HOME/Mail/soft_spam/
# Deliver all other messages
:0:
${DEFAULT}
</code></pre>
<p>I also use the following <code>~/.muttrc</code> configuration to easily report false
negatives/positives and examine my <em>likely spam</em> folder via a shortcut in
<a href="http://www.mutt.org/">mutt</a>:</p>
<pre><code>unignore X-Spam-Level
unignore X-Spam-Status
macro index S "c=soft_spam/\n" "Switch to soft_spam"
# Tell mutt about SpamAssassin headers so that I can sort by spam score
spam "X-Spam-Status: (Yes|No), (hits|score)=(-?[0-9]+\.[0-9])" "%3"
folder-hook =soft_spam 'push ol'
folder-hook =spam 'push ou'
# <Esc>d = de-register as non-spam, register as spam, move to spam folder.
macro index \ed "<enter-command>unset wait_key\n<pipe-entry>spamassassin -r\n<enter-command>set wait_key\n<save-message>=spam\n" "report the message as spam"
# <Esc>u = unregister as spam, register as non-spam, move to inbox folder.
macro index \eu "<enter-command>unset wait_key\n<pipe-entry>spamassassin -k\n<enter-command>set wait_key\n<save-message>=inbox\n" "correct the false positive (this is not spam)"
</code></pre>
<h2 id="Custom_SpamAssassin_rules">Custom SpamAssassin rules</h2>
<p>In addition to the default ruleset that comes with SpamAssassin, I've also
accrued a number of custom rules over the years.</p>
<p>The first set comes from the (now defunct) <a href="https://web.archive.org/web/20100416022759/http://www.rulesemporium.com/rules.htm">SpamAssassin Rules
Emporium</a>.
The second set is <a href="https://salsa.debian.org/debian-listmasters/spamassassin_config">the one that backs <code>bugs.debian.org</code> and
<code>lists.debian.org</code></a>.
Note this second one includes archived copies of some of the SARE rules and
so I only use some of the rules in the <code>common/</code> directory.</p>
<p>Finally, I wrote <a href="https://gist.github.com/fmarier/5e67daffdc7f43c0ed402a3a326d42e4">a few custom rules of my
own</a> based
on specific kinds of emails I have seen slip through the cracks. I haven't
written any of those in a long time and I suspect some of my rules are now
obsolete. You may want to do your own testing before you copy these outright.</p>
<p>In addition to rules to match more spam, I've also written a <a href="https://gist.github.com/fmarier/117bde37015fde22d6639f7d307bc187">ruleset to
remove false positives in French
emails</a>
coming from many of the above custom rules. I also wrote a rule to get a
bonus to any email that comes with a patch:</p>
<pre><code>describe FM_PATCH Includes a patch
body FM_PATCH /\bdiff -pruN\b/
score FM_PATCH -1.0
</code></pre>
<p>since it's not very common in spam emails <img src="https://feeding.cloud.geek.nz/smileys/smile.png" alt=":)" /></p>
<h2 id="SpamAssassin_settings">SpamAssassin settings</h2>
<p>When it comes to my system-wide SpamAssassin configuration in
<code>/etc/spamassassin/</code>, I enable the following plugins:</p>
<pre><code>loadplugin Mail::SpamAssassin::Plugin::AntiVirus
loadplugin Mail::SpamAssassin::Plugin::AskDNS
loadplugin Mail::SpamAssassin::Plugin::ASN
loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
loadplugin Mail::SpamAssassin::Plugin::Bayes
loadplugin Mail::SpamAssassin::Plugin::BodyEval
loadplugin Mail::SpamAssassin::Plugin::Check
loadplugin Mail::SpamAssassin::Plugin::DKIM
loadplugin Mail::SpamAssassin::Plugin::DNSEval
loadplugin Mail::SpamAssassin::Plugin::FreeMail
loadplugin Mail::SpamAssassin::Plugin::FromNameSpoof
loadplugin Mail::SpamAssassin::Plugin::HashBL
loadplugin Mail::SpamAssassin::Plugin::HeaderEval
loadplugin Mail::SpamAssassin::Plugin::HTMLEval
loadplugin Mail::SpamAssassin::Plugin::HTTPSMismatch
loadplugin Mail::SpamAssassin::Plugin::ImageInfo
loadplugin Mail::SpamAssassin::Plugin::MIMEEval
loadplugin Mail::SpamAssassin::Plugin::MIMEHeader
loadplugin Mail::SpamAssassin::Plugin::OLEVBMacro
loadplugin Mail::SpamAssassin::Plugin::PDFInfo
loadplugin Mail::SpamAssassin::Plugin::Phishing
loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2
loadplugin Mail::SpamAssassin::Plugin::RelayEval
loadplugin Mail::SpamAssassin::Plugin::ReplaceTags
loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
loadplugin Mail::SpamAssassin::Plugin::SpamCop
loadplugin Mail::SpamAssassin::Plugin::TextCat
loadplugin Mail::SpamAssassin::Plugin::TxRep
loadplugin Mail::SpamAssassin::Plugin::URIDetail
loadplugin Mail::SpamAssassin::Plugin::URIEval
loadplugin Mail::SpamAssassin::Plugin::VBounce
loadplugin Mail::SpamAssassin::Plugin::WelcomeListSubject
loadplugin Mail::SpamAssassin::Plugin::WLBLEval
</code></pre>
<p>Some of these require extra helper packages or Perl libraries to be
installed. See the comments in the relevant <code>*.pre</code> files or use this
command to install everything:</p>
<pre><code>apt install spamassassin pyzor razor libencode-detect-perl liblog-log4perl-perl libgeoip-dev libmail-dkim-perl libarchive-zip-perl libio-string-perl libmail-dmarc-perl fuzzyocr
</code></pre>
<p>My <code>~/.spamassassin/user_prefs</code> file contains the following configuration:</p>
<pre><code>required_hits 5
ok_locales en fr
# Bayes options
score BAYES_00 -4.0
score BAYES_40 -0.5
score BAYES_60 1.0
score BAYES_80 2.7
score BAYES_95 4.0
score BAYES_99 6.0
bayes_auto_learn 1
bayes_ignore_header X-Miltered
bayes_ignore_header X-MIME-Autoconverted
bayes_ignore_header X-Evolution
bayes_ignore_header X-Virus-Scanned
bayes_ignore_header X-Forwarded-For
bayes_ignore_header X-Forwarded-By
bayes_ignore_header X-Scanned-By
bayes_ignore_header X-Spam-Level
bayes_ignore_header X-Spam-Status
</code></pre>
<p>as well as manual <a href="https://gist.github.com/fmarier/52db8ee8e8af38defa0719b4289657ba">score
reductions</a>
due to false positives, and manual <a href="https://gist.github.com/fmarier/bed781b61959ea90456d9b222e2fc0a2">score
increases</a>
to help push certain types of spam emails over the 12.0 <em>definitely spam</em>
threshold.</p>
<p>Finally, I have the <a href="https://packages.debian.org/stable/fuzzyocr"><code>FuzzyOCR</code>
package</a> installed since it has
occasionally flagged some spam that other tools had missed. It is a little
resource intensive though and so you may want to avoid this one if you are
filtering spam for other people.</p>
<p>As always, feel free to leave a comment if you do something else that works
well and that's not included in my setup. This is a work-in-progress.</p>
Preventing man-in-the-middle attacks on fetchmail and postfixhttps://feeding.cloud.geek.nz/posts/preventing-man-in-middle-attacks-on/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2010-04-30T21:00:00Z
<p>Recent attacks against the DNS infrastructure have exposed the limitations of relying on TLS/SSL certificates for securing connections on the Internet.</p>
<p>Given that typical mail servers don't rotate their keys very often, it's not too cumbersome to hardcode their fingerprints and prevent your mail software from connecting to them should the certificate change. This is similar to how most people use ssh: assume that the certificate is valid on the first connection, but be careful if the certificate changes afterwards.</p>
<h2 id="Fetchmail">Fetchmail</h2>
<p>Here's how to specify a certificate for a POP/IMAP server (Gmail in this example).</p>
<p>First of all, you need to download the server certificate:</p>
<pre><code>openssl s_client -connect pop.gmail.com:995 -showcerts
openssl s_client -connect imap.gmail.com:993 -showcerts
</code></pre>
<p>Then copy the output of that command to a file, say <code>gmail.out</code>, and extract its md5 fingerprint:</p>
<pre>
openssl x509 -fingerprint -<b>md5</b> -noout -in <i>gmail.out</i>
</pre>
<p>Once you have the fingerprint, add it to your <code>~/.fetchmailrc</code>:</p>
<pre>
poll pop.gmail.com protocol pop3 user "<i>remoteusername</i>" is "<i>localusername</i>" password "<i>mypassword</i>" fetchall ssl sslproto ssl3 sslfingerprint "<i>12:34:AB:CD:56:78:EF:12:34:AB:CD:56:78:EF:12:34</i>"
</pre>
<h2 id="Postfix">Postfix</h2>
<p>Similarly, to detect changes to the certificate on your outgoing mail server (used as a smarthost on your local postfix instance), extract its sha256 fingerprint:</p>
<pre>
openssl s_client -connect smtp.gmail.com:587 -starttls smtp -showcerts
openssl x509 -fingerprint <b>-sha256</b> -noout -in <i>isp.out</i>
</pre>
<p>Then add the fingerprint to <code>/etc/postfix/main.cf</code>:</p>
<pre>
relayhost = smtp.gmail.com
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = fingerprint
smtp_tls_mandatory_ciphers = high
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
smtp_tls_fingerprint_digest = <b>sha256</b>
smtp_tls_fingerprint_cert_match =
<i>12:34:AB:CD:56:78:EF:90:12:AB:CD:34:56:EF:78:90:AB:CD:12:34:AB:DD:44:66:DA:77:CF:DB:E4:A7:02:E1</i>
</pre>
mutt's OpenPGP support and FireGPGhttps://feeding.cloud.geek.nz/posts/mutts-openpgp-support-and-firegpg/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2008-10-13T02:51:00Z
<p><a href="http://www.mutt.org/">mutt</a> has excellent <a href="http://tools.ietf.org/html/rfc4880">OpenPGP</a> and <a href="http://www.gnupg.org/">GPG</a> support. However different clients send GnuPG encrypted/signed data in different ways. Lots of people seem to be using <a href="http://getfiregpg.org/">FireGPG</a> these days so that they can use GnuPG within the GMail/Hotmail/Yahoo! web clients.</p>
<p>Here's the <a href="http://www.procmail.org/">procmail</a> rule I used to convert almost all incoming emails to the OpenPGP/MIME format:</p>
<pre><code>:0
* !^Content-Type: multipart/
* !^Content-Type: application/pgp
{
:0 fBw
* ^-----BEGIN PGP MESSAGE-----
* ^-----END PGP MESSAGE-----
| formail \
-i "Content-Type: application/pgp; format=text; x-action=encrypt"
:0 fBw
* ^-----BEGIN PGP SIGNED MESSAGE-----
* ^-----BEGIN PGP SIGNATURE-----
* ^-----END PGP SIGNATURE-----
| formail \
-i "Content-Type: application/pgp; format=text; x-action=sign"
}
</code></pre>
<p>However, this didn't actually work with FireGPG and the way that it puts encrypted messages inline. I eventually <a href="https://help.launchpad.net/ReadingOpenPgpMail">found out</a> that putting this in my <code>~/.muttrc</code> would fix it:</p>
<pre><code>set pgp_auto_decode=yes
</code></pre>
Searching through the contents of emails in mutthttps://feeding.cloud.geek.nz/posts/searching-through-contents-of-emails-in/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2022-10-29T17:04:54Z2008-04-05T10:30:00Z
<p>Mutt has a built-in feature for search the body of emails in the current mailbox (see <code>ESC-b</code>) but it's also possible to make this extremely fast by <a href="http://wiki.mutt.org/?UserStory/SearchingMail">plugging-in external tools</a>. Here's a quick description of my <a href="http://www.rpcurnow.force9.co.uk/mairix/">mairix</a> setup.</p>
<p>After installing the package,</p>
<pre><code>apt-get install mairix
</code></pre>
<p>create a .mairixrc in your home directory containing something like:</p>
<pre><code>base=/home/username/Mail
maildir=inbox:archives*:sent*
mbox=debian*:jokes
mfolder=search
database=/home/username/Mail/.mairix_database
</code></pre>
<p>(This assumes that you store your mail in <code>~/Mail</code> and that you want the search results in a new <code>~/Mail/search</code> maildir folder.)</p>
<p>Then you need to run mairix once to index all of your emails:</p>
<pre><code>mairix
</code></pre>
<p>which you will of course want to run regularly. I created this cronjob in /etc/cron.d/mairix-username:</p>
<pre><code>17,37,57 * * * * username /usr/bin/on_ac_power && /usr/bin/mairix || true
</code></pre>
<p>(To avoid waking up the hard drive needlessly, I prefer to restrict the indexing runs for when I'm running on AC power. Take that out if you're not running it on a laptop.)</p>
<p>Finally, the last bit of setup is in <code>~/.muttrc</code>:</p>
<pre><code>macro index \eb 'c=search'<enter><shell-escape>'mairix ' 'Enter Mairix search term'
</code></pre>
<p>which makes mairix take over the <code>ESC-b</code> shortcut.</p>
<p>If you use GPG, you should also add this to your <code>~/.muttrc</code> to make sure that mairix can see all of your messages by saving them as clear text:</p>
<pre><code>set fcc_clear=yes
bind index s decrypt-save
bind pager s decrypt-save
bind index \ec decrypt-copy
bind pager \ec decrypt-copy
</code></pre>
Handling multiple identities/accounts in mutthttps://feeding.cloud.geek.nz/posts/handling-multiple-identitiesaccounts-in/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2021-06-11T20:43:57Z2008-04-01T19:39:00Z
<p>Since <a href="http://wiki.mutt.org/">mutt</a> seems to be <a href="http://she.geek.nz/archives/482-fixed-mutt-addressbook-failure.html">all the rage</a> <a href="http://dollyfish.net.nz/blog/2008-04-01/mutt-and-vim-custom-autocompletion">these days</a> at work, I might as well document here a few things about my .muttrc that might be useful to others.</p>
<p>(I recommend that you start off with the <a href="http://packages.debian.org/mutt">Debian</a> or <a href="http://packages.ubuntu.com/mutt">Ubuntu</a> package for mutt since a large chunk of necessary customization work has already been done by the Debian maintainers.)</p>
<p>Here is how I handle two user accounts. Let's start off by listing both of these addresses so that mutt can recognize emails using these as being sent by me:</p>
<pre><code>alternates "(fmarier@gmail\.com)|(francois@debian\.org)"
</code></pre>
<p>Now this is to allow me to change the From address within the editor, in case I forgot to set the right one before pressing 'm':</p>
<pre><code>set use_from
</code></pre>
<p>and this one pre-fills the From address when replying to emails based on the email account that received the original mail:</p>
<pre><code>set reverse_name
</code></pre>
<p>Here are the two settings that I'm going to change when switching to a different identity (from address and status bar display).</p>
<pre><code># default profile
set from="francois@fmarier.org"
set status_format="-%r-francois@fmarier.org: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b?%?l? %l?]---(%s/%S)-%>-(%P)---"
</code></pre>
<p>Here are the macros I use to switch from one account to the next (bound to ESC-1 and ESC-2):</p>
<pre><code>macro index \e1 ":set from=francois@fmarier.org\n:set status_format=\"-%r-francois@fmarier.org: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b?%?l? %l?]---(%s/%S)-%>-(%P)---\"\n" "Switch to francois@fmarier.org"
macro index \e2 ":set from=francois@debian.org\n:set status_format=\"-%r-francois@debian.org: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b?%?l? %l?]---(%s/%S)-%>-(%P)---\"\n" "Switch to francois@debian.org"
</code></pre>
<p>And this is the code that uses these macros to switch to the right identity automatically:</p>
<pre><code>folder-hook =inbox 'push \e1oi'
folder-hook =debian-inbox 'push \e2oi'
</code></pre>
<p>(The 'oi' part at the end is there to restore the default sort order.)</p>
<p>Finally, I've got this convenient shortcut which allows me to switch to my inbox quickly and to go from one inbox to the next:</p>
<pre><code>macro index I "c=inbox\n" "Switch to inbox"
folder-hook =inbox 'macro index I "c=debian-inbox\n" "Switch to debian-inbox"'
folder-hook =debian-inbox 'macro index I "c=inbox\n" "Switch to inbox"'
</code></pre>
<p>Next up: <a href="https://feeding.cloud.geek.nz/posts/searching-through-contents-of-emails-in/">indexing your emails using mairix</a>.</p>