Mail server: exim4 and OSX

So, this weekend I had some more work to do on my machines.  I’m aiming to rebuild my main server as a 64-bit server, but it has a lot of stuff running on it and it’s not clear that I remember how I installed it all.  Some of it is a bit untidy, so I decided that I’d move my mail server onto a standalone virtual, and my mythbackend onto a standalone virtual (now that it has HDHomeRun tuners, which are IP-based, I don’t need access to the physical cards, so no real barrier to a virtual).

I’ll talk about the mythbackend install later, but this post is about the mail server.  The mail config has been around for a while, I’m using fetchmail to pull mail down from my ISP, then exim4 as a mail transfer agent (MTA) and dovecot as an IMAP server.  This kind of grew over time:

  • I got a domain name and used the 5 free mail accounts on the ISP, to deal with the problem that whenever we changed ISPs we had to change e-mail address (and the fact we were still paying Bigpond for an e-mail address even though we hadn’t had Bigpond for nearly 3 years, but someone wanted to keep the mail address)
  • I pulled my mail down to the local server using fetchmail for kicks and giggles, so I had exim running
  • Then my partner got sick of webmail, so she wanted it in her outlook client.  I set it up to pull down mail as POP3 from the webmail
  • Then she got an android device, and she wanted the mail shared over both devices, so I installed dovecot as an IMAP client so that the e-mail could be shared.

Anyway, the install was one that grew rather than being planned, and it wasn’t clear that I remembered everything I’d done, so doing the reinstall was a bit worrying.

On Sat morning I started a new build as a new virtual – 64-bit Debian Wheezy.  The install went fine, and I added fetchmail, exim4 and dovecot.  Getting the base configuration in place was also fine, including ntp and nis+ for user accounts.

Things got a bit hairy when it came to authorisation.  The old install simply used local passwords in the exim configuration, but in reading about some of that it turned out it was using the old-style DES encryption, which meant it only looked at the first 8 characters of the password.  That didn’t suit me too well, so I wanted to change to PAM authentication.  After a lot of reading and trying, it turns out that the exim maintainers aren’t big fans of using PAM directly due to potential issues with having to make the shadow file (holding all the passwords) readable to exim, which is a security risk.  They recommend SASL instead.

So then I went down the SASL installation path, and actually got that working reasonably quickly, basing largely on this link: http://wiki.debian.org/Exim

All this time dovecot seemed quite happy to just work with no configuration, which surprised me somewhat given how hard exim was proving to be.

Once I had connectivity working, I then wanted to migrate mail across.  Turns out that I’d sort of set up to have my mail in mbox format in /var/mail, and also had a bunch in mbox format in ~/Mail, but then my Dovecot install was putting imap stuff into those same directories, making a general mess that was sort of halfway between mbox and Maildir.  I decided to make the move to Maildir.

This involved changing the config files to say that I should have Maildir, deleting all the spare bumpf out of the directories, then running dsync to migrate the data from mbox to Maildir, then deleting all the old stuff (keeping a backup offline).

Having done all that, plus some other messing around, I had mail working on the new virtual, and I had thunderbird, 2 iPhones and an iPad all working fine.  But the Macbook Air refused to work.

After a lot of reading, chasing, trying things, it turns out that the OSX Mail client sucks.  It really doesn’t play nice with TLS authentication and exim, and the error finding tools (connection doctor) don’t tell you what’s actually going on.  The first problem was that it didn’t like the certificate (self signed), and wasn’t telling me.  Once I shut mail down, started it again, fiddled settings, deleted and re-added the SMTP connection, it decided to ask me whether I’d like to accept the certificate.  That was enough to get IMAP going, but SMTP was still dead.

The lead was in the exim logs, in /var/log/exim4/mainlog I had the error:

So, this weekend I had some more work to do on my machines.  I’m aiming to rebuild my main server as a 64-bit server, but it has a lot of stuff running on it and it’s not clear that I remember how I installed it all.  Some of it is a bit untidy, so I decided that I’d move my mail server onto a standalone virtual, and my mythbackend onto a standalone virtual (now that it has HDHomeRun tuners, which are IP-based, I don’t need access to the physical cards, so no real barrier to a virtual).

I’ll talk about the mythbackend install later, but this post is about the mail server.  The mail config has been around for a while, I’m using fetchmail to pull mail down from my ISP, then exim4 as a mail transfer agent (MTA) and dovecot as an IMAP server.  This kind of grew over time:

  • I got a domain name and used the 5 free mail accounts on the ISP, to deal with the problem that whenever we changed ISPs we had to change e-mail address (and the fact we were still paying Bigpond for an e-mail address even though we hadn’t had Bigpond for nearly 3 years, but someone wanted to keep the mail address)
  • I pulled my mail down to the local server using fetchmail for kicks and giggles, so I had exim running
  • Then my partner got sick of webmail, so she wanted it in her outlook client.  I set it up to pull down mail as POP3 from the webmail
  • Then she got an android device, and she wanted the mail shared over both devices, so I installed dovecot as an IMAP client so that the e-mail could be shared.

Anyway, the install was one that grew rather than being planned, and it wasn’t clear that I remembered everything I’d done.

On Sat morning I started a new build as a new virtual – 64-bit Debian Wheezy.  The install went fine, and I added fetchmail, exim4 and dovecot.  Getting the base configuration in place was also fine, including ntp and nis+ for user accounts.

Things got a bit hairy when it came to authorisation.  The old install simply used local passwords in the exim configuration, but in reading about some of that it turned out it was using the old-style DES encryption, which meant it only looked at the first 8 characters of the password.  That didn’t suit me too well, so I wanted to change to PAM authentication.  After a lot of reading and trying, it turns out that the exim maintainers aren’t big fans of using PAM directly due to potential issues with having to make the shadow file (holding all the passwords) readable to exim, which is a security risk.  They recommend SASL instead.

So then I went down the SASL installation path, and actually got that working reasonably quickly, basing largely on this link: http://wiki.debian.org/Exim

All this time dovecot seemed quite happy to just work with no configuration, which surprised me somewhat given how hard exim was proving to be.

Once I had connectivity working, I then wanted to migrate mail across.  Turns out that I’d sort of set up to have my mail in mbox format in /var/mail, and also had a bunch in mbox format in ~/Mail, but then my Dovecot install was putting imap stuff into those same directories, making a general mess that was sort of halfway between mbox and Maildir.  I decided to make the move to Maildir.

This involved changing the config files to say that I should have Maildir, deleting all the spare bumpf out of the directories, then running dsync to migrate the data from mbox to Maildir, then deleting all the old stuff (keeping a backup offline).

Having done all that, plus some other messing around, I had mail working on the new virtual, and I had thunderbird, 2 iPhones and an iPad all working fine.  But the Macbook Air refused to work.

After a lot of reading, chasing, trying things, it turns out that the OSX Mail client sucks.  It really doesn’t play nice with TLS authentication and exim, and the error finding tools (connection doctor) don’t tell you what’s actually going on.  The first problem was that it didn’t like the certificate (self signed), and wasn’t telling me.  Once I shut mail down, started it again, fiddled settings, deleted and re-added the SMTP connection, it decided to ask me whether I’d like to accept the certificate.  That was enough to get IMAP going, but SMTP was still dead.

The lead was in the exim logs, in /var/log/exim4/mainlog I had the error:

So, this weekend I had some more work to do on my machines.  I’m aiming to rebuild my main server as a 64-bit server, but it has a lot of stuff running on it and it’s not clear that I remember how I installed it all.  Some of it is a bit untidy, so I decided that I’d move my mail server onto a standalone virtual, and my mythbackend onto a standalone virtual (now that it has HDHomeRun tuners, which are IP-based, I don’t need access to the physical cards, so no real barrier to a virtual).

I’ll talk about the mythbackend install later, but this post is about the mail server.  The mail config has been around for a while, I’m using fetchmail to pull mail down from my ISP, then exim4 as a mail transfer agent (MTA) and dovecot as an IMAP server.  This kind of grew over time:

  • I got a domain name and used the 5 free mail accounts on the ISP, to deal with the problem that whenever we changed ISPs we had to change e-mail address (and the fact we were still paying Bigpond for an e-mail address even though we hadn’t had Bigpond for nearly 3 years, but someone wanted to keep the mail address)
  • I pulled my mail down to the local server using fetchmail for kicks and giggles, so I had exim running
  • Then my partner got sick of webmail, so she wanted it in her outlook client.  I set it up to pull down mail as POP3 from the webmail
  • Then she got an android device, and she wanted the mail shared over both devices, so I installed dovecot as an IMAP client so that the e-mail could be shared.

Anyway, the install was one that grew rather than being planned, and it wasn’t clear that I remembered everything I’d done.

On Sat morning I started a new build as a new virtual – 64-bit Debian Wheezy.  The install went fine, and I added fetchmail, exim4 and dovecot.  Getting the base configuration in place was also fine, including ntp and nis+ for user accounts.

Things got a bit hairy when it came to authorisation.  The old install simply used local passwords in the exim configuration, but in reading about some of that it turned out it was using the old-style DES encryption, which meant it only looked at the first 8 characters of the password.  That didn’t suit me too well, so I wanted to change to PAM authentication.  After a lot of reading and trying, it turns out that the exim maintainers aren’t big fans of using PAM directly due to potential issues with having to make the shadow file (holding all the passwords) readable to exim, which is a security risk.  They recommend SASL instead.

So then I went down the SASL installation path, and actually got that working reasonably quickly, basing largely on this link: http://wiki.debian.org/Exim

All this time dovecot seemed quite happy to just work with no configuration, which surprised me somewhat given how hard exim was proving to be.

Once I had connectivity working, I then wanted to migrate mail across.  Turns out that I’d sort of set up to have my mail in mbox format in /var/mail, and also had a bunch in mbox format in ~/Mail, but then my Dovecot install was putting imap stuff into those same directories, making a general mess that was sort of halfway between mbox and Maildir.  I decided to make the move to Maildir.

This involved changing the config files to say that I should have Maildir, deleting all the spare bumpf out of the directories, then running dsync to migrate the data from mbox to Maildir, then deleting all the old stuff (keeping a backup offline).

Having done all that, plus some other messing around, I had mail working on the new virtual, and I had thunderbird, 2 iPhones and an iPad all working fine.  But the Macbook Air refused to work.

After a lot of reading, chasing, trying things, it turns out that the OSX Mail client sucks.  It really doesn’t play nice with TLS authentication and exim, and the error finding tools (connection doctor) don’t tell you what’s actually going on.  The first problem was that it didn’t like the certificate (self signed), and wasn’t telling me.  Once I shut mail down, started it again, fiddled settings, deleted and re-added the SMTP connection, it decided to ask me whether I’d like to accept the certificate.  That was enough to get IMAP going, but SMTP was still dead.

The lead was in the exim logs, in /var/log/exim4/mainlog I had the error:

  LS error on connection from smoothwall ([192.168.1.28]) [192.168.1.2] (gnutls_handshake): A TLS packet with unexpected length was received.

And then very occassionally I also got:

  SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=smoothwall [192.168.1.2] input="260301"

Turns out that Mail on OSX doesn’t really honour the STARTTLS protocol, so it tries to start before it’s been sent the greeting.  And exim doesn’t like that.

The fix for this was to have exim listen on port 587 as well (setting is in /etc/default/exim4) and to then add a setting telling exim that on port 587 it shouldn’t wait for the greeting, through adding:

  tls_on_connect_ports = 587

I now have a mail server that is running happily in a virtual and serving all our mail to all our devices.  A lengthy grind (other than the time out to visit the Aus/NZ waterski championships on Sat, and a BBQ on Sunday), but that’s one of my biggest risk areas pulled off my main server – and running happily in a virtual on the other server.

So, hopefully this learning will help someone else out there in internet land, at some time in the future.

Advertisements

One thought on “Mail server: exim4 and OSX

  1. Pingback: Part One: Install a secure Debian Wheezy imap mail server into a virtual using Exim, Dovecot, Fetchmail | technpol

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s