assessing the security concerns for running an own email server [closed]

As an ex-admin who then became a penetration tester, I feel your pain. However it turns out that because, like you seem to, I cared enough about the security of my system to find out what things like DKIM and SPF are, and implement them, and use SSH keys etc. it turned out that I was doing most things right. As it sounds like you are ;-)

From your logs, it doesn't look like you are, but one of the most important things is to ensure you are not running an open relay. The following link should allow you to test that: http://mxtoolbox.com/diagnostic.aspx

Also you need to prevent user enumeration. You don't want an attacker to be able to log in to your SMTP server and do: "VRFY bob" or "EXPN joe" and have the mail server reply to tell you whether joe exists on the system or not. It is possible to configure mail servers to not support those features. You also don't want the mail server to respond to "MAIL TO: tom" to immediately tell you that tom exists. You get the idea. Look into smtp/pop3/imap user enumeration and mitigate against it.

Another thing - don't use pop3 or imap without encryption. The password will go over the internet in clear text - Wifi is often easy to hack as well, and this will give away the credentials to unencrypted mail services in a moment. You need transport layer security (TLS).

Similar with SMTP, you want to require authentication for remote users sending mail, but this must also be sent over a TLS connection. Also, don't forget to require authentication for local users sending mail, and prevent open-relaying internally. I recently did a pen-test at a law firm where I could telnet to their mailserver on port 25 and send a mail from the head of the company to an employee, telling him he had a raise or was fired. Obviously I didn't do either, but you don't want to allow the possibility. It is best to have non-repudiation if possible.

As it is clear you are good at finding stuff out for yourself, I won't go into great detail, but suffice to say - you need to know about the relative merits of things like POP3S and POP3 with STARTTLS, the difference between ports 25, 465, 587, etc. IMAP and IMAPS, etc. Requiring client TLS certificates for authentication is probably way over the top, but don't use self-signed server TLS certificates, you should get bona-fide ones. You can get them cheap from go-daddy or similar. Don't use SSL2, SSL3, or TLSv1.0. TLSv1.2 should be all that is enabled if possible.

Also, configure your mail servers not to announce their software version (or any product details if possible) when you connect to the port. A simple generic banner should suffice.

Require strong passwords from your users AND admins!

One more thing, to avoid spam, grey-listing can work very well if configured correctly. I used it for a long time and never had issues with undelivered mail.

Machines on the internet are always being scanned and poked at. If you have configured it securely then I wouldn't worry to much about that (although you could use iptables rate limiting if you are concerned about DoS - or speak to your ISP), BUT on the PLESK etc. control panel side of things and SSH, use iptables to whitelist the IPs that REQUIRE access to those ports, so that attackers cannot even see that they are there. Use nmap on your server to see what attackers will see. If all ports appear filtered to the general public except your mail ports, then great. Only use HTTPS, not HTTP to connect to the web administration stuff and again, only TLSv1.2. Or better still, get it only to listen on localhost so it is inaccessible from the outside, and then use SSH port forwarding to connect to it instead!

Google for "server hardening PRODUCTNAME" for each software item you are using, and also "penetration testing PRODUCTNAME", "PRODUCTNAME exploits" could help.

Very important - Keep all software and the linux OS as patched and up to date as possible.

An often overlooked risk is the risk of your own workstation being compromised. As an administrator you are a prime target. If this happens then an attacker may be able to steal your passwords or SSH keys and find your files that will tell them everything thing they need to know about your config. Once they have that, then they can get into your mail system no matter how well you have secured it, even if it is a hosted Office 365 solution. In fact, in some ways it will be easier for them if it is hosted elsewhere and a more familiar configuration.

Don't log into your machine as root or administrator unless you need to. Don't give out more access to your colleagues than they need. etc. Go very careful with opening e-mail attachments etc - the usual stuff.