Shami’s Blog

I’ve been busy setting up a PHPlist server for my employer. All tests went ok, but as soon as we sent our first newsletter Yahoo! blocked the server. After looking around for a solution people suggested we sign all outgoing emails with DomainKeys and not hammer Yahoo’s servers with consecutive connection.

Using DomainKeys was a simple setup with DKIMproxy, as for throttling, you all know Postfix is one high performance MTA, so that would be hard to do. PHPlist has a throttling feature but I didn’t want to use that because it would slow emails going to all domains and it took about half a day to send messages to about 6,000 users. That is unacceptable.

After some more digging around I found that Postfix 2.5 introduced the perfect solution, here you go:

First, add the following lines to your master.cf

domain1      unix  -       -       n       -       -       smtp
        -o smtp_fallback_relay=
domain2      unix  -       -       n       -       -       smtp
        -o smtp_fallback_relay=
domain3      unix  -       -       n       -       -       smtp
        -o smtp_fallback_relay=

Now, to use those transports add these lines to your transport_maps file

domain1.tld    domain1:
domain2.tld    domain2:
domain3.tld    domain3:

Finally, set the destination_rate_delay for those transports in main.cf

domain1_destination_rate_delay = 10s
domain2_destination_rate_delay = 20s
domain3_destination_rate_delay = 30s

This will effectively send all outgoing messages at full speed, except for messages going to domain1.tld, domain2.tld, and domain3.tld; Postfix will wait 10/20/30 seconds after sending each message to domain1.tld/domain2.tld/domain3.tld.

I’m currently experimenting with PHPlist to use for our corporate newsletters. During the tests I got the following error

Database error 1064 while doing query You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘1)’ at line 3

Tracking this problem down I found the following code segment in “admin/send_core.php”

if (is_array($_POST["criteria_values"])) {
$values = join(", ",$_POST["criteria_values"]);
} else {
$values = $_POST["criteria_values"];
}

$values in this segment will always start with a comma. We just need to add a substr statement to fix that. Just change it to

if (is_array($_POST["criteria_values"])) {
$values = join(", ",$_POST["criteria_values"]);
} else {
$values = $_POST["criteria_values"];
}
if (substr($values, 0, 1) == ",") {
$values = substr($values, 1);
}

That’s it. This will remove the starting comma (If it exists)

A while back one of my clients had an unpatched qmail server configured with local system users, it was set up in a collocation long before I took over. After having to listen to a lot of complaints about slow internet connectivity I found out that 40-50MB attachments were very common. Another thing I didn’t like about this set-up is the fact I had to maintain 2 password databases; An Active Directory for local user login and a shadow file for mail. So a local set-up with Active Directory as a back end was needed.

Postfix comes to the rescue. Having used Postfix for the past 3 years I believe it is the best MTA out there. qmail has it’s merits, but I’m not a big fan. After a lot of arguing the client managed to budge and give me one of the old workstations to use as a server, which, a month later had a hard drive crash and I forgot everything about this. Yesterday I thought I should document this since I didn’t see an easy to follow HOWTO for doing such a set-up.

NOTE: Unless mentioned otherwise, all the samples provided here show the lines you need to change in your configuration files, not the whole contents of those files. Remember to restart each daemon after configuration file changes.

Update: After checking vgumus’s setup I need to mention this. You’ll notice the user part of the email address is the same as the Active Directory user name (mshami and mshami@shami.net). Dovecot expects to get the Active Directory username from Postfix. If you want to use some other address in the “mail” field you have to use the virtual alias maps feature from Postfix to return sAMAccountName.

Update 2: This tutorial isn’t a substitute for reading the manual pages and having the basic skills to perform these operations. Please consult the manuals to get an idea of the configuration options for each software.

Enough with the introduction, lets get down to business. Here is what we’re going to use in order of installation:

FreeBSD 7.0. You can use Linux if you want, but you have to change a few steps. I’m using the ports version that came on the CD.
Dovecot 1.0.10
Postfix 2.4.6

Preparation:
We will need to have an Active Directory environment set up. This is out of the scope of this document
We need a non-privileged user in Active Directory to allow the other programs to authenticate, I’m calling it LDAP, and the password will be qwerty
Test username will be mshami and password will be qazxsw
Domain name is shami.local
Base DN is DC=shami,DC=local
IP addresses for our domain controllers are 192.168.192.210 and 192.168.192.211

FreeBSD:
Start your FreeBSD installation, I like to go with minimal installations and then add the needed components. Just make sure to give /usr about 5GB of space and give /var a LOT of space to hold the logs and the mail files. Then install the ports collection.

Dovecot:
The first time I did this I used Courier-IMAP. Its a good program but here it has a major issue. You have to create the home directories for all your users before they can log in. I wrote a patch for that but you have to apply it on both the IMAP and the POP3 daemons. You also have to patch Maildrop to do the same. So I decided to go ahead with Dovecot which after some research appears to have better performance than Courier-IMAP and more importantly has self-healing capabilities which solves this issue.

First, add a user called vmail (Assuming UID 1001 and GID 1001), this will be responsible for handling the virtual mailboxes. Then install Dovecot from ports

adduser
cd /usr/ports/mail/dovecot/
make
make install

Choose LDAP, LDA, and any other options you want to use
Answer yes when asked to create the group and the user dovecot. Asseming UID and GID of 143.

mkdir /var/vmail
chown vmail:vmail /var/vmail

Configuration:

vi /etc/rc.conf
dovecot_enable="YES"

Configure Dovecot

cd /usr/local/etc
cp dovecot-example.conf dovecot.conf
vi dovecot.conf and change the following stuff:

#We'll be starting with IMAP only, add other protocols when you get your system to start
protocols = imap
#Set all usernames to lowercase before authenticating, because Dovecot will create folders with the mixed case characters.
auth_username_format = %Lu
#Enable non-secure logging for testing
disable_plaintext_auth = no
ssl_disable = yes
#No matter how many domains we have, the usernames will be unique, so save the messages to /var/vmail/username
#Same as default_mail_env
mail_location = maildir:/var/vmail/%n
#Since we have virtual delivery, only the vmail user should be able to deliver, in my case the UID of that user is 1001
first_valid_uid = 1001
last_valid_uid = 1001
#Same thing for groups
first_valid_gid = 1001
last_valid_gid = 1001
#Set this to were you want the messages to reside
valid_chroot_dirs = /var/vmail
#auth default section
##Comment passdb pam
##Commend userdb passwd
##Add ldap passdb and userdb
  passdb ldap {
    # Path for LDAP configuration file, see doc/dovecot-ldap.conf for example
    args = /usr/local/etc/dovecot-ldap.conf
  }
  userdb ldap {
    # Path for LDAP configuration file, see doc/dovecot-ldap.conf for example
    args = /usr/local/etc/dovecot-ldap.conf
  }

Set up the LDAP backend:

cp /usr/ports/mail/dovecot/work/dovecot-1.0.10/doc/dovecot-ldap-example.conf /usr/local/etc/dovecot-ldap.conf
vi dovecot-ldap.conf

hosts = 192.168.192.210 192.168.192.211
dn = CN=LDAP User,OU=Special Users,DC=shami,DC=local
dnpass = qwerty
auth_bind = yes
ldap_version = 3
base = dc=shami, dc=local
user_attrs = sAMAccountName=home
user_filter = (&(ObjectClass=person)(sAMAccountName=%u))
pass_filter = (&(ObjectClass=person)(sAMAccountName=%u))
user_global_uid = 1001
user_global_gid = 1001

auth_bind tells dovecot to try to bind to Active Directory with the username and password clients authenticate with. Since Active Directory won’t let us read the password field then we need to do this. we’re not using Kerberos here.

Testing:

/usr/local/etc/rc.d/dovecot start
telnet localhost 143
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
* OK Dovecot ready.
a LOGIN mshami qazxsw
a OK Logged in.
a EXAMINE INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1206022806] UIDs valid
* OK [UIDNEXT 1] Predicted next UID
a OK [READ-ONLY] Select completed.
a LOGOUT
* BYE Logging out
a OK Logout completed.
Connection closed by foreign host.

If you get that then you’re OK. Otherwise check your logs. You can turn on debugging in dovecot.conf. Also, you can use the Global Catalog port in your queries. The Global Catalog doesn’t use referrals, referrals cause some issues some times.

Now it’s time to get SMTP working

cd /usr/ports/mail/postfix
make
make install

Make sure you choose DOVECOT and OPENLDAP. Also choose any other options you need. No need for any Kerberos options. You can use the default options during the make install operation.

Disable sendmail:

vi /etc/rc.conf
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

vi /etc/periodic.conf
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"

Enable Postfix:

vi /etc/rc.conf
postfix_enable="YES"

Fix the Postfix maps

postalias /etc/aliases

Reboot the system for all settings to take effect, then test:

telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 server ESMTP Postfix
quit
221 2.0.0 Bye

Now that Postfix is running, lets hook it up to Active Directory (This is the complete file)

myhostname=mailhost
mydestination=localhost
mynetworks=127.0.0.1
myorigin=shami.net

virtual_mailbox_base = /var/vmail

virtual_uid_maps = static:1001
virtual_gid_maps = static:1001

smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination

alias_maps = hash:/etc/aliases
command_directory = /usr/local/sbin
daemon_directory = /usr/local/libexec/postfix

virtual_mailbox_domains =
  shami.net

#LDAP Stuff
virtual_mailbox_maps = ldap:ldapvirtual
ldapvirtual_server_host =
  ldap://192.168.192.210
  ldap://192.168.192.211
ldapvirtual_search_base = DC=shami,DC=local
ldapvirtual_bind = yes
ldapvirtual_bind_dn = SHAMI\ldap
ldapvirtual_bind_pw = qwerty
ldapvirtual_query_filter = (sAMAccountName=%u)
ldapvirtual_result_attribute = sAMAccountName
ldapvirtual_version = 3
ldapvirtual_chase_referrals = yes
ldapvirtual_result_format=%s/

Lets test:

telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mailhost ESMTP Postfix
helo localhost
250 mailhost
mail from: mshami@shami.net
250 2.1.0 Ok
rcpt to: mshami@shami.net
250 2.1.5 Ok
data
354 End data with .
hi
.
250 2.0.0 Ok: queued as 92B5911460
quit
221 2.0.0 Bye
Connection closed by foreign host.

If all goes well, Postfix will deliver the message to /var/vmail/mshami/

Using the Dovecot LDA:
Normally the virtual delivery agent is enough, but if you want to apply quota or vacation auto reply you’re going to have to use the Dovecot LDA. Also, the Dovecot LDA updates the mailbox indexes which will give you better IMAP/POP3 performance

vi /usr/local/etc/postfix/master.cf
dovecot   unix  -       n       n       -       -       pipe
    flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/deliver -d ${user}

vi /usr/local/etc/postfix/main.cf
virtual_transport=dovecot
dovecot_destination_recipient_limit=1

vi /usr/local/etc/dovecot.conf and uncomment the following (client section removed):
  socket listen {
    master {
      path = /var/run/dovecot/auth-master
      mode = 0600
      user = vmail
      group = vmail
    }
  }

Test again:

telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mailhost ESMTP Postfix
helo localhost
250 mailhost
mail from: mshami@shami.net
250 2.1.0 Ok
rcpt to: mshami@shami.net
250 2.1.5 Ok
data
354 End data with .
hi
.
250 2.0.0 Ok: queued as 9DBBF1143B
quit
221 2.0.0 Bye
Connection closed by foreign host.

Now check your logs, you should see something like this:

postfix/pipe[904]: 9DBBF1143B: to=, relay=dovecot, delay=6.9, delays=6.4/0.01/0/0.56, dsn=2.0.0, status=sent (delivered via dovecot service)

Great, now we’re ready to enable SMTP authentication:

vi /usr/local/etc/postfix/main.cf
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = /var/run/dovecot/auth-client

vi /usr/local/etc/dovecot.conf
  client {
    path = /var/run/dovecot/auth-client
    mode = 0660
    user = postfix
    group = postfix
  }

Testing:

 telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mailhost ESMTP Postfix
ehlo localhost
250-mailhost
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH PLAIN
250-AUTH=PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH PLAIN AG1zaGFtaQBxYXp4c3c=
235 2.0.0 Authentication successful
quit
221 2.0.0 Bye
Connection closed by foreign host.

Instead of AG1zaGFtaQBxYXp4c3c= you can generate your own username/password combination by using the command

printf '\0username\0password' | mmencode

Where \0 is the null byte.

Enabling quota:
There is no need to go through this here, as the Dovecot wiki explains it clearly.

http://wiki.dovecot.org/Quota

My current employer is switching from FirstClass to Microsoft Exchange. One of the questions I got from an employee was how to make Outlook print the first page of an email. When I went to the print dialog I was surprised that it doesn’t have a page range selection option, which surprised me. Having used Windows since version 3.0 I found that option in all Windows applications.

Anyways, according to Microsoft, you can’t. That’s right, they removed one of the very basic options in email from their flagship email client.

The best alternative I found it is to preview in Internet Explorer and print from there.

The other day I got some complaints from one of my mail users that he can’t send an email to someone. Since I’m the one causing the company to loose money because of my bad servers I had to “fix this issue”.

This happens when a mail server -Usually Microsoft Exchange- announces it has support for 8BITMIME when you send an EHLO message to it. After it accepts your message, it won’t be able to convert it to 7 bit format, then it will email you back with a “Body type not supported by remote host” error.

If you’re using Postfix 2.3 or higher this can be easily fixed as follows:

main.cf:
smtp_discard_ehlo_keyword_maps = hash:/etc/postfix/mta_workarounds
mta_workarounds:
1.2.3.4     8bitmime

Our current setup has Postfix 2.2 which does not support smtp_discard_ehlo_keyword_maps and we were not keen on upgrading. You can dump up postfix with “smtp_never_send_ehlo = yes” in main.cf, which will tell postfix not to ask other servers to tell it what features they support, so postfix just uses the minimal requirements for sending emails.

This solution is too crude for my taste, I mean why would you want to dump up Postfix so that a single recipient can benefit, a poorly configured recipient that is.

What you want to do here is dump up Postfix only when it talks to this recipient, here is how:

master.cf:
brokensmtp      unix  -       -       n       -       -       smtp
    -o smtp_never_send_ehlo=yes

transport:
domain.tld        brokensmtp:

main.cf:
transport_maps = hash:path_to_postfix/transport

Note the smtp part in the brokensmtp line, not smtpd. This creates a dumped up outgoing thread. You only have to route emails to such recipients through this outgoing thread and you’re good.

Ok, so today our CEO said he wanted everybody to stamp their emails with the sending date. It’s very simple to just add the date manually, but as you know, a good engineer is a lazy engineer. So this is how I did it:

Add “$$MYDATE$$” to your signature but without the quotes, then add this simple script to VBA:

Private Sub Application_ItemSend(ByVal Item As Object, Cancel As Boolean)
    If Item.Class = olMail Then
        Dim Signature As String
        Signature = Format(Now(), "d/m/yyyy")
        Item.HTMLBody = Replace(Item.HTMLBody, "$$MYDATE$$", Signature, , 1)
    End If
End Sub

This should do it. This script will replace $$MYDATE$$ with the current date as soon as you hit the “Send” button. So you will still see “$$MYDATE$$” while you’re typing the message.

Just make sure you lower the macro security to medium.

Hope that helps

Now that I think of it, I’ve been using Mozilla Thunderbird for more than 3 years now, it is one of the best email clients I have ever seen, and with the available extensions, it can do almost anything.

At home, I only open Thunderbird when I want to check for new email, but at work, I like to leave it open all the time in case I get any emails from my coworkers. In Windows, Thunderbird has a nice little notification icon that appears in the system tray when a new message arrives, when I recently switched to Linux on my work PC, I started to get annoyed by the fact that this icon doesn’t appear when new messages come, so I have to check my mailbox every few minutes.

I have tried multiple notification plugins, but never found anything interesting, till today. I stumbled upon moztraybiff, which does exactly that, and enables you to minimize Thunderbird to the system tray, which is neat.