Email: Protocols and Background

Hullo!

You sure as hell know what email is, but do you know how it actually works? In today's article I'll talk about how it works, and how best to learn about it in detail.

One of my previous articles, "HTTP Requests in Wireshark", showed that the World Wide Web was based on a standard protocol, HTTP. Web browsers and web servers could talk to each other using this protocol.

It turns out that email is no different. Email clients (e.g. Outlook, Thunderbird, etc) and servers (e.g. Microsoft Exchange, hMailServer, etc) can talk to each other using three standard protocols: IMAP, POP and SMTP. SMTP is used for sending email, while IMAP and POP are used for retrieving email from a server. IMAP is more powerful and complex than POP; POP doesn't even support folders. Each of these protocols uses a set of text commands similar to stuff you might have used in the command line, except that in this case you do it over the network (you'll see how in a minute). The protocols also define ports on which servers should listen, e.g. IMAP uses ports 143 and 993 by default.

You could study these protocols by just reading their respective RFCs, but you can only really learn them properly by seeing them in action. To do this, you will need some software.

First, install a mail server. hMailServer is decent and free, so unless you have a better alternative, go with it. Ideally install this on another computer, otherwise you won't be able to capture requests in Wireshark if your client and server are on the same PC. While installing hMailServer, choose SQL Compact as storage (to keep it simple) and give it an admin password.

When you're done installing hMailServer, you need to create an account. Run hMailServer and enter the admin password to access the Administrator console:

Click on the Add domain... button and invent a domain name. By convention it should end in .local, since this isn't a real domain name. Click the Save button to save it.

Next, find the Accounts folder somewhere under the Domains node.

Click the Add... button. Enter a username and password and click Save to create your account. Notice how the domain is used as part of the email address, for example in my case the email address is user@ranchtest.local.

Great. Now, we need to test this account using an email client. Thunderbird is a popular choice and it's free, although there are many others you can use if you want. Install and run it. Skip the shit that appears upon running:

Press Alt to show the program menu, since Thunderbird seem to have fallen into the horrible practice of hiding it. From the Tools menu, click on Account Settings...:

From the drop-down list, select Add Mail Account...:

Enter your basic account credentials, and then click the Manual config button when it appears:

Now you need tell Thunderbird how to connect to your mail server. Under Server hostname, type the name or IP address of the computer where hMailServer is installed. Using the PC name is better since the IP address might change regularly. If you installed hMailServer on the same machine (not recommended, see above) you can use 127.0.0.1 or localhost as the Server hostname. Other than that, use port 143 for IMAP and port 25 for SMTP; no SSL and Normal password for Authentication. It should look like this:

Click the Re-test button to allow Thunderburd to test those settings. If all is ok, the Done button is enabled, click it to save the settings and create the account. When you do this, a warning appears:

It's a security warning because the client and server will talk to each other on an unencrypted channel. Since you're working on a local test environment, you don't have to worry about it. We actually want the messages to be unencrypted, so that we can capture them with Wireshark.

Finally, install Wireshark so that we can take a look at the email protocols. Set it up for capturing; I explained how to do this in my article "HTTP Requests in Wireshark". After starting the capture session, set the Filter to "imap".

Using Thunderbird, send an email to yourself:

The email is sent to the server via SMTP, and then downloaded to the client via IMAP. After a second you should see some activity in Wireshark. From Thunderbird, click Get Mail to retrieve the new email in your inbox.

In Wireshark, you can see the IMAP commands as well as the email data. You can view this conveniently by right clicking on one of the packets and selecting Follow TCP Stream:

This might seem a bit cryptic, but it's quite easy to learn. In future articles I'll cover the email protocols bit by bit. So check back later for more about email technology! :)