Stress-free Incoming E-Mail Processing with Rails

Posted by Luke
on Friday, June 01

When you need to build a Rails application that accepts e-mail, the framework only goes so far. ActionMailer::Base gives you a nice API for handling the message once you get it, but you need a way to trigger that.

Agile Web Development with Rails suggests something like this procmail rule:

RUBY=/Users/dave/ruby1.8/bin/ruby 
TICKET_APP_DIR=/Users/dave/Work/BS2/titles/RAILS/Book/code/e1/mailer 
HANDLER='IncomingTicketHandler.receive(STDIN.read)' 
:0 c 
* ^Subject:.*Bug Report.* 
| cd $TICKET_APP_DIR && $RUBY script/runner $HANDLER

So does Rails Recipes. So does Slingshot. And the Rails wiki page How To Receive Emails With ActionMailer spends a great deal of space on this technique.

But there are a couple problems with this approach:
  • You need a mail server on your Rails box. And honestly, who wants to configure a mail server? The less time I spend trying to understand Sendmail, the better.
  • Not like I care, but it sort of sucks to do this on Windows.
  • It will fork a Rails process for every e-mail you receive.

The last one is the biggie. If you get hundreds or thousands of e-mails, it is going to fork a ton of processes and that is going to suck.

While most of the space on the Rails wiki is devoted to the procmail-style approach, there are a few tidbits about using POP3 or IMAP to do the fetching, and this comment:

Fetching the mailbox via IMAP is by far the best way of getting messages into Ruby. Piping them through procmail et. al. is very cumbersome, requires access to the unix filesystem and (not in the least) poses a huge security risk. What if someone drops a mailbomb on your system? Not a pretty sight.

When Slantwise needed to build e-mail integration for a client, I asked our hosting provider Engine Yard what they recommended. Ezra Zygmuntowicz replied:

The only production worthy way to receive emails is to have an email daemon that constantly runs in a loop and uses net/imap or net/pop3 to poll the email servers for messages in a loop.

The way they show of using email piped into script/runner is useless because it has to fork a full rails process for each email and then just throw that away. As soon as you start to get any amount of email this solution falls on its face and your servers will hate you for it.

That settled it for us: e-mail checking daemon it is. Using some code adapted from the Rails wiki, we ended up with something like this (this is actually from a different project; some class names and the protocol have been changed to protect the innocent):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#!/usr/bin/env ruby

RAILS_ENV = ARGV[1] || 'development'

require File.dirname(__FILE__) + '/../config/environment.rb'

require 'secure_pop'

if RAILS_ENV == "test"
  SLEEP_TIME = 10
else
  SLEEP_TIME = 60
end

class MailFetcher < Daemon::Base
  @config = YAML.load(IO.read("#{RAILS_ROOT}/config/mail_fetcher.yml"))
  
  def self.start
    puts "Starting Mail Fetcher Daemon"
    
    loop do
      pop = Net::POP3.new(@config[RAILS_ENV]['server'])
      pop.enable_ssl(OpenSSL::SSL::VERIFY_NONE) if @config[RAILS_ENV]['ssl']
      pop.start(@config[RAILS_ENV]['username'], @config[RAILS_ENV]['password'])
      unless pop.mails.empty?
        pop.each_mail do |m|
          IncomingMailHandler.receive(m.pop)
          m.delete
        end
      end
      pop.finish
      
      sleep(SLEEP_TIME)
    end
  end
  
  def self.stop
    puts "Stopping Mail Fetcher Daemon"
  end
end

SmsFetcher.daemonize

We put this file in scripts/mail_fetcher and ran it using monit (extending the Daemon::Base class gives us some PID file stuff for free. You could write that yourself, or use Daemons or dctl which provide similar functionality.). For more examples of how to do this, see Dave Naffis’s post (original here, but his blog seems to be down) and the Rails wiki (scroll down to the bottom).

Using a daemon has some nice advantages. You have a single long running process that fetches your mail. It can be monitored with monit. You don’t have to run your own mail server.

Another way to use IMAP or POP is to use getmail. This is recommended by O’Reilly’s Rails Cookbook and is mentioned on the Rails wiki. Zak Ainsworth wrote a blog post on how to use getmail with ActionMailer, as did Max Dornseif. Using getmail eliminates need to run your own mail server (and as Zak Ainsworth notes in his post, is great when you don’t have root access) but it still will fork a Rails process for every e-mail you receive. It also means you need to install more software on your server (even if it is just a Python program), whereas using a daemon requires only the libraries built-in to Ruby.

After we built two of these daemon things, Dan Weinand realized the code for them was virtually identical. In pseudo-code, they were both something like this:

1
2
3
4
5

def fetch
  establish_connection
  get_messages
  close_connection
end

The only differences were what protocol was used (IMAP versus POP3) and what to do with the messages once they’d been fetched.

Therefore, Dan created the Slantwise Fetcher plug-in, which greatly simplifies creation of a daemon. After installing the plug-in, you generate a daemon skeleton:

./script/generate fetcher_daemon mail

Then configure it to your liking. The fetcher above gets turned into this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'daemon'
require 'yaml'
require 'fetcher'

class MailFetcherDaemon < Daemon::Base
  
  @config = YAML.load_file("#{RAILS_ROOT}/config/mail_fetcher.yml")
  @config = @config[RAILS_ENV].to_options
  
  @sleep_time = @config.delete(:sleep_time)
  def self.start
    puts "Starting MailFetcher"
    # Add your own receiver object below and specify fetcher subclass
    @fetcher = Fetcher::Pop.new({:receiver => IncomingMailFetcher}.merge(@config))

    loop do
      @fetcher.fetch
      sleep(@sleep_time)
    end
  end
  
end

The fetcher plug-in takes care of all the details of downloading the mail from the server, handing it off to the ActionMailer::Base subclass, and then deleting the messages. It also includes some helpful code we’ve needed in our projects: the PLAIN authentication type for IMAP and secure POP3 support (both back-ported from Ruby 1.9 as a monkey patches).

And that’s the stress-free way to receive e-mail with Rails.