Stress-free Incoming E-Mail Processing with Rails

Posted by Luke Francl
on Friday, June 01

When you need to build a Rails application that accepts e-mail, the framework only goes so far. ActionMailer::Base gives you a nice API for handling the message once you get it, but you need a way to trigger that.

Agile Web Development with Rails suggests something like this procmail rule:

RUBY=/Users/dave/ruby1.8/bin/ruby 
TICKET_APP_DIR=/Users/dave/Work/BS2/titles/RAILS/Book/code/e1/mailer 
HANDLER='IncomingTicketHandler.receive(STDIN.read)' 
:0 c 
* ^Subject:.*Bug Report.* 
| cd $TICKET_APP_DIR && $RUBY script/runner $HANDLER 

So does Rails Recipes. So does Slingshot. And the Rails wiki page How To Receive Emails With ActionMailer spends a great deal of space on this technique.

But there are a couple problems with this approach:
  • You need a mail server on your Rails box. And honestly, who wants to configure a mail server? The less time I spend trying to understand Sendmail, the better.
  • Not like I care, but it sort of sucks to do this on Windows.
  • It will fork a Rails process for every e-mail you receive.

The last one is the biggie. If you get hundreds or thousands of e-mails, it is going to fork a ton of processes and that is going to suck.

While most of the space on the Rails wiki is devoted to the procmail-style approach, there are a few tidbits about using POP3 or IMAP to do the fetching, and this comment:

Fetching the mailbox via IMAP is by far the best way of getting messages into Ruby. Piping them through procmail et. al. is very cumbersome, requires access to the unix filesystem and (not in the least) poses a huge security risk. What if someone drops a mailbomb on your system? Not a pretty sight.

When Slantwise needed to build e-mail integration for a client, I asked our hosting provider Engine Yard what they recommended. Ezra Zygmuntowicz replied:

The only production worthy way to receive emails is to have an email daemon that constantly runs in a loop and uses net/imap or net/pop3 to poll the email servers for messages in a loop.

The way they show of using email piped into script/runner is useless because it has to fork a full rails process for each email and then just throw that away. As soon as you start to get any amount of email this solution falls on its face and your servers will hate you for it.

That settled it for us: e-mail checking daemon it is. Using some code adapted from the Rails wiki, we ended up with something like this (this is actually from a different project; some class names and the protocol have been changed to protect the innocent):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/usr/bin/env ruby

RAILS_ENV = ARGV[1] || 'development'

require File.dirname(__FILE__) + '/../config/environment.rb'

require 'secure_pop'

if RAILS_ENV == "test"
  SLEEP_TIME = 10
else
  SLEEP_TIME = 60
end

class MailFetcher < Daemon::Base
  @config = YAML.load(IO.read("#{RAILS_ROOT}/config/mail_fetcher.yml"))
  
  def self.start
    puts "Starting Mail Fetcher Daemon"
    
    loop do
      pop = Net::POP3.new(@config[RAILS_ENV]['server'])
      pop.enable_ssl(OpenSSL::SSL::VERIFY_NONE) if @config[RAILS_ENV]['ssl']
      pop.start(@config[RAILS_ENV]['username'], @config[RAILS_ENV]['password'])
      unless pop.mails.empty?
        pop.each_mail do |m|
          IncomingMailHandler.receive(m.pop)
          m.delete
        end
      end
      pop.finish
      
      sleep(SLEEP_TIME)
    end
  end
  
  def self.stop
    puts "Stopping Mail Fetcher Daemon"
  end
end

SmsFetcher.daemonize

We put this file in scripts/mail_fetcher and ran it using monit (extending the Daemon::Base class gives us some PID file stuff for free. You could write that yourself, or use Daemons or dctl which provide similar functionality.). For more examples of how to do this, see Dave Naffis’s post (original here, but his blog seems to be down) and the Rails wiki (scroll down to the bottom).

Using a daemon has some nice advantages. You have a single long running process that fetches your mail. It can be monitored with monit. You don’t have to run your own mail server.

Another way to use IMAP or POP is to use getmail. This is recommended by O’Reilly’s Rails Cookbook and is mentioned on the Rails wiki. Zak Ainsworth wrote a blog post on how to use getmail with ActionMailer, as did Max Dornseif. Using getmail eliminates need to run your own mail server (and as Zak Ainsworth notes in his post, is great when you don’t have root access) but it still will fork a Rails process for every e-mail you receive. It also means you need to install more software on your server (even if it is just a Python program), whereas using a daemon requires only the libraries built-in to Ruby.

After we built two of these daemon things, Dan Weinand realized the code for them was virtually identical. In pseudo-code, they were both something like this:

1
2
3
4
5
def fetch
  establish_connection
  get_messages
  close_connection
end

The only differences were what protocol was used (IMAP versus POP3) and what to do with the messages once they’d been fetched.

Therefore, Dan created the Slantwise Fetcher plug-in, which greatly simplifies creation of a daemon. After installing the plug-in, you generate a daemon skeleton:

./script/generate fetcher_daemon mail

Then configure it to your liking. The fetcher above gets turned into this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'daemon'
require 'yaml'
require 'fetcher'

class MailFetcherDaemon < Daemon::Base
  
  @config = YAML.load_file("#{RAILS_ROOT}/config/mail_fetcher.yml")
  @config = @config[RAILS_ENV].to_options
  
  @sleep_time = @config.delete(:sleep_time)
  def self.start
    puts "Starting MailFetcher"
    # Add your own receiver object below and specify fetcher subclass
    @fetcher = Fetcher::Pop.new({:receiver => IncomingMailFetcher}.merge(@config))

    loop do
      @fetcher.fetch
      sleep(@sleep_time)
    end
  end
  
end

The fetcher plug-in takes care of all the details of downloading the mail from the server, handing it off to the ActionMailer::Base subclass, and then deleting the messages. It also includes some helpful code we’ve needed in our projects: the PLAIN authentication type for IMAP and secure POP3 support (both back-ported from Ruby 1.9 as a monkey patches).

And that’s the stress-free way to receive e-mail with Rails.

Comments

Leave a response

  1. Matt AimonettiJune 03, 2007 @ 01:01 PM

    good stuff, thanks a lot!

  2. George MoschovitisJune 06, 2007 @ 04:21 AM

    Very interesting. Where can we download the plugin?

  3. ChrisJune 16, 2007 @ 04:26 PM

    Hey I’m trying to get this to work and ran into a wall. If I try to run the script that gets created from the generate then it just throws:

    no such file to load—fetcher (LoadError)

    Am I missing how this is supposed to work? I have installed the plugin and I verified that fetcher is sitting in vendor/plugins/fetcher.

    Thanks for the help!

  4. ChrisJune 16, 2007 @ 05:31 PM

    I tried to leave a comment earlier and it didn’t show up, so here goes again:

    I’m trying to implement this and need some help on getting it going. I’m having trouble running the daemon, I use this command to get it going (this is after generating and customizing):

    ./script/runner script/mail_fetcher start

    Upon doing this it does nothing, it never calls the start method for some reason.

    Then I’m wondering if you have an example receiver for the fetcher, I just don’t know what the structure should look like.

    Thanks for any help!!

  5. Luke FranclJune 19, 2007 @ 03:55 AM

    Hi Chris, I’m AFL (away from laptop…I’ve been mostly offline for the last few weeks) so I’m not sure if I can totally help you debug the problem, but have you tried running the fetcher without using script/runner? I don’t think that’s necessary. Anyway, give it a try and maybe it will work.

    Thanks for trying out the plugin and giving your feed back.

  6. MauChiYuYoJune 21, 2007 @ 12:03 PM

    Hi, nice blog

    well I have an email server made whit ActionMailer::Base

    and I want to implement a resource to import my contact list from Yahoo or Hotmail or Gmail, so for that I have downloaded and installed “contacts” a library that I found but I dont know how to use and I am feeling lost in the code so can some one help me

    Tnx

  7. JoshuaJune 29, 2007 @ 09:10 AM

    Nice! Minor typo? it looks like the last line in the first block of Ruby code should be: MailFetcher.daemonize