Monday, November 11, 2013

Using Mailgun's Event API to Poll for New Messages

If you've ever tried to write email processing logic, you know dealing with IMAP/POP and MIME are all acronyms you'd like to avoid. Having fought that battle in the past, I really did not want to worry about those details. Instead, I want to send and receive messages without worrying about the underlying mechanisms that enable it. This is where Rackspace's Mailgun solution saves the day. Now, mail can be handled using a simple REST API and their servers abstract all the details of MIME, mail protocols, etc. On top of that functionality comes a lot of other great features like mailing lists, campaign tracking, etc. However, my first goal was to tackle sending/receiving mail. Sending turned out to be quite simple, however, receiving required a little more work.

One of the main components of Mailgun are using various webhooks to notify you in real-time that something has happened. As long as you have a publicly accessible server, you can receive the event and act accordingly. However, utilizing this mechanism to accept a message is not always possible. The alternative approach requires you to look for stored events via the Events API end point, extract the message key, and then call the Messages API end point to retrieve the actual message. When polling the events, you'll need to know if you already processed a message so you know not to attempt to retrieve it again. As a result, you need to keep some state information locally to compare against while reading the events. This data only needs to be retained long enough to be outside of the maximum window that you'll use in the Events API call.

For my purposes, I would either look back an hour prior to the last message received or 24 hours, which ever was first. I saved state in a MongoDB collection and used MongoMapper to provide the interface to the database. To access Mailgun, I used HashNuke/mailgun to wrap the REST requests into a nice object layer. The current project did not have functionality to interface with the Events portion of the API, so I forked it and created an additions branch.


   Module Mail

      class Store
         include MongoMapper::Document

         key :message_id,   String
         key :event,        Hash
         key :message,      Hash
         key :last_updated, Time
      end

      def fetch( domain )

         mailgun = Mailgun( :domain=>domain )

         messages = []
         
         # Look at the most recent event to frame the range for the request parameters
         last = Store.where( :last_updated =>{  :$lt => 1.day.ago.utc } ).sort( :last_updated.desc ).first
         
         # Nothing in the store, so fall back to 24 hours
         last ||= 1.day.ago.utc

         # Pad the range by an hour to avoid gaps - overlaps are fine - gaps are not
         parameters = {
             :end   => (last - 1.hour).rfc2822,
             :event   => 'stored'
         }
         
         # Make the request to the Mailgun servers
         recent = mailgun.events.list( parameters )

         # Iterate over all the events
         recent.each do | ev |
            
            # Grab the message ID of the email to verify if we've already
            # processed this event
            msgid = ev['message']['headers']['message-id']
             
            if Store.where( :message_id => msgid ).all.length === 0

               begin
                  # This is indeed a new message, so using the key in the
                  # event, request the message from the server
                  msg = mailgun.messages.fetch_email( ev['storage']['key'] )
                  # grab attachments too ... not shown

                  # Store everything for latter so we don't process this again.
                  Store.create({
                     :message_id => msgid,
                     :event => ev,
                     :message => msg,
                     :last_updated => Time.now.utc
                  })

                  # Add it to the list ...
                  messages.push( msg )
                  
                  # Optionally, remove it from the server
                  mailgun.messages.delete_email( ev['storage']['key'] )
               rescue
                  # Deal with bad news
               end
            end
         end

         messages
      end

   end



This is a pretty good starting point for building a polling solution. Depending on your needs, you may occasionally clean up the Store to remove older events. Since the window is being kept small, you really only need to retain 2 days worth to ensure coverage. Additionally, I left out the sections required to fetch attachments. Those are represented as links in the message back to the resource on the Mailgun server. It would make sense to have that functionality in the Mailgun library, however, it is currently not there and would need to be added. Finally, you still need to actually run this code to perform the polling. I chose to use Resque Scheduler to setup a job that would call the fetch function and then further process the messages. All-in-all, after using it for a little while, Mailgun has been a great solution for managing email. With the API, it turns into just another data-like service which is exactly what I wanted.