Using GMail's Plus-Style Addressing To Track Bounce Back Email Meta Data
Posted April 16, 2010 at 10:13 AM by Ben Nadel
The other day, after some inspiration from Jose Galdamez, I tried tracking bounce back emails with GMail. The approach used GMail's plus-style (+) address functionality to embed email meta data directly into the FailTo address of ColdFusion's CFMail tag. Unfortunately, when you send email using GMail as your SMTP server, custom Return-Path values are ignored; no matter what FailTo address or CFMailParam Return-Path header you define, GMail uses the SMTP login username as the outgoing Return-Path header. This morning, I decided to revisit this approach, this time using SmarterMail rather than GMail as the SMTP server.
To enable more effective filtering, GMail allows its users to use plus-style addressing to append arbitrary values after the username in their email addresses. For example, if I wanted to create some sort of filtering for newsletter subscriptions, I could use the following email address as my sign-up:
ben+newsletters@bennadel.com
Notice here that I am appending "+newsletters" to my username. This won't affect the account to which the email gets delivered; but, it will enable me to create custom filters for TO addresses that contain the phrase, "+newsletters."
If we can use a GMail account as our FailTo address in the ColdFusion CFMail tag, then we can start to build on top of this plus-style functionality to help track bounce backs. The idea here is rather simple - rather than appending something like "newsletters", we're going to append the customer ID to the FailTo email address. Then, when we monitor the FailTo account, we can easily extract the customer ID directly from the IMAP record.
Let's take a look at how this might work. In the following code, we're going to be sending out an email to a "Customer" with an invalid email account. As part of the FailTo address, we're going to include the customer's ID using plus-style notation.
- <!---
- Define the customer ID. This is going to be made part of the
- FailTo address for easier bounce-back tracking.
- --->
- <cfset customerID = "C12345" />
-
- <!---
- Send out a email to an email address we know won't exist. In
- the FailTo address, since we are using a GMail account (hosted),
- we can use PLUS (+) addressing. We are going to append the
- customerID to the FailTo address.
-
- NOTE: I am including a BCC only so I can the outgoing message
- header to see how the return-path is set.
- --->
- <cfmail
- to="the-kiminatrix@hotmail.com"
- bcc="ben+bcc@bennadel.com"
- from="ben+from@bennadel.com"
- failto="coldfusion+#customerID#@bennadel.com"
- subject="This Is A FailTo Test"
- type="html">
-
- <p>
- Hey there, this is a fail to test.
- </p>
-
- <p>
- To: the-kiminatrix@hotmail.com
- </p>
-
- <p>
- FailTo: coldfusion+<strong>#customerID#</strong>@bennadel.com
- </p>
-
- </cfmail>
In the above CFMail tag, I included a BCC address so that I could see what kind of outgoing headers were implemented by the SMTP server. This is an abbreviated version of what that BCC account received:
Delivered-To: ben+bcc@bennadel.com
Return-Path: <coldfusion+c12345@bennadel.com>
Date: Fri, 16 Apr 2010 09:17:32 -0400 (EDT)
From: ben+from@bennadel.com
To: the-kiminatrix@hotmail.com
Subject: This Is A FailTo Test
MIME-Version: 1.0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Mailer: ColdFusion 8 Application Server
As you can see, now that we are using SmarterMail - not GMail - as the SMTP server, our CFMail FailTo attribute successfully set the Return-Path header:
coldfusion+c12345@bennadel.com
With the customer ID - C12345 - embedded directly within the FailTo address, we can now use CFIMAP (ColdFusion 9 only) to easily monitor the bounce back messages in our GMail account:
- <!---
- Set up IMAP email configuration for our FailTo address.
-
- NOTE: This is a GMail hosted email account so we will be able
- to use the PLUS (+) addressing in the email addresses.
-
- NOTE: Since this is using secure IMAP features, this part of the
- code is only available for ColdFusion 9.
- --->
- <cfset imapSettings = {
- server="imap.gmail.com",
- username="coldfusion@bennadel.com",
- password="***********************",
- port="993",
- secure="true"
- } />
-
-
- <!--- Get the headers from the FailTo account. --->
- <cfimap
- name="headers"
- action="getheaderonly"
- attributecollection="#imapSettings#"
- />
-
-
- <!---
- Loop over the headers looking for one that contains a
- plus-style addressing - this will be our FailTo emails.
- --->
- <cfloop query="headers">
-
- <!--- Check for the plus. --->
- <cfif find( "+", headers.to )>
-
- <!---
- Output the customer ID - this will be the part
- of the email address that is between the + and
- the @ sybols.
- --->
- <cfoutput>
-
- Customer ID: #listGetAt( headers.to, 2, "+@" )#
-
- </cfoutput>
-
- </cfif>
-
- </cfloop>
As you can see here, all we need to do is pull down the FailTo account headers using CFIMAP and then look for TO addresses that use the plus-style formatting. Once we have a given address, we can extract the customer ID, treating the email address as a value list delimited by the "+" and "@" characters. When we run the above page, we get the following output:
Customer ID: c12345
While this works well in this particular situation, I have to admit that it took me a long time to figure this out. I spent about an hour this morning trying to send emails to non-existent domains; then, I finally figured out that this seems to only work when the domain name is real (ie. GMail, Hotmail) but the username is not. There must be something special about my SmarterMail SMTP server because, as I demonstrated yesterday, Postmark has no problems tracking bounce back emails sent do non-existent domains.
I really love the idea of using GMail's plus-style addressing to embed meta data directly within the FailTo / Return-Path value; but, it appears that tracking failed emails is a bit more complicated than that. Considering that this won't work if you either use GMail as your SMTP server or if you send an email to a non-existent domain, it starts to make a lot of sense as to why services like Postmark and Campaign Monitor exist.
Reader Comments
Yeah, Postmark aren't kidding when they say:
"You should not have to worry about this stuff, it's an entire industry on its own"
@Seb,
Word up - this stuff seems to have way too many caveats :) All things considered, it's completely lucky that I even tested with non-existent domains; had I started with something real (ie. foobar@yahoo.com), it would have worked fine and I would not even have know there were potential complications.
Besides, I'd be very surprised if foobar@yahoo.com doesn't actually exist!
@Seb,
Ha ha, good point.... and now the spiders know alllll about it!
Ben did you also know that you can use periods in your email address such as e.r.n.e.s.t.b.r.e.a.u@gmail.com and it works!!!
A little off topic, but a great tidbit nonetheless.
@Ernest,
Yeah, that's a really interesting feature. I guess this is another thing that could theoretically be used for filtering? I am not sure what the practical purpose is.
Having nothing to do with practical, sometimes, I find it simply more "classy" to use a period:
ben.nadel@gmail.com
I am not sure why I find it classier, but I do :P
Hey Ben, looks great, I need to start building something along these lines soon. But tell me, do we need to use the imap capabilities of CF9? We're still stuck on 8 over here.
Can we acheive the same thing with CFPOP or is that more of a gmail restriction?
Cheers
Jim
@JimC,
You can use CFPOP on CF8 to connect to GMail, but it takes a tiny bit of finagling. You need to connect over SSL which doesn't officially exist in the CFPop tag until CF9. However, you can turn on SSL in the Java layer before you use the CFPOP tag and it should work.
The hack is like 3 lines of code, but it should work. Check out Jacob Munson's blog for the details:
http://techfeed.net/blog/index.cfm/2008/3/28/No-SSL-With-ColdFusions-CFPOP
Awesome, cheers Ben. I think that was a reply in 30 secs. Do you ever sleep?
@JimC,
... as I drink my 2nd Monster of the day - not nearly enough sleep as I'd like :)
Ben, I just recently reworked the bounce back system where I'm at so it is interesting to see your posts on the same subject. I thought about using + addressing, but after some testing found that my success rate of using a regex to find the email address in the bounce message body was very good.
The larger challenge for me in this process was determining if the message sent to the bounce account was really a bounce and if so, a soft or hard bounce. For this part, I used the signature file that is part of Email Bounce Detector, http://bouncedetector.riaforge.org/.
After some tuning of the signature file I'm pretty happy with the results.
As for the your issue with SmarterMail and not receiving a bounce when sending to non existent domain. It sounds like a configuration issue with SmarterMail. I'm using Exchange as my smtp server and do receive bounces for non existent domains. My server sends a message stating the domain could not be found in dns or something similar.
The bounce process is a strange beast with the all of the different configurations options on the different email servers out there as well as the vast array of administrative decisions that are made. I hope to do a blog post of my own on the trials and tribulations I went through managing bounces at the organization I work at.
Somewhat OT, but I use the + modifier on my gmail address when I sign up for new services to see if spam that I receive contains it.
I will usually make my address "me+theSiteName@gmail.com". If I receive spam on that address, then I know that the site either sold or otherwise gave up(willingly or unwillingly) my email address.
Hi Ben,
I feel a bit stupid for asking this: I know that ben+newsletters@gmail.com will get delivered to ben@gmail.com, but how is emails to ben+newsletters@bennadel.com not get rejected by the mail server as unknown user?
Did you create an alias? Or maybe something about the gmail set up I don't understand?
Thanks!
ps: love your blog!
@Uly - Plus addressing is a feature that some email servers support. Some info can be found here: http://en.wikipedia.org/wiki/E-mail_address#Sub-addressing
@John,
Yeah, the bounce process is definitely a beast! I'll check out that project on RIAForge, thanks.
@JAlpino,
That's a cool idea. I really like the idea of the plus addressing, but have not tried it yet for anything other than this bounce back experimentation.
@Uly,
Totally good question - bennadel.com uses a hosted GMail account. So while the address is @bennadel.com, it's actually using GMail under the hood; as such, I get to use all the little tricks that GMail does.
I am finishing up development of a new Mass Email application and figuring out how to track bounces was the only piece I had left to completely figure out.
I had tested downloading the bounced emails from the server into my mail program using CFPOP so the bounce could be analyzed and fixed locally. The bounces will be stored in a local database table, linked to the broadcast that the email came from and removed from the server once I verify the message was downloaded. The email admin can go in and look at the bouced email. If you use coldfusion to send the mail you can also review the email logs that are created daily. If you use an offsite server to send the email then you have to look at the bounced message details.
This way the bounce account does not have a chance to grow to large. In my testing CFPOP would freeze my coldfusion service if the email box got to large, over 5k emails.
I will agree that this is a bit more data intensive than it needs to be but once the data was stored locally I could parse the body for the error message and the bounce from invalid domain, incorret emails, domains marked for spam etc... I use smartermail for production and testing.
One side note, I use CF_Mail by http://www.zrinity.com, it seems to be more feature rich and more stable than CFMail by adobe. I am using CF8 and will be upgrading to 9.x in the next month or two to take advantage of its improved integration with excel and some other features that would be useful for my new app.
I am 98% finished with my mass email program, it should be out of beta next month.
It is in use at several sites now and has successfully sent 40,000+ emails since it has been in beta. Let me know if anyone is interested in seeing it when I release it, I would love some feedback from other developers.
@Kevin,
Sounds like you have a pretty comprehensive approach to bounce-back protection going on there. I know this comment was made a while back - has this been launched? How is it working out?
Be careful with sending too many emails to the same gmail account at the same time. I have found gmail will refuse emails when it looks like your email account is being bombarded with emails. So if your email campaign has lots of bounces you might not get them all in your gmail account if gmail starts refusing those messages. I do not recall if gmail message gives a temporary failure or permanent failure. Just watch out for this. Temp failures should eventually get to you and might just be delayed.
@Jeff,
Wow, that's an awesome heads-up to have. I had no idea that it might do that.
Actually I was wrong. I got the error message today...it has nothing to do with Google. Sorry!
@Jeff,
No worries. Hope you got your errors sorted out.
nice article Ben.
You can actually send email from a different address using GMAIL which may also solve your FAILTO addressing issues as well, here is how.
http://www.askdavetaylor.com/configure_google_gmail_to_have_a_different_sender.html
Also your problem with no receiving bounces when sending email to non existent domains or email addresses it quite normal.
A lot of Mail servers have the option to simply ignore email sent to a non existent user or domain rather than sending back a bounce message, as this causes what is called back scatter which causes a ton of problems for hosts and ISP's.
More info on that here: http://spamlinks.net/prevent-secure-backscatter.htm
@Russ,
Bad ass!! I actually have two email addresses set up in my email account; but, it never occurred to me that those emails would be considered valid for CFIMAP/CFPOP style outgoing mail. Brilliant :)
@Kevin Duncan,
Did you ever finish your project? I've been working on a new email delivery solution myself and I read your comment about yours nearly being done (in 2010). I would be curious to check out what you came up with if you still have it laying around.
