What is A/B Testing?


A/B testing is a term used in technology and marketing which refers to sending out two different versions of a message or product and then comparing the metrics to see which version performs better. 


In terms of emails, this means sending out an email with two different variations on a feature (an "A" email and a "B" email) and then comparing their stats to see which has a better open rate or gets you more volunteers, money, etc. Then you can use the more effective method for future emails. 


Some things that may be worth A/B testing include:

  • The body of your email, including different pitches, colors, fonts, etc.
  • The email subject line
  • The "name" the email is coming from
  • The day or time of day an email is sent


To gain insights out of A/B testing, it is good practice to make only one change at a time. For example, you do not want to A/B test the subject line and the email content in the same email because you won't be able to tell whether the performance variation was driven by one change or the other. A/B Test one feature at a time, and keep the changes relatively small. 


A/B testing also works better at scale. Generally, save your testing for emails that go out to a large, relatively diverse group so you get a sense of how the changes play among your whole universe of voters. 


ActionNetwork allows you to test some of these features in their tool. Below is a step by step guide to A/B testing in ActionNetwork


How to Use ActionNetwork's A/B Testing Feature


Open up a new email draft. You'll notice that certain fields have a green plus next to them. Choose which feature you'd like to test and click the plus next to that field


This will bring up a second box which will allow you to type a new variation of the content. For example, you could A/B test the From field as follows- A: Your Name B: Your Organization's Name.


From there, create, target, and send your email as normal. ActionNetwork will randomly send half of your targets email A and the other half email B


Once the email is sent and people start taking actions (opening the email, clicking on links, signing up for volunteer shifts, giving money, etc), go to the emails tab. You'll see two separate sections for the same email - one for version A and one for version B. From there you can open each and compare their stats to one another. 


After reviewing the stats, you can choose to take actions based on that information. For example, if the open rate with the "From" line as your organization is higher than your name, you may want to use the organization's name from now on. 


You can also A/B test manually, sending two different emails with different features, such as the same email at different times. ActionNetwork's built-in feature is just meant to make the process easier, but you can do the same thing manually if you'd like. 


                                                                                                                                                                                                


Questions or feedback? Please email us at voterfile@dfl.org 


If you have suggestions for other articles that you’d like to see in the knowledge base, submit your request here.