A fundamental decision email marketers face is choosing what offer to include in a campaign. Will offering a % discount entice customers to buy, or will a small gift card be more effective? More importantly, which will drive greater sales performance? Today, we highlight the recent experience of one of our e-commerce customers to demonstrate how using A/B testing can shed light on what kinds of offers deliver the best results.
Setting up the campaign
For this particular campaign, the online store wanted to test offering two different types of promotions to their customers. The store had experience running different types of promotions, but was unsure of which offers resulted in the best performance. To answer this question, the store generated a target segment of customers on which to run the test, and sent one group a % discount offer and the other group a small gift card for a fixed dollar amount. The sizes of the offers were set such that they represented similar value given average basket sizes for the store.
The store also defined how they were going to measure success before running the campaign. First, they wanted to compare traditional email engagement such as open rates and click rates. Going further, they sought to measure performance in terms of purchase conversion and total sales.
Approximately one week after running the campaign, we measured the open rate, click rate, and purchase conversion rate. We compared these metrics to a recent email newsletter that was sent to a broader list and did not contain a special promotional offer:
Compared to the newsletter, open rates and purchase rates were significantly higher with the promotional campaigns, but click rates fell short. This is intuitive, given the newsletter was designed to provide compelling and clickable articles, while the promotional offers were intended to drive sales. Comparing the two promotional offers, it appears that the gift card campaign was the clear winner, with better open and click performance and a 3.4x lift on purchase conversion. The next step was to assess actual sales performance:
The sales data tells a very different story. As expected, the promotional campaigns generated greater absolute sales (up to 3.5x) than the standard newsletter. However, contrary to the story that the conversion rate data told, the % discount offer actually generated greater sales per email and overall total sales than the gift card campaign. Diving further in the data, we discovered that while the gift card enticed more customers to make a purchase, the average basket size for the purchases generated by the % discount were over 6.2x higher.
This case study brings about lessons that all e-commerce stores can benefit from:
- Conversion alone isn’t enough; track total sales too. Tracking %-based measures such as open, click, and purchase rates can be interesting, but the real story is told by the total sales figures. Sometimes these sets of metrics can tell contradictory stories – go with absolute sales or profit in such cases.
- Pick the offer to match the result. If driving volume and repeat customers is the goal, this example implies a fixed-dollar discount is best. However, if increasing average purchase size or moving high-value inventory is the goal, a % discount campaign is likely the better choice.
Running A/B tests on segmented email campaigns can help online stores determine which types of offers work best and why. Every store is different – consider structuring your own experiments to learn more about how your customers behave and spend – and don’t forget to let us know your own success story!