Preparing for the Holidays – Part II: Know the Revenue Impact of Page Speed Improvements with A/B Tests

The Official 2010 Vancover Olympic Store was looking to both reduce site abandonment and increase conversions. By conducting an A/B test (also known as a split test), it was decided to direct 50% of site traffic to a single page checkout (test), and other 50% to the original two page checkout (control), which at the time, yielded sub-par conversion rates. After 606 transactions, it was discovered that the single-page checkout outperformed the original checkout by 21.8%. The store enjoyed increases in web conversions and decreases in cart abandonments. It is vital that eCommerce sites adopt effective conversion rate testing projects (Load Testing, A/B Tests) in preparation for the Holiday season, just like the Vancouver Olympic store did in preparation for heavy traffic flow from the event. 

Part 1 of this series discusses how eCommerce sites can add revenue to load tests. In this segment, we will be discussing how businesses can quantify how page speeds impact conversions with A/B Testing.

What is A/B Testing?

In a nutshell, an A/B test makes it easy for eCommerce retailers to quantify the impact of site changes. This is because new site designs or webpage element changes are placed against current ones to determine which of the two best improves specific website metrics. Metrics include sign-ups, white paper downloads, purchases, etc. Both versions of the site, the current (A) version, called the control, and the “new” (B) version, called the variation, are tested with live traffic. Websites usually split traffic flow 50/50 or 60/40 and assign a cookie to each visitor so that their experience on the website stays consistent. This keeps the integrity of the test. Now, data can be collected from the A/B test and whichever segment that gives the best conversion rate wins! (Or does it?)

Some Popular (and Effective) A/B Tests

There are many different ways to optimize conversion rates using an A/B tests. Here are a few popular favorites:

Call-To-Action Buttons

For landing pages, this single rectangle (or whatever shape you choose) is the most important element. Whether it be an “Add to Cart” button or a “Sign Up for a Newsletter” button, improving click through rates on these increases business value. Run an A/B test where you place the button above the fold and below the fold to see which yields the most clicks. You may be surprised with the results. While every website is different, it is interesting to note that saw that placing a CTA below the fold increased conversions by 304%. You can also change around the wording of the CTA or the color of the button. Content Verve noticed in an A/B test that changing the wording from “You” to “My” within a “Start (Your/My) 30-Day Free Trial” button increased CTR by 90 percent.

Pricing Structures

For eCommerce sites, it is important to strategically outline pricing in a way that both increases conversions and makes shoppers believe they are receiving good value. After all, repeat customers are vital for any business. Practical eCommerce states that the top 1% of customers will spend as much as the bottom 50% over time. Consider running an A/B test that tries pricing items out a variety of ways (4 for $x, 6 for $x) and see what works best for you. Again, every eCommerce site is different, so it is important to not assume that what works for one site will work for yours.

Improving User Experience

This one is huge. There are variety of different ways to improve UX. It is not only important to understand the value of increasing site performance, but also the importance of aesthetics. Run an A/B test that improves certain aesthetics (image quality, navigation, etc) or increases site performance and see how those may effect web conversions. However you may choose to tackle UX, it is important to understand that site speed and conversion rates go hand in hand. 

I’ve done an A/B test. Now what?

You need to analyze the results and quantify the impact your site variation has on conversions. From there, a formidable business argument can be made supporting or neglecting new site changes. Hopefully the variation achieves your goals, but it oftentimes does not and you will have to go back to the drawing board. Do not get discouraged.

The Blind Spot of A/B Testing

For eCommerce sites, it is oftentimes difficult to identify the full breadth of opportunity that site B has over site A. The blind spot occurs when site B converts slightly higher overall but site A has a much better conversion rate for users with page loads faster than “x” seconds. From there, a decision may be made to keep Site A because might have the better potential with some performance tuning.

Blue Triangle can eliminate the blind spots in A/B testing and quantify how page speeds impact A/B testing results.