Testing Myopia - short sightedness in interpreting results of A/B Tests

A/B Testing is an effective way of continually optimizing a website where all the changes made to a website are backed by supporting evidence in the form of the results of tests.

The first requirement of a well thought-out A/B test is a hypothesis that clearly outlines the expected outcome that the intended change will bring about. This well-defined hypothesis then gives us the success metrics that will help us decide whether the proposed change is more effective. If the hypothesis and success metrics are not defined right at the beginning, there is a high likelihood that multiple stakeholders would interpret the test results in a way that resonates better with their own beliefs/ objectives.

Assume that we run an A/B Test to optimize the masthead/ main navigation on the website. The image below shows an example of the current state of a masthead and a proposed alternate layout.

Let us assume 2 scenarios for the A/B test. In the visual depictions that follow, grey icons indicate the visitors to the page/ pages where the test is running. For convenience, let us assume that there is only 1 visit by each visitor. The purple icons indicate the visitors who used the masthead. Visitors who used the masthead and placed an order are shown in green. Again, to keep this illustration simple, assume that the visitors, who do not use the masthead, do not place an order.

In scenario 1, the usage of masthead is higher in the alternate layout (5 out of 10 visitors for the tested variation vs 4 out of 10 visitors for the current state) but the conversion rate for users of the masthead is much lower (1 order from 5 masthead users in the tested variation vs 2 orders from 4 masthead users in the current state). While assessing the test results, some might say that the masthead usage has increased and hence the tested variation is a winner. Some might argue that the conversion of masthead users is lower and hence the tested variation is not a winner.

Both these views are "Myopic"or short-sighted. The key comparison will be what percentage of visitors arriving at the tested page/ pages eventually placed an order. I use the term "Effective Conversion"  to represent this. Effective conversion is the number of orders facilitated by the element being tested (masthead in this case) as a percentage of visitors that qualify for the experience where the variation is active. This quantifies the role played by the element in translating a visit into a revenue generating one.

Looking at Effective Conversion also helps us overcome, what I call,  "Convenience Segmentation" approach of looking at the performance. I define Convenience Segmentation as the approach of focusing on the performance of only a smaller segment of visitors and concluding that any improvement in the performance of this segment is good enough to say that the overall experience is more effective (even if there is no visible improvement in success metrics at a broader level).

Let us now look back at scenario-1. We get the following values for "Effective Conversion".

Effective Conversion at the bottom of the table clearly shows that the proposed variation for the masthead is less effective in driving visitors to conclude a purchase.

Now, let's visit scenario 2. Here, assume that the usage of masthead is lower as compared to the current state but the conversion rate for the users of masthead is higher.

Again, if we look at engagement with the masthead alone (3 out of 10 visitors for the tested variation vs 4 out of 10 visitors for the current state), we will conclude that the change is ineffective while focusing on the conversion rate for masthead users  (2 orders from 3 masthead users in the tested variation vs 2 orders from 4 masthead users in the current state) will lead us to believe that the change works well.

Here again, if we do not want to be myopic, what we need to look at is the effective conversion for the masthead.

Effective conversion for both the current state and the proposed variation is the same, which tells us that the masthead is effective in driving conversion rate to the same extent in both the cases.

Although a fairly simplified illustration, it still highlights the importance of looking at the larger picture while interpreting results of A/B tests. An ideal approach to optimize an element will be to work towards both increased engagement as well as increased efficacy of the element in driving conversion rate higher and we need to look at "Effective Conversion" to make an appropriate conclusion.

From Cart to Conversion - a simple framework for an effective checkout

Cart abandonment has been a challenge for ecommerce websites for a long time. There are numerous estimates of industry cart abandonment rate - this summary from Baymard gives a number of such statistics.  All these statistics indicate that abandonment rate remains to be an area of concern for most, if not all, eCommerce websites or online retailers.

Multiple surveys and studies in the past have tried to explore the key reasons behind users adding a product to cart and then not going ahead to complete the purchase. Some of the top reasons, as quoted by Forrester Research, were high shipping and handling costs, user was just comparing the prices or the product price was very high.

If we look at the long tail of the reasons, we come across many reasons that websites can address relatively easily 

• Shipping and handling costs were listed too late during the checkout process
• I didn't want to register with a site
• Site was asking for too much information
• Checkout process was too long or confusing
• Website didn't have the payment option I wanted to use

Cart abandonment can be be lowered by focussing on some of these many, but easy to address, reasons that customers quote for not completing a purchase. While the segment of shoppers spending time on multiple websites for research may not always have a high intent to complete the purchase immediately, the journey should be made as convenient as possible for visitors with a serious intent of purchase. 

In my opinion, "Convenience" is one of the key factors that makes eCommerce so popular - be it the convenience of comparing multiple products with the least effort, convenience of shopping from home or sometimes the convenience of returning a product without being asked for an explanation. 



A simple framework for Checkout success

What I have outlined below is a simple 4-element framework that any eCommerce website can use to minimize the number of visitors who do all the hard work of deciding what product to buy but then have to give up due to the hurdles they face in checkout.


1. SIMPLIFY

Checkout pages should reduce, as Steve Krug put it in "Don't make me think", “cognitive workload” for the visitors. A good checkout should seek only the most relevant information, keep the forms simple and make moving forward very convenient. Let's see some instances where websites, driven by another objective, overlook the simplification factor.

Some websites offer a "Guest Checkout" action but make it very difficult to spot on the page. While this may seem to be a great idea to encourage more sign-ups, in my opinion, this is an example of being myopic and limiting the number of people who complete the process thereby giving the website an opportunity to re-target them at a later point in time. Figure 1 has 2 images - the one on the left shows an example of an attempt to make "Guest Checkout" almost obscure, while the one on the right shows what I believe is in line with the principle of simplification. 

  

Fig. 1 Prominence of the "Guest Checkout" call-to-action

Figure 2 shows, on the left, an example of overwhelming customers by seeking a lot of information on the checkout pages and then on the right, a easy-to-fill form that simplifies the checkout journey for customers.

Fig 2. Minimize the number of fields on checkout pages

A single page checkout process is probably a direction worth pursuing. While companies offer one-click checkout or express checkout for registered users, what I am indicating is the need to condense the entire checkout section to a single page even for users who do not want to register. The other alternative that comes closest to this is an accordion layout where the experience is smoother and users do not have to navigate through multiple pages in checkout.


2. INFORM 

Cart as well as Checkout pages should give all the necessary information relevant to the purchase but at the same time do not distract with excess information.

Figure 3 shows two examples. The image on the right is similar to the one on the left with an additional piece of information - the accepted modes of payment. Showing the right information at the right time makes a big difference. By showing the payment options on the cart page, a website can prevent users being surprised on the payment page when they do not find their preferred payment mode. 

While showing the payment options on the cart page might lead to higher exits on the cart page, it is a better alternative to not showing this information at all. Why I say this is that a user still has the option of saving his cart and coming back later to complete his purchase but when he/ she goes right up to the payment page and does not find the preferred mode, it is highly likely that the person will never come back given that there, usually, is no option to save cart later in checkout.  

Fig 3. Mention the accepted payment options on cart page

The image on the right in Figure 4 shows a checkout summary which is a great idea to eliminate any anxiety that a user may have on checkout pages. By showing all the relevant information at all times, the need to press the "Back" button in checkout is obviated.

Fig 4. A summary on all the pages of checkout

3. ASSURE

Checkout pages often ask for a lot of information making the prospective customers wonder why do they need to reveal so much about themselves for an order. A good checkout process will ask for only the information that is critical to order fulfilment. 

A good example of this is a message that users know that the checkout process is completely secure and they need not worry about their information being misused, as shown in the second image below where a line of text placed below the "call-to-action" button offers assurance.

4. RESUME

A good website always offers the customers who abandon cart, an option to continue their journey on the website from where they left off.

An example as shown below is offering a direct path to the cart or reminding the user about his/ her product selection when he/ she comes back to the website having added a product to cart in the previous visit.

Well drafted cart abandonment mails that remind users to come back to the website at any point in time and resume their purchase process is another effective tool in making life easier for visitors when they come to the website at a later date.
Optimizing cart and checkout with this broader framework in perspective, will be definitely helpful in achieving a more effective checkout and reduce cart abandonment significantly.