Testing Myopia - short sightedness in interpreting results of A/B Tests

A/B Testing is an effective way of continually optimizing a website where all the changes made to a website are backed by supporting evidence in the form of the results of tests.

The first requirement of a well thought-out A/B test is a hypothesis that clearly outlines the expected outcome that the intended change will bring about. This well-defined hypothesis then gives us the success metrics that will help us decide whether the proposed change is more effective. If the hypothesis and success metrics are not defined right at the beginning, there is a high likelihood that multiple stakeholders would interpret the test results in a way that resonates better with their own beliefs/ objectives.

Assume that we run an A/B Test to optimize the masthead/ main navigation on the website. The image below shows an example of the current state of a masthead and a proposed alternate layout.

Let us assume 2 scenarios for the A/B test. In the visual depictions that follow, grey icons indicate the visitors to the page/ pages where the test is running. For convenience, let us assume that there is only 1 visit by each visitor. The purple icons indicate the visitors who used the masthead. Visitors who used the masthead and placed an order are shown in green. Again, to keep this illustration simple, assume that the visitors, who do not use the masthead, do not place an order.

In scenario 1, the usage of masthead is higher in the alternate layout (5 out of 10 visitors for the tested variation vs 4 out of 10 visitors for the current state) but the conversion rate for users of the masthead is much lower (1 order from 5 masthead users in the tested variation vs 2 orders from 4 masthead users in the current state). While assessing the test results, some might say that the masthead usage has increased and hence the tested variation is a winner. Some might argue that the conversion of masthead users is lower and hence the tested variation is not a winner.

Both these views are "Myopic"or short-sighted. The key comparison will be what percentage of visitors arriving at the tested page/ pages eventually placed an order. I use the term "Effective Conversion"  to represent this. Effective conversion is the number of orders facilitated by the element being tested (masthead in this case) as a percentage of visitors that qualify for the experience where the variation is active. This quantifies the role played by the element in translating a visit into a revenue generating one.

Looking at Effective Conversion also helps us overcome, what I call,  "Convenience Segmentation" approach of looking at the performance. I define Convenience Segmentation as the approach of focusing on the performance of only a smaller segment of visitors and concluding that any improvement in the performance of this segment is good enough to say that the overall experience is more effective (even if there is no visible improvement in success metrics at a broader level).

Let us now look back at scenario-1. We get the following values for "Effective Conversion".

Effective Conversion at the bottom of the table clearly shows that the proposed variation for the masthead is less effective in driving visitors to conclude a purchase.

Now, let's visit scenario 2. Here, assume that the usage of masthead is lower as compared to the current state but the conversion rate for the users of masthead is higher.

Again, if we look at engagement with the masthead alone (3 out of 10 visitors for the tested variation vs 4 out of 10 visitors for the current state), we will conclude that the change is ineffective while focusing on the conversion rate for masthead users  (2 orders from 3 masthead users in the tested variation vs 2 orders from 4 masthead users in the current state) will lead us to believe that the change works well.

Here again, if we do not want to be myopic, what we need to look at is the effective conversion for the masthead.

Effective conversion for both the current state and the proposed variation is the same, which tells us that the masthead is effective in driving conversion rate to the same extent in both the cases.

Although a fairly simplified illustration, it still highlights the importance of looking at the larger picture while interpreting results of A/B tests. An ideal approach to optimize an element will be to work towards both increased engagement as well as increased efficacy of the element in driving conversion rate higher and we need to look at "Effective Conversion" to make an appropriate conclusion.