When A/B Testing Isn’t Enough: Introducing Control Groups
We already discussed here at App Growth Experts the importance of selecting the best metrics for each objective and A/B Testing to validate the results. But, sometimes, it’s not enough to be sure we are on the right track.
The definition of good metrics instead of vanity metrics and utilization of A/B Testing is a good starting point. However, are we able to define the real results of a Push Notification strategy (not only one message, but the whole communication strategy)?
We can start the discussion with the scenario presented in my previous article:
It’s clear that Variation A is driving more results and even more revenue. It’s a good story, and certainly will be very good to use those numbers in a presentation…
Both messages were sent on the same time frame, Friday night, the day part with the best Conversion Rate. The question that arises is: was the revenue from those Push Notifications really incremental? And, if those messages were not sent anymore, will it cause some impact on our business?
This kind of question cannot be answered without Control Groups.
What to expect when we use Control Groups?
Sometimes we face problems like launching a new strategy to communicate with Users and don’t see an increase in revenue, but, all our tools are indicating good results on conversion and revenue sides. This should be simple, if we received $10,000 with a message, this is the increase we expect to see in our reports. Then why isn’t this happening?
The answer is simple: because part of this conversion or any other behaviour will happen organically, even without the message.
Part of your Users will remain dormant without incentives to open your app. These users will not make a purchase if you don’t trigger them to open your app and check the new hot deals. Other users are more likely to open your app every Friday night, even if you are not sending push messages to them.
To enhance your model, you should use Control Groups, remaining part of your user base clear, not receiving any kind of impact. These Users will help you to understand the results of variations compared with the “zero” incentive, so you can understand what will happen if you stop sending those messages.
So, let’s recap the first example, now with the information from our Control Group:
Now we have more data to check the result of Variation A, our old winner. We can see that Users who received the message have exactly the same Conversion Rate of users without any message (Control Group). It’s an interesting result in this specific case, because the Open Rate is high at 8% (generating new sessions) but not good enough to increase the Conversion Rate.
Considering the revenue, we can see that Users from Variation A are generating $500 more in revenue than the Control Group. So, considering the revenue aspect, this is the increment we are getting with this message. It’s also interesting to see that Variation B is showing a lower Conversion Rate, and also with $500 below in revenues.
If you have this kind of visibility into the numbers, you can understand (now with the correct numbers) the true impact of Variation A on revenues and also remove or change Variation B, since it’s not driving incremental results…when comparing using a Control Group.
Considering that part of our job is to understand the results and make experiments that drive more benefits for our business, you should consider Control Groups as part of your marketing strategy. Here, we are talking about Push Notifications, but you can do the same for any other types of external triggers, like Email Messages.
Important: If the Control Group is winning, have we to disable all the campaigns?
Be careful about abrupt decisions when working with Control Groups! Predictions about the results are very difficult and error prone. For some types of apps or communication strategies, the results can appear after a longer time period.
It’s also very important to consider the bigger picture. For example, for Push Notifications, the main objective is keeping the User engaged, and keep the app in the User’s memory, avoiding uninstall and increasing the Retention Rate. With this point in mind, you should also observe the impact of your messages on these metrics.
In my current experience, with e-commerce mobile apps, it’s very common to see Push Notifications with a low Click-Through Conversion Rate and Revenues from converters (sometimes lower than Control Groups), even when we have good Open Rates. However, the whole strategy plays an important role on fighting against uninstalls.
One last tip is to include Retention and Uninstall Rates in your book of important metrics and compare variations of A/B Tests against Control Groups. This way, you will be aware of how intrusive or annoying your messages are for your Users and your business.
