Mobile A/B Testing Best Practices

Buse Kanal
September 28, 2022
0 Comments

If you want to stay on top of the fierce competition in app stores, you need to know and apply mobile A/B testing best practices.

Simply trusting your gut feeling or blindly making decisions will only lead to missing great opportunities for app growth. A/B tests will give you the right data to make better changes on your app page.

What is A/B testing?

A/B testing is comparing the performance of two or more alternative versions of one thing and seeing which of them would give you the best results.

In the world of mobile app marketing, you create alternative versions of whatever you want to test on your app listing. It could be your app store screenshots, videos, icons, descriptions, and subtitles, just to name a few.

Just to give you an example, you can A/B test your screenshots. Create two or more versions of your screenshots. From there, equally, divide your target audience by the number of your screenshot versions. You then let each group see specific screenshot versions and determine which version gives the highest conversion or engagement rate.

A:B Testing

Why should you perform A/B testing?

A/B experiments help you make data-driven decisions and choose which versions of your app store listing you should use. An effective A/B experiment can help you identify the best configuration of your app store page and continually improve it.

A/B testing in Play Store and App Store

Now that you know what A/B testing is, it’s time to learn how to do the right A/B test in the best way possible.

There are two major app stores today — Google Play Store and Apple App Store. The goal of A/B testing in the Play Store and App Store is essentially the same, but how to perform them is different.

So, for the purpose of this discussion, let’s take a look at how A/B tests work on each of these app stores.

Google Play Store A/B Testing Best Practices

On the Play Store, A/B testing is referred to as store listing experiments. The main goal of these experiments is to help you determine the best graphics and localized text for your app page. You can perform A/B tests on your main page or the custom store listing pages.

Store Listing Experiments

Image source: Google Play

If you already have a published app, you can create variants of your current app page and see which of these variants converts the most based on your data.

There are two main types of experiments you can do on Google Play Console:

Default graphics – this experiment is used for testing graphics in the default store listing language of your app.
Localized – use this if you want to test your app’s feature graphics, icons, promo videos, screenshots, and descriptions. You can perform tests in up to five languages.

To start with your experiment, you simply need to log into your Play Console account, and go to Grow > Store presence > Store Listing experiments page. From there, select “Create experiment” and follow the prompts.

When doing experiments on Play Store, remember the following best practices:

The icons, screenshots, and videos have the biggest impact on your app store listing. Be sure to test them regularly to obtain the best results.
Don’t test multiple assets simultaneously. This will only complicate things. As much as possible, test one asset at a time to obtain a clear result.
Make sure you run your experiments for at least seven days. This will ensure that you get data not just during weekdays but also on the weekends.
Take advantage of the email notifications of Play Console. Opt in and get alerted as soon as new results are available from your experiments.
A/B tests aren’t a one-time activity. It should be done consistently to see whether you are still using the best assets or not. Remember, seasons, users, locations, and the Play Store’s ecosystem constantly changes.

Apple App Store A/B Testing Best Practices

A/B testing is referred to as Product Page Optimization (PPO) in the Apple App Store. With PPO, you can compare screenshots, app icons, and app previews.

Product Page Optimization

Image source: Apple Developer

The Product Page Optimization feature allows you to create up to three product page versions (also called “treatments”). You can then compare these treatments to your current page version. All data and results go to App Analytics where you can determine which of the versions you have tested resonate the most with your target audience.

You can use PPO to know the following:

Learn which of your value propositions allows you to get more users.
See if highlighting a culturally relevant feature would boost downloads.
Know whether changing your app icon, using different screenshots, or having an alternative motif of your app page listing would result in better installs.
Determine if adding seasonal content would lead to a spike in downloads.

Apple runs A/B tests by showing a particular treatment to a specific set of people. They would see the same treatment across the App Store to ensure you get an accurate result. From there, you can go to the App Analytics and see how each treatment performs compared to your current product page.

To optimize your App Store A/B tests, here are some of the best practices to apply:

Use a descriptive reference name, which can be up to 64 characters. You should write your test name in such a way that you can easily identify what type of test you performed by just looking at the name. Here’s a good example: “Summer 2022 Screenshot order with 3 variations.”
Use even traffic proportion. If you have one treatment and one original product page, you can divide the traffic proportion by 50%.
Determine test duration. Apple gives you an estimated time of how long your test would run. As a rule of thumb, you don’t want to run your test for more than 90 days. If the estimate shows you more than three months, try to change traffic allocation or the number of treatments.
Run your test until you reach 90% confidence in the results.
Limit the number of test treatments. You can use up to three treatments during a test. However, it doesn’t mean you should maximize this number. Remember that the more treatments you use, the longer your test will run. Moreover, using too many treatments can make your data more congested, making it difficult to analyze results.

Perform better A/B tests

Now that you know the best practices in A/B testing, you should already have a better way to create the best app page listing. However, if you want to perform professional A/B testing, you can always get the help of ShyftUp and let their team of experts improve your app performance.

ShyftUp is among the best user acquisition agencies that will help you perform efficient A/B experiments. They have the necessary experience and expertise to deliver the best results.

Give them a call today!