Oliver Kern is an advisor at Mobilize.
Generally speaking, knowing what 1,000 players do - or don't do - in your game isn't a sample size that should prompt a massive shift in how you pursue app marketing or map out your gameplay.
But, when it comes to retention, 1,000 players is a sample size you need to understand.
For 1,000 players can predict the success of your game.
The trick is to know the actual number of players you need in order to prove what you need to know about your game.
This is critical in the soft launch phase, where missing a step, or misreading your retention metrics at key points (namely, Day 1, Day 7 and Day 30), could rob your game of success.
But you don't just risk your chance to make it big with your game; you also squander time and budget spending too much to acquire more users when a sample of just 1,000 players can provide you everything you want to know at that stage.
While I know first-hand that some game developers are convinced they "just need to add another few thousand players" in order to gain higher confidence in what they are observing in their game, mounting evidence tells exactly the opposite story.
Do the math
The clearest way of seeing what I mean is to road test our Sample Size Calculator.
The free tool from IQU Mobile/Mobilize lets developers calculate precisely how many users they need to validate the retention and conversion rates they are getting in soft launch.
The response from indies who have taken it out for a spin allow me to expose three big myths about what 1,000 players can really tell you about your app.
- MYTH #1 More is always better
When it comes to proving or disproving what you think you know about your game or its users, the popular belief is that more is always better.
However, there are many scenarios where acquiring more users will only increase your level of confidence, while burning cash in the process.
When you are in soft launch you have to know when to draw the line.
Imagine you are aiming for a confidence level of 90% and a 5% +/- margin error.
If you expect 36% retention, then you'd fall between 31% and 41%.
If your results lie outside of that window, adding another 1,000 players certainly won't significantly change that result.
- MYTH #2 Good things take time
That might be the case for wine, but not for your game.
When you are in soft launch you have to know when to draw the line and that is when you have enough people in your sample to prove what you want to know about your app.
Continuing to spend to get more users without a clear goal is not capital efficient.
- MYTH #3 You need a big marketing budget to understand if your game is any good
More players don't necessarily give you better results. They give you more confidence - but at a price.
So if your Day 1 retention turns out to be only 25% and you spend on 1,000 additional players, allowing a certain margin error, you can bet it's still going to remain in the region of 25%.
The *big* difference is that you are not going to have a confidence level hovering around 90% as it will have shot up to as high as 97%.
The question is: Did you really need to achieve this at such an expense?
More players don't necessarily give you better results. They give you more confidence.
Would it not have been smarter to stop testing, improve the game and then run another test to see if your Day 1 retention has significantly improved?
Step back and it all depends on what you want to do based on bigger numbers and greater certainty.
Let's assume you have an investor that will unlock the second tranche of money if you reach 36.8% conversion rate on Day 1, and you are currently hovering around 35%.
In that case, you obviously want to make sure you are literally on the money, which means that spending on 5,000 additional well targeted players to interact in your game is worth it.
But if you need to reach 36.8% because it's what you think you need and you are in the region of 31%, then don't spend your time (and money) trying to get additional 3,000 users.
Instead, focus on improving your game since it's clear that another build, and not just additional marketing spend, is going to get you to your goal.
How much is enough?
While some may argue you need $100,000 to understand your game and its conversion rate, that is, in my view, the biggest myth of all.
The post postulates that that you need 33,000 downloads at $3 per download. Do the math based on this premise, and it does indeed add up to $100,000 - proving you need to spend a bundle (and acquire lots of users) to have data on 1,000 paying customers (assuming a conversion rate of 3%).
I would advise to break down your effort into multiple steps.
You may end up spending $100,000 but when you are already confident that you have a really valuable game in your hands.
Start with 500 random players and see how they behave. If it's not good enough, stop spending and re-start once you have a new and hopefully improved build.
If you have done that a few times and your retention has not dramatically improved, you have spent a fraction of the mentioned costs to come to the conclusion to kill it.
If your retention rates have improved and are now very competitive, you can go on and understand where your ARPDAU sits and if it's worth improving some low hanging fruits before you go into launch.
What retention tells you
As game publisher Nexon CEO Owen Mahoney reminds us in this post, what really matters (and what we need to measure) is conversion and value.
Mahoney points out the retention rate tells you about the fun potential of your game - and observation that underlines the importance of retention as the metric that speaks volumes (and with high certainty) about the commercial potential of your game.
Soft launch is not the stage where you want to be focusing on LTV.
Read between the lines, and the key indication that you are providing your users value in the early and decisive stages of your app is retention.
Conversion is your focus when you're sure you have a hit game on your hands.
Before you ask yourself if you need more players for your game, ask yourself if you are really at the advanced stage of your game lifecycle when you want to be calculating LTV anyhow.
The soft launch is not the stage where you want to be focusing on this metric - but it is the stage when you need to read a low conversion rate as a red flag.
Don't read it as a sign that you need to spend more money to acquire more or better players in order to have more confidence in your observations.
Take it as strong proof that you need to focus investments on improving your game and making it an app that can attract and retain a loyal audience.
You can find out more about Mobilize's Pilot soft launch testing program here.