There’s a proven approach to boosting game metrics: A/B testing and data-driven decisions. And while they can be very beneficial, along with things like alerts and automatisation, there's a catch - they're not the be-all and end-all in every situation.

Sometimes, especially post-launch, you need to push the pedal to the metal: roll out new features, update rapidly, and see the results unfold without getting bogged down in excessive testing.

In this post Azur Games CRO, Maxim Kozhnov shares his insight on how analysts can optimise their workflow at various stages, ensuring they hit the sweet spot of productivity without overdoing it.

---

Are tests always a must?

A/B tests take up significant time, require heaps of data, and involve the whole team, so it’s smart to weigh their necessity.

Take the soft launch phase, for instance. This stage opens up your game to a market segment to gather initial, real-world metrics. In a way, it’s the continuation of the technical launch, but it's more than just checking server pings or ensuring smooth matchmaking in PvP. It's also when you start zeroing in on vital metrics like early retention rates. Together with your team, you should prioritise which metrics are key and which can take a backseat. If R1 is your goal, focus on that, even at the expense of, say, short-term monetisation. The aim is to implement crucial product changes swiftly to increase player engagement, with monetisation tweaks coming later.

Therefore, until your game hits the global stage, investing heavily in A/B testing might not be the most resourceful approach. Initially, you're looking to iron out technical kinks, capture the first set of metrics, and prep for a global release where substantial user data for analysis will be available. However, this hinges on your ability to attract users during the soft launch. If you’re flush with data at this stage, then A/B testing becomes a valuable tool for making informed and confident decisions.

Post-global release: Are tests the go-to strategy?

Even post-global release, A/B tests aren't necessarily your first port of call. In fact, they can sometimes be a drag on development speed. Picture this: your game’s R1 is at a dismal 10% (hypothetically speaking, to grasp the gravity of the situation). Now, you’re at a crossroads – either drop the project or swiftly experiment with game-changing hypotheses. A/B testing becomes truly beneficial when your game is already stable and profitable. That’s when you're wary of losing revenue through missteps and have ample data to work with. Then, A/B testing shines – minor tweaks can boost monetisation or retention by 5%. But let's be honest: a 5% improvement won’t save a project that needs a 30% lift just to stay afloat.

In a freshly launched game, where monetisation is still picking up and the future's hazy, minor adjustments won’t cut it. If things don’t go according to plan, you need bold, immediate changes, and there’s neither the time nor the justification for A/B testing in these scenarios.

So, how do you make decisions when data alone isn’t enough? The answer lies in two things: internal expertise and astute observation. Plus, there are proven solutions that don't need pre-testing. Take a shooter game, for example. It’s pretty clear that including a battle pass feature is a no-brainer. It’s a tried and tested feature that just works – so integrate it and keep moving forward. Rapid updates are key. If these changes substantially boost game retention, that's fantastic – and you’ll notice it even without A/B tests. If, however, updates don't make a significant difference, A/B tests likely wouldn't have been the magic bullet anyway.

Distractions for analysts (and others too)

Dashboards: Daily essentials for analysts, but they can turn chaotic. It sounds like a no-brainer, but keep only what you actually use on dashboards. How do we know what’s essential? Simple – if you haven't glanced at certain charts in months, you probably don’t need them.

I remember coming across a dashboard cluttered with a graph tracking grenade throws per battle. Interesting? Maybe. Essential for constant monitoring? Unlikely. For specific tweaks, a one-off calculation suffices. Analysts might keep these one-time analyses handy on their personal laptops or separate dashboards for easy reference, but these aren’t daily must-sees.

I stick to four main dashboards:

1. Product and monetisation metrics like Retention, ARPU, Payers Conversion, etc.
2. In-game economy metrics, tracking general player earnings and in-game currency expenditure.
3. Technical metrics, ideally broken down by servers.
4. Gameplay metrics, like average battles per player per day, session counts, and playtime.

Sometimes, I'll add a fifth one – a daily overview pulling key stats from the others.

Alerts: Here’s the thing: too many alerts, and they lose their impact. They should be meaningful, not noise.

For us, alerts aren’t for analysts alone. Other departments benefit, like monetisation, where alerts flag dips or spikes in network revenue. The UA team gets alerts for campaign spending fluctuations and abrupt shifts in ROI.

We don't categorise alerts into 'important' and 'less important'. We aim for every alert to be critical. If it pops up, it demands immediate attention.Excessive alerts lead to overload, and soon, you're just tuning them out, especially after being distracted by trivial ones repeatedly.

This segues into my next point.

Is automation always the answer?

Here's a thought: if a task takes just five to ten minutes daily, automating it might seem pointless, right? After all, setting up the automation could eat up days, even weeks. But there's a real value in automating those everyday tasks. It's not just about saving time. When you do something repeatedly, your attention might wander, and you risk missing crucial details. Automation and alerts aren't just for efficiency; they're there to catch what might slip past you.

On the flip side, if a task takes two hours a month, but automating it requires a 40-hour commitment, it's worth pondering – is this automation necessary? Is it worth taking a week off your regular work to automate something you don't do that frequently? Plus, tasks that aren’t daily tend to be less prone to human error due to the "blurred eye" effect.

Take our company, for example. We compile spreadsheets for project spends, financial reports, invoices, and then collate them. We considered automating this, but the data comes in varied formats, even PDFs. The math didn't add up – the automation didn’t justify the cost. Keep in mind that automation snags can sidetrack a specialist from more pressing tasks.

But marketing bids? That's a different story. Automating the daily grind of analysing ad campaign performance, making adjustments, and ensuring ROI has cut down on manual labour for our UA managers significantly. Now they have more bandwidth for new channels and campaign types. In this instance, automation pays off.

However, remember that process automation typically benefits large companies with steady revenue streams. If you're a smaller outfit, consider whether automated procurement is truly essential or whether your resources can be better allocated elsewhere. Maybe there's more to learn in procurement before you set those automated wheels in motion.

So, the game plan is: build your product, grow the business, and then think about automating processes. You can't keep adding to your team indefinitely. In my view, companies don't become successful just because they've automated everything. They automate because they're successful.

The human element

Game designers and producers frequently approach us analysts with various tasks that usually become our priority. These pros have a keen sense of what the game needs at any given moment. Over time, analysts also hone their expertise in this realm. So, before diving into any sudden analytical request, it’s crucial to understand the ‘why’ behind it. You might spend hours crunching numbers only to realise the insights you've gathered don't quite hit the mark. A senior-level analyst will ask about the specific problem or goal at hand. With experience, an analyst becomes adept at recommending the most effective data-driven solution, whether that's a deep dive into certain metrics or comprehensive research.

The endgame for analysts isn’t just to cut down on tasks. It’s about being more impactful, generating insights of your own accord, and doing research that can elevate a project's metrics.

Brief summary

Here's a quick rundown on how to avoid overdoing it in project management:

• A/B testing might not be the best use of time for a young project not yet generating revenue. Align with your team on a step-by-step plan and focus on key metrics. The goal is to move swiftly towards a global launch.
• When a critical metric is alarmingly low, opt for swift, significant action rather than waiting for A/B test data. Roll out updates one after another, aiming to significantly improve the struggling metric.
• A/B testing becomes truly valuable when the project is stable and profitable. At this stage, even small tweaks based on test results can significantly enhance retention and ensure the project's longevity.
• Keep your dashboards streamlined. Stick to the 3-4 most crucial ones for daily monitoring.
• Limit alerts to only the most critical events and for monitoring automated processes.
• Automate routine daily tasks. If a task only consumes a couple of hours each month, manual handling might be more efficient. If you’re still a small-scale operation, think twice before investing heavily in automation. Sometimes, the resource is better spent elsewhere.

Edited by Paige Cook