Comment & Opinion

KPIs are important but don't fall into the Ludic Fallacy

Solve the problem not the symptom says Luke Dicken

KPIs are important but don't fall into the Ludic Fallacy

Analytics are one of a development team's greatest weapons. What do you need to know about your game?

There's a really good chance that if you're logging the right things, you can find out absolutely anything, and with a solid platform behind you and a sufficiently large player base, you can A/B(/C/D...) test to streamline your operation post launch.

We can't talk about analytics without bringing up Key Performance Indicators - KPIs. KPIs give you a sense of control and at least awareness over your current situation.

Is your retention where you need it to be? What's you CPI like? How about your ARPU? How does that compare to your ARPPU?!

And of course the big question - is your LTV high enough to continue to employ this many people? It's enough to make your head spin.

The Ludic Fallacy

In his seminal book The Black Swan, Nassim Nicholas Taleb introduced a concept that he called “The Ludic Fallacy”, which broadly states that when we reduce a problem down to a mathematical abstraction, we can expend an awful lot of time trying to “solve” that system, or put another way to win the “game” we have constructed for ourselves.

We become blinkered by this and forget that this is just an abstraction, so winning doesn’t necessarily translate back.

Trying to solve your retention problems by hanging on to those players who felt a need to leave is an act of desperation.

Taleb's example for this is a Las Vegas casino, intent on minimising financial losses.

With this as their goal they became obsessed with stamping out cheating at the gaming tables, but even a cursory look at their recent financial history revealed huge losses due to paperwork mistakes, bomb threats, a kidnap ransom payment and so on - their biggest losses were not gaming related.

Why their obsession with gambling then? Because it could be easily modelled, it was well understood, it probabilistically behaved itself - and then the process became tied intrinsically to this model rather than the real goal, and became significantly less useful.

How does knowing this help us as developers? Well lets go back to our analytics and KPIs.

These give us a fantastic way to model players’ behaviour in our games; they can tell us how the game is being used but even then, maybe only after an insightful A/B test. They're incomplete. Consequently, our whole analysis process tell us "what" but not "why”.

That’s a really important distinction.

Analytics in practice

Lets say that you've launched your game, but your retention is low. Conventional wisdom tells you to this is a big problem and some of the largest conferences in the field have recently presented sessions showcasing for example how you can identify people who might leave your game using some classification theory techniques over your data and then... Well let's be honest, that same conventional wisdom tells you to bribe them or to nag them and hope they don't leave.

But this has all the elegance of hitting someone in the face with a sock full of pennies!

Trying to solve your retention problems by hanging on to those players who felt a need to leave is an act of desperation. It's trying to reattach a severed limb with a band aid. These folks didn't enjoy your game! Time and time again I see teams who try to fix the problems their KPIs report by chasing better numbers.

Always always remember, low retention - or any other under-performing KPI - isn’t a problem, it’s a symptom..

This is the Ludic Fallacy at its most insidious because it's making you spin your wheels treating the obvious problems rather than the underlying causes - that your game design didn’t engage these people.

You can try to move the needle a bit by hanging on to them, but if they’re not into it, it’s a temporary solution.

A better solution

So what you can do instead? Using analytics to determine problem points - whether that be retention, conversion, demographic skew or whatever - is really great. But focusing on the analytics is where the problem lies. Step back, look at the big picture.

I heard at GDC Europe about an analytics team that functions as a very tightly knit group, passing their insights around each other, but in a process that doesn’t go anywhere near the game designers.

That’s a problem, and really highlights the issue. Talk to the whole team! If some part of your game isn't firing on all cylinders, maybe another approach to that bit would be better.

Always always remember, low retention - or any other under-performing KPI - isn’t a problem, it’s a symptom.

Luke Dicken is a Game AI specialist, founder of Robot Overlord Gamesand Chair of the IGDA Foundation.

He’s a member of Full Indie UK, a friendly group of UK Indie Game Developers who have teamed up to become friends, help each other out, and make great games. Find more info here.


PocketGamer.biz regularly posts content from a variety of guest writers across the games industry. These encompass a wide range of topics and people from different backgrounds and diversities, sharing their opinion on the hottest trending topics, undiscovered gems and what the future of the business holds.