Data & Research

Reading List: The Signal and the Noise

Nate Silver’s classic on prediction versus forecast

Reading List: The Signal and the Noise

Springing to fame on the back of his correct predictions for every state in the 2012 US presidential election - and 31 of 33 Senate seats - Nate Silver has become the poster boy for the power of big data to tell us what the future holds.

Of course, the truth, as Silver himself is at pains to explain in his book The Signal and the Noise, is much more complex than that.

For one thing, presidential politics - and his other love, baseball - are relatively easy problems to solve. There is a lot of historic data to mine and the working of the system are transparent and bounded by fixed rules.

In contrast, when considering topics such as climate change, earthquakes or the US economy, Silver points out that not only is the data very complex - there’s a lot of noise in the signal - we don’t even have a good fundamental understanding of the mechanics that drive the systems.

For this, and other reasons, Silver prefers the term ‘forecast’ to ‘prediction’. A forecast is prediction of different outcomes contextualised within their probability of occurrence.

Predicting the future in terms of certainties is what gets us into trouble.

Deeper magic

Interesting stuff, for sure, but what’s the relevance to the mobile games industry, you might be thinking?

To start with, there are some general conclusions that it would be sensible to consider.

Experts have a strong tendency to overestimate their ability to forecast future outcomes.

Silver finds that experts in all fields have a strong tendency to overestimate their ability to forecast future outcomes. And the more data they have, the more confident they become.

This is particularly the case when people or companies construct models based on past data to work out how they think the future will look. Not only does such activity rely on the accuracy of the models being used, but typically these will be based on an incomplete data set, which is likely to miss out key trends, resulting an exercise of retrofitting an overly complex model to substandard data.

As Silver put it, it’s not clear that in sectors such as economics, earthquake prediction, climate change or stock picking, more data has led to better predictions (or forecasts), although he argues this has been the case in weather forecasting.

Yet, people are becoming increasingly over confident in their analysis purely because their datasets are now so large.

In other words, beware statisticians boasting of their ‘x’ billion data points. Instead, ask them how many good data points they have.

Spotting the sucker

Where the book is more directly relevant to mobile developers and their inhouse quants, however, is Silver’s consideration of stock investment and more colourfully, online poker players.

When it comes to stocks, the argument between the perfect (or efficient) market theorists and those who highlight the importance of behavioural traits is ongoing, but the more important point for Silver is if you think you can do better than the market, you don’t just have to be smart, understand numbers and have a considerable sum to invest (and strong nerves), you also have to out-perform all the other smart, numerate and rich investors, not to mention all the hedge fund companies and the merchants bankers, with their MBAs, NYSE co-located servers, billions of dollars and (perhaps) insider information.

It’s a lesson Silver learnt to his cost as a professional online poker player when market liquidity dried up following a change of US legislation in 2006.

Success in the F2P market is now highly leveraged around the effectiveness of your user acquisition funnel.

As studies show, the winners of the online poker world are funded by the losers (obviously), but significantly, while the losers’ money is spread across many winners, it’s the best players who generate the vast bulk of the winnings.

So, when US legalisation tightened the online poker market, the inexperienced losers stopped playing.

Not only did this mean much less cash in the pot to be won, but also that previously half decent players - and semi-successful players - become the relatively worst players, whose money was taken by the top players.

Relative failure

And this provides a sobering point when transposed to free-to-play games: not something that Silver does himself, I should point out.

This isn’t because of legislative pressure, which may reduce net cash inflows to the market, but because of the conclusion that it’s the less experienced, less innovative and less well funded F2P developers who will be the ‘losers’ in that situation.

But even if you ignore this possibility, success in the F2P market is now highly leveraged around the effectiveness and reach of your user acquisition funnel; that is attracting and retaining a large number of players.

So, in that regard, how well do you think you can compete with the likes of Supercell, GungHo Online, Kabam or King, who are looking to attract exactly the same players as you and likely using the same - if not better - tools and analytics?

Of course, they have a lot more money too.

In this situation, the smartest people (as Silver himself did, having lost over $100,000) quickly realise they are not the world’s best poker players - or have the world’s greatest UA funnel - and cash out to look for better opportunity.

And that might be a sensible move for many companies currently trying to make a living in the F2P mobile gaming space.

The bigger picture

As for Silver, he’s more interested in whether we should be skeptical about climate change (yes about the models, no about the actuality, although to-date warming has been lower than predicted), or the chances of a terrorist attack in the US that kills 10,000 or 100,000 people during the next decade.

More data in-and-of itself will not provide a more accurate forecast.

Yet, even and above his obvious enthusiasm for the subject matter and his skill in explaining diverse subjects, the key to The Signal and the Noise is that in all spheres of life, we should be more thoughtful about the data we increasingly rely on for our decision making.

More data in-and-of itself will not provide a more accurate forecast. That relies on a better understanding of the environment we are engaged in, better data, improved accuracy from the models which we use to simplify highly complex real-world situations, and the need to always be alert to our many mistakes.

After all, discovering these and using them to improve our data and models is the only way we will improve our understanding, even if we never get to the situation where we can perfectly predict the future.

The Signal and the Noise is available from various sites including Amazon US, priced from $15.73 or Amazon UK, priced from £6.99.


Contributing Editor

A Pocket Gamer co-founder, Jon is Contributing Editor at PG.biz which means he acts like a slightly confused uncle who's forgotten where he's left his glasses. As well as letters and cameras, he likes imaginary numbers and legumes.