From feast to famine.

Following two weeks of hype it's game over for Pokemon GO, apparently.

Not really, but the narrative of data stories about its phenomenal rise has switched to peak Pokemon GO and decline.

First up was Verto Analytics, which depending on what you were measuring reckoned Pokemon GO peaked during the week starting 11 July.

SurveyMonkey Intelligence puts the peak date more precisely as Thursday 14 July, when over 25 million people in the US played the game.

By Wednesday 20 July, that number was down to 23ish million.

Of course, all companies generate their numbers in different ways.

Some like Verto Analytics and Slice Intelligence have millions of US consumers with installed software that tracks what they are doing on their devices.

Clearly only a small subset of the 320 million total US population, at least these companies' estimates are based on hard figures.

Devils and detail

There are some significant caveats though.

Verto and Slice only sample adults over 18, which cuts out a large part of the Pokemon GO-playing audience. Survey Monkey says it scales down to players aged 13 and over (which is likely difficult to do under the COPPA rules).

Also Survey Monkey has only recently been releasing such reports, where Verto and Slice are 100% focused on their consumer surveys.

However, the bigger issue is whether any of these samples are representative of what they are try to measure.

In particular, given they are general samples, typically balanced for mobile shopping not gaming, how well do they capture the behaviour of the Pokemon GO-playing audience?

And how the companies are statistically manipulating their raw numbers to try to bridge that gap.

One example of this was Slice's conclusion that Pokemon GO accounted for 47% of the total US mobile games market by revenue over its launch.

It could be correct, but only if the total mobile game spending Slice was seeing from its 4 million online shoppers matched the overall US market, and that could be heavily skewed if it didn't have enough high level players from games such as Game of War and Mobile Strike.

More likely, it was over-estimating the spend from Pokemon GO players because it didn't have enough high spending mobile RPGs players in its online shopper sample and hence was under-estimating the total daily spend.

Did they predict Brexit?

Of course, this sort of thing is effectively this is what political opinion pollers have been doing - more or less successfully - for years.

Yet as with current issues ranging from Brexit in the UK to the rise of Donald Trump in the US, there's much more volatility in the sample population at the moment.

That makes it very hard to accurately measure rare events because you don't have the same level of historical data, and hence can't create accurate sample groups.

We're also seeing something similar in the case of climate change, as what were previously thought to be once-in-100-year events - high tides, flash rainfall totals etc - are being upgraded to once-in-30-year events.

In some ways, this situation is less problematic for data companies, which don't consider what individuals are doing, but look at app store numbers etc and then try to reverse engineer from known positions.

Yet as App Annie found when the likes of Clash of Clans and Mobile Strike were first lighting up the top grossing charts, knowing the download or revenue totals for games in top 50 isn't really helpful when trying to work out the #1 position.

This is because like the rise of Donald Trump, the #1 grossing total for Clash of Clans had never been seen before so you couldn't extrapolate from historical data.

And it's a similar thing with Pokemon GO.

The only people who know the real figures are Niantic Labs, The Pokemon Company and Nintendo, plus their various tech partners.

Over time, that data will leak out in financial filings or as people change jobs etc and the system will reset its estimates taking into account the latest high water mark.

The fun starts now

As for the wider issues of peak Pokemon GO, in many ways it's irrelevant.

Clearly given the hype surrounding the game's launch, there was always going to be an explosion in players, which would drop off over the first couple of weeks.

This happens with every free-to-play game.

What is much more interesting is what will be Pokemon GO's stable active player base: 15 million isn't impossible during the holidays if the game is updated properly and the weather is good. 

And even 10 million daily players would be a phenomenon.

So when it comes to significant data points from Pokemon GO, we're not even at the end of beginning, let alone the beginning of the end.

Much like Donald Trump, in fact.