In the wake of Game Of Thrones season 4's first episode yesterday, we've seen the inevitable follow-up statistics claiming that Australia downloads more pirate copies of Game Of Thrones than any other country in the world. And as usual, these claims are deeply suspect and essentially unjustified.
Torrent news site TorrentFreak estimates that more than a million copies of the episode were downloaded on the day of its debut (which was Monday Australian time, Sunday in the US). It based that claim on examining torrents that were being openly shared, which probably means it's an under-estimate (since encrypted and private torrents wouldn't be included). But while that figure is rubbery, the figures about which countries are downloading are even more questionable.
Here's how TorrentFreak explained its methodology:
In addition to the downloads, we also looked at the countries people were sharing from. A sample of 18,333 IP-addresses collected over the day shows that Australia takes the crown with 11.6% of the total. The United States is a good second with 9.3%, followed by the United Kingdom with 5.8%.
That sample of 18,333 addresses represents less than 2 per cent of the claimed total downloads. We don't know how the addresses were selected. More importantly, we don't know what time they were selected. As we noted earlier in the year when examining similar statistics about the Breaking Bad finale, Australia would be over-represented if you checked statistics between 6pm and 7pm our time, as people returning home from work began their downloads for an evening's viewing. By that point, many people in the US would have finished their downloads already.
All of the other points we raised in that article apply here too. In particular, identifying country of origin by IP address isn't perfect, so making absolute claims is difficult.
Here's the main thing: the claim that Australia tops the world for Game Of Thrones piracy is ultimately based on the behaviour of just 2103 people (11.6 per cent of the 18,133 sampled addresses). I'd actually be amazed if the number of Australians who pirated the show was anywhere near as low as that — but that's the figure on which the claim is based. Statistically, it's nonsense.
Yes, conventional TV ratings are based on small sample sizes too. But those groups are selected to be representative of broader demographic trends, and the methodology used is public. Neither is true in TorrentFreak's case. We have no idea how those addresses were selected or what period of time they were collected over. There's simple not enough evidence for the claim about Australian piracy to stand up.
A final thought. The claimed million downloads is only one-sixth of the 6 million people who actually watched the show in the US on HBO. Piracy is clearly a problem, but it still doesn't represent the majority of the audience for the show.
Update: After this post was published, TorrentFreak added a note to its article saying "the IP-address sample was collected during the first 12 hours, which means that there’s a geographical bias" — the point we made in this and the previous article.
Also for the commenters: when I wrote "Statistically, it's nonsense" I was referring to the overall flaws with the study and its methodology, not just the sample size. As the very next sentence notes, small sizes can work but the way you select the sample is also important — and that's something that's still largely opaque here. It's simply not a set of numbers you can treat with any confidence.