Scarcely a day passes when I don't receive a report from some analyst or research organisation informing me of how a new product has saved a bunch of companies a massive sum of money, or how a product has been identified as a leader or innovator in their chosen market niche. But can we trust these reports?
We all know the Mark Twain quote - "There are three types of lies; lies, damned lies, and statistics" - but I think the trustworthiness of those reports goes well beyond the statistical analysis. I think we need to be a lot more sceptical of the data we are being flooded with.
I start by looking at who paid for the research and how they are referenced in the report. For example, I recently received a report on workplace collaboration. It had the usual data you'd expect such as how a robust network is important, how the successful deployment of collaboration tools has delivered a measurable - and always substantial - benefit, and some further information about secondary benefits.
But the money shot came from how a particular vendor's solution would deliver those benefits and more. Once I read a little further and looked at some of the fine print, I found the report was commissioned by - you guessed it - the vendor that was reported on favourably.
In contrast, the Credit Suisse report I referenced was not vendor sponsored and presented an independent view that offered balance. It didn't just paint one scenario. It looked at different potential outcomes.
The other key criteria I look at is the quality of the data and analysis. Many surveys pull their respondents from limited pools. While not all vendor sponsored research is bad - Ponemon Institute and Verizon's annual security reports are pretty solid in my view - I find the data is more valuable if it comes from a diverse range of respondents and that the researchers make that clear.
For example, some reports try to extrapolate the views of 200 small businesses that use a particular product as a case for how their product could be used in enterprises. I also look for respondents that represent different sectors and locales and that the report details how their sample was chosen.
You don't need to be a master statistician to make a call on the veracity of a report's data.
Finally, what's the motivation of the analysts? There are some often-quoted reports that are made from data that is provided by sponsors or clients who pay for inclusion to a report.
That doesn't mean the reports aren't useful. But it does mean you need to understand why someone gets ranked highly in a report.
Analyst reports and research are incredibly valuable. And just because a report is sponsored by a vendor doesn't automatically discount its value. But don't accept the conclusions of a report without critically evaluating the source of the data, who sponsored the report and the motivation of the researcher.