How To Gauge The Trustworthiness Of Analyst Reports

Scarcely a day passes when I don't receive a report from some analyst or research organisation informing me of how a new product has saved a bunch of companies a massive sum of money, or how a product has been identified as a leader or innovator in their chosen market niche. But can we trust these reports?

We all know the Mark Twain quote - "There are three types of lies; lies, damned lies, and statistics" - but I think the trustworthiness of those reports goes well beyond the statistical analysis. I think we need to be a lot more sceptical of the data we are being flooded with.

When I receive a new report - Iike the Credit Suisse report I looked at when discussing Amazon's impact on local retail - there are a few criteria I use to see if my bullshit meter starts crackling.

I start by looking at who paid for the research and how they are referenced in the report. For example, I recently received a report on workplace collaboration. It had the usual data you'd expect such as how a robust network is important, how the successful deployment of collaboration tools has delivered a measurable - and always substantial - benefit, and some further information about secondary benefits.

But the money shot came from how a particular vendor's solution would deliver those benefits and more. Once I read a little further and looked at some of the fine print, I found the report was commissioned by - you guessed it - the vendor that was reported on favourably.

In contrast, the Credit Suisse report I referenced was not vendor sponsored and presented an independent view that offered balance. It didn't just paint one scenario. It looked at different potential outcomes.

The other key criteria I look at is the quality of the data and analysis. Many surveys pull their respondents from limited pools. While not all vendor sponsored research is bad - Ponemon Institute and Verizon's annual security reports are pretty solid in my view - I find the data is more valuable if it comes from a diverse range of respondents and that the researchers make that clear.

For example, some reports try to extrapolate the views of 200 small businesses that use a particular product as a case for how their product could be used in enterprises. I also look for respondents that represent different sectors and locales and that the report details how their sample was chosen.

You don't need to be a master statistician to make a call on the veracity of a report's data.

Finally, what's the motivation of the analysts? There are some often-quoted reports that are made from data that is provided by sponsors or clients who pay for inclusion to a report.

That doesn't mean the reports aren't useful. But it does mean you need to understand why someone gets ranked highly in a report.

Analyst reports and research are incredibly valuable. And just because a report is sponsored by a vendor doesn't automatically discount its value. But don't accept the conclusions of a report without critically evaluating the source of the data, who sponsored the report and the motivation of the researcher.


Comments

Be the first to comment on this story!

Trending Stories Right Now