Big Data Is Scientific, And That’s Why Most Businesses Fail At It

Big Data Is Scientific, And That’s Why Most Businesses Fail At It

Much of big data comes from people. Web logs, mobile phone usage, financial transactions, insurance claims, you name it: it’s being recorded for potential further analysis to generate business value and improved customer experience.

Lab picture from Shutterstock

It goes by the name of customer analytics, and large retailers and service providers, at least in the US, are obsessed with it.

Online businesses are significantly ahead of traditional bricks and mortar businesses when it comes to leveraging data to drive business value. The major reasons are cultural, social and operational.

These online businesses are much closer to a truly scientific culture in which every idea or proposition is automatically considered a hypothesis subject to testing rather than a heavenly insight for which the burden of evidence can be waived. They not only have an obsession with measurement, but also with experimentation.

A scientific approach to business

The design of experiments, data collection, analysis and understanding are what characterise scientific enterprise. So, in order to embrace big data, it’s necessary to embrace science, meaning its values, culture and new methods based on machine learning, which is the automation of hypothesis generation (from data) and testing (against data).

Yet, a scientific culture is not what you will find in a typical bricks and mortar business.

The top online companies have researchers and scientists who seriously understand science and its new machine learning method. Google recently hired Geoffrey Hinton, the father of neural networks and deep learning (instances of machine learning). It has also just reportedly acquired, for US$500 million, a startup comprised of deep learning experts.

Facebook followed suit by catching Yan LeCun, who pioneered the use of neural networks to solve large-scale real-world problems.

Extracting value from data requires not only the right tools but also the right leaders to build the right teams to use these tools (and build the ones that still don’t exist).

Bricks and mortar businesses in general do not have such people on board, although those who are ahead are desperately trying to hire them. The bad news is that the demand for those people is way, way beyond the supply.

Another crucial point is that those giant online properties have an operational model in which the results of the science can make their way into every decision that results in some intervention, with relatively small cost. This is in contrast to traditional businesses that are burdened with a range of channels each with legacy IT systems and human processes.

Data analysis is itself innocuous unless it drives some form of action. Internet companies have mastered this trade through computational advertising. The causal business effect of interventions such as displaying an ad in a webpage is quantified precisely by how much an advertiser has bid for having the ad displayed or clicked on.

The user’s feedback (in general through clicking or not) is then automatically sent back to a machine learning algorithm that learns how profitable that ad is (per customer). The loop is then closed. The system that determines the intervention allocation policy monitors the business outcomes of every intervention and from that updates the policy automatically so as to maximise the business value of future allocations.

The offline world

What to say of existing bricks and mortar businesses in this regard? Josh Wills, director of data science at Cloudera, a leading big data solutions provider for enterprise, claims no one is doing this automated closed-loop revenue generation mechanism apart from the giant online properties.

Maybe he is right, maybe not. But even if there are others doing this, there is certainly a long way to go. Granted, there are existing data-driven policies for marketing, credit scoring, pricing and other activities in big service providers like banks, telecoms and insurance companies.

But even in the US such large corporations suffer with the operational issues of legacy systems, as well as cultural and technological silos that simply make it too hard to integrate data science and intervention policy in a closed loop across a variety of business areas.

So, what’s the solution? I don’t think there is any silver bullet. The best bet I would place is simply to follow what has worked for online businesses: work as fast as possible on acquiring the right culture, people and operations model. In the US and Europe, some large retailers and service providers have been moving fast.

Walmart has had for years a large team dedicated to data science to leverage the historical purchase data to better tailor offers to its customers. Retailer Target made headlines two years ago when New York Times reporter Charles Duhigg brought to the public’s attention the now famous incident of one of Target’s analytics models predicting a teenager’s pregnancy before her father did.

One of the world’s largest mobile carriers, Telefonica from Spain, has several years ago established a scientific research group in machine learning.

Although Australian companies are in general significantly behind, in the past two years a few large corporations have started to make moves on the people side by succeeding in hiring data scientists. A notable domestic event was Woolworths recently acquiring a 50 per cent stake in data analytics company Quantium.

Whether such and other large retailers and service providers will go a step beyond by realising a cultural and operational shift is also required remains to be seen.

Tiberio Caetano is Principal Researcher, Machine Learning Research Group at NICTA. He is affiliated with NICTA and its subsidiary, Ambiata Pty Ltd, as well as with the Australian National University. His role at Ambiata focusses on growing data-rich businesses through use of large-scale machine learning systems.

The ConversationThis article was originally published on The Conversation. Read the original article.


  • To me, this sounds more like “mechanistic” than “scientific”. I’d be hesitant to align the entire business – and particularly the people in the business – around data science, as most people don’t work or think that way. The idea of approaching business operations by employing a “truly scientific culture” is not new – scientific management as an approach is over 100 years old, and was largely disregarded by the 1960s (though many modern business management principles and techniques evolved from it). That’s not to say there aren’t potential benefits from applying more rigour – I’m definitely in favour of enterprises employing approaches drawn from experimentation – but there are pitfalls. The rules and processes around scientific approaches to business can often act to stifle innovation, for example. It also narrows you’re cultural fit and diversity levels – and consequently, introduce pools of group-think – because it reduces your talent pool to a very specific type of person. So you’d definitely have to compromise some of the data science with some social science.

    At this stage, successfully leveraging big data seems to be more about scale than approach. A scientific approach to it works at very large scales because you’re dealing with very, very large aggregates, so that the volume of people and data you’re are dealing with average out the human factors to the point where they don’t have to be considered all that much – basically removing the “social” from the “science”. Trying to push that down to smaller enterprises and businesses – and particularly bricks and mortar establishments – would be tricky, because the environment is more volatile and the organisations more susceptible to change. I imagine, though, that if you’re a consultant seeking business, marketing an ability to utilise some large data sets to help small businesses better target their efforts would be a lucrative line of work. Not saying this is the case here, but certainly if you want to make a lot of money fast, you could definitely pick a worse approach.

Comments are closed.

Log in to comment on this story!