Much of big data comes from people. Web logs, mobile phone usage, financial transactions, insurance claims, you name it: it's being recorded for potential further analysis to generate business value and improved customer experience.
Lab picture from Shutterstock
It goes by the name of customer analytics, and large retailers and service providers, at least in the US, are obsessed with it.
Online businesses are significantly ahead of traditional bricks and mortar businesses when it comes to leveraging data to drive business value. The major reasons are cultural, social and operational.
These online businesses are much closer to a truly scientific culture in which every idea or proposition is automatically considered a hypothesis subject to testing rather than a heavenly insight for which the burden of evidence can be waived. They not only have an obsession with measurement, but also with experimentation.
A scientific approach to business
The design of experiments, data collection, analysis and understanding are what characterise scientific enterprise. So, in order to embrace big data, it's necessary to embrace science, meaning its values, culture and new methods based on machine learning, which is the automation of hypothesis generation (from data) and testing (against data).
Yet, a scientific culture is not what you will find in a typical bricks and mortar business.
The top online companies have researchers and scientists who seriously understand science and its new machine learning method. Google recently hired Geoffrey Hinton, the father of neural networks and deep learning (instances of machine learning). It has also just reportedly acquired, for US$500 million, a startup comprised of deep learning experts.
Facebook followed suit by catching Yan LeCun, who pioneered the use of neural networks to solve large-scale real-world problems.
Extracting value from data requires not only the right tools but also the right leaders to build the right teams to use these tools (and build the ones that still don't exist).
Bricks and mortar businesses in general do not have such people on board, although those who are ahead are desperately trying to hire them. The bad news is that the demand for those people is way, way beyond the supply.
Another crucial point is that those giant online properties have an operational model in which the results of the science can make their way into every decision that results in some intervention, with relatively small cost. This is in contrast to traditional businesses that are burdened with a range of channels each with legacy IT systems and human processes.
Data analysis is itself innocuous unless it drives some form of action. Internet companies have mastered this trade through computational advertising. The causal business effect of interventions such as displaying an ad in a webpage is quantified precisely by how much an advertiser has bid for having the ad displayed or clicked on.
The user's feedback (in general through clicking or not) is then automatically sent back to a machine learning algorithm that learns how profitable that ad is (per customer). The loop is then closed. The system that determines the intervention allocation policy monitors the business outcomes of every intervention and from that updates the policy automatically so as to maximise the business value of future allocations.
The offline world
What to say of existing bricks and mortar businesses in this regard? Josh Wills, director of data science at Cloudera, a leading big data solutions provider for enterprise, claims no one is doing this automated closed-loop revenue generation mechanism apart from the giant online properties.
Maybe he is right, maybe not. But even if there are others doing this, there is certainly a long way to go. Granted, there are existing data-driven policies for marketing, credit scoring, pricing and other activities in big service providers like banks, telecoms and insurance companies.
But even in the US such large corporations suffer with the operational issues of legacy systems, as well as cultural and technological silos that simply make it too hard to integrate data science and intervention policy in a closed loop across a variety of business areas.
So, what's the solution? I don't think there is any silver bullet. The best bet I would place is simply to follow what has worked for online businesses: work as fast as possible on acquiring the right culture, people and operations model. In the US and Europe, some large retailers and service providers have been moving fast.
Walmart has had for years a large team dedicated to data science to leverage the historical purchase data to better tailor offers to its customers. Retailer Target made headlines two years ago when New York Times reporter Charles Duhigg brought to the public's attention the now famous incident of one of Target's analytics models predicting a teenager's pregnancy before her father did.
One of the world's largest mobile carriers, Telefonica from Spain, has several years ago established a scientific research group in machine learning.
Although Australian companies are in general significantly behind, in the past two years a few large corporations have started to make moves on the people side by succeeding in hiring data scientists. A notable domestic event was Woolworths recently acquiring a 50 per cent stake in data analytics company Quantium.
Whether such and other large retailers and service providers will go a step beyond by realising a cultural and operational shift is also required remains to be seen.
Tiberio Caetano is Principal Researcher, Machine Learning Research Group at NICTA. He is affiliated with NICTA and its subsidiary, Ambiata Pty Ltd, as well as with the Australian National University. His role at Ambiata focusses on growing data-rich businesses through use of large-scale machine learning systems.