Before there was big datasets - there were surveys. Before there was data science - there was market research.
The entire debate as to whether or not machines will replace market researchers is an odd one – nay, it is ridiculous. Machine Learning (ML) is no threat; on the contrary, over the course of 40 years market research has embraced it all.
Pedro Domingos (in The Master Algorithm, 2015 ) described the Five Tribes of ML. Each tribe (or school) draws its inspiration from social or natural science. At the centre of each tribe resides its general-purpose algorithm, which includes a mathematical proof that given enough data, it can learn anything. Collectively they offer different angles at which to attack the same problem. Through these lenses, we can appreciate the extent to which ML informs how we go about unpicking the complexities of consumer behaviour.
They are:
‘Symbologists’
Taking their inspiration from logisticians and philosophers, the symbologists employ inverse deduction. This is a fancy of way of saying ‘work backwards to fill in the gaps’. We devise a set of general rules that best describes our data. Decision trees predict market segments and model the influence and interaction of factors that drive decision-making. What are the triggers and barriers to customer loyalty? Our weapon of choice, the mighty Random Forests.
‘Bayesians’
This is the statistician’s equation – the tribe close to our hearts. This equation calculates the likelihood of something happening, given the chances of you thinking it might happen before you see any evidence. As the evidence mounts, you update your opinion. Based upon this real world approach to probabilistic inference, Markov chains hide behind Natural Language Processing, Hierarchical Bayes Regression behind conjoint. Herein lies almost everything we do in the fields of brand equity and consumer choice prediction. What will happen if I add a new feature and up the price tag? We have all to thank the Bayesians.
‘Connectionists’
The Backpropagation algorithm emulates neuroscience as a mathematical expression of neurons and synapses. We train layer upon layer to form the ultimate prediction machine – the very versatile Artificial Neural Network. Traditionally used in segmentation and self-organising maps, a personal favourite. New applications of ‘deep learning’ emerge on a monthly basis: image recognition and video transcription have the potential to unlock new data sources and possibilities.
‘Evolutionaries’
Biology. Here genetic programming imitates evolution. Starting with random answers to a problem, better answers are ‘bred’, their attributes combined. Generations of solutions (think racehorses) are created and appraised, then combined again, until we discover the most effective. This lends itself to the simulation of design. Knowing what our customers covet and avoid, we breed into existence the best possible combinations, which adapt effortlessly to real world cost/price/competitive constraints. We use genetic algorithms for product, service, brand and messaging optimisation.
‘Analogisers’
Favouring analogical reasoning (psychology), its algorithm, the Support Vectors Machine (SVM), studies similarity. What makes one group different from the next? How do I decide? Frontiers drawn between one group and another, shred through most pattern-recognition problems. This lends itself to direct marketing and is a very useful technique for time series forecasting. What can I do to up my response rate?
Quantitative market research is frequentist statistics. That is, our own general-purpose algorithm is ‘counting stuff’. We sample and estimate, but this is not all we do – it is just where we begin. Those who spell our eventual demise at the hands of ML do not realise that, as an industry, we are among the most widely rehearsed – tech stragglers we are not. Rather, as each ML tribe matures, market research will thrive for we are already, and shall continue to be, a welcoming home to the Five Tribes.
