Posted by Ray Poynter, 21 June 2019
To market research and insight professionals this may sound like a silly question, the received wisdom is that surveys produce quantitative information. However, over the last couple of years I have, on several occasions, heard data scientists refer to the results from 1000 surveys (or even in one case, 10,000 surveys) as ‘qualitative’. My initial assumption was that these data scientists simply did not understand the principle of sampling, i.e. that a sample of 1000 people, drawn randomly from a normal distribution, will produce estimates of the population mean that, 95% of the time, are within plus or minus 3% of the true value. Because of this lack of appreciation of the science of sampling, the data scientists were failing to see that samples of 1000 surveys were more like their data set of 10 million than they were like 16 depth interviews.
However, more recently, I have started to wonder whether there is another way of addressing this issue. Previously, I used the definition of qual and quant common to social and market researchers. Broadly this relates to qual being the creation of descriptions of the world by applying a subjective assessment of observed phenomena, and quant being the application of algorithms to discover/measure things.
However, there are other ways of looking at qualitative. For example, qual is about describing, it is about assessing the nature of things, it is about judging what matters. Let’s think about a U&A survey for a supermarket and compare it with the big data / data science approach. The big data approach measures things that require little interpretation, for example what was purchased, how was it paid for, what things were purchased in combination. This data can be processed to find a wide range of things, such as what happens when prices and choices change, or whether somebody’s home circumstances have changed (e.g. somebody has moved in, or their income has gone down, or they have become vegetarian).
If and when we market researchers use a U&A, we tend not to focus on what people do (although that is part of what we do), the focus is how they feel about what they do, why do they do what they do. If we take a question like “How likely are you recommend this supermarket to friends and family on a scale 0-10?” are we really measuring the sort of thing that a data scientist would measure? People do not actually have a number in their mind in the range 0-10. The shopper has a set of emotional relationships to the supermarket that are in flux, they are engaging in a qualitative exercise when they give us a number – they are constructing the number to represent how they feel. Not only is this number not ‘real’ in the sense that ‘how many tins of beans they bought’ is real, it is not stable. If we ask people the same question again it can and does changes – in the sense of the data scientists it is not a characteristic of the person – in the way that their bank record is a characteristic.
It could therefore be argued that a large part of social and market research is focused on understanding qualitative topics, for example what do people think, why do they think it, how strongly do they believe it etc. Although the tools used, from averages to structural equation modelling are quantitative tools, the essence of what is being researched is qualitative. Perhaps this is part of the language gap between Data Science and Market Research? Market Researchers might be using the terms qual and quant to talk about how the measurements are conducted. But data scientists might be focusing more on the nature of what is being measured?
As a simplification of the argument:
- In market research qual the participant and the researcher make a qualitative assessment about something.
- In market research quant the participant makes a qualitative assessment about something.
- In data science quant nobody makes a qualitative assessment about the action being measured.
Thoughts?