The End Of The Line?
Is the online survey dying?
Another day and another research agency in trouble. No doubt my old friends at YouGov will turn things around but I can't help wondering whether we are seeing the beginning of the end of online quantitative research.
Validity
When I joined YouGov, online surveys were in their infancy. I spent a lot of time reassuring clients about the validity of the online approach by doing parallel runs with face-to-face and telephone methods, pointing to our accuracy in predicting elections and even discussing how we correctly predicted that Will Young rather than Gareth Gates would win Pop Idol!
Fast-forward 20 years and I would be very reluctant to give those same assurances. So it's probably just as well I no longer work there!
Why so reluctant? For a start, online surveys are still nowhere near representative. We do clever things with sampling and weighting and even apply margin of error measures (a nonsense in itself) but this can't disguise that samples are simply not representative. And I'm not just talking about those who lack internet access. I'm talking about communities whose voice is historically underrepresented in society. This could be due to their gender, ethnicity, age or simply because they don't believe their views are valued.
Panel quantity not quality
There is also the problem that agencies have tried to achieve a broader reach by growing their panels. But all this has done is fill panels with the same unrepresentative people. Quite literally when you consider that respondents sign up to multiple agencies. The push for size has been at the expense of quality and this this can only lead to poor quality data. It's become a numbers game. The bigger the panel, the more surveys you can sell and the more money you make.
Many survey respondents make things up about themselves and their lifestyle because they are motivated by rewards. These so called 'Walter Mitty' panellists are a huge issue because they make already unreliable data even more unreliable. The industry doesn't talk about these enough.
Where do we go from here?
So am I calling for a return to face-to-face and telephone surveys? No, not at all. These methods are just as problematic.
What needs to happen is a more honest conversation about survey quality and the recognition that they are only ever indicative of what a subset of society think, rather than being a precise measure on which you can base business decisions.
More fundamentally, I think we need to be smarter in what we do with the data we already have. I honestly believe that many research questions can be answered by a deep-dive analysis of information that already exists within a business, especially when combined with publicly available consumer data and time-series economic and financial data.
Techniques such as predictive modelling, using verifiable datasets, can give more robust and reliable insights than would be possible through online surveys.
And then there's AI and who knows where that will take us?
So is the online survey dying? Possibly.
