Faking it : Can marketers trust surveys?

The online survey is an important tool for the marketer and is routinely used in measuring awareness, evaluating campaigns and for generating some useful PR.  However, the rise of online survey platforms, including DIY, has also brought about significant challenges, particularly concerning the reliability of the data collected.

Survey companies do a good job in selecting their samples and weighting the data to make them look representative. But scratch beneath the surface and all is not what it seems.  One of the biggest but least spoken about issues is the prevalence of "Walter Mitty" panellists - respondents who misrepresent themselves or provide unreliable data, usually because they are chasing incentive payments but sometimes out of malice or to appear more interesting.  These respondents, named after the fictional character known for living in a fantasy world, can severely distort survey results. And because so many people are on multiple panels, the problem is widespread. One investigation found that half of all respondent answers were so poor they couldn't be used.

This issue raises a critical question for marketers: can you truly trust the results of online surveys, especially when these insights are pivotal in guiding marketing strategies?

The impact of unreliable data on marketing decisions

At the heart of marketing lies the need to understand what the customer wants, a task that becomes increasingly complex as consumer behaviours are influenced by a variety of digital channels. Accurate data is crucial for mapping the customer journey, from initial awareness to final purchase. However, if the data collected through online surveys is compromised by respondents who are not who they say they are then marketers may find themselves basing strategies on flawed information. This not only risks derailing the buyer journey but can also lead to ineffective campaign strategies that fail to resonate with the target audience.

The problem is further compounded by the fact that more people are now living on mobile devices, where survey engagement can be fleeting and responses less considered. The ease with which respondents can access surveys on mobile platforms increases the risk of dishonest or insincere answers, particularly when incentives are involved. These factors contribute to the prevalence of false data, making it difficult for marketers to identify the genuine story that will drive decision-making.

Survey data is often used to gauge consumer sentiment, brand perception and the effectiveness of marketing messages. Yet, if survey participants are not providing genuine responses, the resulting data can paint a misleading picture. For example, a survey might suggest that a particular campaign has resonated well with its target audience, prompting further investment in that strategy. However, if the responses were derived from "Walter Mitty" panellists, the campaign's perceived success might be nothing more than an illusion, leading to wasted resources and missed opportunities. Similarly, a survey might indicate that a new logo is well-received, leading to its rollout across all brand materials. If this insight is based on respondents embellishing their situation, the brand may unwittingly weaken its market position, alienating consumers who did not, in fact, respond positively to the change. This scenario underscores the broader issue of making sense of the data—when marketers cannot trust the results, they risk making decisions that could harm their brand.

Steps to ensure data reliability

Given the significant impact that unreliable survey data can have on marketing strategies, it is important to take steps to ensure the reliability of the data they use. One critical measure is the careful selection of survey provider. As part of your selection process asks questions of your provider. Ask how many of their panellists are also on other panels (a good provider will know) and what steps they take to reduce how often multi panellists appear in samples.

In addition, get confirmation of the data validation techniques they use.  Ask how they identify inconsistencies and flag potentially unreliable responses, and how they detect patterns of dishonesty  among respondents. Ask what percentage of responses they discard as unreliable and be wary if this number is either too high or too low. If they can't give you a number, take your business elsewhere because it means they're not taking survey quality seriously.

A good tip for weeding out dishonest respondents at questionnaire stage is to include a false brand on a brand list and delete all responses that select the fake one. There should be no problem getting this automated so that the dishonest respondent is replaced as part of the quota.

Marketers should also consider diversifying data sources. Relying solely on survey data can be risky, particularly if the sample is compromised. Cross referencing survey results with other forms of consumer data—such as ratings and review sites, social media analytics, sales data and website traffic— can provide a more holistic view of consumer behaviour, reducing the impact of any one flawed data set.

Finally, treat online surveys as indicative rather than precise. They will give you insight into how things are but business critical decisions shouldn't be based on survey data alone.