What is Response Bias in Online Studies? (and how to slay it)

Business man tempted by angel and devil on his shoulder, signifying response bias

You’ve got it all ready: vetted panel, crafted questionnaire, and an even enterprise software platform from a trusted provider. You’re ready for that data that will forward your market research to blissful success. Everything should be fine, right?

Wrong. A lot can go wrong.

One is response bias, or respondent bias. It’s a major issue in any survey methodology.

The other issue is respondent fatigue—addressed fully in our article Empathy for the Devil that is Writing Survey Questionnaires, which also addresses how to remove this other potential adversity for methods of data collection.

But back to the first issue. What exactly is response bias?

In Personality and Individual Differences, Adrian Furnham states:

Response bias is a general term for a wide range of cognitive biases that influence the responses of participants away from an accurate or truthful response. These biases are most prevalent in the types of studies and research that involve participant self-report, such as structured interviews or surveys.

The Encyclopedia of Survey Research Methods further explains:

Response bias is a general term that refers to conditions or factors that take place during the process of responding to surveys, affecting the way responses are provided. Such circumstances lead to a nonrandom deviation of the answers from their true value. Because this deviation takes on average the same direction among respondents, it creates a systematic error of the measure, or bias. The effect is analogous to that of collecting height data with a ruler that consistently adds (or subtracts) an inch to the observed units. The final outcome is an overestimation (or underestimation) of the true population parameter.

In brief, response bias is the reality that participants bring a lot baggage to surveys (and attempt to hide a lot of this baggage by playing it safe alongside the proverbial pack). Furthermore, respondents have the innate desire to please studies they’re participating in, and therefore have the tendency in answering questions as the researcher might want instead of answering honestly.

Even briefer, humans are humans: complex in public as they are in private.

Yet there are research techniques that can decrease response bias. Some of these procedures paradoxically mean being more human and less scientific!


Don’t Lead Your Respondent



In the law arena, it’s akin to the prosecutor loaded question: “Where were you on the night you murdered your wife?”

In a subconscious way, researchers frequently do the same. It’s also referred to as inherent bias. As one researcher wrote:

For example, a satisfaction survey may ask the respondent to indicate where she is satisfied, dissatisfied, or very dissatisfied. By giving the respondent one response option to express satisfaction and two response options to express dissatisfaction, this survey question is biased toward getting a dissatisfied response.

The answer to this is ensure that your questions are balanced. In addition, verify every questionnaire with other colleagues to always ensure no personal partiality contaminates the study.


Give Them a Way Out



As mentioned, respondents have a natural desire to assist the studies they’re involved in. They also naturally will put pressure on themselves to offer the best possible answers. This pressure may (naturally AND mathematically) create skewed answers. Alleviating this pressure will ensure more honest responses.

An op-out choice is an effective manner way to relive any pressure. As one authority on surveys explained:

That is why it is imperative that every question has an opt-out choice. This is usually in the form of a “Don’t Know,” “Not Sure” or “Undecided.” Not only will adding the opt-out choice eliminate a lot of inaccurate answers from your study, but it will also provide you with valuable information.

The same source states that a notion called “social desirability” is potentially present in surveys—and that is the resistance of respondents to answer sensitive questions due to an intrinsic fear of being exposed to society. The solution is to stress the anonymity of the study beforehand, as well as reminders throughout the survey in the form of text reminders on the screen before a section (as an example).


Offer Questions in a Dynamic Manner



When respondents are presented with a steady pattern of inquiry, they typically answer based on the previous question or subject arrangement. (This can likewise cause respondent fatigue.)

Researcher Sam Mcfarland found that: “When you start with a closed question, you may affect how the respondent will answer a subsequent open-ended question on the same topic because the earlier question has primed them to focus on that issue.”

A White Paper dealing with response bias in surveys, issued by the University of Jerusalem, further stated solutions:

Other ways of circumventing or revealing response bias could be, for example, presenting items on separate screens when using computerized versions, instead of presenting all items simultaneously on one page. The same can be done with traditional PP questionnaires, though the procedure is much more cumbersome. Moreover, the use of a computerized questionnaire enables simple manipulation of the visual presentation of both items and scales. For instance, the computerized questionnaire can present a small number of items simultaneously. Using such techniques might reduce response bias by hindering participants’ attempts to rely on answers to previous items, or on the visual pattern of their answers that is visible when using PP questionnaires. This predicted reduction in response bias is expected to result in lower measures of internal consistency for the computerized versions of questionnaire.

To wit, keep it exciting, mix it up and scatter the topics in the questionnaire.

Most experts on survey procedures, including most of the ones quoted in this piece, agree that these additional techniques can go a long way in mitigating response bias:

–  Use clear and simple language in market research questions.
–  Do not used loaded/lightning rod terms or words unless necessary (e.g.: environmentalist, terrorist, politician, etc.).
–  Avoid negatives like “not,” or at least highlight them so the respondent understands their context.
–  Be as transparent and communicative with the panel throughout the process.

With all if this in mind, response bias can be alleviated to much less than a malady. Just as positive, humans can be humans while research data can become divine for market research.



Back to Blog

Latest insights and trends on market research and surveys

Facebook IconYouTube IconTwitter IconVisit Our LinkedInVisit Our LinkedInVisit Our LinkedIn