Tag Archives: questionnaire design

The Nine Circles Of Survey Research Hell

 

When Dante Alighieri was composing the Inferno section of his epic poem, the Divine Comedy, he was surely thinking of online survey content and execution. Okay, maybe he was thinking of something else. Nonetheless, Dante’s visionary landscape of falling into a place where everything around you burns to ruin can apply to various situations. It certainly applies to how shoddy survey research can incinerate your market research.

Let’s keep it heavenly then, by avoiding these survey circle hells.

 

First Circle (Limbo)

 

 

This place (or state of being) is not that bad. It’s full of nice gardens where pagans like Plato, Virgil and Julius Caesar hang out. They never had a chance to convert to Dante’s religion, but get a pass for being notable and thus hang out in blandness for eternity.

Here on earth, that’s the problem when it comes to market research. Nothing happens. You’ve released a survey, and it’s as quiet as a Nickelback internet fan site. Response rates are low. Why is this happening?

How to get out of this hell: There are many explanations, as you will see, found by plunging deeper into the rest of the survey circle hells.

 

Second Circle (Lust)

 

 

In Dante’s poem, the condemned here float continually by the fury of strong winds. It’s an allegory of what happens when one lacks emotional control. Helen of Troy is imprisoned here, and surely Bill Clinton and Don Draper will join her in time.

For researchers: You’ve allowed your passion for the project to pollute the survey. You have a bias, and it’s often unconscious. As one researcher wrote:

One of the leading causes of misleading survey data is researcher bias that comes directly from the survey writer. This bias is sneaky. It’s caused by survey creators who innocently influence the results to reach an outcome they hope or expect to reach. It’s sneaky because survey creators are typically unaware it’s happening.

How to get out of this hell: Allow others to edit the questionnaire; keep questions concise and neutral; offer opt-out questions so other souls don’t drink (and later spew) your Kool-Aid.

 

Third Circle (Gluttony)

 

 

Here the three-headed mutt, Cerberus, makes sure that all gluttons linger in frozen, dirty slush (people in Chicago can relate when commuting in the wintertime).

For researchers: You’ve overstuffed your questionnaire. As our research reveals, respondent fatigue sets in at approximately 20 minutes of a survey. Respondent fatigue may result in respondents exerting less effort and spending less time thinking about their answers.

How to get out of this hell: Keep questionnaires as short as possible; keep your vocabulary simple or at least at the level of your audience; place important questions at the beginning.

 

Fourth Circle (Greed)

 

 

Greedy and hoarders end up stranded in this dimension, forever carrying bags of money they can never enjoy, under the control of Pluto (the Roman god of death, not Mickey’s dog).

For researchers: You’re pressured by a budget and end up skimping on respondent incentives. Bad move.

As we’ve reported, incentives have these benefits:

–  Response rates that are 10-15% higher
–  Improved response rates from hard-to-reach groups
–  Increase efficiency, especially when it comes to non-response follow-ups

How to get out of this hell: Reward your survey participants, bro. With online surveys flooding the market these days, it might not be an option—unless you want to carry around bags of non-filled surveys for Pluto.

 

Fifth Circle (Anger)

 

 

For some reason, Dante envisioned this place being the River Styx with the condemned floating on or under its icy waters. We do find a city called Dis that is about as attractive as Gary, Indiana.

For researchers: You won’t let anyone edit your survey, or you place too many trap questions (which we revealed can harm survey data—by shifting the thinking of respondents to critical thinking from “optimal thinking,” the state of mind they reason as they normally would in daily life). No one better get in the way of your research project, or it’s Khan wrath, baby!

How to get out of this hell: Place fewer trap questions and make sure there is a second or third set of eyes on your questionnaire. Yes, these circles overlap like an infernal slinky.

 

Sixth Circle (Heresy)

 

 

Welcome to an eternal landscape of flaming tombs. You should avoid this place at any cost unless you’re doing a heavy metal video.

For researchers: What is worse a heresy in market research than not checking your data? As examples, you don’t notice that some respondents are finishing surveys in less than 30%-50% of the median time; that others are marking the same line in each question; or that some are even leaving a Christmas Tree pattern in the survey (it does happen!).

How to get out of this hell: Double-check, my friend, like we keep saying, and don’t go at it alone.

 

Seventh Circle (Violence)

 

 

Witness a place where all violent thugs spend time beating each other up, under the governance of a Minotaur, harpies and whatever monster Dante must have found in his ancient copy of World of Warcraft. It’s a cosmic Fight Club!

For researchers:  Your survey research may not be threatening you, but a lack of empathy can be just as perilous. As qSample’s president Rudly Raphael stated:

Companies need to have more empathy for the research participant. The person(s) who writes the survey instrument should ask themselves if they could sit through that survey for 25-30 minutes. Companies should make surveys fun and engaging, regardless of the topic. They should test their surveys over and over again to identify the fatigue points in the survey. This is usually the area where data integrity is compromised.

How to get out of this hell: Did you not hear Rudly through the din of the harpies? Cultivate empathy. Picture yourself taking the survey, perhaps on a smartphone waiting for a bus in the rain. Picture the vocabulary of your audience as well as their habits.

 

Eighth Circle (Fraud)

 

 

Some monster called Geryon goes around in this place harassing all manner of fraudsters. We’re sure that this includes the founders of the startup you invested in back in 2012.

For researchers: It’s still surprising that so many researchers don’t communicate to participants the length of the questionnaire or clearly explain the purpose of the survey. They avoid placing progress bars or timers. They don’t engage the respondents after the survey. It may not be a scam, but it certainly makes survey participants provide lesser data quality.

How to get out of this hell: Like the figures who guided Dante in Inferno, take a journey with your respondents. That in itself lets them know that they’re actually heading to a paradise.

 

Ninth Circle (Treachery)

 

 

This level is the sucky-suck of all places. It’s hot, and you’ve got the company of such fiends as Cain, Mordred, and Judas. Need we say more? Should we include Carrot Top to seal the deal?

For researchers: We don’t have to say anything else, really, pointing to the vibe of the Seventh and Eighth Circle. Your survey ultimately represents your brand or client. You are truly your respondent’s keeper. Anything but utmost professionalism can go beyond compromising survey data and response rates. It may hinder future surveys.

How to get out of this hell: If you’re this low, ignored all the advice from the other circles, then may it’s time switch to selling insurance or swindling people by founding a startup. There is little hope.

 

Conclusion

 

 

At the end of Inferno, Dante crawls through the center of the earth and enters Purgatory. If you avoid these circles in your survey research projects, you won’t have to worry about any purgation. You’ll be already rising through heavens of quality data.

 

Devils Advocate Advertisement

The Five Greatest Survey Questions Ever

Author Thomas Pynchon wrote: “If they can get you asking the wrong questions, they don’t have to worry about answers.”

This quote is socially charged. Nevertheless, in this or any event, market research comes to the rescue. Asking the right question is central to the industry—part of the continued epic of understanding the intimate desires and motivations of various demographics.

qSample has provided research on keeping questionnaires for surveys as concise and straightforward as possible—all to understand the complex behaviors of humans.

Effective questionnaires are especially vital in an era where mobile devices erode both time and space for executing surveys. The mobile ecosystem is just smaller and consumer attentions spans are getting shorter. Mobile surveys are on the rise, so it’s prudent to embrace this new tech normal. As one market researcher put it:

The trend is clear that more and more surveys are being taken on smartphones and tablets. Desktop computers still account for the majority of responses, but for any given project market researchers can now expect up to 40% response on smartphones. This percentage will likely continue to increase in coming years.

So keep it simple, and start with simple. As Leonardo da Vinci once said: “Simplicity is the ultimate sophistication.”

To assist with this approach for suitable marketing questionnaires, thought leaders in marketing have revealed essentially questions that can become the cornerstone for any type of survey (and beyond).

 

1.  What’s happening in your life that brought you here today?

 

This wording is found in noted copywriter Joanna Wiebbe’s book, Where Stellar Messages Come From. You can modify this open-ended question depending on your brand or forum. It may seem odd in a provided online survey, but the phrasing can be edited to inquire at what brought a person to a particular store or website.

The question is most effective in brief pop-up surveys, though. It takes an immediate snapshot of an individual’s state of mind, as well as how it relates to a brand. Information about a social media link or frustration at another product bringing them to your fiefdom can be pure gold.

 

2.  Were you able to complete your task today?

 

This straightforward inquiry also unlocks several dimensions of a respondent or potential customer. As bestselling author and marketing guru, Avinash Kaushik, states in his blog:

An extremely simple question that asks the survey takers to self report their own perception of your website’s effectiveness in helping them complete their tasks….We have the customers voice telling us exactly how well the website is performing when it comes to delivering the goods.

Again, this can be applied to surveys. It’s surveying the survey. Market researchers should always be auditing their own work, and who better than their respondents fresh in the study.

 

3.  Would you recommend us to a friend?

 

This question may sound obvious, yet marketers sometimes miss this elephant in the research room.

In fact, there might not be another better question in marketing. Bestselling loyalty expert Fred Reichheld agrees with this, calling it the ultimate question. He expands this notion fully in his book entitled (surprise) The Ultimate Question.

As with the other questions and repeating myself, the wording and context may be altered to suit the needs of the brand.

Sean D’Souza of Copyblogger offers some variable and follow-ups:

– What was the obstacle that would have prevented you from buying this product?
– What did you find as a result of buying this product?
– What specific feature did you like most about this product?
– What would be three other benefits about this product?
– Would you recommend this product? If so, why?
– Is there anything you’d like to add?

No, I can’t add anything else to the sound advice in this section.

 

4.  Any question that isn’t just yes/no or either/or

 

Not allowing options like “I don’t know” or an open-ended space in survey questionnaires is dangerously close to a leading question. Some experts say it is a leading question.

Inc.’s Jeff Haden is one such example:

Either/or questions, just like leading questions, assume some answer. Instead of sharing options, just state the problem. Then ask “What do you think?” Or “What would you do?” Or “How should we handle this?

Haden further states there is always an undisclosed option, an idea researchers might be missing, and it’s important to leave a space to let it flow out of respondents.

 

5.  The questions you ask yourself before the survey

 

This thought leadership insight comes from qSample’s own president, Rudly Raphael. He states: “In my opinion, there are three questions that a researcher should ask before any study.” These are:

Who is your audience?
What is the objective of your survey/research?
– What are your biases (and what can you do to avoid them from polluting) the survey?

In other words, you have to know yourself as intimately as you expect to know your audience. If you assume you don’t have a bias, according to Raphael, you probably do and it’s time you get some feedback on others in the company or project.

 

Conclusion

 

These five questions (or more like question narratives) incorporate the same the keynotes: empathy, directness and engagement. They tend to spark the best possible data from respondents.

They can also be utilized by sales people.

Obviously, market researchers are required to mine deeper and into other spectrums of the respondent’s mind. As Raphael says:

The best questions for online surveys are challenging because it depends on the topic. Researchers design surveys for a myriad of a myriad of verticals like products, services, intelligence, etc. An IT question about network adapters will not be helpful to someone writing a finance survey.

Nevertheless, these questions serve as cornerstones for surveys or at least tools for that essential simplicity in a mobile era.

They are the right questions to gaining the right answers in your research.

Contact Us

Infographic of the Week: 4 Steps to Writing the Best Online Survey

qSample is known for its online panels. We also are known for consulting with clients throughout a research project, all to ensure the best possible study is executed. And we make sure to share these insights on various channels including social media, articles, and (of course) our weekly infographic. Here we go again.

This infographic originally appeared in our post Empathy for the Devil that is Writing Survey Questionnaires. It’s been rebooted it for your edification. We hope it assists you in your efforts at sterling online research…or at least makes you reach for the nearest Rolling Stones tune in your music device.

4 Steps to writing best survey

 

button white papers

 

Download this infographic.

Embed Our Infographic On Your Site!