Getting the best possible survey data is a heroic endeavor for anyone performing online research. It makes for a better market and academic world. Sadly, there are those respondents who impede this adventure, spawned from the nether regions of River Sample™ and causing bad survey data. By viewing these respondents through the lenses of archetypal villains, a researcher’s champion task is not only easier but perhaps even entertaining (even the most stoic of us researchers get blurry-eyed glancing at survey data all day).
Get your Data Superhero tights on, because here are the six Survey Supervillains you need to watch for:
1. The Speeder: The name says it all, and this Quicksilver or Flash-persona blasts through surveys at rates that defy logic (or more like enforce logic, since they’re obviously not paying attention to questions). The Speeder is easily detected because he or she finishes surveys in less than 30%-50% of the median time. For a more technical way to defeat the Speeder, an article on Research Access recommends:
Check the median time to completion and establish rules that you feel comfortable with – I often flag those taking <1/3 of median time with a “1” (“speeder”), and those taking < 1/4 of the median time with a “2” (“super speeder”). You might consider removing outliers (at the slow end) before calculating your median.
Yes, there is a Super Speeder too! No superhero saga is complete without reboots and higher-level bosses.
2. The Flat-Liner: Also known in as the Straight-Liner, the Flat-Liner is the grim reaper of market research, the Thanos of survey methodologies. When reading data, this Survey Supervillain’s death-mark manifests as the same line answered each time. The fiend is just not paying attention or has chosen a specific line in order to quickly complete a survey. The Flat-Liner may take the shape of the Speeder to take you off your superhuman game, but they typically finish at an average rate. One market researcher proposes a solution to defeat (or at least nudge) the dreaded Flat-Line:
For this reason it is important that any multi-numeric list used to measure straight-lining has questions that one would reasonably expect respondents to have varying opinions on (e.g. asking a respondent to agree with both positive and negative statements about the same product).
3. The Lord of Silence: Not communicating in surveys is doom for survey research, but easy to remove. However, claptrap or sloppy answers are just as dangerous yet harder to detect. That’s what the Lord of Silence does. A tell-tale sign this Survey Supervillain is invading your research is time discrepancies—such as claiming he/she was born in 1997 but then saying they have been at their current job for 20 years.
A data research blogger further advises: “Scan through open-end responses (and other-specify responses) for gibberish or excessively vague and short answers.”
4. Bizarro Grinch: Yes, some Survey Supervillains hurt you by leaving loot. Bizarro Grinch will leave a Christmas-Tree pattern in the questionnaire multiple choice section. Often, this nemesis gets cuter and leaves other “artwork” shapes in the survey. Unlike his associate, the Flat-Liner, it’s harder to expose this Survey Supervillain when viewing a pattern on an Excel or other spreadsheet. As one researcher explains:
Christmas-Tree responses are a little more difficult to find, however, if you’ve used numeric reporting values, you should be able to easily spot ascending or descending patterns in Excel and remove these responses.
Take this mission with all of your heart, as this is worse than a pair of Old Navy socks under the tree on December 25th!
5. The Duplicator: Similar responses may appear in a survey, and that could mean this Survey Supervillain is committing a heist in your survey data. He or she is the sibling of the Lord of Silence. On the other hand, doppelganger information can frequently be the infamous maxim of “We have met the enemy and the enemy is us.”
In other words, there is a glitch in the software replicating data, or a respondent is taking the survey a second time. Your Spidey Senses should be clear that this replica data needs to be destroyed.
6. The Revolutionary: Sadly, the Revolutionary is one your own citizens revolting because you didn’t keep the peace as the superhero in your survey neighborhood. Essentially, you didn’t write the best possible questionnaire, and thus unlocked a Zombie Apocalypse of respondent fatigue and respondent bias (Hint: both links grant the kryptonite you need to repeal these uprisings, or at least provide the best survey questions and survey sampling).
As with any responsible champion of good data, your task will be to win them over again.
If you follow these steps (before they autodestruct in five seconds), you will be a Data Superhero. In reality, panel providers these days offer excellent sample. Yet as the Research Access piece stated it’s inevitable that “1 and 5% of survey data from panel sample is garbage.”
It’s just the way it is. Yet a simple gallant inspection and a few Superman techniques can ensure you rout these percentages rather quickly. After all, it’s widely known that 70% of online surveyors clean their data before analyzing.
Afterward, it’s always prudent to inform panel provider of these discrepancies. Then they can make sure to minimize Survey Supervillains or glitches like Duplicators. Forward them the ID’s of the respondents you banished from your study Metropolis. They’ll take action on their side to warn and/or remove these panelists from their database (as we do every week at qSample).
Furthermore, many Survey Supervillains are actually computer bots. It’s just part of the internet to encounter them. Panel providers can tweak software to eliminate the Rise of the Machines.
With that in mind, you can continue with your research to become survey savvy and a true Data Superhero. Your data results will look more like The Avengers and less like The Watchmen.