In his article in GreenBook, Is Online Sample Quality A Pure Oxymoron?, Scott Weinberg presents a “state of the union” on the present online survey industry. The news is not good, according to him. Weinberg boasts more than 12 years of experience in the field with some lauded companies, so his alarming declaration carries some weight.
The article is spanning and often byzantine with evidence on seemingly the glaring issues facing online panel providers, from methodology to business philosophy—almost to the point of being apocalyptic. The online survey industry is booming, but according to the author it is in danger of catastrophic irrelevance.
Here are some examples that are endemic not only with established online panel providers but also in research departments in marketing companies:
– Tainted panels, such as doctor panels with non-doctors included.
– Highly-paid salespeople without a rudimentary knowledge of sampling terminology.
– Panels regularly infiltrated by Chinese hackers trying to make a quick buck.
– Offering studies with close to 100 questions that raises profits but forces panelists into unnecessary marathon surveys.
– Submitting specialized panels filthy with river sample (which in the industry basically means the sludge of online sampling, not vetted in any meaningful way).
–A huge disconnect between the back and front ends of operations, with the client in the middle having nowhere to go but down.
The article states that companies are making money, so heads are turned away while dollars are counted, and the online survey apocalypse gets closer.
This Armageddon can be halted, though. Weinberg both captures the problem and offers the solution with this Zen-like quote (which he actually heard from a colleague):
When is the last time you washed your rental car?
That may not make sense at first, except when one replaces “rental car” with “online panelists.” Then things change. After all, market research companies don’t own their online panelists, but rent them from the public and then rent them to their clients.
Unlike car rental companies, many don’t “wash” them—meaning they are not properly screening, nurturing, or engaging online panels. This would be unacceptable to anyone doing business with Hertz or Enterprise, yet somehow it’s being overlooked to an extent in the online panel industry.
So how do you “wash” online panels? The article offers some suggestions, and these are already being embraced by many in the industry:
Better surveys: Many online panel providers put the onus of surveys on their clients, and leave it at that. Weinberg writes in a (sadly) comical way:
I’ve seen literally hundreds of surveys that have been presented to online panelists. I’ve been a member of numerous panels as well. Half of these surveys are flat out laughable. Filled with errors. Missing a ‘none of the above’ option. Requiring one to evaluate a hotel or a restaurant they’ve never been to.
Online panel providers could act more as consultants. Surveys should be as trim and direct as possible to keep panelists involved. (Weinberg further suggests minimizing matrices, calling them “the laziest type of survey writing.”)
Better incentives: Weinberg wastes no breath explaining the cost-cutting culture of online panel providers. Every company wants to save money, of course, but you get what you pay for and information is too important in this era to be ever diluted. A panelist without enough of an incentive is a panelist that will potentially lie or rush; and that will be one piece of information that may injure a company in its quest to please its customers/clients.
Furthermore, the effect of depressing prices leaves less room for companies to invest in security software and other quality assurance that can remove the blight of hackers and other unfit respondents.
Better employees: Weinberg doesn’t mean fire those who manage online panels, but invest more in the back end operations; create an environment where all ends of the company/department are in constant communication. He states that the industry has become akin to a high tech deli where the sales team is all about “slinging sample by the pound, and let the overworked and underappreciated sample managers handle the cleanup and backroom topoffs.”
Better mobile: Weinberg specifically states that smartphone surveys are the way of the future that should be embraced right now. He doesn’t go into detail, but the advantages of mobile surveys are generally known in the industry:
– Easier to administer and reach target audiences.
– Reaches the coveted millennials and businesspeople.
– Utilizes GPS technology that shows actual respondents.
– More versatile and real-time.
– Broader in its use of various media.
If these issues are not addressed (and we and many other companies address and solve every day), it only means less effective online sampling and less impactful information for market research. It may not be an actual apocalypse, but it’s just plain bad business.
For those seeking quality market research via online sampling, it’s obvious from the above to look for these three issues:
– Does the online company/department assist with the survey, and is present throughout the process?
– Does it prices seem too good to be true, and do they not have propriety panels? (Weinberg suggests online survey providers use more “invite only” panels, so this might be a relevant questions when shopping around).
–Do the employees work together, as with difference sectors communicating with the client throughout the process?
– Does the company optionally provide mobile technology for online surveys?
If these questions are answered positively, then a prospective an online panel provider likely has “washed their rental cars,” and you should drive it for better market research.
More on mobile surveys: