Tag Archives: surveys

Seven Sins to Avoid When Designing a Survey

A year ago, we published a series of best practice for questionnaire development. This year, we would like to highlight some common mistakes that research practitioners should avoid, to ensure their data collection effort is not wasted. We are calling them The Seven Deadly Sins Of Questionnaire Design. The Seven Deadly Sins emerged from an ecclesiastic era, and since then have evolved as broader ethic markers for those who prefer disinfected consciences. Gwyneth Paltrow lost her head over them in the movie Seven, and the secular world has incorporated them as business credos (an example being The Seven Deadly Sins of Management, from the Harvard Business Review).

As long as people are dropping the ball in their professions, the Seven Deadly Sins work as a values template. They certainly work in market research, specifically when it comes to designing online survey questionnaires. Without a heavenly questionnaire, a survey will plunge into the deepest recesses of hell.

Below are the Seven Deadly Sins of Questionnaire Design that every researcher should avoid.

LUST

You are passionate about your project. You let that lust pollute your wording, even allow bias to possess the questions like a Linda Blair dream. Sometimes you don’t even know you’re doing it! As one research expert put it: “Bias is the mortal enemy of all surveys, and as a survey creator, it’s important to guard against it to make sure you get reliable results.”

To avoid this Hades, keep your language neutral and dry; employ a sensible number of opt-outs and open-ended questions; and make sure you use a second and even third set of eyes while crafting questions.

GLUTTONY

You crave that data or have a reprehensible voracity for it. You write an extremely long questionnaire, which ultimately results in crappy data collection.  You ignore the fact that respondents don’t care much for long, boring surveys, or your greediness for the data causes tunnel vision in your data collection methodology.

As some of our own research has shown, respondent fatigue sets in after 20 minutes of a survey. This may result in respondents exerting less effort and spending less time thinking about their answers as they journey deeper into the survey. A survey over 35 minutes is an indicator that your craving for data is approaching its peak with little consideration for the survey participant.

Some had predicted that online surveys completed from mobile devices would approach 50%. In an era where mobile devices are displacing computers, long survey questionnaires are just a sin.

SLOTH

Put it simply, edit, edit, edit! Mistakes won’t make you look slothful to respondents, just demonic. Furthermore, on the side effects of sloth, a good research practitioner explained:

But the problem is rarely “bad respondents” – instead the problem is lazy researchers.  When people discover that the survey they just agreed to take is boring, tedious, repetitive, or too long, they either quit altogether or they stop providing good answers. As I’ve stated many times in the past, when it comes to data quality, the burden should always be on the researcher first.

WRATH

You want to defend your research project or the sanctity of your data. Full of wrath, you will strike down upon thee with great vengeance and furious anger those unsuitable respondents who access your holy survey. Okay, maybe not Ezekiel 25:17, but you’re going to place a lot of screeners and trap questions to weed out the unfaithful.

Recent studies indicate the methodology of trap questions for surveys may not be as effective as originally thought. In reality, trap questions might have unexpected results—such as shifting the thinking of respondents to critical thinking from “optimal thinking” (that is the state of mind they reason as they normally would in daily life, which is typically necessary for reliable data).

There are other analytical ways to evaluate respondent data that don’t include placing land mines in your questionnaire.

GREED

Budgets are rarely fun, unless you’re working on the next Marvel film or you’re a Congressman(not from the state of Illinois). At the same time, a sense of greed within you assumes that the internet ought to make research economical.

Therefore, you skimp on incentives. Bad move.

Some reports claim that 175,000 online surveys are conducted a day. This volume has influenced a drop in participation rates to historic lows, which some estimates to be at 2%. On the other hand, studies have shown that the proper incentive will have a positive effect on survey response rate. We touched base on this topic on a previous blog.

This form of avarice can be avoided by rewarding your respondents properly for their time, and never assume they care about your brand as much as you do.

ENVY

Okay, you’ve done all this work and soon respondents will joyfully complete surveys, while vying for an iPad, that trip to Hawaii or Starbucks gift card. You say to yourself: “this is a lot for a 40min IDI or telephone interview and it’s all from my blood, sweat and tears”.

This attitude of envy will harm your research project. You’re not just envious, you lack empathy—the key ingredient for a successful online survey questionnaire. As stated in a previous blog, empathy is significant.

“Companies need to have more empathy for the research participant. The person(s) who writes the survey instrument should ask themselves if they could sit through that survey for 25-30 minutes. Companies should make surveys fun and engaging, regardless of the topic. They should test their surveys over and over again to identify the fatigue points in the survey. This is usually the area where data integrity is compromised.”

This quote addresses all the other deadly sins, mind you, because they all overlap. It certainly overlaps with:

PRIDE

Pride is also known as vainglory. It basically means you think you know better than everyone, including study participants. Pride has become a positive quality in western culture, but don’t let it fool you.

Pride also tends to stifle the ability to be open-minded. With all the tech innovations changing market research this year, such as eye tracking technology, social listening etc., don’t let your pride assume your ways are absolute and unchangeable.

The opposite of the Seven Deadly Sins are the Seven Heavenly Virtues. We’ll discuss them pertaining to online surveys in the future. Right now, though, avoiding the above list will likely create a paradise for your next online survey undertaking. Hell might not freeze over, but neither will your data.

The Nine Circles Of Survey Research Hell

 

When Dante Alighieri was composing the Inferno section of his epic poem, the Divine Comedy, he was surely thinking of online survey content and execution. Okay, maybe he was thinking of something else. Nonetheless, Dante’s visionary landscape of falling into a place where everything around you burns to ruin can apply to various situations. It certainly applies to how shoddy survey research can incinerate your market research.

Let’s keep it heavenly then, by avoiding these survey circle hells.

 

First Circle (Limbo)

 

 

This place (or state of being) is not that bad. It’s full of nice gardens where pagans like Plato, Virgil and Julius Caesar hang out. They never had a chance to convert to Dante’s religion, but get a pass for being notable and thus hang out in blandness for eternity.

Here on earth, that’s the problem when it comes to market research. Nothing happens. You’ve released a survey, and it’s as quiet as a Nickelback internet fan site. Response rates are low. Why is this happening?

How to get out of this hell: There are many explanations, as you will see, found by plunging deeper into the rest of the survey circle hells.

 

Second Circle (Lust)

 

 

In Dante’s poem, the condemned here float continually by the fury of strong winds. It’s an allegory of what happens when one lacks emotional control. Helen of Troy is imprisoned here, and surely Bill Clinton and Don Draper will join her in time.

For researchers: You’ve allowed your passion for the project to pollute the survey. You have a bias, and it’s often unconscious. As one researcher wrote:

One of the leading causes of misleading survey data is researcher bias that comes directly from the survey writer. This bias is sneaky. It’s caused by survey creators who innocently influence the results to reach an outcome they hope or expect to reach. It’s sneaky because survey creators are typically unaware it’s happening.

How to get out of this hell: Allow others to edit the questionnaire; keep questions concise and neutral; offer opt-out questions so other souls don’t drink (and later spew) your Kool-Aid.

 

Third Circle (Gluttony)

 

 

Here the three-headed mutt, Cerberus, makes sure that all gluttons linger in frozen, dirty slush (people in Chicago can relate when commuting in the wintertime).

For researchers: You’ve overstuffed your questionnaire. As our research reveals, respondent fatigue sets in at approximately 20 minutes of a survey. Respondent fatigue may result in respondents exerting less effort and spending less time thinking about their answers.

How to get out of this hell: Keep questionnaires as short as possible; keep your vocabulary simple or at least at the level of your audience; place important questions at the beginning.

 

Fourth Circle (Greed)

 

 

Greedy and hoarders end up stranded in this dimension, forever carrying bags of money they can never enjoy, under the control of Pluto (the Roman god of death, not Mickey’s dog).

For researchers: You’re pressured by a budget and end up skimping on respondent incentives. Bad move.

As we’ve reported, incentives have these benefits:

–  Response rates that are 10-15% higher
–  Improved response rates from hard-to-reach groups
–  Increase efficiency, especially when it comes to non-response follow-ups

How to get out of this hell: Reward your survey participants, bro. With online surveys flooding the market these days, it might not be an option—unless you want to carry around bags of non-filled surveys for Pluto.

 

Fifth Circle (Anger)

 

 

For some reason, Dante envisioned this place being the River Styx with the condemned floating on or under its icy waters. We do find a city called Dis that is about as attractive as Gary, Indiana.

For researchers: You won’t let anyone edit your survey, or you place too many trap questions (which we revealed can harm survey data—by shifting the thinking of respondents to critical thinking from “optimal thinking,” the state of mind they reason as they normally would in daily life). No one better get in the way of your research project, or it’s Khan wrath, baby!

How to get out of this hell: Place fewer trap questions and make sure there is a second or third set of eyes on your questionnaire. Yes, these circles overlap like an infernal slinky.

 

Sixth Circle (Heresy)

 

 

Welcome to an eternal landscape of flaming tombs. You should avoid this place at any cost unless you’re doing a heavy metal video.

For researchers: What is worse a heresy in market research than not checking your data? As examples, you don’t notice that some respondents are finishing surveys in less than 30%-50% of the median time; that others are marking the same line in each question; or that some are even leaving a Christmas Tree pattern in the survey (it does happen!).

How to get out of this hell: Double-check, my friend, like we keep saying, and don’t go at it alone.

 

Seventh Circle (Violence)

 

 

Witness a place where all violent thugs spend time beating each other up, under the governance of a Minotaur, harpies and whatever monster Dante must have found in his ancient copy of World of Warcraft. It’s a cosmic Fight Club!

For researchers:  Your survey research may not be threatening you, but a lack of empathy can be just as perilous. As qSample’s president Rudly Raphael stated:

Companies need to have more empathy for the research participant. The person(s) who writes the survey instrument should ask themselves if they could sit through that survey for 25-30 minutes. Companies should make surveys fun and engaging, regardless of the topic. They should test their surveys over and over again to identify the fatigue points in the survey. This is usually the area where data integrity is compromised.

How to get out of this hell: Did you not hear Rudly through the din of the harpies? Cultivate empathy. Picture yourself taking the survey, perhaps on a smartphone waiting for a bus in the rain. Picture the vocabulary of your audience as well as their habits.

 

Eighth Circle (Fraud)

 

 

Some monster called Geryon goes around in this place harassing all manner of fraudsters. We’re sure that this includes the founders of the startup you invested in back in 2012.

For researchers: It’s still surprising that so many researchers don’t communicate to participants the length of the questionnaire or clearly explain the purpose of the survey. They avoid placing progress bars or timers. They don’t engage the respondents after the survey. It may not be a scam, but it certainly makes survey participants provide lesser data quality.

How to get out of this hell: Like the figures who guided Dante in Inferno, take a journey with your respondents. That in itself lets them know that they’re actually heading to a paradise.

 

Ninth Circle (Treachery)

 

 

This level is the sucky-suck of all places. It’s hot, and you’ve got the company of such fiends as Cain, Mordred, and Judas. Need we say more? Should we include Carrot Top to seal the deal?

For researchers: We don’t have to say anything else, really, pointing to the vibe of the Seventh and Eighth Circle. Your survey ultimately represents your brand or client. You are truly your respondent’s keeper. Anything but utmost professionalism can go beyond compromising survey data and response rates. It may hinder future surveys.

How to get out of this hell: If you’re this low, ignored all the advice from the other circles, then may it’s time switch to selling insurance or swindling people by founding a startup. There is little hope.

 

Conclusion

 

 

At the end of Inferno, Dante crawls through the center of the earth and enters Purgatory. If you avoid these circles in your survey research projects, you won’t have to worry about any purgation. You’ll be already rising through heavens of quality data.

 

Devils Advocate Advertisement

The Fascinating Founders Of Market Research

 

In a past article, I created the quote: “Those who do not learn from market research history are doomed to average data.” As far as I know, the quote hasn’t made it to any social media memes or posters with cats. I still stand by it. I’ve advocated the importance of retaining some understanding (and context) of the history of market research—on this blog and in such sites like The American Marketing Association. As with the pioneering concepts of Edward Bernays, many market research ideas can actually go full circle and become relevant today.

In this article, I present a summary of all the principal Founders of Market Research. I drew from various resources, but mainly from Paul A. Scipione’s A Nation of Numbers: The Development of Marketing Research in America. It’s an excellent book we included in our article The Best Books On Market Research.
Get ready to journey on a Tardis of market research chronicles (or perhaps modified DeLorean if you have more American time travel sensibilities). I believe you will find that these fathers of market research to be far more interesting (and even controversial) than you could have imagined.

 

George H. Gallup (1901-1984)

 

 

Most Americans are aware of the Gallup Poll, but few know about the individual it’s named after, George Gallup, who also happens to be the chief developer of public polling.

Gallup began as a journalism and advertising academic, teaching in such hallowed institutions as Columbia University and Northwestern University. His breakthrough happened in 1932 when he helped his mother-in-law, Ola Babcock Miller, during her election for the position of Iowa for Secretary of State. No Democrat had won statewide office in Iowa since the Civil War. Gallup’s polling of the counties gifted Miller the insights to understand voters. The only question on his questionnaire was, “What are you most worried about?” Miller calibrated her political message depending on the majority answer of each county. This resulted in a landslide victory.

The Iowa election not only propelled Gallup to nationwide celebrity, it eventually led him to found the American Institute of Public Opinion, which soon became a perennial bellwether for national politics. His one black eye would be his calling for Thomas Dewey against Harry Truman in 1948 (Gallup blamed the blunder on closing polling too early).

Market Research Legacy: Gallup was a champion for dry quantitative research, claiming one should focus on the “what” instead of the “why” in research. He held little patience for research that delved into the subconscious motivations of respondents, as well as focus groups in general.

Quote: “I could prove God statistically.”

 

Arthur C. Nielsen (1898-1980)

 

 

Nielsen is also a household name like Gallup. His story is one of how adversity can create opportunity. During the Great Depression, Nielsen’s firm ACN (which tested and measured technologies) struggled like most businesses.

Struggling for income, Nielsen came up with two innovations in 1936: the Nielsen Drug Index (NDI) in 1933 and the Nielsen Radio Index (NRI). These innovations were based on his idea that he could develop a nationwide probability sample of drugstores. His company began auditing key brand statistics like purchase invoices and shelf stock, and then plugging those numbers into a proprietary equation. This analysis produced sales and mark share statistics for competitive brands of the over-the-counter drugs.

Large companies like Johnson & Johnson and Procter & Gamble soon began paying Nielsen for his insights. This gave birth to the power of price reductions, sales, and promotions within stores. In time, ACN became the first market research firm to exceed a billion dollars in annual revenue. Later on, of course, Nielsen entered the television market to become truly immortal.

Nielsen always claimed he “accidentally” entered market research.

Market Research Legacy: Beyond his sample statistic innovations, in 1948 Nielsen’s company built the first ever general-purpose computer, the Univac, for $150,000. Steve Jobs and Bill Gates would later work on cheaper products, thankfully. Moreover, Nielsen Ratings have been the standard-bearer for marketing television for decades.

Quote: “The price of light is less than the cost of darkness.”

 

Alfred Politz (1902-1982)

 

 

In this pantheon, Alfred Politz balances the dry countenance of Nielsen. His life is closer to that of a movie star or international spy.

Politz worked as a marketer in Germany in the 1930s, even though he was a physicist by education. He fled his homeland to the US, aided by chewing gum magnate William Wrigley. The reason was that Politz feared that Hitler would force him to work on weapons of mass destruction.

Once in America, Politz worked in advertising, claiming he wanted a job where he could make the most money for the least amount of thinking. His fiery personality led him through many companies, producing great marketing work but also great enemies. Eventually, he founded Alfred Politz Research, Inc., which would become a chief market research firm. His company dominated the development and implementation of new quantitative methods for advertising research for nearly 30 years. During his meteoric rise, he lived the life of a rock star: flamboyant and hedonistic (and often entertaining to the press).

Market Research Legacy: His advocacy of random probability sampling shaped market research. His influential, high-visibility studies for Life Magazine became legendary in marketing circles.

Quote: “Repeat exposure without repeat payment.”

 

Ernest Dichter (1911-1991)

 

 

We provided an exposition in Freud and The Intriguing History of the Focus Group. Like Politz, Dichter was a complex and fascinating character, a press celebrity who even appeared on The Tonight Show. Furthermore, the Viennese Dichter also came to America to avoid Hitler, but the reason was that he was a Jew.

Ironically, Dichter and Politz were lifelong enemies, their antagonism a symbol of the yin and yang tension of qualitative and quantitative polarities.

Certainly, Dichter was a trailblazer of qualitative research, as well as a Freudian who believed in manipulating the “beast mode” forces bubbling in the human subconscious. From phallic-looking cigarettes to the legendary tiger in the gas tank, Dichter played with symbols and archetypes to manipulate consumers.

Market Research Legacy: Developed and perfected the focus group (and coined the term), permanently injected psychology into market research.

Quote: “What people actually spend their money on in most instances are psychological differences, illusory brand images.”

 

Charles Coolidge Parlin (1872 – 1942)

 

 
Like Nielsen, you could say Charles Parlin accidentally entered market research.

A Wisconsin schoolteacher with no marketing experience, Parlin was hired by Curtis Publishing Company in 1911. He didn’t even receive a job title. His job was to provide better advertising insights on an agriculture magazine the company had recently bought.

Parlin created market research out of thin air, from data gathering to best practices, interviewing subjects and breaking down the findings in sober numbers. Almost overnight after his groundbreaking work for Curtis Publishing, logical rule replaced intuition in the marketing world. He eventually started his own firm, National Analyst, and further pioneered market research as we know it.

Market Research Legacy: Founded the first market research company in history, parlayed one set of simple insights into an entire set of standards that still govern market research and general marketing to this day.

Quote: “Advertising, properly applied, reduces selling cost. The consumer pays the expense of advertising which therefore justifies itself.”

 

Conclusion

 

 

If there is a main takeaway to the lives of these individuals, it would be that market research is often as accidental as it is analytical. It’s a cauldron of invention as it is a laboratory of data, producing a $24 billion a year industry that employs more than 150,000 workers nationwide. Also, market research doesn’t have to by dry and rigid, or at least perceived as such, and that often common sense rules the day for individuals trying to make sense of market forces.

Most of us probably won’t appear on The Tonight Show or have to worry about fleeing despotic governments. Nevertheless, we can have a good time and change the flow of this nation of numbers.

 

A version of this article appears in Quirks

 

Cutting Costs on High Tech Gear and Electronics

Consumers desire the latest electronics and high tech gear, but unfortunately cost can be prohibitive. Savvy consumers have blogs, websites and books dedicated to saving money both for the home and business. Here are a few non-traditional and creative ways to save money when it comes time to buy the needed items to outfit an office or the latest technology gadget.

 

Online Auctions

Online auctions claim to save buyers anywhere from 25%-75% on all sorts of items. One online auction generating attention is Quibids It’s free and easy to create an account but in order to bid, one must first purchase 100 bids at 60 cents a bid. Smaller bid packages may be purchased after the initial purchase. Items can be placed on a watch list to track selling prices at certain times. We saw two remote-controlled cars sell for vastly different amounts. In the early afternoon it sold for one bid (60 cents) and later that evening sold for 240 bids.

 

If one loses a bid on an item, QuiBids will sometimes offer the item at a Buy Now price, and will subtract the bids already placed on that item. For instance a $50 gift card that had $40 worth of bids already might be available to purchase at the remaining cost of $10.

 

Auctions are not a guarantee of lower prices, but often a creative way to try to buy high priced items at lower costs. All items on QuiBids are new and items ship for free within the U.S.

 

Use Online Shopping Carts

Another way to save money is to shop on-line at sites like Amazon. Items can be placed in a shopping cart for later and will adjust as the price either drops or rises. Amazon Prime membership will also save money on the item price and get two-day free shipping on Prime items. Overstock.com often offers cheap shipping as well.

 

In addition, if you place items in an online shopping cart but do not complete the purchase, you may receive an email within a couple of days offering you a discount on the items in your shopping cart. It’s the store’s way of following up to close a sale.

HiTechmoney

Planning and Patience

A tried and true way to save money on electronics and high tech gear is to plan ahead. During the year there are times when high priced items are on sale. Televisions, cell phones, iPods, digital cameras, computers and more have yearly sale times, especially when new models come out.

 

It’s good to research before purchasing. Consumer Reports has a list of which items commonly go on sale and when so that you can better plan in advance. For instance, computers are a popular sale item before school starts and TVs in January. You can also avoid spending hundreds at once when you buy an item by pre-buying gift cards in smaller amounts to save up for a particular item. Purchase gift cards at many retail outlets as well as online via Google Play, Apple, Best Buy, Amazon and more.

 

Gift cards can also be used anytime, including during Black Friday sales weekend. Stores advertise specials on everything, including electronics. Long lines may deter consumers from Black Friday sales, but it’s worth researching what items stores are selling at discounted prices as often you can get items for pennies on the dollar. Buying gift cards ahead saves you spending the full amount all at one time.

 

Ask

A great way to have the latest smartphone and save on monthly bills is to ask the phone carrier if there are deals, upgrades or discounts available. Customer service agents work hard to create customer loyalty by offering discounts, deals, and reviewing bills to make sure consumers are getting the best rates. Many big phone companies offer deals and discounts to keep customers happy (this also applies to cable/satellite and internet services) and you will never know-unless you ask. Remember the person on the other end of the phone wants to sign new people up and make sales.

 

Saving money is a strategy that can be applied to every aspect of business, including keeping up on the latest products that help save time, money and manpower. High priced technology and electronics can be made more affordable by research, planning and patience.

Will Office for iPad Make Tablets the Tool for Work?

Last month, Microsoft shook the tablet world by finally launching its Office for iPad apps, and the reverberations may be felt for years to come.

While tablets have become extremely popular, with many users preferring smaller to bulky and more awkward laptops, they have yet to truly find their niche in the business world. Even with bluetooth keyboards, and writing tools like Google Docs and iWork, many businesses refuse to adopt the device as a serious business tool. In the academic arena, many students prefer tablets for note taking and research, but find it awkward to use for papers and presentations. The launch of Microsoft’s Office for iPad apps is designed to change all that.

We conducted a survey with our general consumer panel to gauge their level in the new Microsoft Office for iPad app. The survey was fielded in less than 2 days during the first week of April, with more than 400 respondents sharing their insights on this new product and what it means to them professionally.

Survey results clearly indicate that tablets are still very popular with the general populaltion. Unsurprisingly, Apple was king among those devices with  almost half (40.45%) of our panel indicated they own a version of the iPad, but a mere 26.18% claim their primary uses for those tablets are for work and school. Confirming our suspicions, we found that entertainment rules the tablet world, with a staggering 69.12% of our panel logging on for fun. As always, internet surfing, watching videos, and updating social network pages continue to be a staple of tablet use.

Clearly this isn’t a hardware issue, and many who own Apple devices, such as the iPad, prefer to use the Microsoft Office software. This suggests that the new apps will be very well received by tablet owners, but there is a lot more involved than ease of use. Our survey showed that there was a strong positive reaction to the Office for iPad apps, as 63.02% said that they plan on using the new apps now that they are available, yet 76.56% didn’t feel that the apps were worth the $99 subscription fee. This is likely due to the abundance of less powerful, but free, programs/apps that will allow users to run similar tasks. Price does seem to play an enormous role, as 64.58% of our panel is considering the free Microsoft Office smartphone apps as a serious alternative. This poses the question of whether or not smartphones may find a place as a document editing tool in the business world as well.

The smartphone apps may be free, but lack many important features that are available on the iPad and laptop versions. In addition to features, portability seems to be a factor. Today’s “on-the-go” lifestyle means that document editing on a smartphone may be preferable to carrying a bulky laptop, or even a tablet. On the down side, small screens and lack of features may discourage users from choosing apps like Office for smartphones, even if they’re free. Our panel was also concerned about storage space on their devices. These apps can take up a considerable amount of space on users’ iPads, and 67.71% claimed that this alone would discourage them from downloading the software.

Cloud services have been available for some time, but there is no question that Microsoft has arrived very late to the game. This may be due to the company’s efforts to streamline their products for unconventional devices, or a simple lack of attention to the tablet market. In either case, this is definitely an interesting move for Microsoft. One pitfall of releasing the apps for the iPad is that this may have a negative effect on the sales of non-Apple tablets, including Microsoft’s own Surface. In addition, the Office for iPad apps are significantly better looking and much more streamlined than other versions. This may influence which tablets businesses buy for their employees, and ultimately hurt Microsoft’s Surface sales as well. Apple will receive a percentage of Office 365 subscription fees sold through iTunes, which will also add to the funding of their competition. All of these factors make the late release of the Office for iPad apps a very interesting decision. Only time will tell if Microsoft’s new launch was a brilliant strategy, or too-little-too-late, but in any case, the world is taking a second look at the tablet as a serious business tool.

Microsoft Office for iPad Infographic (3)