Mar
31
Thanks to the Internet, many tools once exclusive to professionals are now widely available at low or no cost.
Nowadays, just about anyone with a computer can design a page, retouch a photo, or even compose music. But if you’ve ever had to hide crossed fingers behind your back while muttering something nice about a friend’s would-be masterpiece, you know that tools don’t ensure competence.
This is particularly true when it comes to conducting online research. An array of affordable web-based tools makes it a cinch to build and distribute an interactive survey. But there remains an art, even a science, to designing the questions. The results of an unwittingly but poorly framed question can be costly. Besides wasted time and expense, there is a danger of basing important strategic decisions on “facts” which later turn out to be fiction.
Even the seasoned research pro must exercise ongoing vigilance to avoid ending up with abundant-but-useless feedback. Here are some guidelines to keep in mind:
1. Some pitfalls are less obvious than others. For instance, even the order in which questions are listed has been documented to influence answers. So has a survey’s background color. To ferret out hidden problems, try randomly mixing things up. Perhaps each respondent can receive the questions in a different order, in different colors, on different days, etc.
2. Word choice matters. Spotting and pruning potentially biasing words is trickier than it seems. Though technically the same, people may have a different reaction to a question about a “rule” than about a “regulation.” For that matter, more people would rather “not lose” than “save” or even “gain.”
3. Never ask respondents to predict or even report their behavior. They can’t do it. At least, not with any degree of reliability. Yet if you ask, they will try to answer, and they will be sincere in what they tell you. The problem is that people perform poorly when it comes to correctly identifying what they do today, much less predicting what they’ll do tomorrow.
4. Caveat to the prior question: Unless, that is, you want to reveal something else. Asking people to predict their behavior won’t tell you anything about what they will or won’t do, but it can tell you a lot about how they see themselves.
5. An online survey may not be representative of your market. For one thing, only customers who are online can participate. Responses will only come from those who care to and have the time to participate. Thus your data may be weighted toward people with complaints, people with time on their hands, people who are bored, or people who enjoy completing forms.
6. Provide a “does not apply” option. This keeps people from checking a box, any box, just to keep you happy.
7. Don’t overlap. Say if you want to know how many employees a company has. Do not offer choices like “up to 100” and “100 or more.” Make the categories distinct: “Under 100” and “100 or more.”
8. Keep it simple, but not too simple. Too many questions, too many steps between “strongly agree” and “strongly disagree,” too-long pull-down menus, and too many essay-type questions can lead respondents to rush or simply quit. On the other hand, too few options won’t get to the nub. Seek for a balance.
9. The more respondents, the more reliable the data. I have seen companies that should know better than to base big decisions on opinions from 10 or 20 people. While no response is 100 percent reliable, try for at least 5,000 respondents. And do your best to ensure that they represent a good cross-section of your market.
10. Research is not invincible. Take care as to how you use survey results. Treat them as suggestive, not conclusive. If they lead you to a hunch, test the hunch before betting the farm. A little caution now can spare you a lot of trouble and expense later on.