Survey This!

This article was published in the original Podium magazine on Dec 1, 2001, with the tag line, "A critique of the practice of Internet or web-based surveying."

by Raywat Deonandan

Back in 1997, Microsoft Canada garnered a lot of media attention for it's ballyhooed 24-hour on-line Internet poll. Internet users were asked to visit the Microsoft web site and answer the nine questions posted there by that champion of Canadian surveyors, the Decima Corporation. Fifty-five thousand Canadians responded, half of whom were from Ontario. According to the Microsoft press release, it was a "demonstration of the power of Internet technology."

And how was this power used? It was used to answer such gripping questions as "If the Earth's axis shifted and lengthened the day by one hour, how would you spend the extra time?" and "Which of the following famous T.V. bosses reminds you most of your own boss?" More precisely, it was a demonstration of the power of Microsoft's marketing team.

Admittedly, the survey did garner interesting information about Internet usage, such as the insights gleaned from the survey question, "What is the single most important function that the Internet serves for you at work?" But that seems to be the only useful result of such an endeavour, at least for now, because a sample of people who use the Internet is only representative of other people who use the Internet.

In other words, one still cannot use the Internet as a survey tool to make inferences about the great unwashed masses. Despite what marketers tell us, most people, even here in Canada, remain unwired.

Luckily, the majority of on-line pollsters are sophisticated enough to realize this, and have directed all their survey questions toward an electronic end. For example, Dr. Kimberly Young of the University of Pittsburgh maintains a very long on-line questionnaire that attempts to detail a psychological profile of each respondent. Her actual research question, however, has to do with the phenomenon known as "Internet addiction," so she has probably targeted the right audience. In fact, in order to spend so much on-line time completing her survey, one would have to be addicted to the Internet.

Another truism of surveying is that you usually attract a certain kind of person, whether intentionally or accidentally. For example, Microsoft wanted to sample Canadian Internet users, but the very language of the survey --English-- excluded every francophone Canadian.

A company called Easyscopes runs a regular monthly survey from its website and has found a typical proportion of female respondents to be around 80%! This, of course, greatly exceeds what one might expect, especially when the Microsoft data suggested a 1997 female on-line presence of 20%. But then one must realize that the Easyscopes site is where many people go to read their horoscopes, supposedly a predominantly female pastime.

One must therefore consider a respondent's motivation for being at the website to begin with. In the case of Microsoft, a total prize package of $500,000 was the bait for completing their survey. Other on-line surveyors offer cheaper but more subtle incentives. The inevitable and somewhat ubiquitous sex surveys tempt one to completion by letting each respondent view the cumulative results immediately upon completion of each survey. It's a temptation that's hard to resist, especially if --as the polls would suggest-- you're a middle-aged middle-class unmarried heterosexual man who gets most of his social interaction through a computer screen.

The more academic investigators are definitely at a loss in this compensatory sense. And there are quite a few of them out there, including a University of Brighton graduate student who is investigating the effects of the Internet on users' quality of life. But few graduate students have sufficient funds to offer half a million dollars in prize money, or a research topic sexy enough to lure respondents who are otherwise beckoned by the luridness of explicit sex surveys and the deep pockets of billionaire software magnates.

So if even academics can't make effective use of the "power of Internet technology," Microsoft's claims start to sound a little hollow. However, one final benefit may yet be wrought from the growing trend of Internet surveying. It may, someday soon, noticeably reduce the number of annoying telephone polls.