July 13, 2009

The wisdom of crowdsourcing

In conjunction with the death of some middle-aged pop star, the Boston Globe is running a visual trivia contest to name seven of the 45 stars in a photo taken for the 1985 taping of “We are the World.” It’s a fun exercise for anyone who is/was a fan of 80s music.

I got two wrong — one because I couldn’t see the singer, and one because there were two (somewhat obscure) artists closely associated with each other, and I guessed the wrong one. In the latter case, out of 20,000+ respondents, the correct answer had the lowest % of right answers of the entire quiz (65.2%). The quiz allowed people to peek before answering, which may have inflated the correct answer count.

I used to watch Who wants to be a millionaire? and it was remarkable how often (in response to a “lifeline”) the audience was right, particularly on the obscure questions. Still, I was amused when audience either split on a plausible answer or even got it wrong; with a large enough sample, a wrong answer would suggest some sort of systematic bias (e.g. towards a more famous actor or place).

From a strategic standpoint, it suggests to me that there are two types of crowd-sourcing contexts. In one, it’s helpful (or fun) to get the right answer, but it’s not the end of the world if you don’t. In other cases (the $1 million question, diagnosing your child’s infection) mistakes have consequences, and only the right answer will do.

I think the crowd-sourcing literature needs to make more of this distinction. For user innovations, the assumption (probably correctly) is the more the merrier — do a good job of ideation and the firm can sift through the ideas to get the right one. If you’re relying on Wikipedia, IMDB or other user-generated content to be accurate, then you want the correct answer. (Perhaps that’s why WikiDoctor is a cybersquatter rather than a real website).

It seems to me that several a great opportunities for experimental research here. First, if the recipient of the crowd-sourced data wants accuracy, are there ways (e.g. weighting) to design the idea generation or filtering process to improve accuracy?

Secondly, does the nature of the UGC/crowd source request (either implicitly or explicitly) change what the crowd does? For example, if you were surveying nurses, doctors or EMTs, would a simple manipulation (“life threatening” vs. “not life threatening”) change how the contributor approached the contribution process?

No comments: