Thursday, January 12, 2006

Beware the experts in the campaign's last days (Not me, of course, those other guys)

VICTORIA - Most of the experts telling you what's going on in this election are likely no more astute than the average lab rat.
That's the conclusion - perhaps slightly overstated - of an American book that looks at the performance of political commentators, and it's an important thing to remember as the campaign enters its final days. People who plan to vote strategically to block a Liberal or Conservative government are likely relying on the experts' analysis, if only to assess the closeness of the local and national races. Other voters may be interested in pundits' predictions on how the parties would behave if elected.
But Philip Tetlock's' book, Expert Political Judgment: How Good Is It, shows those voters could be led badly astray. (I haven't read the book, just a fine review by Louis Menand in the New Yorker. At least I confess my knowledge gaps.)
Tetlock, from the University of California, has spent 20 years studying the accuracy and quality of experts' opinions. He's tracked the pronouncements of the academics and pundits who show up on the news offering their analysis - or write columns like this.
The experts are no more insightful or accurate than the average citizen, he found. In fact the more well-known they are, and the more frequently quoted, the less likely their predictions and judgments are to be accurate.
Tetlock set up experiments to test the insight and acumen of 284 experts over a long period, asking for predictions on topics within their areas. Then he measured their success, and found you would do better by writing the options on a wall and throwing a dart blindfolded.
So why do these smart people, paid for their insight and expertise, get it so wrong?
A bunch of reasons. Experts feel pressure to come up with predictions and opinions that are clever and somewhat surprising. You don't get invited back to a TV panel or asked to write more newspaper op-ed pieces if you state the obvious. I do radio and TV commentary from time to time. Often the interviewer winds up the interview by asking what's going to happen next on the topic of the day, and I sense disappointment if I say I have no idea, or state the obvious. No one likes to disappoint.
So experts look for obscure angles, or complicated ideas that are novel, interesting and reinforce their status as smart observers. Because they've studied the issues, they can grab bits of information to support their view. The problem is that the most obvious analysis is often the right one.
That's why lab rats can be better at making predictions. Tetlock writes about a Yale study where rats were placed in a T-shaped maze, with food at the end of one of the arms. The food placement appeared random, but it was placed at the end of the left arm 60 per cent of the time. The rats figured that out, and started going left most often, playing the percentages. Yale students were asked to observe the experiment, and say where they thought the food would be each time. They looked for patterns, and trends, and did worse than the rats at predicting because they ignored the obvious in favour of complex theories.
And experts, once they develop those theories, tend to stick to them and bend the facts to fit their views.
None of this means experts are irrelevant. It simply reminds us to be skeptical, to test the analysis and predictions against our own knowledge as well as the comments of others. It's a good time for that reminder. Many voters will be making complex calculations over the next few days, about the likely outcome of the election and the real agendas of the parties. It's reasonable to look for the experts for help.
But it's important to remember that they are quite likely to be wrong.
Footnote: The other trend is the emergence of the psuedo-expert, the pundits with close ties to each of the parties who gather to engage in predictable partisan bickering under the guise of commentary. The panels have become a media staple, even though they add little fresh or original to the public debate.

1 comment:

Anonymous said...

Another recent book that questions the wisdom of expterts is "The Wisdom of Crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies and nations" by James Surowiecki.
As well, you can just remember the media predictions before every election (BC, Canada, US) in the lst few years.