Experts Say Experts Can’t Be Trusted

29.08.07

Categorie: Extremists, Ideas, Research |

experts1_cover.jpgWho’s better at predicting threats, conflicts and their outcomes: policymakers and pundits, or a bunch of average slobs? According to one recent study, they’re all about the same:  

The research can have serious consequences for foreign policy and business. [Co-author Kesten] Green says, “Political leaders in the West are pondering how best to deal with the threat of the Iranian government’s nuclear ambitions. Forecasting problems such as this are the stuff of not only international relations but also of takeover battles, commercial competition, and labor-management disputes. In most cases, experts use their judgment to predict what will happen. How good are their forecasts?

“The short answer is that they are of little value in terms of accuracy. In addition, they lead people into false confidence.” …

The case studies were diverse: they included a hostile takeover attempt, nations preparing for war, a controversial investment proposal, a nurses’ strike, an action by football players for a larger share of the gate, an employee resisting the downgrading of her job, artists demanding taxpayer funding, and a new distribution arrangement that a manufacturer proposed to retailers. …

Analysis of additional data produced similar results. In one instance, the authors attempted to determine if veteran experts would be more likely to make accurate forecasts than less experienced experts. “Common sense expectations did not prove to be correct,” they write. “The 57 forecasts of experts with less than five years experience were more accurate (36%) than the 48 forecasts of experts with more experience (29%).”

This would come as no surprise to James Surowiecki, whose excellent book The Wisdom of Crowds contends that individuals, regardless of their supposed expertise, are usually dumber than the aggregated wisdom of a loose network of diverse amateurs who all approach a problem from different perspectives. The magic is in the aggregating. Get that right and you can do a pretty good job of predicting almost anything.

For some reason, I’m reminded of something an officer friend of mine, a recent Iraq veteran, said in a bar one night. ”If you ask anyone who understands Iraq if they understand Iraq, they say, ’No.’ Anyone who knows anything about Iraq knows they don’t know anything.”   

(Before you go flaming me in the comment section: yes, I am aware of the irony of experts saying experts can’t be trusted.)

|

8 Responses to “Experts Say Experts Can’t Be Trusted”

  1. 111 says:

    Whenyou got the beer goggles on, fat chicks look good. When you are sober and wear thick glasses. and take em off you cannot see 2 feet in front of you.

    Inexperienced , experienced

  2. This is one more reason why universal broadband access should be provided. Provides a way for the “aggregated wisdom of a loose network of diverse amateurs who all approach a problem from different perspectives” to be heard.

  3. RC says:

    I’d argue that certitude loses to skepticism. The most intriguing statistic was that those with a lower confidence score(41%) for their answer were more accurate than those with a high confidence score(28%).

    I’d also argue that calling yourself an expert does not make you an expert.

    There where many things that I didn’t see discussed in this paper. One I would have like to see is if individual experts/novices were more accurate than the group as a whole. It also would have been great if these individuals had filled out the questionnaire once before reading the data to get a true measure of the chance score.

    A study that I read involving stock trading came to the conclusion that only 25% of the traders were basing the trading on knowledge. This has everything to say on how individuals make decisions. Do people follow the crowd or operate from observation.

  4. Jim S. says:

    “The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts.”
    - Bertrand Russell

  5. Kris says:

    Philip Tetlock wrote a very good book called Expert Political Judgment that makes this argument in with a good bit of science to back him up. His bottomline? When it comes to forecasting, how you think is more important than what you know.

  6. Bubba says:

    He who speaks, does not know. He who knows, does not speak. – Chinese proverb

  7. Tech Talk says:

    Forward Bias…

    On Fridays, you’ll be sampling other bits of the blogosphere courtesy of a new Tech Talk feature we’re calling Forward Bias. The name comes from the definition of the term: connecting the voltage source (that’s you, dear reader) to the material, …

  8. Anonymous says:

    Many experts who have left, or otherwise have expressed dissatisfaction with Wikipedia, fall into two categories: Those who have had repeated bad experiences dealing with jackassses, and are frustrated by Wikipedia’s inability to restrain said jackasses; and those who themselves are jackasses. Wikipedia has seen several recent incidents, including one this month, where notable scientists have joined the project and engaged in patterns of edits which demonstrated utter contempt for other editors of the…

Leave a Reply