Science of Failure, Failure of Science

Apparently, science is a whole lot less scientific than we imitators of science like to think.

Dunbar came away from his in vivo studies with an unsettling insight: Science is a deeply frustrating pursuit. Although the researchers were mostly using established techniques, more than 50 percent of their data was unexpected. (In some labs, the figure exceeded 75 percent.) “The scientists had these elaborate theories about what was supposed to happen,” Dunbar says. “But the results kept contradicting their theories. It wasn’t uncommon for someone to spend a month on a project and then just discard all their data because the data didn’t make sense.  T

There were models that didn’t work and data that couldn’t be replicated and simple studies riddled with anomalies. “These weren’t sloppy people,” Dunbar says. “They were working in some of the finest labs in the world. But experiments rarely tell us what we think they’re going to tell us. That’s the dirty secret of science.”

How did the researchers cope with all this unexpected data? How did they deal with so much failure? Dunbar realized that the vast majority of people in the lab followed the same basic strategy. First, they would blame the method. The surprising finding was classified as a mere mistake; perhaps a machine malfunctioned or an enzyme had gone stale. “The scientists were trying to explain away what they didn’t understand,” Dunbar says. “It’s as if they didn’t want to believe it.”

As this blog has consistently reported, cognitive consistency is a problem.  So, I guess I should then take pride in changing my mind along my career.  A few examples:

• One of my first pieces argued that federalism is problematic as it helped to doom the Soviet Union, Yugoslavia, and Czechoslovakia by providing politicians with narrower audiences and thus incentives to play up ethnic divides.  A subsequent piece based on a complete dataset found that federalism is not so problematic at all.
• Further unpublished work focused on how relative size of groups interacted with federal institutions--so groups empowered by federalism are less violent and those that are weakened are more violent.
• This lead to a particularly fun moment when a guest speaker at McGill posted a slide with various scholars and their takes on political institutions and my name was in two different (and opposing) boxes in the table.  
•  In my first efforts to understand ethnic conflict, I built upon the IR concept of the security dilemma--that competition among countries (groups) for security leads to each being worse off.  I developed this argument in graduate school (independently of Posen's work) and then eventually applied it in some of my work.  The basic idea is that ethnic groups will compete to control the government, leading to its destruction.  However, over the course of time, I realized that this approach, particularly in the hands of others, tended to downplay the role of governments.  This is important since governments do most of the mass killings in the world.  So, I worked with others on developing an alternative model based on another set of IR theories--deterrence. 
• My current project reflects another significant departure.  At first, to understand multilateral military operations, we sought to focus on NATO.  But we quickly learned that the real driver of NATO operations happens to be the policies of each contributing country.  And then, despite our preference to focus solely on institutions, we have found that individuals matter.  

So, I take away this piece from Wired some satisfaction that despite my desire to the right and expected answer, I have been willing to follow the data even when it leads me astray.  And I do encourage my students to study the anomalies, the outliers, as they can tell us a great deal.  I am not alone in this as the methods textbooks of the day do encourage focusing on the outliers.

While the scientific process is typically seen as a lonely pursuit — researchers solve problems by themselves — Dunbar found that most new scientific ideas emerged from lab meetings, those weekly sessions in which people publicly present their data. Interestingly, the most important element of the lab meeting wasn’t the presentation — it was the debate that followed. Dunbar observed that the skeptical (and sometimes heated) questions asked during a group session frequently triggered breakthroughs, as the scientists were forced to reconsider data they’d previously ignored. The new theory was a product of spontaneous conversation, not solitude; a single bracing query was enough to turn scientists into temporary outsiders, able to look anew at their own work.

This is not surprising, as I remind my students that ours is a social science, emphasis on the social. That we are constantly interacting with others to persuade them and for them to persuade us.  And the most interesting part of most presentations, especially job talks, is the question and answer session.  I also find myself thinking harder and more clearly about my students' work when I talk to them about it, rather than during the reading of their chapters/proposals/articles.

And before folks think that this Wired piece means that science is a delusion, they should read its punchline--that incidental collaboration lead to a new discovery and a Nobel prize.