"It Makes Sense, but Not Science"

That’s a quote from my mother (hi mom!), describing the phenomenon of causal explanations that seem intuitively correct, but upon scientific investigation, turn out to be wrong.

I’m bringing it up because Laura Seay at Texas in Africa has an excellent series of posts up on how social scientists think about evidence, causation, and uncertainty:

  1. how social scientists think: anecdotes aren’t evidence
  2. how social scientists think: what your driver says isn’t evidence
  3. how social scientists think: correlation is not causation
  4. how social scientists think: we’re not completely sure about much

She’s writing in an attempt to explain why social scientists and activists seem to produce such divergent assessments of the causes of conflict in places like the DRC and such contradictory prescriptions of what should be done to help. She argues that while academics are trained to think “in systematic ways that explain causal relationships between phenomena,” advocates “are trained to stir emotions and to draw personal connections between international events and Western students, consumers, and families.”

I don’t disagree that advocacy’s focus on awareness raising contributes to over-simplification of analysis (see Amanda and me not disagreeing here and here), however, I don’t think that’s what’s at the root of the difference in approach.

The kind of advocacy that Laura is talking about, even when not performed by human rights groups, is an outgrowth of the human rights movement. Yes, there are other philosophical sources for this type of activism, but the approach is heavily dependent on the methodology developed by human rights lawyers over the last half century. This methodology is characterized by meticulous documentation of violations of human rights laws, naming and shaming of perpetrators, and advocacy for (1) compliance with the laws, and (2) punishment of non-compliers.

Which is to say: lawyers don’t argue from evidence the way social scientists do. As Laura explains, social scientists are primarily concerned with using evidence to explain why certain events occurred and to predict future events. Lawyers use evidence to prove that certain events occurred and sometimes to prescribe or proscribe future action.

Consequently, lawyers (and those who employ their methodology) and social scientists are interested in patterns in evidence for very different reasons. When human rights researchers collect evidence, they are trying to provide enough support to legitimize a claim that serious violations are occurring. The goal is therefore quantity and reliability.

When social scientists collect evidence, they are trying to establish a causal story. As Laura explains, it is incredibly difficult to get from showing that two things happened to proving that one thing caused another thing. This means that they have to think about connections between pieces of evidence, and control for other potential explanations of the evidence they observe, that aren’t relevant for the lawyers’ approach.

Neither of these approaches is better than the other. The evidence-gathering methods of the human rights movement are extremely well suited to the strategy they were designed to support: advocacy for the cessation of violations of human rights laws.

The problem is that the strategies of some advocacy groups have outpaced their methodology. What Laura is highlighting is a move towards making recommendations that rely on causal claims that their evidence-gathering methods are not able to support.

(Stay tuned for my next mom-quote inspired post: “Not All Fruit Is Perfect, Just Eat It and Shut Up”)

10 thoughts on “"It Makes Sense, but Not Science"

  1. Good point. But the traditional approach of human rights groups (ie HRW) also steers clear of certain messier (but important) aspects of human relations that can't be resolved — or aren't most effectively resolved — by calling for enforcement or establishment of the rule of law. Peace processes, for instance. Or policies that promote reconciliation amongst peoples at odds.

    Also, doesn't the human rights approach still wade into assigning causality when calling for the policies they think are most likely to bring redress to violations of the rule of law, a process which involves the implicit weighing of tradeoffs without significant evidence (do we call for this $X we believe we can get from Y donor to go toward SSR or toward an early warning system for civilian protection?)?

  2. Poff, yes, these are good points. I'd argue that they're indicative of the fact that the human rights approach, adapted to the case of the strong-state rights violator (where the recommendation is "stop doing that!"), doesn't easily lend itself to addressing rights abuses arising out of weak state situations (where, as you note, any recommendation is likely to involve some assignment of causality).

    As you say, some human rights groups have tried to avoid the thornier questions, while others have waded in willy-nilly, but I don't think either approach is viable long-term. The nature of human rights crises has changed from what it was 25 years ago; I think the movement needs to change with it.

  3. This is also one of the central points made in Hart and Honoré's Causation in the Law, some twenty something years ago, to explain why causal concepts in legal discourse are markedly different from the ones developed in the sciences (hard or soft). Things have evolved since — think of scientific evidence in medical malpractice suits — but it is still the case that evidence and causal notions in the law are not about establishing statistical regularities applicable in general terms ('what are the main causes of internal armed conflict in Africa?'), but rather about proving what did in fact occur in a given case that is legally salient ('Did conflict diamonds provided by A actually facilitate/cause conflict X?').

  4. "When social scientists collect evidence, they are trying to establish a causal story"

    I don't think this is always the case – there's a lot of descriptive social science out there. The whole business of poverty measurement is unoncerned with causality, and quite concerned with precision of its estimates. I think most of the advocacy crowd is concerned with neither.

  5. Linking policy advocacy to legal advocacy is suggestive because it invites comparing the activist's stance relative to the issue of concern with the lawyer's commitment to arguing one side only.

    In other words, if I hire a lawyer (as my wife has just done, although not against me—I'm in her corner!), then I can reasonably expect her to take my side without qualification and to throw everything except the kitchen sink at the other side. My lawyer has good reason to hold back the kitchen sink, by the way: if she advances an obviously bad argument on my behalf, her strong arguments lose credibility. Nevertheless, she had better not make any arguments intended to weaken my case.

    Mustn't we say of many policy advocates that, like legal advocates, their minds are made up regarding which side to defend? A crucial difference, however, is that a lawyer is by definition a professional and therefore someone who should recognize when an argument is too shaky to put forward. In contrast, many policy advocates are amateurs—literally, lovers, whose passion for a cause is wedded to inability to see the kitchen sink's unsuitedness for throwing.

  6. This is an awesome addition to the great series of posts Laura is doing. Thanks, Kate.

    What you say about law and proof makes me think of a narrower group of scientists: statisticians. I'm not one, but I'm fascinated by them. One of my favorite interviews ever was with Patrick Ball, chief genius of the Human Rights and Data Analysis Program at Benetech. He had really interesting things to say about testifying at the ICTR about data he'd collected and whether that data "proved" that ethnic cleansing happened. I refer you out to the article I wrote, not to self-promote but because he really says cool stuff. http://www.csmonitor.com/World/2008/0207/p20s01-wogn.html

  7. This just seems like a muddle to me. Social scientists and human rights advocates are both trying to get at the same thing: truth. But good social scientists and good human rights advocates realize that truth is a slippery concept and one can only document a version of truth – the essential is to be methodical and rigorous in the documentation.

    The fact is causality is very difficult to establish. I am an epidemiologist and would never think of social scientists as demonstrating (let alone "proving") causality because they don’t (generally) set up randomized controlled trials. But RCTs are also contested conceptually as the only means of establishing causality (see Bayesian critiques, for example). There is not one type of causality, there are many. Fetishizing causality has led to a paralysis in the application of common sense (smoking doesn’t cause cancer, it is correlated with cancer).

    Increasingly behavioral economists are fixated on experimental designs but it’s hard to talk about behavioral tendencies in specific settings as causal relationships. Certainly the generalizability is often suspect.

    Human rights advocates have both a documentation role and an advocacy role. In documenting they may be looking at causal relationships, as a way of reflecting truth. This violation was due to discrimination. Of course, this is not a RCT. As advocates, human rights practioners look for ways to convey how what they have documented relates to international law, which only sometimes requires causality, but almost always requires simplicity. Anyone who has read a 350 page anthropology text will recognize that simplicity is rarely the goal of (at least some) social scientists.

  8. Speaking solely on the lawyer vs. social scientist issue: Much of what "traditional" lawyers (i.e. litigators) do is almost antithetical to the social scientist's project of "using evidence to explain why certain events occurred and to predict future events." One of the most basic rules of evidence, for example, is that in general you can't introduce the fact that a person did acts A, B, and C in the past as proof of the fact that he did act D at a later point in time. Many of the evidentiary skirmishes in criminal trials — which, although not dispositive, have a huge impact on a case — focus on motions for blanket exclusion of categories of forbidden "propensity evidence."

    This is a long-winded way of saying that it's not surprising that human rights groups, which often have an affinity for common law litigation, have a hard time viewing the facts as a social scientist would. That said, I'm not sure I agree with the generalization that advocates — and here I really mean lawyer/advocates — "are trained to stir emotions and to draw personal connections between international events and Western students, consumers, and families." I don't think that portraying the choice as between facts (social scientists) and emotions (advocates) does us any favors. It begs the question as to which methodology is superior (who's going to argue that "stirring emotions" is the best way forward?). The difference between a good advocate and a good social scientist is in how one uses the facts at hand. The important thing is to understand the limitations of each way of thinking.

  9. Pingback: On Luvungi, and the Problem of Evidence in Advocacy | Wronging Rights

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>