«

»

May 03

Conceptual Replication

Here is the short commentary that I wrote about conceptual replication for the May issue of The Psychologist (PDF), published by the British Psychological Society (which is why they spell “behaviour” that way — although as a Canadian, I don’t really object). They have an entire section in the issue devoted to questions of replication in psychology (I’ll have a bit more to say about that later), but they are all worth a read.

 

There is no substitute for direct replication – if you cannot reproduce the same result using the same methods then you cannot have a cumulative science. But conceptual replication also has a very important role to play in psychological science. What is conceptual replication? It’s when instead of replicating the exact same experiment in exactly the same way, we test the experiment’s underlying hypothesis using different methods.

One reason conceptual replication is important is that psychologists aren’t always dealing with objectively defined quantities like 2mg of magnesium; instead we have to operationalise conceptual variables in a concrete manner. For instance, if we want to test the effects of good mood on helping behaviour, what counts as a good mood and what counts as helping behaviour? These are not objectively defined quantities, so we have to decide on something reasonable. For example, in the 1970s Alice Isen found that people were far more likely to help someone who had dropped some papers after they had found a dime in a phone booth (Isen & Levin, 1972). But we have even more confidence in this result now that it’s been conceptually replicated: helping has been found to increase not only after finding money, but after reflecting on happy memories, doing well on a test, or receiving cookies. Numerous different ways of measuring helping behaviour have been used as well. In other words, even if nobody had ever tried to directly replicate the original research, the conceptual replications give us confidence that the underlying hypothesis – that a positive mood increases helping behaviour – is correct.

In the recent debate (see tinyurl.com/cfkl2gk and tinyurl.com/7ffztux) about the failure to replicate John Bargh’s finding that priming the elderly stereotype leads people to later walk more slowly as they leave the experiment, there has been a great deal of misunderstanding because of confusion between direct replication and conceptual replication. While it is true that there have been few direct replications of that finding, the underlying hypothesis that ‘activating a stereotype in the perceiver without the perceiver’s knowledge would increase the perceiver’s own behavioural tendencies to act in line with the content of that stereotype’ has been replicated many times. It was even conceptually replicated in the very same series of published studies in which the original ‘slow walking’ study was published, as well as by independent researchers at different universities around the world. Taking these conceptual replications into account, most psychologists are not nearly as troubled by a single failure to replicate the result as it may appear that they should be.

 

In brief, the commentary focuses on the inherent subjectivity involved in addressing psychological questions — we can’t always study human behavior the same way we study physics. Conceptual replication helps us test a hypothesis in several different ways to make sure that we’re really testing what we think we’re testing. Direct replication is certainly an important part of the process, but if we rely solely on reproducing the same result over and over again then we run the risk of making the same mistakes over and over again, too. Conceptual replication helps us mitigate that risk, as I cover in this post on the history of cognitive dissonance.

Related posts:

4 comments

5 pings

Skip to comment form

  1. Emily

    I wonder your thoughts on questions of replication in fields like anthropology or even developmental psychology where there is a lot of “judgement” in data production (I’m thinking Mark Hauser). I think in the Hauser case there was a defense, if a weak one, that he was really good at figuring out what the monkeys were looking at, so the reason it wasn’t replicable was because everyone else was worse at monkey observation. One possible standard for replication is that a reasonable person in the field should be able to replicate your results — it shouldn’t just be limited to you.

    1. Dave Nussbaum

      Hi Emily, I think the basic standard that a reasonable person in the field should be able to replicate your results is a good one. If nobody else can replicate your results then you’re not really doing science — with very rare exceptions. Even if you wanted to play devil’s advocate on Hauser’s behalf, he should at least be able to describe the methods he uses to do his “figuring out” so that someone else could do it too. If it can’t be replicated, then it doesn’t mean it’s not useful or interesting — many of the most important contributions to science were not scientific in and of themselves, but were by people good at figuring things and people out — but I don’t think it’s science.

  2. Mike

    A very interesting analysis of the differentiation between “conceptual Replication” and “replication”. I would love to hear more about a deeper designation of the Conceptual Replication. We currently have ” It’s when instead of replicating the exact same experiment in exactly the same way, we test the experiment’s underlying hypothesis using different methods.” I imagine by “different methods” you intend to maintain the more important conceptual items under analysis of a particular hypothesis. So for some hypothesis “Given X, then Y”, we want to probe further into the hypothesis and find out what variables can be replicated in different ways. I imagine, in-so-doing, we run the risk of simplifying certain psychological concepts further than we might like. I do not suggest that this will always be the case. However, I maintain that a discipline like psychology would be prone to such a result. A very interesting and fun read. Thank you Dave.

    1. Dave Nussbaum

      Thanks for the comment Mike! I think I follow you until you say that we “run the risk of simply
      fying certain psychological concepts further than we might like.” I’m not sure exactly what you meant by that, so if you could say a little more it would be very helpful. Thanks!

  1. Replicating Dissonance » Random Assignment

    [...] « Conceptual Replication [...]

  2. Dave Nussbaum on “Conceptual Replication” « ethicalonion

    [...] http://www.davenussbaum.com/conceptual-replication-part-i/ Share this:ShareEmailRedditTwitterFacebookLinkedInLike this:LikeBe the first to like this post. [...]

  3. The Scienceblogging Weekly (May 11, 2012) | The Best Parties Site

    [...] Conceptual Replication by Dave Nussbaum: There is no substitute for direct replication – if you cannot reproduce the same result using the same methods then you cannot have a cumulative science. But conceptual replication also has a very important role to play in psychological science. What is conceptual replication? It’s when instead of replicating the exact same experiment in exactly the same way, we test the experiment’s underlying hypothesis using different methods… [...]

  4. The Scienceblogging Weekly (May 11, 2012) - Interior Designer

    [...] Conceptual Replication by Dave Nussbaum: There is no substitute for direct replication – if you cannot reproduce the same result using the same methods then you cannot have a cumulative science. But conceptual replication also has a very important role to play in psychological science. What is conceptual replication? It’s when instead of replicating the exact same experiment in exactly the same way, we test the experiment’s underlying hypothesis using different methods… [...]

  5. What counts as a successful or failed replication? « The Hardest Science

    [...] empirical results vs. conclusions about theories — goes to the heart of the discussion about direct and conceptual replication. Direct replication means that you reproduce, as faithfully as possible, the procedures and [...]

Leave a Reply