Appendix G — Kahneman’s Open Letter to Priming Researchers
G.1 Background
This letter has been archived by Nature, and can be currently found here. If the link stops working, please email me and I’ll fix it.
As mentioned in Chapter 14 , this letter was sent by psychologist Daniel Kahneman to psychologist John Bargh and others in the field of priming research. The letter was sent at a time where the entire field of not only priming, but psychology in general came under intense scrutiny in the form of a Replication Crisis, which since encouraged other fields to deal with their own crises of reproducibility, transparency, and replication. Today, the STEM fields as well are dealing with their own similar crises, particularly around fabrication or manipulation of images, which have become easier to detect with image recognition software and keen scientific integrity consultants like the formidable Dr Elisabeth Bik.
The letter sent by Kahneman is instructive and prescient. He correctly predicted the train wreck that would become the Replication Crisis, and while his daisy chain proposal may not have been realized among labs studying priming, it was reflected in similar large replication efforts such as Many Labs and Many Economists that continue to this day. Below, you will find the letter in full.
G.2 The Letter in Full
From: Daniel Kahneman
Sent: Wednesday, September 26, 2012 9:32 AM
Subject: A proposal to deal with questions about priming effects
Dear colleagues,
I write this letter to a collection of people who were described to me (mostly by John Bargh) as students of social priming. There were names on the list that I could not match to an email. Please pass it on to anyone else you think might be relevant.
As all of you know, of course, questions have been raised about the robustness of priming results. The storm of doubts is fed by several sources, including the recent exposure of fraudulent researchers, general concerns with replicability that affect many disciplines, multiple reported failures to replicate salient results in the priming literature, and the growing belief in the existence of a pervasive file drawer problem that undermines two methodological pillars of your field: the preference for conceptual over literal replication and the use of meta-analysis. Objective observers will point out that the problem could well be more severe in your field than in other branches of experimental psychology, because every priming study involves the invention of a new experimental situation.
For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. These people have now attached a question mark to the field, and it is your responsibility to remove it.
I am not a member of your community, and all I have personally at stake is that I recently wrote a book that emphasizes priming research as a new approach to the study of associative memory – the core of what dual-system theorists call System 1. Count me as a general believer. I also believe in a point that John Bargh made in his response to Cleeremans, that priming effects are subtle and that their design requires high-level skills. I am skeptical about replications by investigators new to priming research, who may not be attuned to the subtlety of the conditions under which priming effects are observed, or to the ease with which these effects can be undermined.
My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction.
I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is self-defeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field.
The following is just an example of such a protocol:
- Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants.
- Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate.
- The board makes a public commitment to these five specific effects
- Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc.
- Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed).
- Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work
- Run enough subjects to guarantee power (probably more than in the original study)
- Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others.
- Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others.
This is something you could do quickly, and relatively cheaply. The main costs are 10 trips, and funds to cover these costs would be easy to get (I have checked). You would have to be careful in selecting laboratories and results to maximize credibility, and every step of the procedure should be open and documented. The unusually high openness to scrutiny may be annoying and even offensive, but it is a small price to pay for the big prize of restored credibility.
Success (say, replication of four of the five positive priming results) would immediately rehabilitate the field. Importantly, success would also provide an effective challenge to the adequacy of outsiders’ replications. A publicly announced and open effort would be credible among colleagues at large, because it would show that you are sufficiently confident in your results to take a risk.
More ambiguous results would be painful, of course, but they would still protect the reputations of scholars who sincerely believe in their work – even if they are sometimes wrong.
The protocol I outlined is just an example of something you might do. The main point of my letter is that you should do something, and that you must do it collectively. No single individual will be able to overcome the doubts, but if you act as a group and avoid defensiveness you will be credible.
All best,