Skip to main content
Move the World.
What Happens If Scientists Lose Faith in Their Own Research?
Credit: George Hodan

In September 2016, the psychologist Dana Carney came forward with a confession: She no longer believed the findings of a high-profile study she co-authored in 2010 to be true. The study was about “power-posing” — a theory suggesting that powerful stances can psychologically and physiologically help one when under high-pressure situations. Carney’s co-author, Amy Cuddy, a psychologist at Harvard University, had earned much fame from power poses, and her 2012 TED talk on the topic is the second most watched talk of all time.

Amy Cuddy demonstrating her theory of
"power posing," purportedly altering
hormone levels through "authoritative"
postures.
Amy Cuddy demonstrating her theory of "power posing," purportedly altering hormone levels through "authoritative" postures.

Carney, now based at the University of California, Berkeley, had, however, changed her mind. “I do not believe that 'power pose' effects are real,” she wrote on her website in 2016. The reason, she added, was that “since early 2015 the evidence has been mounting suggesting there is unlikely any embodied effect of nonverbal expansiveness.” Other researchers, it turned out, could not replicate the power pose results, and withering scrutiny of the Carney and Cuddy study by fellow scientists mounted.

Carney’s assertions and Cuddy’s responses were widely covered in the media. (Earlier this year, Forbes reported that Cuddy had successfully refuted criticism of the power-posing study.) And despite her own eventual refutation of the findings, Carney did not believe the original paper warranted a full retraction, because it “was conducted in good faith based on phenomena thought to be true at the time,” she told the research integrity blog Retraction Watch.

But in many researchers' minds, Carney's change of heart and the pointed questions surrounding power pose research typifies the replication crisis that has dogged fields like social psychology for years. Some researchers have gone so far as to suggest that most published research findings are false.

A 2016 survey by Nature of more than 1,500 scientists found that more than 70 percent of researchers failed to successfully reproduce another researcher’s work and more than half failed to reproduce their own. Psychology is one of the most affected disciplines, with studies suggesting that wearing red makes one more attractive, or that smiling makes people happier, proving difficult for follow-up researchers to reproduce.

Of course, some researchers have argued that the replication crisis is exaggerated. But even if that is the case, there really is no effective way for scientists to quickly and publicly inform colleagues that they are no longer confident in their published work. Public declarations like Carney’s are one way to go, but they are often difficult to track.

So an ambitious new effort, motivated by Carney’s move, is encouraging psychologists to own up to shortcomings in their published work via a website in the form of official loss-of-confidence statements — published at a single online clearinghouse for such confessions called the Loss of Confidence Project.

“An Idealistic Approach”

The aim is to simplify how such statements are reported as opposed to the current process, in which researchers bicker in back-and-forth commentaries and rebuttals, says Julia Rohrer, who studies personality psychology at the International Max Planck Research School on the Life Course in Berlin, Germany, and is one of three researchers working on the project.

“People will defend their scientific claims until their death,” Rohrer said. “As scientists, we should be aware that people are often wrong.” Carney’s move, for example, was generally well-received by psychologists who welcomed her transparency, she noted.

Rohrer and her colleagues, Tal Yarkoni of the University of Texas at Austin and Christopher Chabris, at the Geisinger Health System in Pennsylvania, are currently accepting submissions of loss-of-confidence statements, focusing on psychology studies — and with some ground rules: Authors submitting a loss-of-confidence statement, for example, are expected take primary responsibility for methodological or theoretical problems with their paper — otherwise, the entry goes into whistleblowing territory and is not eligible for publication. The researchers eventually plan to publish the statements in an academic paper, Rohrer said.

Only a handful of psychologists have submitted proposals so far, Rohrer said, despite an initial show of enthusiasm for the initiative. The dearth of submissions might be because it’s a bold step, Rohrer suggested, or because researchers don’t have the time to sift through all of their past work. Co-authors of a paper may also disagree over the decision to submit a statement, she adds — especially if they are at different stages in their career.

“We do know that more people have these stories,” Rohrer noted, but they are reluctant to share them. She also admitted that her project is an “idealistic approach to scientific self-correction,” since it relies on researchers being honest about their own work.

Still, more emphasis on self-correction will help to address the fact that some researchers whose work has not held up feel they have been treated unfairly, Rohrer said. Researchers are often too cautious about admitting errors because they assume the consequences will be negative, she adds, even though reactions to cases like Carney’s have been predominantly positive.

A Warning, Not a Punishment

Rohrer’s supervisor, Stefan Schmukle, a psychologist at the University of Leipzig, has submitted a loss-of-confidence statement based on a 2007 study he co-authored about variations in the lengths of ring and index fingers in men and women.

“It is important to communicate to other scholars that I have lost confidence in these specific results so that other researchers are aware that it might not make sense to follow this line of research,” says Schmukle.

Like Carney, Schmukle also feels his paper shouldn’t be retracted even though he doubts the study’s main findings are reproducible. “In my view, a retraction would be appropriate if the data were faked, if the statistical analysis was wrong, or something like that,” Schmukle says. The problem with his paper, however, is that some of the results, which in hindsight are important for understanding it, were not reported.

Participating in Rohrer’s project doesn’t necessarily imply that authors should retract or correct their papers, she noted — adding that she would consider it dysfunctional if papers were retracted every time authors admitted to mistakes, because that would be a disincentive for coming forward in the first place.

Marcus Munafò, a biological psychologist at the University of Bristol in the U.K. who has pulled a paper after spotting an error, made a similar case. “Whether or not to retract a paper is a tricky issue,” he said. “But I wouldn’t retract papers that report results that are almost certainly wrong but that were conducted in good faith.”

(For its part, Retraction Watch credits authors who retract their own papers by acknowledging them for “doing the right thing.")

Munafò, who co-authored a manifesto for reproducible science, described the new loss-of-confidence project as an ambitious effort with the potential to introduce a valuable self-correction mechanism. “The only real drawback is that it might not take off, perhaps because it’s a bit ahead of its time,” he said.

Although science is often described as self-correcting, Munafò added, many of the means of self-correction — such as replicating studies and reporting null results — are not currently prioritized.

 · · · 

“Take the Chance of Disclosing Now”

Some researchers are tackling loss-of-confidence incidents on their own. Rebecca Willén, an independent psychology and meta-science researcher who divides her time between Indonesia and Europe, publishes disclaimers about her studies on her own website. “I believe there is a difference between loss-of-confidence and retroactive disclosure statements,” said Willén, who added that she plans to participate in Rohrer’s project.

One difference, she noted, is that unlike loss-of-confidence statements, retroactive disclosures can also just say everything was reported transparently in the study, as Willén does for some of her papers listed on her site. “A few of my disclosure statements were made public already in my Ph.D. thesis, while a couple of others were not published until just recently.”

Another psychologist who has acknowledged shortcomings in his published work is Will Gervais of the University of Kentucky. In 2012, a flashy study published in the journal Science by Gervais and colleagues — which attracted several media headlines — claimed that thinking analytically can make people less religious.

But when outside researchers took a second look at his paper, it failed to hold up. The replicators praised Gervais and his team for their openness with their data, and Gervais published a statement about the replication study and his personal perspective on his own website. In it, he wrote that his “methodological awakening” started around 2012.

Gervais says the loss-of-confidence project would seem to offer researchers an opportunity to publicly distance themselves from previous work that they've come to believe is not robust. “This is certainly better than just holding your breath and hoping nobody tries to replicate it,” he said.

And Willén agreed, adding that this is a good time to consider coming forward with such admissions, because the atmosphere is charitable. “Take the chance of disclosing now,” she said. “Once this phase is over, it’s more likely that retroactive disclosures will result in negative consequences for your career.”

Daniël Lakens, an experimental psychologist at Eindhoven University of Technology in the Netherlands, included a loss-of-confidence statement within a commentary he wrote after a meta-analysis and a large replication study raised doubts about findings in his own paper. “Scientific papers are regrettably not like software,” he said. “I can update software I write, and the latest version is always the best version.”

Opening Up the “File Drawer”

In recent years, some academic publishers have made efforts to turn scholarly articles into living documents but most papers remain static, often containing outdated or refuted information. One problem psychology and many other disciplines suffer from is publication bias, where academic journals favor positive results over negative ones.

This means negative findings are stashed away by researchers and never published — also known as the “file-drawer problem” — resulting in a skewed picture of whether a phenomenon exists in scholarly literature.

To solve that problem, a journal called Meta-Psychology was launched last year. The journal, which has no publication fee, welcomes papers reporting negative findings — dubbed “file-drawer reports” — and replication reports in which authors confirm or refute their own or somebody else’s results.

Whatever the method, owning up to mistakes is not easy. Willén, for instance, said she became unpopular in her department for struggling against what she calls “questionable practices” during her Ph.D. years. “Publishing my retroactive disclosure statements were simply the result of many years of inner struggle," she said. "Follow my conscience or be loyal to people I deeply respect and would love to continue working with? My conscience finally won that battle.”

Dalmeet Singh Chawla is a freelance science journalist based in London. Between January 2016 and January 2017, he worked as a full-time reporter for Retraction Watch. This article was first published on Undark.

More About

Disease X
Preparing for Outbreak: Fighting the World's Deadliest Diseases
Preparing for Outbreak: Fighting the World's Deadliest Diseases
Watch Now
Disease X
Preparing for Outbreak: Fighting the World's Deadliest Diseases
The World Health Organization has compiled a list of the most dangerous diseases that could strike next year.
Watch Now

Recently the media was abuzz with talk of ‘Disease X,’ a mysterious illness that could spread across the world. And then...nothing happened. It turns out that people had misinterpreted the World Health Organization’s List of Blueprint priority diseases, which identifies the world’s most dangerous diseases in terms of potential for outbreaks or epidemics. There are diseases that are already known - like Ebola, MERS or SARS - and then a…

Dispatches
Paralyzed Mice Walk Again After Breakthrough Treatment
Paralyzed Mice Walk Again After Breakthrough Treatment
Dispatches
Paralyzed Mice Walk Again After Breakthrough Treatment
One small step for a mouse, perhaps one giant leap for treating spinal injuries.
By Dan Bier

One small step for a mouse, perhaps one giant leap for treating spinal injuries.

Superhuman
Hunting Down His Son’s Killer
Hunting Down His Son’s Killer
Watch Now
Superhuman
Hunting Down His Son’s Killer
For years, there was no diagnosis, no treatment, and no cure — because his son's disease had never been seen before. That wasn't going to stop this dad.
Watch Now

What do you do when there are no experts to turn to? For computer scientist Matt Might, the answer was obvious: you become the expert. When doctors couldn't figure out his son's disease, he found a way to crack the code. Matt's son, Bertrand, suffers from an extremely rare genetic disease, called NGLY1 deficiency, which causes chronic seizures, liver problems, and developmental delays. In fact, it was so rare that…

Dispatches
We Found the Oldest Human Virus: It's Familiar (but Weird)
We Found the Oldest Human Virus: It's Familiar (but Weird)
Dispatches
We Found the Oldest Human Virus: It's Familiar (but Weird)
The discovery cracks open a 7,000-year history of human-virus warfare. And it's raising weird questions.
By Dan Bier

The discovery cracks open a 7,000-year history of human-virus warfare. And it's raising weird questions.

Dispatches
Glowing Cancer Cells Could Find Hidden Tumors (And Replace Mammograms)
Glowing Cancer Cells Could Find Hidden Tumors (And Replace Mammograms)
Dispatches
Glowing Cancer Cells Could Find Hidden Tumors (And Replace Mammograms)
A new pill can make cancer cells glow under infrared light, and it could eliminate for mammograms.
By Dan Bier

A new pill can make cancer cells glow under infrared light, and it could eliminate for mammograms.

The New Space Race
Can We Make It In Space?
Can We Make It In Space?
Watch Now
The New Space Race
Can We Make It In Space?
What if one day, everything in space was made in space? 3D printing may hold the answer.
Watch Now

NASA intern turned Silicon Valley entrepreneur, Jason Dunn, saw what was holding humans back from colonizing outer space...and decided to do something about it. With his company Made in Space’s cutting-edge 3D printer, astronauts can break their reliance on costly resupply missions from Earth and—for the first time ever—build new supplies for themselves in space. Dunn and his team believe their invention will usher in a new era of dramatic…

Superhuman
Open Sourcing the Brain
Open Sourcing the Brain
Watch Now
Superhuman
Open Sourcing the Brain
Open BCI has developed a 3D-printed headset that allows your brain to interact with computers in amazing ways.
Watch Now

OpenBCI has developed a 3D-printed headset that allows our brains to interact with software. Want to measure the effect of meditation on your brain? It's possible. Want to control a prosthetic limb with your mind? It's possible. Right now, the only thing OpenBCI's tech can't do are the things we haven't thought of.