Skip to main content
Move the World.
What Happens If Scientists Lose Faith in Their Own Research?
Credit: George Hodan

In September 2016, the psychologist Dana Carney came forward with a confession: She no longer believed the findings of a high-profile study she co-authored in 2010 to be true. The study was about “power-posing” — a theory suggesting that powerful stances can psychologically and physiologically help one when under high-pressure situations. Carney’s co-author, Amy Cuddy, a psychologist at Harvard University, had earned much fame from power poses, and her 2012 TED talk on the topic is the second most watched talk of all time.

Amy Cuddy demonstrating her theory of
"power posing," purportedly altering
hormone levels through "authoritative"
postures.

Amy Cuddy demonstrating her theory of
"power posing," purportedly altering
hormone levels through "authoritative"
postures.

Carney, now based at the University of California, Berkeley, had, however, changed her mind. “I do not believe that 'power pose' effects are real,” she wrote on her website in 2016. The reason, she added, was that “since early 2015 the evidence has been mounting suggesting there is unlikely any embodied effect of nonverbal expansiveness.” Other researchers, it turned out, could not replicate the power pose results, and withering scrutiny of the Carney and Cuddy study by fellow scientists mounted.

Carney’s assertions and Cuddy’s responses were widely covered in the media. (Earlier this year, Forbes reported that Cuddy had successfully refuted criticism of the power-posing study.) And despite her own eventual refutation of the findings, Carney did not believe the original paper warranted a full retraction, because it “was conducted in good faith based on phenomena thought to be true at the time,” she told the research integrity blog Retraction Watch.

But in many researchers' minds, Carney's change of heart and the pointed questions surrounding power pose research typifies the replication crisis that has dogged fields like social psychology for years. Some researchers have gone so far as to suggest that most published research findings are false.

A 2016 survey by Nature of more than 1,500 scientists found that more than 70 percent of researchers failed to successfully reproduce another researcher’s work and more than half failed to reproduce their own. Psychology is one of the most affected disciplines, with studies suggesting that wearing red makes one more attractive, or that smiling makes people happier, proving difficult for follow-up researchers to reproduce.

Of course, some researchers have argued that the replication crisis is exaggerated. But even if that is the case, there really is no effective way for scientists to quickly and publicly inform colleagues that they are no longer confident in their published work. Public declarations like Carney’s are one way to go, but they are often difficult to track.

So an ambitious new effort, motivated by Carney’s move, is encouraging psychologists to own up to shortcomings in their published work via a website in the form of official loss-of-confidence statements — published at a single online clearinghouse for such confessions called the Loss of Confidence Project.

“An Idealistic Approach”

The aim is to simplify how such statements are reported as opposed to the current process, in which researchers bicker in back-and-forth commentaries and rebuttals, says Julia Rohrer, who studies personality psychology at the International Max Planck Research School on the Life Course in Berlin, Germany, and is one of three researchers working on the project.

“People will defend their scientific claims until their death,” Rohrer said. “As scientists, we should be aware that people are often wrong.” Carney’s move, for example, was generally well-received by psychologists who welcomed her transparency, she noted.

Rohrer and her colleagues, Tal Yarkoni of the University of Texas at Austin and Christopher Chabris, at the Geisinger Health System in Pennsylvania, are currently accepting submissions of loss-of-confidence statements, focusing on psychology studies — and with some ground rules: Authors submitting a loss-of-confidence statement, for example, are expected take primary responsibility for methodological or theoretical problems with their paper — otherwise, the entry goes into whistleblowing territory and is not eligible for publication. The researchers eventually plan to publish the statements in an academic paper, Rohrer said.

Only a handful of psychologists have submitted proposals so far, Rohrer said, despite an initial show of enthusiasm for the initiative. The dearth of submissions might be because it’s a bold step, Rohrer suggested, or because researchers don’t have the time to sift through all of their past work. Co-authors of a paper may also disagree over the decision to submit a statement, she adds — especially if they are at different stages in their career.

“We do know that more people have these stories,” Rohrer noted, but they are reluctant to share them. She also admitted that her project is an “idealistic approach to scientific self-correction,” since it relies on researchers being honest about their own work.

Still, more emphasis on self-correction will help to address the fact that some researchers whose work has not held up feel they have been treated unfairly, Rohrer said. Researchers are often too cautious about admitting errors because they assume the consequences will be negative, she adds, even though reactions to cases like Carney’s have been predominantly positive.

A Warning, Not a Punishment

Rohrer’s supervisor, Stefan Schmukle, a psychologist at the University of Leipzig, has submitted a loss-of-confidence statement based on a 2007 study he co-authored about variations in the lengths of ring and index fingers in men and women.

“It is important to communicate to other scholars that I have lost confidence in these specific results so that other researchers are aware that it might not make sense to follow this line of research,” says Schmukle.

Like Carney, Schmukle also feels his paper shouldn’t be retracted even though he doubts the study’s main findings are reproducible. “In my view, a retraction would be appropriate if the data were faked, if the statistical analysis was wrong, or something like that,” Schmukle says. The problem with his paper, however, is that some of the results, which in hindsight are important for understanding it, were not reported.

Participating in Rohrer’s project doesn’t necessarily imply that authors should retract or correct their papers, she noted — adding that she would consider it dysfunctional if papers were retracted every time authors admitted to mistakes, because that would be a disincentive for coming forward in the first place.

Marcus Munafò, a biological psychologist at the University of Bristol in the U.K. who has pulled a paper after spotting an error, made a similar case. “Whether or not to retract a paper is a tricky issue,” he said. “But I wouldn’t retract papers that report results that are almost certainly wrong but that were conducted in good faith.”

(For its part, Retraction Watch credits authors who retract their own papers by acknowledging them for “doing the right thing.")

Munafò, who co-authored a manifesto for reproducible science, described the new loss-of-confidence project as an ambitious effort with the potential to introduce a valuable self-correction mechanism. “The only real drawback is that it might not take off, perhaps because it’s a bit ahead of its time,” he said.

Although science is often described as self-correcting, Munafò added, many of the means of self-correction — such as replicating studies and reporting null results — are not currently prioritized.

 · · · 

“Take the Chance of Disclosing Now”

Some researchers are tackling loss-of-confidence incidents on their own. Rebecca Willén, an independent psychology and meta-science researcher who divides her time between Indonesia and Europe, publishes disclaimers about her studies on her own website. “I believe there is a difference between loss-of-confidence and retroactive disclosure statements,” said Willén, who added that she plans to participate in Rohrer’s project.

One difference, she noted, is that unlike loss-of-confidence statements, retroactive disclosures can also just say everything was reported transparently in the study, as Willén does for some of her papers listed on her site. “A few of my disclosure statements were made public already in my Ph.D. thesis, while a couple of others were not published until just recently.”

Another psychologist who has acknowledged shortcomings in his published work is Will Gervais of the University of Kentucky. In 2012, a flashy study published in the journal Science by Gervais and colleagues — which attracted several media headlines — claimed that thinking analytically can make people less religious.

But when outside researchers took a second look at his paper, it failed to hold up. The replicators praised Gervais and his team for their openness with their data, and Gervais published a statement about the replication study and his personal perspective on his own website. In it, he wrote that his “methodological awakening” started around 2012.

Gervais says the loss-of-confidence project would seem to offer researchers an opportunity to publicly distance themselves from previous work that they've come to believe is not robust. “This is certainly better than just holding your breath and hoping nobody tries to replicate it,” he said.

And Willén agreed, adding that this is a good time to consider coming forward with such admissions, because the atmosphere is charitable. “Take the chance of disclosing now,” she said. “Once this phase is over, it’s more likely that retroactive disclosures will result in negative consequences for your career.”

Daniël Lakens, an experimental psychologist at Eindhoven University of Technology in the Netherlands, included a loss-of-confidence statement within a commentary he wrote after a meta-analysis and a large replication study raised doubts about findings in his own paper. “Scientific papers are regrettably not like software,” he said. “I can update software I write, and the latest version is always the best version.”

Opening Up the “File Drawer”

In recent years, some academic publishers have made efforts to turn scholarly articles into living documents but most papers remain static, often containing outdated or refuted information. One problem psychology and many other disciplines suffer from is publication bias, where academic journals favor positive results over negative ones.

This means negative findings are stashed away by researchers and never published — also known as the “file-drawer problem” — resulting in a skewed picture of whether a phenomenon exists in scholarly literature.

To solve that problem, a journal called Meta-Psychology was launched last year. The journal, which has no publication fee, welcomes papers reporting negative findings — dubbed “file-drawer reports” — and replication reports in which authors confirm or refute their own or somebody else’s results.

Whatever the method, owning up to mistakes is not easy. Willén, for instance, said she became unpopular in her department for struggling against what she calls “questionable practices” during her Ph.D. years. “Publishing my retroactive disclosure statements were simply the result of many years of inner struggle," she said. "Follow my conscience or be loyal to people I deeply respect and would love to continue working with? My conscience finally won that battle.”

Dalmeet Singh Chawla is a freelance science journalist based in London. Between January 2016 and January 2017, he worked as a full-time reporter for Retraction Watch. This article was first published on Undark.

Up Next

Genetics
This Insect Swiped a Gene From Plants To Help Itself Survive
gene transfer
Genetics
This Insect Swiped a Gene From Plants To Help Itself Survive
A plant gene has been discovered in whiteflies, marking the first known example of horizontal gene transfer of a functional gene between plants and insects.

A plant gene has been discovered in whiteflies, marking the first known example of horizontal gene transfer of a functional gene between plants and insects.

Citizen Science
Citizen Scientists Helped NASA Map Our Cosmic Backyard
brown dwarfs
Citizen Science
Citizen Scientists Helped NASA Map Our Cosmic Backyard
NASA was able to create a highly detailed 3D map of nearby brown dwarfs thanks to the efforts of 150,000 citizen scientists.

NASA was able to create a highly detailed 3D map of nearby brown dwarfs thanks to the efforts of 150,000 citizen scientists.

Health
New Lasers See Into The Brain to Detect Concussions
concussion test
Health
New Lasers See Into The Brain to Detect Concussions
Researchers are attempting to improve the age-old concussion test with a device that can measure CCO levels in the brain using infrared lasers.

Researchers are attempting to improve the age-old concussion test with a device that can measure CCO levels in the brain using infrared lasers.

Future Forward
Reversing Extinction for the Northern White Rhino
Reversing Extinction for the Northern White Rhino
Watch Now
Future Forward
Reversing Extinction for the Northern White Rhino
The Northern white rhino is functionally extinct, meaning the two living females of the species are infertile. This scientist from the San Diego Zoo may have the solution to save the last of this extinct species, by tapping into her frozen zoo.
Watch Now

The Northern White Rhino is functionally extinct - but scientists at the San Diego Zoo are launching an unprecedented effort to recreate the species and save it from extinction. The earth is undergoing an unprecedented wave of extinctions, with estimates that between 200 and 2,000 species go extinct every day. One on the brink is the Northern White Rhinoceros, which after waves of poaching only has two known members in the...

Dispatches
Personal Genetics Might Solve the Opioid Crisis – and the Pain Crisis
Personal Genetics Might Solve the Opioid Crisis – and the Pain Crisis
Dispatches
Personal Genetics Might Solve the Opioid Crisis – and the Pain Crisis
Why does pain hurt more for some people? Why do others feel nothing at all?
By Erin Young

Why does pain hurt more for some people? Why do others feel nothing at all?

Dispatches
Can DIY Science (Finally) Cut the Cost of Insulin?
Can DIY Science (Finally) Cut the Cost of Insulin?
Dispatches
Can DIY Science (Finally) Cut the Cost of Insulin?
A century after its discovery, insulin is still incredibly expensive, but DIY bio-manufacturing could change that...

A century after its discovery, insulin is still incredibly expensive, but DIY bio-manufacturing could change that in a big way.

Dispatches
We Found the Oldest Human Virus: It's Familiar (but Weird)
Ancient Human Viruses Weird and Familiar
Dispatches
We Found the Oldest Human Virus: It's Familiar (but Weird)
The discovery cracks open a 7,000-year history of human-virus warfare. And it's raising weird questions.

The discovery cracks open a 7,000-year history of human-virus warfare. And it's raising weird questions.

The New Space Race
A Delivery Service for the Moon
A Delivery Service for the Moon
Watch Now
The New Space Race
A Delivery Service for the Moon
This startup wants to offer the world insanely accurate shipping to the moon.
Watch Now

Landing on the moon has always been an inaccurate pursuit. But Astrobotic has fixed that problem. The company’s unique GPS system allows it to land spacecraft within meters—rather than kilometers—of the intended target. And now they’re using the tech to offer the world’s first delivery service to the moon.