Skip to main content
Move the World.

In 2014, a groundbreaking study was released. Researchers had followed activists from the Los Angeles LGBT Center as they knocked on doors of California voters. The study showed that gay canvassers were able to significantly increase support for gay marriage by simply having a single, face-to-face conversation. And, the study claimed, not only were those conversations surprisingly effective, the impact was also long lasting.

The study got a lot of attention from national media and made waves in the research world where the prevailing thought is that changing beliefs is notoriously difficult.

Among the researchers who took notice were Josh Kalla at Berkeley and David Broockman at Stanford. Kalla and Broockman were fascinated by the study and wanted to run a version of it themselves. They partnered with the LA LGBT Center to launch a new study in Florida to test if canvassers could shift opinion on transgender rights. To design their own study, they dove into the original research and methodology in detail. And as they did, they made a startling realization: the results of the original study were completely fabricated. Kalla and Broockman found inconsistencies and inaccuracies across the methodology and the results. When they released their findings, Donald Green, co-author of the original study immediately retracted the paper. Michael LaCour, the paper’s other co-author, has stood by his research.

The retraction resulted in a frenzy of corrections by media outlets. While the collapse of the original research grabbed headlines, Kalla and Broockman continued to work on the LGBT Center’s Florida canvassing campaign -- essentially doing the research the original study claimed to do. In a fascinating twist, their study revealed that, while important details from the original “findings” weren’t accurate, the idea that a conversation can make a significant and lasting impact on a person’s opinion on an issue, is backed up by the numbers.

We sat down with Josh Kalla to talk through their research and how they think it fits into the current conversation on how people’s opinions evolve on some of most hotly debated issues.

Headquarters of the Los Angeles LGBT Center

Michael O’Shea: Your story related to the work that you did with Dave Fleischer and the LA LGBT center has a lot of interesting twists and turns, but let's start by focusing on the research that you published about the canvassing effort. What exactly did you set out to find with your research?

Josh Kalla: Attitude change is something that psychologists and political scientists, and the social sciences more broadly, have really been interested in exploring since the rise of modern social science, in the early 20th century. It's something that's been debated, whether it's easy, whether it's hard, what can cause it, when can it happen.

For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice.

One area where it's been hotly researched is on prejudice, really beginning with studies around racism, but you also have a lot of interest in prejudice reduction against handicapped people, against Muslims, against transgender people. For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice in American society, or society more generally. These are deeply ingrained beliefs that are personally held. You tend to acquire them in childhood. You're socialized into them. Your religion, your social status, all reinforce them. These are things that are hard, if not impossible, to change. Our best hope to reduce prejudice in society is generational replacement. Maybe our children will be better, but people are born with these prejudices. They develop them. They're hard to change. That's the academic background.

Along comes Dave Fleischer with this innovative technique and story to try to reduce prejudice. David Broockman, my colleague, and I were fascinated by this work. Before we got started, we went to canvas with Dave and his team and thought there was really something special going on there, something worth studying. We set out to conduct this academic experiment to see whether or not the work that the Leadership LAB was doing was actually effective at reducing prejudice towards transgender people.

MO:  How do you go about actually measuring that?

JK: Measuring prejudice is hard. Most people don't want to openly admit that they're prejudiced, particularly if they know that they're being studied. They tend to give the "socially desirable" answer. So we conduct this academic experiment where we really separate out the research side of things from the work that Dave Fleischer and his group were doing.

First we send around 60,000 people in South Florida a piece of mail inviting them to participate in a survey sponsored by our universities, Berkeley and Stanford. This survey had a whole range of questions in them. Everything from "What's your favorite beach in Miami?", to "What do you think of the latest films?", to, at the time, I think there were questions about the early 2016 presidential election. Sprinkled in there were a few questions about transgender rights. There might have been half a dozen questions, out of 50 or so questions in that survey, to make it not seem like this was a survey about transgender rights. Then, among the people who took this survey, we got their baseline measures, and we randomly assigned half of them to be canvassed by Dave and his group and to get the full, 10 or so minute conversation about transgender people and reducing prejudice. The other half of the people got what we call a placebo. This is a group that we're not trying to change their attitudes about transgender people or prejudice, we just want to know, who are the people we could have talked about that, but let's talk to them about something else. Then, afterwards, we follow up with an additional survey that lets us compare whether the change over time in the group that got the trans rights conversation is different than the group that got the other, unrelated conversation. We do it both immediately and then over time, to see whether this was a one-off effect (which is a big fear in the prejudice reduction literature) or whether this is a longer term, deeper change in attitudes.

Dave Fleischer in the New York Times Magazine

MO: So what did you guys find?

JK: We found very large and long lasting effects. We found that the conversations reduced the amount of prejudice towards transgender people. It's hard to put it on a scale that's comprehensible, but the most comprehensible scale is the average reduction that occurred in society, more generally, towards gay men and lesbians from, I want to say, 1996 to 2012. That general reduction is what we see happening in that Miami study, with just a 10 minute conversation.

MO: It seems a lot of people grasped onto the “findings” of the original study because it makes sense on the surface. Talking to somebody of a particular group could reduce prejudice towards that group. So since that’s not the whole story, what's the insight here? It's just simply that having a face to face conversation and presenting people with ideas that are counter to what they already believe, regardless of who it comes from, increases empathy?

The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy.

JK: The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy. The conversation was around getting the canvasser to share a story of a time that they faced prejudice and discrimination. If they were transgender, they may talk about it in terms of that. If they were LGB, they may talk about it in terms of that. If they were an ally, they might talk about all the other prejudice that exists in society. Whether you're a woman, whether you're a minority, whether you are friends with people who face prejudice. Prejudice is so widespread that almost everyone can talk about a personal experience. Getting the person being canvassed to share a similar story and trying to, through analogy, make that connection between, "Hey, here's a time when you felt discriminated against or when a loved one felt discriminated against. It feels really lousy. We all, collectively, think this is bad and we want to avoid this. Here's a group that faces a whole bunch of prejudice in their lives. You probably haven't learned much about this group. You probably haven't heard much about this group." This might have been the first conversation that people had ever had about transgender people or transgender rights. I think part of the effect might have just been educational, of demystifying this group you had never heard of before, then the other part is this empathy building.

Dave Fleischer and his team of canvassers had 15,000 one-on-one conversations with voters

MO: It seems like a lot of things that I've read about research that's adjacent to your work seems to take a much more pessimistic view of our ability to change our minds. There’s a lot of talk about confirmation bias and people seeking out opinions that are the same as their own. Often, the feeling that you're left with is, and as you said this was the prevailing thought for a long time, that it is very hard to change people's minds. Your research goes in direct conflict with that. Can you talk about how you think this research fits into that larger discussion about how people form and change their opinions?

It's always hard in science to try to explain why something happens.

JK: I think it's important to keep in mind the issue of transgender rights, where it is a fairly new issue for a lot of people, as opposed to racism, or abortion, or whether or not you're a Democrat or Republican. Those, I think, are probably conceptually very different issues for many people, particularly at the time when we were doing the study. I think that's one more straight forward explanation as to why this study was different. It's just this particular issue was not something that people had much exposure to before. It was not tied to other parts of their identity in a way that other issues are. David and I have another paper with the LA LGBT Center, trying to change attitudes around abortion and reducing stigma towards women who have had abortion and there we find no effect. We find that similar strategy, it's the same group, same leadership, they were not able to change attitudes around abortion. Again, there are a whole lot of explanations you could give. Maybe it wasn't the right script. Maybe if you were to do it again, it would be more effective. It's always hard in science to try to explain why something happens, but at least the two data points that we have is trans rights, big effects, abortion, currently no effects. Abortion is this old issue that's been on the political agenda for a long time. Most people in American society have thought about abortion, or pregnancy, and sex, and birth control, and that constellation of issues at some point in their lives. But trans rights are different. Me just defining what it means to be transgender might be the first time that has ever happened for someone, in a way that abortion, when you're talking to a middle aged person, is probably not the first time that's ever happened. I think that's one way to reconcile the broader literature with these findings, but I also think Dave Fleischer and his group is potentially on to something with their canvassing. I spend a lot of time with various political groups, and go canvassing with them, and look at their persuasive efforts, and no one really does what LA LGBT Center does.


Related video:

Up Next

Public Health
More Than $1 Million in Prizes For Coronavirus Solutions
coronavirus solutions
Public Health
More Than $1 Million in Prizes For Coronavirus Solutions
Emergent Ventures is awarding more than $1 million in coronavirus prizes to people who make significant progress in combating COVID-19.

Emergent Ventures is awarding more than $1 million in coronavirus prizes to people who make significant progress in combating COVID-19.

#Fixingjustice - Sentencing Reform
A Mother’s Wish for Her Son’s Killer
A Mother’s Wish for Her Son’s Killer
#Fixingjustice - Sentencing Reform
A Mother’s Wish for Her Son’s Killer
DeVitta Briscoe never had a chance to request a lighter sentence for the man who shot her son.

DeVitta Briscoe never had a chance to request a lighter sentence for the man who shot her son.

#fixingjustice - Policing
Do We Need More Police Or Better Police?
Do We Need More Police Or Better Police?
#fixingjustice - Policing
Do We Need More Police Or Better Police?
American cities are safer than they used to be, but they’re still quite violent, and many economists think they’re...

American cities are safer than they used to be, but they’re still quite violent, and many economists think they’re under-policed. More police could help reduce crime, but only if people trust them to do a good job.

#fixingjustice - Policing
Community Policing is Back in Vogue. But Does it Work?
Community Policing is Back in Vogue. But Does it Work?
#fixingjustice - Policing
Community Policing is Back in Vogue. But Does it Work?
As police departments look for ways to rebuild trust with their communities, an increasing number are turning to...
By Natalia Megas

As police departments look for ways to rebuild trust with their communities, an increasing number are turning to new community policing programs. But are they effective? As with most things, it depends on what you measure.

Dispatches
Why a Third of Antidepressants Are Prescribed for "Off-label" Problems
Why a Third of Antidepressants Are Prescribed for
Dispatches
Why a Third of Antidepressants Are Prescribed for "Off-label" Problems
The "secret life of antidepressants" could open up a host of new treatments.
By Leah Shaffer

The "secret life of antidepressants" could open up a host of new treatments.

Inside the Cajun Navy: How Volunteers Are Training to Rescue Hurricane Victims
Inside the Cajun Navy: How Volunteers Are Training to Rescue Hurricane Victims
Watch Now
Inside the Cajun Navy: How Volunteers Are Training to Rescue Hurricane Victims
Can civilian-led rescue be part of future disaster recovery efforts?
Watch Now

A Look Inside the Cajun Navy Before Hurricane Katrina hit, the "Cajun Navy" didn't even exist. But in the aftermath of the storm, a group of volunteers that helped rescue thousands flood victims stranded in their homes and vehicles have come together again. Their goal? To work together to better assist people in times of need directly following a natural disaster. They call themselves the Cajun Navy, and their work...

Dispatches
Unlike Smoking, Vaping Won't Mess With Your Microbiome
Unlike Smoking, Vaping Won't Mess With Your Microbiome
Dispatches
Unlike Smoking, Vaping Won't Mess With Your Microbiome
Smoking kills off good bacteria and upsets the balance of power your gut.

Smoking kills off good bacteria and upsets the balance of power your gut.

Relentless
How Hamish Brewer is Changing the Game
The Tattooed, Skater Principal Making Education Fun Again
Watch Now
Relentless
How Hamish Brewer is Changing the Game
The Tattooed, Skater Principal Making Education Fun Again
Watch Now

To hear Hamish Brewer speak is to be inspired. About education. About life. About the possibilities of it all. And for disadvantaged youths that come from poverty and broken homes, the New Zealander with an infectious energy works hard to encourage them to excel, not only in school, but in life. About Hamish Brewer With a shaved head, tattoos, and one of those loud, larger-than-life personalities, Hamish Brewer is hard to...

Crossing the Divide
Americans Are Divided. These People Are Doing Something About It.
Americans Are Divided. These People Are Doing Something About It.
Crossing the Divide
Americans Are Divided. These People Are Doing Something About It.
Amidst our most intense religious, political, and cultural conflicts, there are people around the country who are...
By Dan Hayes

Amidst our most intense religious, political, and cultural conflicts, there are people around the country who are working tirelessly to forge connections