Skip to main content
Move the World.

In 2014, a groundbreaking study was released. Researchers had followed activists from the Los Angeles LGBT Center as they knocked on doors of California voters. The study showed that gay canvassers were able to significantly increase support for gay marriage by simply having a single, face-to-face conversation. And, the study claimed, not only were those conversations surprisingly effective, the impact was also long lasting.

The study got a lot of attention from national media and made waves in the research world where the prevailing thought is that changing beliefs is notoriously difficult.

Among the researchers who took notice were Josh Kalla at Berkeley and David Broockman at Stanford. Kalla and Broockman were fascinated by the study and wanted to run a version of it themselves. They partnered with the LA LGBT Center to launch a new study in Florida to test if canvassers could shift opinion on transgender rights. To design their own study, they dove into the original research and methodology in detail. And as they did, they made a startling realization: the results of the original study were completely fabricated. Kalla and Broockman found inconsistencies and inaccuracies across the methodology and the results. When they released their findings, Donald Green, co-author of the original study immediately retracted the paper. Michael LaCour, the paper’s other co-author, has stood by his research.

The retraction resulted in a frenzy of corrections by media outlets. While the collapse of the original research grabbed headlines, Kalla and Broockman continued to work on the LGBT Center’s Florida canvassing campaign -- essentially doing the research the original study claimed to do. In a fascinating twist, their study revealed that, while important details from the original “findings” weren’t accurate, the idea that a conversation can make a significant and lasting impact on a person’s opinion on an issue, is backed up by the numbers.

We sat down with Josh Kalla to talk through their research and how they think it fits into the current conversation on how people’s opinions evolve on some of most hotly debated issues.

Headquarters of the Los Angeles LGBT Center

Michael O’Shea: Your story related to the work that you did with Dave Fleischer and the LA LGBT center has a lot of interesting twists and turns, but let's start by focusing on the research that you published about the canvassing effort. What exactly did you set out to find with your research?

Josh Kalla: Attitude change is something that psychologists and political scientists, and the social sciences more broadly, have really been interested in exploring since the rise of modern social science, in the early 20th century. It's something that's been debated, whether it's easy, whether it's hard, what can cause it, when can it happen.

For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice.

One area where it's been hotly researched is on prejudice, really beginning with studies around racism, but you also have a lot of interest in prejudice reduction against handicapped people, against Muslims, against transgender people. For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice in American society, or society more generally. These are deeply ingrained beliefs that are personally held. You tend to acquire them in childhood. You're socialized into them. Your religion, your social status, all reinforce them. These are things that are hard, if not impossible, to change. Our best hope to reduce prejudice in society is generational replacement. Maybe our children will be better, but people are born with these prejudices. They develop them. They're hard to change. That's the academic background.

Along comes Dave Fleischer with this innovative technique and story to try to reduce prejudice. David Broockman, my colleague, and I were fascinated by this work. Before we got started, we went to canvas with Dave and his team and thought there was really something special going on there, something worth studying. We set out to conduct this academic experiment to see whether or not the work that the Leadership LAB was doing was actually effective at reducing prejudice towards transgender people.

MO:  How do you go about actually measuring that?

JK: Measuring prejudice is hard. Most people don't want to openly admit that they're prejudiced, particularly if they know that they're being studied. They tend to give the "socially desirable" answer. So we conduct this academic experiment where we really separate out the research side of things from the work that Dave Fleischer and his group were doing.

First we send around 60,000 people in South Florida a piece of mail inviting them to participate in a survey sponsored by our universities, Berkeley and Stanford. This survey had a whole range of questions in them. Everything from "What's your favorite beach in Miami?", to "What do you think of the latest films?", to, at the time, I think there were questions about the early 2016 presidential election. Sprinkled in there were a few questions about transgender rights. There might have been half a dozen questions, out of 50 or so questions in that survey, to make it not seem like this was a survey about transgender rights. Then, among the people who took this survey, we got their baseline measures, and we randomly assigned half of them to be canvassed by Dave and his group and to get the full, 10 or so minute conversation about transgender people and reducing prejudice. The other half of the people got what we call a placebo. This is a group that we're not trying to change their attitudes about transgender people or prejudice, we just want to know, who are the people we could have talked about that, but let's talk to them about something else. Then, afterwards, we follow up with an additional survey that lets us compare whether the change over time in the group that got the trans rights conversation is different than the group that got the other, unrelated conversation. We do it both immediately and then over time, to see whether this was a one-off effect (which is a big fear in the prejudice reduction literature) or whether this is a longer term, deeper change in attitudes.

Dave Fleischer in the New York Times Magazine

MO: So what did you guys find?

JK: We found very large and long lasting effects. We found that the conversations reduced the amount of prejudice towards transgender people. It's hard to put it on a scale that's comprehensible, but the most comprehensible scale is the average reduction that occurred in society, more generally, towards gay men and lesbians from, I want to say, 1996 to 2012. That general reduction is what we see happening in that Miami study, with just a 10 minute conversation.

MO: It seems a lot of people grasped onto the “findings” of the original study because it makes sense on the surface. Talking to somebody of a particular group could reduce prejudice towards that group. So since that’s not the whole story, what's the insight here? It's just simply that having a face to face conversation and presenting people with ideas that are counter to what they already believe, regardless of who it comes from, increases empathy?

The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy.

JK: The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy. The conversation was around getting the canvasser to share a story of a time that they faced prejudice and discrimination. If they were transgender, they may talk about it in terms of that. If they were LGB, they may talk about it in terms of that. If they were an ally, they might talk about all the other prejudice that exists in society. Whether you're a woman, whether you're a minority, whether you are friends with people who face prejudice. Prejudice is so widespread that almost everyone can talk about a personal experience. Getting the person being canvassed to share a similar story and trying to, through analogy, make that connection between, "Hey, here's a time when you felt discriminated against or when a loved one felt discriminated against. It feels really lousy. We all, collectively, think this is bad and we want to avoid this. Here's a group that faces a whole bunch of prejudice in their lives. You probably haven't learned much about this group. You probably haven't heard much about this group." This might have been the first conversation that people had ever had about transgender people or transgender rights. I think part of the effect might have just been educational, of demystifying this group you had never heard of before, then the other part is this empathy building.

Dave Fleischer and his team of canvassers had 15,000 one-on-one conversations with voters

MO: It seems like a lot of things that I've read about research that's adjacent to your work seems to take a much more pessimistic view of our ability to change our minds. There’s a lot of talk about confirmation bias and people seeking out opinions that are the same as their own. Often, the feeling that you're left with is, and as you said this was the prevailing thought for a long time, that it is very hard to change people's minds. Your research goes in direct conflict with that. Can you talk about how you think this research fits into that larger discussion about how people form and change their opinions?

It's always hard in science to try to explain why something happens.

JK: I think it's important to keep in mind the issue of transgender rights, where it is a fairly new issue for a lot of people, as opposed to racism, or abortion, or whether or not you're a Democrat or Republican. Those, I think, are probably conceptually very different issues for many people, particularly at the time when we were doing the study. I think that's one more straight forward explanation as to why this study was different. It's just this particular issue was not something that people had much exposure to before. It was not tied to other parts of their identity in a way that other issues are. David and I have another paper with the LA LGBT Center, trying to change attitudes around abortion and reducing stigma towards women who have had abortion and there we find no effect. We find that similar strategy, it's the same group, same leadership, they were not able to change attitudes around abortion. Again, there are a whole lot of explanations you could give. Maybe it wasn't the right script. Maybe if you were to do it again, it would be more effective. It's always hard in science to try to explain why something happens, but at least the two data points that we have is trans rights, big effects, abortion, currently no effects. Abortion is this old issue that's been on the political agenda for a long time. Most people in American society have thought about abortion, or pregnancy, and sex, and birth control, and that constellation of issues at some point in their lives. But trans rights are different. Me just defining what it means to be transgender might be the first time that has ever happened for someone, in a way that abortion, when you're talking to a middle aged person, is probably not the first time that's ever happened. I think that's one way to reconcile the broader literature with these findings, but I also think Dave Fleischer and his group is potentially on to something with their canvassing. I spend a lot of time with various political groups, and go canvassing with them, and look at their persuasive efforts, and no one really does what LA LGBT Center does.


Related video:

More About

Wildfires
A Community at the Frontlines of the War on Wildfire
A Community at the Frontlines of the War on Wildfire
Watch Now
Wildfires
A Community at the Frontlines of the War on Wildfire

When facing literal hellfire, this community effort may be the only way to protect homes and lives from devastating wildfires.

Watch Now

California is burning like never before, with wildfires damaging nearly one million acres in 2018 alone. Despite the State’s best efforts, preventing forest fires doesn’t seem possible from the top down. With climate change accelerating the frequency and scale of these wildfires at an alarming rate, experts are turning to private citizens to help with forest fire prevention and control. Or as we like to say, to help fight fire…

#fixingjustice - Prosecution
Exclusive Interview: How Miriam Krinsky is Leading Us to a Smarter & More Equitable Justice System
Exclusive Interview: How Miriam Krinsky is Leading Us to a Smarter & More Equitable Justice System
#fixingjustice - Prosecution
Exclusive Interview: How Miriam Krinsky is Leading Us to a Smarter & More Equitable Justice System

Her organization is bringing together a new generation of prosecutors with a shared vision of fair, compassionate, and responsible criminal…

By Dan Bier

Her organization is bringing together a new generation of prosecutors with a shared vision of fair, compassionate, and responsible criminal justice reform.

#fixingjustice - Policing
Civilian Oversight Is a Solution to Police Misconduct. But is it Effective?
Civilian Oversight Is a Solution to Police Misconduct. But is it Effective?
#fixingjustice - Policing
Civilian Oversight Is a Solution to Police Misconduct. But is it Effective?

Creating a civilian review board to oversee police conduct seems like a straightforward solution to disciplinary issues on the force.…

By Andrew Denney

Creating a civilian review board to oversee police conduct seems like a straightforward solution to disciplinary issues on the force. But why is it so hard to implement?

Sponsored
Rebuilding Coal Country
Rebuilding Coal Country
Watch Now
Sponsored
Rebuilding Coal Country

Meet the nonprofit bringing jobs back to coal country.

Watch Now

Coalfield Development is a nonprofit on a mission to unlock the resources and potential of coal country. Partnering with local entrepreneurs, they've helped launch six new businesses and trained hundreds of workers across Appalachia. They provide education and training for unemployed workers, from college courses to trade skills, and partner with businesses to help them succeed on the job. The coal jobs might not be coming back, but they believe…

Wrong
Did the Food Pyramid Make Us Fat?
Did the Food Pyramid Make Us Fat?
Watch Now
Wrong
Did the Food Pyramid Make Us Fat?
Watch Now

To address rising health problems in the 1980s, scientists and policymakers developed the US Food Pyramid to encourage healthy eating. But despite a massive public education campaign, obesity and diabetes continued to rise. What happened? How did we get it so… wrong?

Relentless
The Tattooed, Skater Principal Making Education Fun Again
The Tattooed, Skater Principal Making Education Fun Again
Watch Now
Relentless
The Tattooed, Skater Principal Making Education Fun Again

The Tattooed, Skater Principal Making Education Fun Again

Watch Now

To hear Hamish Brewer speak is to be inspired. About education. About life. About the possibilities of it all.

And for disadvantaged youths that come from poverty and broken homes, the New Zealander with an infectious energy works hard to encourage them to excel, not only in school, but in life.

About Hamish Brewer

With a shaved head, tattoos, and one of those loud, larger-than-life personalities, Hamish Brewer is hard…

Criminal Justice
Can This Robot Stop Violence at Traffic Stops?
Can This Robot Stop Violence at Traffic Stops?
Criminal Justice
Can This Robot Stop Violence at Traffic Stops?

A Duke robotics PhD student and his partner think they have a way ease tensions while deep-rooted differences are hashed…

By Michael O'Shea

A Duke robotics PhD student and his partner think they have a way ease tensions while deep-rooted differences are hashed out.

On The Cusp
This Week in Ideas: Saying Goodbye to Lab Rats and Replacing Bees with Drones
This Week in Ideas: Saying Goodbye to Lab Rats and Replacing Bees with Drones
On The Cusp
This Week in Ideas: Saying Goodbye to Lab Rats and Replacing Bees with Drones

Breakthrough could mean the end of test animals, violent crime nearly cut in half, and drones that pollinate flowers.

By Michael O'Shea

Breakthrough could mean the end of test animals, violent crime nearly cut in half, and drones that pollinate flowers.