Skip to main content
Move the World.

In 2014, a groundbreaking study was released. Researchers had followed activists from the Los Angeles LGBT Center as they knocked on doors of California voters. The study showed that gay canvassers were able to significantly increase support for gay marriage by simply having a single, face-to-face conversation. And, the study claimed, not only were those conversations surprisingly effective, the impact was also long lasting.

The study got a lot of attention from national media and made waves in the research world where the prevailing thought is that changing beliefs is notoriously difficult.

Among the researchers who took notice were Josh Kalla at Berkeley and David Broockman at Stanford. Kalla and Broockman were fascinated by the study and wanted to run a version of it themselves. They partnered with the LA LGBT Center to launch a new study in Florida to test if canvassers could shift opinion on transgender rights. To design their own study, they dove into the original research and methodology in detail. And as they did, they made a startling realization: the results of the original study were completely fabricated. Kalla and Broockman found inconsistencies and inaccuracies across the methodology and the results. When they released their findings, Donald Green, co-author of the original study immediately retracted the paper. Michael LaCour, the paper’s other co-author, has stood by his research.

The retraction resulted in a frenzy of corrections by media outlets. While the collapse of the original research grabbed headlines, Kalla and Broockman continued to work on the LGBT Center’s Florida canvassing campaign -- essentially doing the research the original study claimed to do. In a fascinating twist, their study revealed that, while important details from the original “findings” weren’t accurate, the idea that a conversation can make a significant and lasting impact on a person’s opinion on an issue, is backed up by the numbers.

We sat down with Josh Kalla to talk through their research and how they think it fits into the current conversation on how people’s opinions evolve on some of most hotly debated issues.

Headquarters of the Los Angeles LGBT Center

Michael O’Shea: Your story related to the work that you did with Dave Fleischer and the LA LGBT center has a lot of interesting twists and turns, but let's start by focusing on the research that you published about the canvassing effort. What exactly did you set out to find with your research?

Josh Kalla: Attitude change is something that psychologists and political scientists, and the social sciences more broadly, have really been interested in exploring since the rise of modern social science, in the early 20th century. It's something that's been debated, whether it's easy, whether it's hard, what can cause it, when can it happen.

For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice.

One area where it's been hotly researched is on prejudice, really beginning with studies around racism, but you also have a lot of interest in prejudice reduction against handicapped people, against Muslims, against transgender people. For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice in American society, or society more generally. These are deeply ingrained beliefs that are personally held. You tend to acquire them in childhood. You're socialized into them. Your religion, your social status, all reinforce them. These are things that are hard, if not impossible, to change. Our best hope to reduce prejudice in society is generational replacement. Maybe our children will be better, but people are born with these prejudices. They develop them. They're hard to change. That's the academic background.

Along comes Dave Fleischer with this innovative technique and story to try to reduce prejudice. David Broockman, my colleague, and I were fascinated by this work. Before we got started, we went to canvas with Dave and his team and thought there was really something special going on there, something worth studying. We set out to conduct this academic experiment to see whether or not the work that the Leadership LAB was doing was actually effective at reducing prejudice towards transgender people.

MO:  How do you go about actually measuring that?

JK: Measuring prejudice is hard. Most people don't want to openly admit that they're prejudiced, particularly if they know that they're being studied. They tend to give the "socially desirable" answer. So we conduct this academic experiment where we really separate out the research side of things from the work that Dave Fleischer and his group were doing.

First we send around 60,000 people in South Florida a piece of mail inviting them to participate in a survey sponsored by our universities, Berkeley and Stanford. This survey had a whole range of questions in them. Everything from "What's your favorite beach in Miami?", to "What do you think of the latest films?", to, at the time, I think there were questions about the early 2016 presidential election. Sprinkled in there were a few questions about transgender rights. There might have been half a dozen questions, out of 50 or so questions in that survey, to make it not seem like this was a survey about transgender rights. Then, among the people who took this survey, we got their baseline measures, and we randomly assigned half of them to be canvassed by Dave and his group and to get the full, 10 or so minute conversation about transgender people and reducing prejudice. The other half of the people got what we call a placebo. This is a group that we're not trying to change their attitudes about transgender people or prejudice, we just want to know, who are the people we could have talked about that, but let's talk to them about something else. Then, afterwards, we follow up with an additional survey that lets us compare whether the change over time in the group that got the trans rights conversation is different than the group that got the other, unrelated conversation. We do it both immediately and then over time, to see whether this was a one-off effect (which is a big fear in the prejudice reduction literature) or whether this is a longer term, deeper change in attitudes.

Dave Fleischer in the New York Times Magazine

MO: So what did you guys find?

JK: We found very large and long lasting effects. We found that the conversations reduced the amount of prejudice towards transgender people. It's hard to put it on a scale that's comprehensible, but the most comprehensible scale is the average reduction that occurred in society, more generally, towards gay men and lesbians from, I want to say, 1996 to 2012. That general reduction is what we see happening in that Miami study, with just a 10 minute conversation.

MO: It seems a lot of people grasped onto the “findings” of the original study because it makes sense on the surface. Talking to somebody of a particular group could reduce prejudice towards that group. So since that’s not the whole story, what's the insight here? It's just simply that having a face to face conversation and presenting people with ideas that are counter to what they already believe, regardless of who it comes from, increases empathy?

The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy.

JK: The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy. The conversation was around getting the canvasser to share a story of a time that they faced prejudice and discrimination. If they were transgender, they may talk about it in terms of that. If they were LGB, they may talk about it in terms of that. If they were an ally, they might talk about all the other prejudice that exists in society. Whether you're a woman, whether you're a minority, whether you are friends with people who face prejudice. Prejudice is so widespread that almost everyone can talk about a personal experience. Getting the person being canvassed to share a similar story and trying to, through analogy, make that connection between, "Hey, here's a time when you felt discriminated against or when a loved one felt discriminated against. It feels really lousy. We all, collectively, think this is bad and we want to avoid this. Here's a group that faces a whole bunch of prejudice in their lives. You probably haven't learned much about this group. You probably haven't heard much about this group." This might have been the first conversation that people had ever had about transgender people or transgender rights. I think part of the effect might have just been educational, of demystifying this group you had never heard of before, then the other part is this empathy building.

Dave Fleischer and his team of canvassers had 15,000 one-on-one conversations with voters

MO: It seems like a lot of things that I've read about research that's adjacent to your work seems to take a much more pessimistic view of our ability to change our minds. There’s a lot of talk about confirmation bias and people seeking out opinions that are the same as their own. Often, the feeling that you're left with is, and as you said this was the prevailing thought for a long time, that it is very hard to change people's minds. Your research goes in direct conflict with that. Can you talk about how you think this research fits into that larger discussion about how people form and change their opinions?

It's always hard in science to try to explain why something happens.

JK: I think it's important to keep in mind the issue of transgender rights, where it is a fairly new issue for a lot of people, as opposed to racism, or abortion, or whether or not you're a Democrat or Republican. Those, I think, are probably conceptually very different issues for many people, particularly at the time when we were doing the study. I think that's one more straight forward explanation as to why this study was different. It's just this particular issue was not something that people had much exposure to before. It was not tied to other parts of their identity in a way that other issues are. David and I have another paper with the LA LGBT Center, trying to change attitudes around abortion and reducing stigma towards women who have had abortion and there we find no effect. We find that similar strategy, it's the same group, same leadership, they were not able to change attitudes around abortion. Again, there are a whole lot of explanations you could give. Maybe it wasn't the right script. Maybe if you were to do it again, it would be more effective. It's always hard in science to try to explain why something happens, but at least the two data points that we have is trans rights, big effects, abortion, currently no effects. Abortion is this old issue that's been on the political agenda for a long time. Most people in American society have thought about abortion, or pregnancy, and sex, and birth control, and that constellation of issues at some point in their lives. But trans rights are different. Me just defining what it means to be transgender might be the first time that has ever happened for someone, in a way that abortion, when you're talking to a middle aged person, is probably not the first time that's ever happened. I think that's one way to reconcile the broader literature with these findings, but I also think Dave Fleischer and his group is potentially on to something with their canvassing. I spend a lot of time with various political groups, and go canvassing with them, and look at their persuasive efforts, and no one really does what LA LGBT Center does.


Related video:

Up Next

Frontier Exploration
What Lies Beneath: An Interview With Underground Author Will Hunt
Will Hunt Finds Humanity Underground
Frontier Exploration
What Lies Beneath: An Interview With Underground Author Will Hunt
In Will Hunt’s book Underground, the journalist and "urban explorer" details his expeditions through caves, mines, tunnels, catacombs, and sewer systems around the world and the depths of the information he's found.

In Will Hunt’s book Underground, the journalist and "urban explorer" details his expeditions through caves, mines, tunnels, catacombs, and sewer systems around the world and the depths of the information he's found.

The Sound of Science
This Musician Transforms Scientific Data Into Elaborate Melodies
This Musician Transforms Scientific Data Into Elaborate Melodies
The Sound of Science
This Musician Transforms Scientific Data Into Elaborate Melodies
When we convert complex data into sound and listen to it, quite often what emerges is something we can understand through sound, even though we could never understand it visually.
By Teresa Carey

If you think staring at rows of numbers and graphs seems humdrum, these musicians agree. They are on a mission to expose new scientific information through sound, by turning flat datasets into musical scores --- creating the soundtrack for science: Listen to Mark Ballora’s sonification of singularity with flutes and electronics: Jenni Evans first met Mark Ballora at a Penn State social gathering. Both were professors at...

Catalysts
A Home Helping Teen Moms Break the Cycle of Generational Poverty
A Home Helping Teen Moms Break the Cycle of Generational Poverty
Watch Now
Catalysts
A Home Helping Teen Moms Break the Cycle of Generational Poverty
Hope House in Denver is changing lives for generations to come.
Watch Now

Teen pregnancy is highly linked to poverty and lower educational outcomes. Only about 50% of teen mothers receive a high school diploma by age 22, compared with 90% of women without a teen birth. This is a vulnerable population who often have nowhere to turn. Early and effective intervention can give mothers the tools to continue their education and become self-sufficient, giving them a path to self-sufficiency changes...

Thinking Differently
Is Autism a Mysterious Medical Condition or a Disability That's Part of the...
Is Autism a Mysterious Medical Condition or a Disability That's Part of the Spectrum of Human Diversity?
Thinking Differently
Is Autism a Mysterious Medical Condition or a Disability That's Part of the...
Rethinking Autism: Interview with NeuroTribes Author Steve Silberman

Rethinking Autism: Interview with NeuroTribes Author Steve Silberman

Thinking Differently
The Joy of Being Wrong
The Joy of Being Wrong
Watch Now
Thinking Differently
The Joy of Being Wrong
Can practicing intellectual humility make us smarter and happier? Science says yes.
Watch Now

Arguments on social media are notorious. People often naturally form an echo chamber of people with similar beliefs, and when people outside it start arguments, the discussion often becomes antagonistic. Is there a better way? Science suggests that a good starting point is by practicing intellectual humility. By admitting the possibility that we ourselves could be wrong, we’re able to better evaluate arguments and construct...

Inside the Cajun Navy: How Volunteers Are Training to Rescue Hurricane Victims
Inside the Cajun Navy: How Volunteers Are Training to Rescue Hurricane Victims
Watch Now
Inside the Cajun Navy: How Volunteers Are Training to Rescue Hurricane Victims
Can civilian-led rescue be part of future disaster recovery efforts?
Watch Now

A Look Inside the Cajun Navy Before Hurricane Katrina hit, the "Cajun Navy" didn't even exist. But in the aftermath of the storm, a group of volunteers that helped rescue thousands flood victims stranded in their homes and vehicles have come together again. Their goal? To work together to better assist people in times of need directly following a natural disaster. They call themselves the Cajun Navy, and their work...

Wrong
Is Vitamin C a Total Sham
Is Vitamin C a Total Sham
Watch Now
Wrong
Is Vitamin C a Total Sham
In the heart of cold and flu season, it's natural to reach for the Vitamin C. But we may want to think twice. While...
Watch Now

In the heart of cold and flu season, it's natural to reach for the Vitamin C. But we may want to think twice. While it has ballooned into a billion dollar business and crystallized as an all-powerful supplement in our collective conscious, countless studies have showed that Vitamin C’s ability to cure is questionable at best, and may even increase your risk for certain diseases. Did we all really get Vitamin C totally...wrong?

Crossing The Divide
Crossing the Divide Trailer
Crossing the Divide Trailer
Watch Now
Crossing The Divide
Crossing the Divide Trailer
77% of people think the country is more divided than it’s ever been. We set out to meet the people trying to change that.
Watch Now

It feels like we've never been more divided. Yet amidst our most intense religious, political, and cultural conflicts, there are people around the country who are working tirelessly to forge connections. It’s not easy and the odds of success are far from certain, but for some, accepting things as they are just isn't an option. Freethink presents a new original series, “Crossing the Divide.”