Skip to main content
Move the World.

In 2014, a groundbreaking study was released. Researchers had followed activists from the Los Angeles LGBT Center as they knocked on doors of California voters. The study showed that gay canvassers were able to significantly increase support for gay marriage by simply having a single, face-to-face conversation. And, the study claimed, not only were those conversations surprisingly effective, the impact was also long lasting.

The study got a lot of attention from national media and made waves in the research world where the prevailing thought is that changing beliefs is notoriously difficult.

Among the researchers who took notice were Josh Kalla at Berkeley and David Broockman at Stanford. Kalla and Broockman were fascinated by the study and wanted to run a version of it themselves. They partnered with the LA LGBT Center to launch a new study in Florida to test if canvassers could shift opinion on transgender rights. To design their own study, they dove into the original research and methodology in detail. And as they did, they made a startling realization: the results of the original study were completely fabricated. Kalla and Broockman found inconsistencies and inaccuracies across the methodology and the results. When they released their findings, Donald Green, co-author of the original study immediately retracted the paper. Michael LaCour, the paper’s other co-author, has stood by his research.

The retraction resulted in a frenzy of corrections by media outlets. While the collapse of the original research grabbed headlines, Kalla and Broockman continued to work on the LGBT Center’s Florida canvassing campaign -- essentially doing the research the original study claimed to do. In a fascinating twist, their study revealed that, while important details from the original “findings” weren’t accurate, the idea that a conversation can make a significant and lasting impact on a person’s opinion on an issue, is backed up by the numbers.

We sat down with Josh Kalla to talk through their research and how they think it fits into the current conversation on how people’s opinions evolve on some of most hotly debated issues.

Headquarters of the Los Angeles LGBT Center

Michael O’Shea: Your story related to the work that you did with Dave Fleischer and the LA LGBT center has a lot of interesting twists and turns, but let's start by focusing on the research that you published about the canvassing effort. What exactly did you set out to find with your research?

Josh Kalla: Attitude change is something that psychologists and political scientists, and the social sciences more broadly, have really been interested in exploring since the rise of modern social science, in the early 20th century. It's something that's been debated, whether it's easy, whether it's hard, what can cause it, when can it happen.

For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice.

One area where it's been hotly researched is on prejudice, really beginning with studies around racism, but you also have a lot of interest in prejudice reduction against handicapped people, against Muslims, against transgender people. For the longest time, the general academic viewpoint, not to oversimplify things, was that it's really hard to reduce prejudice in American society, or society more generally. These are deeply ingrained beliefs that are personally held. You tend to acquire them in childhood. You're socialized into them. Your religion, your social status, all reinforce them. These are things that are hard, if not impossible, to change. Our best hope to reduce prejudice in society is generational replacement. Maybe our children will be better, but people are born with these prejudices. They develop them. They're hard to change. That's the academic background.

Along comes Dave Fleischer with this innovative technique and story to try to reduce prejudice. David Broockman, my colleague, and I were fascinated by this work. Before we got started, we went to canvas with Dave and his team and thought there was really something special going on there, something worth studying. We set out to conduct this academic experiment to see whether or not the work that the Leadership LAB was doing was actually effective at reducing prejudice towards transgender people.

MO:  How do you go about actually measuring that?

JK: Measuring prejudice is hard. Most people don't want to openly admit that they're prejudiced, particularly if they know that they're being studied. They tend to give the "socially desirable" answer. So we conduct this academic experiment where we really separate out the research side of things from the work that Dave Fleischer and his group were doing.

First we send around 60,000 people in South Florida a piece of mail inviting them to participate in a survey sponsored by our universities, Berkeley and Stanford. This survey had a whole range of questions in them. Everything from "What's your favorite beach in Miami?", to "What do you think of the latest films?", to, at the time, I think there were questions about the early 2016 presidential election. Sprinkled in there were a few questions about transgender rights. There might have been half a dozen questions, out of 50 or so questions in that survey, to make it not seem like this was a survey about transgender rights. Then, among the people who took this survey, we got their baseline measures, and we randomly assigned half of them to be canvassed by Dave and his group and to get the full, 10 or so minute conversation about transgender people and reducing prejudice. The other half of the people got what we call a placebo. This is a group that we're not trying to change their attitudes about transgender people or prejudice, we just want to know, who are the people we could have talked about that, but let's talk to them about something else. Then, afterwards, we follow up with an additional survey that lets us compare whether the change over time in the group that got the trans rights conversation is different than the group that got the other, unrelated conversation. We do it both immediately and then over time, to see whether this was a one-off effect (which is a big fear in the prejudice reduction literature) or whether this is a longer term, deeper change in attitudes.

Dave Fleischer in the New York Times Magazine

MO: So what did you guys find?

JK: We found very large and long lasting effects. We found that the conversations reduced the amount of prejudice towards transgender people. It's hard to put it on a scale that's comprehensible, but the most comprehensible scale is the average reduction that occurred in society, more generally, towards gay men and lesbians from, I want to say, 1996 to 2012. That general reduction is what we see happening in that Miami study, with just a 10 minute conversation.

MO: It seems a lot of people grasped onto the “findings” of the original study because it makes sense on the surface. Talking to somebody of a particular group could reduce prejudice towards that group. So since that’s not the whole story, what's the insight here? It's just simply that having a face to face conversation and presenting people with ideas that are counter to what they already believe, regardless of who it comes from, increases empathy?

The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy.

JK: The mechanism that we think might be reducing the prejudice is this perspective taking. It's the fancier way of saying empathy. The conversation was around getting the canvasser to share a story of a time that they faced prejudice and discrimination. If they were transgender, they may talk about it in terms of that. If they were LGB, they may talk about it in terms of that. If they were an ally, they might talk about all the other prejudice that exists in society. Whether you're a woman, whether you're a minority, whether you are friends with people who face prejudice. Prejudice is so widespread that almost everyone can talk about a personal experience. Getting the person being canvassed to share a similar story and trying to, through analogy, make that connection between, "Hey, here's a time when you felt discriminated against or when a loved one felt discriminated against. It feels really lousy. We all, collectively, think this is bad and we want to avoid this. Here's a group that faces a whole bunch of prejudice in their lives. You probably haven't learned much about this group. You probably haven't heard much about this group." This might have been the first conversation that people had ever had about transgender people or transgender rights. I think part of the effect might have just been educational, of demystifying this group you had never heard of before, then the other part is this empathy building.

Dave Fleischer and his team of canvassers had 15,000 one-on-one conversations with voters

MO: It seems like a lot of things that I've read about research that's adjacent to your work seems to take a much more pessimistic view of our ability to change our minds. There’s a lot of talk about confirmation bias and people seeking out opinions that are the same as their own. Often, the feeling that you're left with is, and as you said this was the prevailing thought for a long time, that it is very hard to change people's minds. Your research goes in direct conflict with that. Can you talk about how you think this research fits into that larger discussion about how people form and change their opinions?

It's always hard in science to try to explain why something happens.

JK: I think it's important to keep in mind the issue of transgender rights, where it is a fairly new issue for a lot of people, as opposed to racism, or abortion, or whether or not you're a Democrat or Republican. Those, I think, are probably conceptually very different issues for many people, particularly at the time when we were doing the study. I think that's one more straight forward explanation as to why this study was different. It's just this particular issue was not something that people had much exposure to before. It was not tied to other parts of their identity in a way that other issues are. David and I have another paper with the LA LGBT Center, trying to change attitudes around abortion and reducing stigma towards women who have had abortion and there we find no effect. We find that similar strategy, it's the same group, same leadership, they were not able to change attitudes around abortion. Again, there are a whole lot of explanations you could give. Maybe it wasn't the right script. Maybe if you were to do it again, it would be more effective. It's always hard in science to try to explain why something happens, but at least the two data points that we have is trans rights, big effects, abortion, currently no effects. Abortion is this old issue that's been on the political agenda for a long time. Most people in American society have thought about abortion, or pregnancy, and sex, and birth control, and that constellation of issues at some point in their lives. But trans rights are different. Me just defining what it means to be transgender might be the first time that has ever happened for someone, in a way that abortion, when you're talking to a middle aged person, is probably not the first time that's ever happened. I think that's one way to reconcile the broader literature with these findings, but I also think Dave Fleischer and his group is potentially on to something with their canvassing. I spend a lot of time with various political groups, and go canvassing with them, and look at their persuasive efforts, and no one really does what LA LGBT Center does.


Related video:

Up Next

Catalysts
Old Skool Café Provides an Alternative to Gangs and Violence
Old Skool Café Provides an Alternative to Gangs and Violence
Catalysts
Old Skool Café Provides an Alternative to Gangs and Violence
As a former prison guard, Teresa Goines watched kids drift in and out of the justice system. Now she runs a jazz-themed eatery that hires at-risk youth and gives them a place to learn and grow.
By Esther Landhuis

As a former prison guard, Teresa Goines watched kids drift in and out of the justice system. Now she runs a jazz-themed eatery that hires at-risk youth and gives them a place to learn and grow.

Thinking Differently
How to Change the World
How to Change the World
Watch Now
Thinking Differently
How to Change the World
Olivia Leland, founder and CEO of Co-Impact, on how we should approach solving the world’s most complex problems.
Watch Now

Do you want to change the world? Olivia Leland, founder and CEO of Co-Impact, shares 3 lessons learned from organizations, philanthropists, and social change leaders that move the world forward. When we think about changing the world, we make a few mistakes over and over again. We think it’s a matter of voting, or giving to charity, or starting a business - but historically, many of the world’s biggest changes have come...

Culture of Change
Drone Racers Are A Thing and They’re Amazing
Drone Racers Are A Thing and They’re Amazing
Watch Now
Culture of Change
Drone Racers Are A Thing and They’re Amazing
Blistering speed. Big money. 11-year-old world champions. Is drone racing the next big sport?
Watch Now

Yes, drone racers are a thing and they’re amazing. The Drone Racing League has gone from a dream to ESPN in a few short years. We met world champion Paul Nurkalla, aka Nurk FPV, and got an inside look at how the DRL is striving to be the next big sports league. Somewhere between esports, NASCAR and Star Wars sits drone racing, also known as FPV racing. It’s a breathtakingly fast spectacle where drone pilots fly quadcopters...

Catalysts
How This Non-Profit Weaves a Family of Support for Struggling Teens in Baltimore
How This Non-Profit Weaves a Family of Support for Struggling Teens in Baltimore
Catalysts
How This Non-Profit Weaves a Family of Support for Struggling Teens in Baltimore
“Everyone - no matter their age, race, or background - needs a network of supportive relationships to help them...

“Everyone - no matter their age, race, or background - needs a network of supportive relationships to help them thrive.”

Helping Prisoners Overcome Trauma with Dance
Helping Prisoners Overcome Trauma with Dance
Watch Now
Helping Prisoners Overcome Trauma with Dance
Women in prison are regaining a sense of freedom — through dance.
Watch Now

Dance to Be Free is a program helping female prisoners overcome trauma with dance. While the inmates at the Commonwealth of Virginia Department of Corrections are physically incarcerated, the freedom that comes through dance helps them open up, enjoy themselves, and regain self-confidence. Founder Lucy Wallace began teaching dance in prison in order to help inmates, who often had unaddressed PTSD from physical or emotional...

3 Ways To Think About the Future
3 Ways To Think About the Future
Watch Now
3 Ways To Think About the Future
Why don’t we have a moon base? A cartoonist and a scientist explain.
Watch Now

Zach Weinersmith, creator of the Saturday Morning Breakfast Cereal webcomic and Kelly Weinersmith, professor and scientist, explain how to think about the future - a focus of their new illustrated book “Soonish: Ten Emerging Technologies That'll Improve and/or Ruin Everything.” The New York Times best-selling book has been praised for its exciting and nuanced looks at ten technologies that could change the world...

The Violinist Playing for Freedom in Venezuela
The Violinist Playing for Freedom in Venezuela
Watch Now
The Violinist Playing for Freedom in Venezuela
Venezuelan violinist Wuilly Arteaga has been beaten and arrested, but it hasn’t stopped him from using music to help bring freedom to his country
Watch Now

When Wuilly Arteaga, a Venezuelan violin virtuoso, joined protests with just his instrument, he was beaten and arrested. But that hasn’t stopped him from using music to help bring freedom to his country.

Pop Revolution
Skatepunks of Myanmar
Skatepunks of Myanmar
Watch Now
Pop Revolution
Skatepunks of Myanmar
Is the booming skating scene in Myanmar a sign of a more hopeful future for the once isolated country?
Watch Now

After years of repression, smartphones and social media are giving Myanmar a taste of the outside world. And young people are eager to catch up on what they missed. We join a group of skaters who are reimagining life in the once isolated society and shaping its future in the process.

Culture
This Week in Ideas: Unveiling Google Earth VR, China Goes All in on CRISPR, Cuba's...
This Week in Ideas: Unveiling Google Earth VR, China Goes All in on CRISPR, Cuba's Cancer Vaccine
Culture
This Week in Ideas: Unveiling Google Earth VR, China Goes All in on CRISPR, Cuba's...
Google releases some beautiful VR, human trials of gene-editing technology CRISPR, and importing Cuba's cancer...
By Mike Riggs

Google releases some beautiful VR, human trials of gene-editing technology CRISPR, and importing Cuba's cancer vaccine.