Skip to main content
Move the World.
experience synesthesia

Lead Image © Vassily Kandinsky / Wikimedia Commons

When Russian artist Vasily Kandinsky was born, the first computer hadn't even been built yet.

Now, more than 75 years after Kandinsky's death, an online tool powered by AI is allowing people to experience his paintings the same way he did: as music.

Hear Colors, See Sounds

Kandinsky had a neurological condition called synesthesia, in which the stimulation of one sense triggers another.

In Kandinsky's case, seeing certain colors caused him to hear certain sounds and feel certain emotions. Yellow sounded like trumpets and felt cheeky, for example, while red sounded like a violin and felt restless.

"When Kandinsky painted, two senses worked systematically together: hearing and sight," Serge Lasvignes, president of the Centre Pompidou, an art museum, library, and research center in Paris, wrote in a Google blog post.

"Colors and shapes translated into sounds, harmonies, and vibrations made up lines and patterns," he continued.

Connecting colors and sounds is just one of as many as 80 different ways to experience synesthesia, though. Other people associate letters with colors or words with flavors.

No one knows for sure how many people have synesthesia — depending on your source, it could be one in 2,000 people, one in 300 people, or (to some extent) everyone — but famous "synesthetes" include musician Kanye West, author Vladimir Nabokov, and composer Richard Wagner.

For people who don't experience synesthesia naturally, the concept can be hard to understand — how can a color have a sound? — so Google Arts & Culture and the Centre Pompidou collaborated on "Play a Kandinsky."

This online tool lets users hear what Kandinsky might have heard when he painted his masterpiece, Yellow-Red-Blue, in 1925 — and all of the music in the experiment was written by AI.

Recreating a Masterpiece

The first step to creating "Play a Kandinsky" was understanding exactly how Kandinsky experienced synesthesia, so the researchers studied his writings about his experiences.

Next, they needed the music.

"By inviting the machine to express itself, the music kind of emerged by itself."

Antoine Bertin

Experimental musicians NSDOS and Antoine Bertin trained a Google AI to compose music for parts of Yellow-Red-Blue based on what they'd learned about Kandinsky's synesthesia and the phonographs in the artist's personal collection.

"We trained Google Transformer, a neural network that was developed by the Google Magenta team, on music from Kandinsky's time and invited this machine to generate new scores to form a new perspective on what the painting may sound like from today's point of view," Bertin said.

"By inviting the machine to express itself, the music kind of emerged by itself," he added. "And it felt that, as artists, we were more the intermediary from Kandinsky's time to today rather than the interpreters of the theory ourselves, and that was very exciting."

Your Chance to Experience Synesthesia

Now, anyone who wants to experience synesthesia the way Kandinsky might have can visit the "Play a Kandinsky" experiment, which is broken into four chapters.

The first chapter introduces the concept of synesthesia and how Kandinsky experienced it.

In the second, users click on one of three shapes — a yellow rectangle, a red cross, or a blue circle — to hear a snippet of music similar to what Kandinsky might have heard and felt when he looked at that shape.

"When Kandinsky painted, two senses worked systematically together: hearing and sight."

Serge Lasvignes

The third chapter is meant to let users experience synesthesia the way Kandinsky might have while looking at Yellow-Red-Blue.

Google broke the painting down into seven regions, and by clicking on one, users trigger a musical movement based on the shapes and colors in that region.

The region with a yellow rectangle at its center, for example, triggers the "Yellow Movement," described as "simple and bright shapes which echo earthly madness and passionate joy."

In the final chapter, users make their own musical movement based on Yellow-Red-Blue.

They start by choosing two emotions from a list, and then listen to those emotions conveyed through the painting. By clicking on various parts of the painting, they can add or subtract additional sounds.

Whether "Play a Kandinsky" achieves its goal of allowing users to experience synesthesia the way the artist might have is hard to say.

However, it does make the abstract concept easier to grasp, and bringing AI into the experiment could entice a new audience to explore the works of one of the greatest artists of a time before digital technology.

We'd love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].

Up Next

Coded
Meet the Artist and Activist Who Wants You to Erase Your DNA
Meet the Artist and Activist Who Wants You to Erase Your DNA
Coded
Meet the Artist and Activist Who Wants You to Erase Your DNA
Heather Dewey-Hagborg wants to make sure people understand the hidden secrets in the DNA they leave behind...
By Michael O'Shea

Heather Dewey-Hagborg wants to make sure people understand the hidden secrets in the DNA they leave behind everywhere they go.

Accessibility
Helping People with Disabilities Become Working Artists
Helping People with Disabilities Become Working Artists
Watch Now
Accessibility
Helping People with Disabilities Become Working Artists
Only 20% of people with disabilities participate in the workforce; Art Enables helps those with special needs turn art into a career.
Watch Now

Art Enables helps artists with special needs create and sell their artwork. Only 20% of people with disabilities participate in the workforce. This program counters that by providing both an art studio for people with disabilities to make art and career and vocational training to market and sell their artwork. It helps these artists express themselves creatively, socialize, and earn flexible income to increase their...

Art That Heals
This Artist Heals Wounds of Violence With Tattoos
This Artist Heals Wounds of Violence With Tattoos
Art That Heals
This Artist Heals Wounds of Violence With Tattoos
Brian Finn tattoos over scars from self-harm, violence, or human trafficking for free or at a discounted rate in order to help people heal.

Brian Finn tattoos over scars from self-harm, violence, or human trafficking for free or at a discounted rate in order to help people heal.

The Sound of Science
This Electronic Musician Is Transforming the Soundscape of Hospitals
This Electronic Musician Is Transforming the Soundscape of Hospitals
The Sound of Science
This Electronic Musician Is Transforming the Soundscape of Hospitals
Trained as a classical pianist, ambient artist Yoko Sen is on a mission to redesign medical devices' beeps and buzzes and save us from alarm fatigue.

Trained as a classical pianist, ambient artist Yoko Sen is on a mission to redesign medical devices' beeps and buzzes and save us from alarm fatigue.

Ancient Origins
Meet the Artist Who Paints With DNA Clues
Meet the Artist Who Paints With DNA Clues
Ancient Origins
Meet the Artist Who Paints With DNA Clues
When Maayan Harel paints a portrait, her subject isn't sitting in front of her. She doesn't even have a photograph...

When Maayan Harel paints a portrait, her subject isn't sitting in front of her. She doesn't even have a photograph to work from. Instead, she looks at clues from ancient human DNA. Last month, scientists from the Hebrew University of Jerusalem made headlines when they revealed Harel's portrait of a Denisovan, an extinct group of archaic humans that may have lived with Neanderthals. The first Denisovan remains were...

The Sound of Science
This Musician Transforms Scientific Data Into Elaborate Melodies
This Musician Transforms Scientific Data Into Elaborate Melodies
The Sound of Science
This Musician Transforms Scientific Data Into Elaborate Melodies
When we convert complex data into sound and listen to it, quite often what emerges is something we can understand through sound, even though we could never understand it visually.

If you think staring at rows of numbers and graphs seems humdrum, these musicians agree. They are on a mission to expose new scientific information through sound, by turning flat datasets into musical scores --- creating the soundtrack for science: Listen to Mark Ballora’s sonification of singularity with flutes and electronics: Jenni Evans first met Mark Ballora at a Penn State social gathering. Both were professors at...