New Google tool lets you experience synesthesia

Dive into the mind of the abstract artist who could hear colors with “Play a Kandinsky.”

When Russian artist Vasily Kandinsky was born, the first computer hadn’t even been built yet.

Now, more than 75 years after Kandinsky’s death, an online tool powered by AI is allowing people to experience his paintings the same way he did: as music.

Hear Colors, See Sounds

Kandinsky had a neurological condition called synesthesia, in which the stimulation of one sense triggers another.

In Kandinsky’s case, seeing certain colors caused him to hear certain sounds and feel certain emotions. Yellow sounded like trumpets and felt cheeky, for example, while red sounded like a violin and felt restless.

“When Kandinsky painted, two senses worked systematically together: hearing and sight,” Serge Lasvignes, president of the Centre Pompidou, an art museum, library, and research center in Paris, wrote in a Google blog post.

“Colors and shapes translated into sounds, harmonies, and vibrations made up lines and patterns,” he continued.

Connecting colors and sounds is just one of as many as 80 different ways to experience synesthesia, though. Other people associate letters with colors or words with flavors.

No one knows for sure how many people have synesthesia — depending on your source, it could be one in 2,000 people, one in 300 people, or (to some extent) everyone — but famous “synesthetes” include musician Kanye West, author Vladimir Nabokov, and composer Richard Wagner.

For people who don’t experience synesthesia naturally, the concept can be hard to understand — how can a color have a sound? — so Google Arts & Culture and the Centre Pompidou collaborated on “Play a Kandinsky.”

This online tool lets users hear what Kandinsky might have heard when he painted his masterpiece, Yellow-Red-Blue, in 1925 — and all of the music in the experiment was written by AI.

Recreating a Masterpiece

The first step to creating “Play a Kandinsky” was understanding exactly how Kandinsky experienced synesthesia, so the researchers studied his writings about his experiences.

Next, they needed the music.

By inviting the machine to express itself, the music kind of emerged by itself.

Antoine Bertin

Experimental musicians NSDOS and Antoine Bertin trained a Google AI to compose music for parts of Yellow-Red-Blue based on what they’d learned about Kandinsky’s synesthesia and the phonographs in the artist’s personal collection.

“We trained Google Transformer, a neural network that was developed by the Google Magenta team, on music from Kandinsky’s time and invited this machine to generate new scores to form a new perspective on what the painting may sound like from today’s point of view,” Bertin said.

“By inviting the machine to express itself, the music kind of emerged by itself,” he added. “And it felt that, as artists, we were more the intermediary from Kandinsky’s time to today rather than the interpreters of the theory ourselves, and that was very exciting.”

Your Chance to Experience Synesthesia

Now, anyone who wants to experience synesthesia the way Kandinsky might have can visit the “Play a Kandinsky” experiment, which is broken into four chapters.

The first chapter introduces the concept of synesthesia and how Kandinsky experienced it.

In the second, users click on one of three shapes — a yellow rectangle, a red cross, or a blue circle — to hear a snippet of music similar to what Kandinsky might have heard and felt when he looked at that shape.

When Kandinsky painted, two senses worked systematically together: hearing and sight.

Serge Lasvignes

The third chapter is meant to let users experience synesthesia the way Kandinsky might have while looking at Yellow-Red-Blue.

Google broke the painting down into seven regions, and by clicking on one, users trigger a musical movement based on the shapes and colors in that region.

The region with a yellow rectangle at its center, for example, triggers the “Yellow Movement,” described as “simple and bright shapes which echo earthly madness and passionate joy.”

In the final chapter, users make their own musical movement based on Yellow-Red-Blue.

They start by choosing two emotions from a list, and then listen to those emotions conveyed through the painting. By clicking on various parts of the painting, they can add or subtract additional sounds.

Whether “Play a Kandinsky” achieves its goal of allowing users to experience synesthesia the way the artist might have is hard to say.

However, it does make the abstract concept easier to grasp, and bringing AI into the experiment could entice a new audience to explore the works of one of the greatest artists of a time before digital technology.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].

OpenAI and Microsoft are reportedly planning a $100B supercomputer
Microsoft is reportedly planning to build a $100 billion data center and supercomputer, called “Stargate,” for OpenAI.
Can we stop AI hallucinations? And do we even want to?
“Making stuff up” and “being creative” may be two sides of the same coin — but you have to be able to tell the difference.
When AI prompts result in copyright violations, who has to pay?
Who is responsible for copyright violations when they’re produced by generative AI? The technology is outpacing the law.
Google’s Deep Mind AI can help engineers predict “catastrophic failure”
How vulnerable is the electrical grid to a malicious attacker who destroys select substations? Google’s Deep Mind can help predict the answer.
Does AI need a “body” to become truly intelligent? Meta researchers think so.
We’re finally starting to see what can happen when we put an advanced AI “brain” in a state-of-the-art robot “body” — and it’s remarkable.
Up Next
virtual patients
Subscribe to Freethink for more great stories