New Google tool lets you experience synesthesia

Dive into the mind of the abstract artist who could hear colors with “Play a Kandinsky.”

When Russian artist Vasily Kandinsky was born, the first computer hadn’t even been built yet.

Now, more than 75 years after Kandinsky’s death, an online tool powered by AI is allowing people to experience his paintings the same way he did: as music.

Hear Colors, See Sounds

Kandinsky had a neurological condition called synesthesia, in which the stimulation of one sense triggers another.

In Kandinsky’s case, seeing certain colors caused him to hear certain sounds and feel certain emotions. Yellow sounded like trumpets and felt cheeky, for example, while red sounded like a violin and felt restless.

“When Kandinsky painted, two senses worked systematically together: hearing and sight,” Serge Lasvignes, president of the Centre Pompidou, an art museum, library, and research center in Paris, wrote in a Google blog post.

“Colors and shapes translated into sounds, harmonies, and vibrations made up lines and patterns,” he continued.

Connecting colors and sounds is just one of as many as 80 different ways to experience synesthesia, though. Other people associate letters with colors or words with flavors.

No one knows for sure how many people have synesthesia — depending on your source, it could be one in 2,000 people, one in 300 people, or (to some extent) everyone — but famous “synesthetes” include musician Kanye West, author Vladimir Nabokov, and composer Richard Wagner.

For people who don’t experience synesthesia naturally, the concept can be hard to understand — how can a color have a sound? — so Google Arts & Culture and the Centre Pompidou collaborated on “Play a Kandinsky.”

This online tool lets users hear what Kandinsky might have heard when he painted his masterpiece, Yellow-Red-Blue, in 1925 — and all of the music in the experiment was written by AI.

Recreating a Masterpiece

The first step to creating “Play a Kandinsky” was understanding exactly how Kandinsky experienced synesthesia, so the researchers studied his writings about his experiences.

Next, they needed the music.

By inviting the machine to express itself, the music kind of emerged by itself.


Antoine Bertin

Experimental musicians NSDOS and Antoine Bertin trained a Google AI to compose music for parts of Yellow-Red-Blue based on what they’d learned about Kandinsky’s synesthesia and the phonographs in the artist’s personal collection.

“We trained Google Transformer, a neural network that was developed by the Google Magenta team, on music from Kandinsky’s time and invited this machine to generate new scores to form a new perspective on what the painting may sound like from today’s point of view,” Bertin said.

“By inviting the machine to express itself, the music kind of emerged by itself,” he added. “And it felt that, as artists, we were more the intermediary from Kandinsky’s time to today rather than the interpreters of the theory ourselves, and that was very exciting.”

Your Chance to Experience Synesthesia

Now, anyone who wants to experience synesthesia the way Kandinsky might have can visit the “Play a Kandinsky” experiment, which is broken into four chapters.

The first chapter introduces the concept of synesthesia and how Kandinsky experienced it.

In the second, users click on one of three shapes — a yellow rectangle, a red cross, or a blue circle — to hear a snippet of music similar to what Kandinsky might have heard and felt when he looked at that shape.

When Kandinsky painted, two senses worked systematically together: hearing and sight.


Serge Lasvignes

The third chapter is meant to let users experience synesthesia the way Kandinsky might have while looking at Yellow-Red-Blue.

Google broke the painting down into seven regions, and by clicking on one, users trigger a musical movement based on the shapes and colors in that region.

The region with a yellow rectangle at its center, for example, triggers the “Yellow Movement,” described as “simple and bright shapes which echo earthly madness and passionate joy.”

In the final chapter, users make their own musical movement based on Yellow-Red-Blue.

They start by choosing two emotions from a list, and then listen to those emotions conveyed through the painting. By clicking on various parts of the painting, they can add or subtract additional sounds.

Whether “Play a Kandinsky” achieves its goal of allowing users to experience synesthesia the way the artist might have is hard to say.

However, it does make the abstract concept easier to grasp, and bringing AI into the experiment could entice a new audience to explore the works of one of the greatest artists of a time before digital technology.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].

Related
AI is helping to ID victims in Holocaust photos
The new website From Numbers to Names uses facial recognition tech to identify previously anonymous faces in Holocaust photos.
“Robot rooms” could be the future of homes and offices 
Integrating robots into walls, ceilings, furniture, and appliances could radically change our indoor spaces.
Cruise robotaxi service is quickly expanding
General Motors subsidiary Cruise is quickly expanding its commercial robotaxi service to more cities in the US.
AI could help cancer patients avoid a deadly recurrence
A new study found that AI can use a patient’s initial skin cancer growth to predict their risk of melanoma recurrence.
AI-discovered drug for ALS enters human trials in just four years
Verge Genomics has begun a trial of their AI-found ALS drug candidate, one of the first tests for this new method of drug discovery.
Up Next
virtual patients
Subscribe to Freethink for more great stories