Researchers at Georgia Tech have been working to improve a musical robot called Shimon. Now Shimon doesn’t simply play music, he also writes the words to his own songs – and sings them.
Shimon is a four-armed robot with a ball-like head. He holds small mallets in his “hands” to play a kind of xylophone called a marimba. As Shimon plays, his head moves around in time to the music.
(Source: Gil Weinberg, Georgia Tech.)
Shimon has been around for quite a while as a musical robot. Even back in 2015, he was playing with other musicians at the Kennedy Center in New York. What Shimon could do back then was already pretty cool.
Using “deep learning”, Shimon was taught to write his own music. Deep learning – also known as Artificial Intelligence (AI) – means that computer programs sort deeply through huge amounts of information. This allows them to find patterns humans may not have noticed. The programs can then use those patterns in new and surprising ways.
(Source: Gil Weinberg, Georgia Tech.)
For Shimon, that meant he could not only make up his own music, he could do it in real time, while playing with other musicians. This is called “improvising”.
Now Shimon is back with a whole bunch of new tricks. He can write the words to his own songs, and sing them.
(Source: Gil Weinberg, Georgia Tech.)
Shimon learned to write the words for the songs the same way he learned to write music – by being “fed” thousands and thousands of examples. Shimon was trained on the lyrics (words) to 50,000 songs.
Georgia Tech professor Gil Weinberg, who leads the Shimon project, gets Shimon going with a starting idea. Shimon then writes the lyrics based around that idea. Mr. Weinberg usually provides much of the music for the songs, but Shimon helps out there, too.
(Source: Gil Weinberg, Georgia Tech.)
Richard Savery, one of the researchers, says, “You’ll get a word like ‘storm’, and then it’ll generate a whole bunch of related words, like ‘rain’.” He says Shimon comes up with many ideas, keeps the good ones, and builds on them.
Singing the songs is another story. To give Shimon a voice, the Georgia Tech team worked with experts at Pompeu Fabra University in Spain. The voice was created using AI and sounds very much like a man.
Shimon’s face has also gotten new features. Shimon’s mouth now moves smoothly in time as he sings. He also has eyebrows, which allow his face to show more emotion.
(Source: Gil Weinberg, Georgia Tech.)
To show off Shimon and his new abilities, the researchers have recorded one of Shimon’s songs, called “Into Your Mind”. They have made a video for the song, which is out on YouTube.
Later this spring, the team plans to put out an album of about 8-10 of Shimon’s songs. The album will be released on the music streaming service Spotify. There are also plans for Shimon to go on tour with a band to play and sing his songs live.
(Source: Gil Weinberg, Georgia Tech.)
For Mr. Weinberg, that’s the main goal behind the Shimon project – not to have robots take over, but to have robots and humans make something beautiful together.
Did You Know…?
By chance, Shimon’s new song has some words of hope for people stuck at home because of the coronavirus : “Home is where the rainbow comes.”
(Source: Gil Weinberg, Georgia Tech.)