Robots are generally presumed to be coming for our manual jobs, but lately it seems that even more creative endeavors aren’t safe from automation. Shimon the robot is a marimba-mashing musical maestro we first saw back in 2017, and now he’s learned to sing, dance, write lyrics and compose his own melodies. Better still, he’s about to go on tour to promote his first album.
Last time we checked in, Shimon was more focused on the marimba, bashing out simple tunes of his own creation. Now it looks like he’s become way more versatile, and his career is taking off as a result. Truth be told, Shimon’s first single, Into Your Mind, is actually pretty damn catchy. Take a listen:
Shimon learns his craft the same way other creative robots do – by being fed huge amounts of data from existing human examples. In this case, that meant 50,000 lyrics from jazz, prog rock and hip-hop music.
With that foundation laid, Shimon can then begin assembling his own lyrics from the rules he’s learned. But there’s more to it than just repeating “yeah baby tonight” – the team made sure that Shimon understood some of the other ingredients that make music so appealing.
“There are lots of systems that use deep learning, but lyrics are different,” says Richard Savery, a researcher on the team. “The way semantic meaning moves through lyrics is different. Also, rhyme and rhythm are obviously super important for lyrics, but that isn't as present in other text generators. So, we use deep learning to generate lyrics, but it's also combined with semantic knowledge.”
Shimon’s creator, Georgia Tech Professor Gil Weinberg, starts off by giving him a theme, such as space, and the robot will then write lyrics based on that theme.
“You'll get a word like ‘storm,’ and then it'll generate a whole bunch of related words, like ‘rain’,” says Savery. “It creates a loop of generating lots of material, deciding what's good, and then generating more based on that.”
The resulting lyrics definitely sound like something an AI would come up with. They’re made up of words and phrases that sound like sentences at a glance, but don’t really make any sense, like “there may be music or a star into your mind,” or “home is where the rainbow comes.” Then again, plenty of human-written songs are just nice-sounding nonsense anyway, so maybe Shimon isn’t that far off the mark.
The robot can also chip in with some melodies of its own making, but for the most part, those are composed by Weinberg to fit Shimon’s lyrics.
It’s not just during the production phase that Shimon has upskilled – he now has a much stronger stage presence than he used to. He still plays the marimba of course, but he now does so faster and varies from soft to strong dynamic range.
Best of all, Shimon now sings his own lyrics himself. A team at Pompeu Fabra University developed a unique voice for him using their own machine learning algorithms, trained on hundreds of songs.
Shimon has also been given a facelift. Where he once just looked like an expressionless metal ball on a robot arm, Shimon now has eyebrows that waggle, a mouth that syncs with his singing, and head movements that let him convey emotion better. He’ll bob his head to the music, bend down towards the marimba during more intense parts of the song, or look around at his human bandmates.
And this robot-human collaboration is just the beginning. The team says that Shimon and the band have an album coming out in the next few months, comprised of eight to 10 songs. After that, there’s even a tour planned.
While much of that turnout will probably be thanks to people wanting to watch a robot croon along with humans, the researchers say that AI-aided art should be able to stand on its own.
“I think we have reached a level where I expect the audience to just enjoy the music for music’s sake,” says Weinberg. “This is music that humans, by themselves, wouldn't have written. I want the audience to think, ‘There's something unique about this song, and I want to go back and listen to it, even if I don't look at the robot’.”
The team explains how Shimon works in the video below.
Source: Georgia Tech