Augmented reality has previously allowed us to place furniture in a room before buying, bring photos to life and go gaming on the streets. Now you can have a life-like avatar perform an upbeat version of a Beethoven classic in your front room.
Online music sensation The Piano Guys have partnered with AR/VR developers Massive Technologies to seat the band's Jon Schmidt at the piano in your home, if you have one, and at a virtual one if you haven't.
Massive Technologies has generated a realistic avatar of Schmidt to perform "with total visual and musical accuracy." Smartphone or tablet users just need to fire up the AR Pianist iOS/Android app, point the device's rear camera at any piano and the app will put Schmidt at the keyboard.
The app uses machine learning to place the virtual virtuoso at the piano within an accuracy of a centimeter, and the device holder can move around the space with six degrees of freedom to view his playing from any angle. If you want to get closer to the action, you can zoom in, and should you want to use the experience to learn from a master, playback can be slowed down by up to 100 times.
Sheet music highlighting notes as they are played also scrolls in the app to aid learning, and there's a view which lights up and color codes the keys onscreen to allow students to "jam with Jon" on their own piano. The app can use the mobile device's microphone to listen out for pitch discrepancies and can provide real-time feedback.
The keys of the real-world piano won't move of course, and the instrument will stay silent, but onscreen the playing area comes to life and the speakers of the mobile device ring with the notes of the personal at-home concert. That said, if you have a MIDI-equipped acoustic or digital piano, the app can wirelessly trigger notes to sound on your home piano. Clever stuff. And if you don't have a piano at home, the app will generate a concert grand right in front of you... well, on the device screen anyway.
"Any 88-key keyboard instrument, acoustic or digital, will work, but it is no obstacle if one does not have a piano at all," explained Craig Knudsen, video production manager and official "technology guy" for The Piano Guys. "In this case, AR Pianist will recognize a clear floor space or tabletop and provide a detailed 3D model of the Yamaha C7 Neo concert grand featured on so many of The Piano Guys’ recordings and album covers."
The track chosen by The Piano Guys for their first outing on the AR Pianist is cut from their latest album – titled 10 – and is a lively rendition of Beethoven's Für Elise, just in time for the classical composer's 250th birthday next month. The app also includes a growing library of pieces performed by world-class players, but only the Für Elise Jam features a motion-captured 3D avatar of Jon Schmidt. The others feature an avatar of a generic player.
The app is reported compatible with most smartphones and tablets released in the last few years, but those devices packing LiDAR technology, such as the iPhone 12, or ToF sensors, like the Samsung Galaxy S20, will have an improved experience – as anyone stepping into the field of view can actually be placed in the room with Schmidt, or even sat on the piano bench next to him.
The AR Pianist app is free to download, but unlimited access to all of the songs and premium features is available for a weekly subscription of US$2.99, or $59.99 annually. Future plans for the app include a full AR/VR version that will work with headsets like the Oculus Quest. The video below has more.
Sources: Massive Technologies, The Piano Guys
