A watch-like device that helps people feel the music

Called Muss-Bits (Musical Sensory Substitution Bits) it has also been named a finalist in the Designers Institute of New Zealand Best Design awards in the Public Good category.

It was developed by Associate Professor Suranga Nanayakkara and his team in the Augmented Human Lab at the Auckland Bioengineering Institute.

Muss-Bits is comprised of two parts. A “sensor bit” that can be placed on or plugged into various sound sources such as musical speakers or an instrument, and which includes an audio jack, a contact microphone, a signal processing unit and a wireless module.

It also has a “display bit’”, which is basically a vibrotactile motor and LED lights encased in what looks like a watch. 

The development of Muss-Bits was partly inspired by renowned deaf multi-percussionist and performer, Dame Evelyn Glennie, who feels music through her feet.

The device draws on a similar principle, but worn like a watch so people feel music through their wrist and arm. “It is honestly life changing,” says Marama Bowler who has been trialling the device for the past five months.

Ms Bowler lost her hearing three years ago at the age of 19 and was connected to Dr Nanayakkara though the Hearing House, an Auckland charity that helps deaf children and adults. He invited her to try the device, and if she liked it take it home.

“Suranga asked me choose a song on YouTube,” she recalls. “The first song that came into my head was Uptown Funk by Bruno Mars. The minute it started with its first beat I started crying. It was like I was fully listening to the song.”

It has brought her a lot of joy, particularly when driving the car and she can turn up the stereo, and has helped her recall and feel music she used to be able to hear. “I remember a lot of RnB music, like Usher, Alicia Keys, Mario and Nelly … basically stuff from 2015 and before.”

“With Muss-Bits I'm able to somewhat hear music again. I watch a lot of karaoke on YouTube and sing along and everything. Now I can watch the video, and the beats that come off it make me understand when the song starts without needing the karaoke version.”

Muss-Bits could be used to help deaf children learn music by introducing them to the beat and rhythm of music early on. It could be used to enhance the feedback loop of auditory information for deaf musicians, and help them evaluate their own performance.

Dr Nanayakkara and his team have have created a number of devices aimed at making lives easier. This includes the Finger Reader, which allows the visually impaired to read text through a device they wear on their fingers, and ChewIt, which allows for discreet and hands-free interaction with a phone, computer, smartwatch and so on.

He and his team are in regular contact with Ms Bowler, and now looking to adapt Muss-Bits in response to her feedback.

That includes incorporating a speech-recognition system that converts voice to text in real time, but also a microphone array that allows the user to identify the direction of the voice. “So that if you’re in a meeting, you can read what is being said, but also see who’s saying it,” says Dr Nanayakkara.

He and his team have a driving philosophy, focused on creating human-computer interfaces that learn and adapt to people instead of the other way around, to develop technologies that actually make life easier, particularly for those who face challenges due to sensory deficits in hearing or vision.

“We want to have a meaningful impact on people’s lives,” he says. 

There are many ways you can make a difference and support the Auckland Bioengineering Institute - you can make a donation or bequest, or even subscribe to our newsletter.