MussBits++

A Sensory Substitution System to Bridge the Gap Between Hearing Loss and Music Perception

Hearing loss and deafness can create social and emotional barriers, including difficulty experiencing music. MuSS-Bits++ aimed to bridge this gap, using sensory substitution technology to enhance musical experiences for people with hearing impairments.

MuSS-Bits++ employs a wearable system consisting of a "Sensor-Bit" and "Display-Bit." The Sensor-Bit captures audio from various sources, analyzes the signal in real-time, and extracts key musical elements. This information is wirelessly transmitted to the Display-Bit, which translates musical components into patterns of light and vibration for the user to experience.

The project involved significant work across hardware, software, and user interface design. Key elements included custom electronic systems optimized for low power consumption, precision, and wearability. This includes the use of capacitive touch interfaces for user interaction. On the software side MussBits++ required on device real-time signal processing algorithms (like Fast Fourier Transform), ultra-low latency wireless communication protocols, and a modular design for future expansion.

MuSS-Bits++ was developed in close collaboration with the Hearing House in New Zealand and the School for the Deaf in Sri Lanka. This user-centered approach shaped the design and ensured it would address the specific needs of individuals with hearing impairments. Its deployment in both locations since 2017 provides tangible proof of its potential for real-world impact. Furthermore, recognition in prominent conferences (CHI, ASSETS), and awards like the Public Good Gold Award at the Best Design Awards in New Zealand highlight the project's innovative nature, further driving the field of inclusive technology forward.
  • Software Engineering
  • Hardware System Design
  • Rapid Prototyping
  • Machine Learning
  • Project Management
  • C
  • C++
  • C#
  • Unity 3D
  • Unreal Engine
  • Android
  • Python
  • Java
  • Kotlin
  • Assembly
  • HDL
  • Embedded Linux
  • Altium Designer
  • LTspice
  • Cadence OrCAD
  • Adobe Illustrator
  • Adobe Photoshop
  • Adobe Premiere Pro
  • Figma

Augmented Human Lab
Website
Public Good Award - Designers Institute of New Zealand Best Design Awards (2019)
Link| Video
ACM Conference on Human Factors in Computing Systems (CHI 2018)
PDF| Video
Assistive Augmentation. Cognitive Science and Technology. (Springer 2017)
PDF
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2016)
PDF
Australian Conference on Computer-Human Interaction (OzCHI 2016)
PDF

< Back to Projects