Shimi

WE recently revived an old social robot from the lab, Shimi, to help people who are deaf and hard of hearing experience music

Shimi expresses music in multiiple forms: gestures, two types of haptics, and an accompanying app. As different parts of the songs change, Shimi’s forms of output will change as well

We worked with 12 people who are deaf and hard of hearing, and asked them their opinions on features/ways that would be beneficial to assist them experience music.

Shimis gestures change based on the genre of the song. It will dance more intensely based on the energy. As the song changes section, shimi will generate a new dance. Shimis hand, which used to be a phone doc, now provides tactile feedback based on the tempo and key signature of the song. The hand will tick like a clock and reset every measure. Shimi also has a base with vibrational haptics that are a filtered version of the song. We amplified the lower frequencies to make the feeling more pronounced. LAstly, Shimis app changes the color based on the emotion of the song, and displays lyrics information