A deaf person walks into a bar. That isn’t the beginning of a joke, but a potentially frustrating situation—unless the bartender happens to know sign language. That’s where Hand Talk comes in. It translates spoken words into sign language that an avatar then conveys on a smartphone screen.

For now, Hand Talk can only translate Portuguese into Libras, the sign language used in Brazil—the home of the program’s creator, Ronaldo Tenório. But Brazil alone has at least 10 million deaf people, one million of whom have downloaded Hand Talk’s mobile app.

The users hold up their smartphone to a hearing person, who sees a message on the screen that says “Speak to translate.” As soon as the person starts talking, an animated avatar named Hugo begins signing.

Turning the audio into animations of gestures requires laborious programming because everything has to be exactly right, all the way down to Hugo’s facial expressions, which also carry meaning in sign language. Tenório and his team feed their program thousands of example sentences every month and match them with 3-D animations of sign language. They constantly push these improvements out through app updates.

Tenório plans to roll out different versions of the avatar in the future so users can switch the gender or race of their Hugo in an effort to broaden the appeal and accessibility of having a virtual translator in one’s pocket. 

—Julia Sklar