One brain implant decodes speech and gestures together

A single brain implant has translated intended speech and gestures at the same time, allowing two people with severe paralysis to communicate through a personalized animated avatar within seconds.

The proof-of-concept system uses 253 electrodes placed on the surface of the sensorimotor cortex, a brain region involved in speech and body movement. Customized machine-learning models interpret electrical activity and turn it into on-screen text while prompting the avatar to wave, nod, shrug or make other gestures.

One participant had impaired speech and movement after a brainstem stroke. The other had amyotrophic lateral sclerosis, a motor neuron disease that progressively affected speech and movement. During testing, they attempted or imagined phrases and gestures, both separately and in combinations such as saying “hello” while waving.

For the first participant, the system identified intended gestures with 88% accuracy and speech with 84% accuracy. For the second, the results were 66% and 70%, respectively, compared with roughly 9% expected by chance.

The researchers say the study is an early step rather than a device ready for daily use. It involved only two people and a limited selection of phrases and gestures, and accuracy must improve. Still, decoding both forms of expression together could eventually make assisted communication more natural and nuanced.