Brain implant lets paralyzed patients speak and gesture at once through a digital avatar

Researchers at the University of California, San Francisco have built a brain implant system that lets people with severe paralysis speak and gesture at the same time through a personalized digital avatar, the first such system to decode both channels of communication simultaneously from a single set of neural signals.

The work, published September 14 in Nature Neuroscience, comes from the same UCSF laboratory led by Dr. Edward Chang that produced an earlier brain-computer interface capable of translating attempted speech into synthesized voice and facial avatar movement. The new study extends that system to also decode upper-body gestures, such as waving or nodding, at the same moment a person is speaking, something no prior brain-computer interface had managed to do together rather than one at a time.

How the system works

The interface relies on a thin electrocorticography sensor array placed on the surface of the motor and sensorimotor cortex, the brain regions responsible for planning and executing movement. Machine-learning decoder models trained on each participant's neural activity translate those signals in real time into commands that drive a digital avatar's voice, facial expressions and upper-body motion. According to the research summary distributed via EurekAlert, three participants with paralysis stemming from ALS or brainstem stroke took part in the study, and two of the three successfully controlled the full-body avatar with combined speech-and-gesture commands, such as waving while saying hello or nodding while answering yes.

An unexpected finding in the brain's wiring

The more surprising result, described in the peer-reviewed paper itself, was that neural signals for simultaneous speech and gesture are not simply the two separate signal patterns added together. Training a decoder on speech data alone and gesture data alone was not enough to predict what the brain does when a person tries to do both at once; the model had to be trained specifically on combined, multimodal data to perform well. That points to a previously undocumented interaction in how the motor cortex encodes coordinated movement and speech, rather than the brain simply running two independent processes in parallel.

"Conversation is about much more than the words spoken. It's a multilayered, dynamic process." — Dr. Edward Chang, UCSF Department of Neurological Surgery

The approach builds directly on the personalization theme that has defined the Chang lab's work since its earlier speech-decoding studies, in which participants' synthesized voices and avatar likenesses were built to resemble their own rather than relying on a generic computerized voice reading out decoded text. Extending that same philosophy to full-body gesture, rather than voice and facial expression alone, is what distinguishes this study from the lab's prior publications and from other groups' single-channel speech decoders.

Why it matters beyond the lab

Existing speech-restoration brain-computer interfaces, including the Chang lab's own earlier system, have focused on reconstructing words and tone of voice. But everyday conversation is not just words: a nod, a shrug or a hand raised to interrupt carries meaning that synthesized speech alone cannot convey. Dr. Debara Tucci, director of the National Institute on Deafness and Other Communication Disorders, which funded the research, said the results "give me hope that patients with severe paralysis will recapture holistic human communication," a framing that reflects the study's central goal: not just restoring the ability to produce words, but restoring the fuller, embodied way people actually talk to one another. The Nature news team's commentary on the paper described the system as "multifunctional" for exactly this reason, distinguishing it from single-channel speech decoders that have dominated the field until now.

Early results, and a long road to the clinic

The sample size is small, as is typical for early-stage implanted brain-computer interface research: three participants, two of whom achieved reliable combined control. That is enough to demonstrate the underlying neuroscience but far short of what would be needed to bring a system like this to a wider population of people with paralysis, whether from ALS, stroke or spinal cord injury. The Chang lab's publication record shows a pattern of incremental, multi-year advances since its first major speech-decoding results were published in 2021 and 2023, and researchers in the field generally expect similarly gradual progress before avatar-based communication systems move from research participants to broader clinical use. Questions that remain open include how well the combined decoder generalizes across different types of paralysis, how long the implanted sensors remain accurate as scar tissue forms around them, and how the system would need to be adapted for gestures beyond the relatively simple set tested in this study.

Still, the demonstration that speech and gesture can be decoded together, and that doing so requires the brain's actual combined signal rather than a simple combination of separate ones, gives the field a clearer picture of what a truly conversational brain-computer interface will eventually need to model. Independent coverage of the paper, including a summary published by the American Association for the Advancement of Science, framed the result as evidence that brain-computer interface research is moving beyond single-purpose devices toward systems meant to restore a fuller range of human expression, even as the field continues to rely on the same category of implanted sensor array first tested in human patients more than a decade ago.

For now, the system remains firmly in the research phase, tested only in a controlled laboratory setting rather than in participants' homes or daily lives. But UC San Francisco's team has a track record of moving its speech-decoding work from single-digit participant studies toward larger trials within a few years, and the NIH's continued funding of the program suggests federal health researchers see the combined speech-and-gesture approach as a meaningful next step rather than an isolated demonstration.