Machine translation of cortical activity to text with an encoder-decoder framework

A decade after speech was first decoded from human brain signals, accuracy and speed remain far below that of natural speech. Here we show how to decode the electrocorticogram with high accuracy and at natural-speech rates. Taking a cue from recent advances in machine translation, we train a recurre...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Nature neuroscience Ročník 23; číslo 4; s. 575 - 582
Hlavní autori: Makin, Joseph G, Moses, David A, Chang, Edward F
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: United States Nature Publishing Group 01.04.2020
Predmet:
Accuracy
Adult
Algorithms
Brain - physiology
Brain-Computer Interfaces
Cerebral cortex
Coders
Decoding
Electrocorticography
Female
Humans
Machine translation
Middle Aged
Neural networks
Neural Networks, Computer
Recurrent neural networks
Representations
Speech
Speech Perception
Transfer learning
Translation
Words (language)
ISSN:1097-6256, 1546-1726, 1546-1726
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:A decade after speech was first decoded from human brain signals, accuracy and speed remain far below that of natural speech. Here we show how to decode the electrocorticogram with high accuracy and at natural-speech rates. Taking a cue from recent advances in machine translation, we train a recurrent neural network to encode each sentence-length sequence of neural activity into an abstract representation, and then to decode this representation, word by word, into an English sentence. For each participant, data consist of several spoken repeats of a set of 30-50 sentences, along with the contemporaneous signals from ~250 electrodes distributed over peri-Sylvian cortices. Average word error rates across a held-out repeat set are as low as 3%. Finally, we show how decoding with limited data can be improved with transfer learning, by training certain layers of the network under multiple participants' data.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:1097-6256
1546-1726
1546-1726
DOI:10.1038/s41593-020-0608-8