Decoding sign language finger flexions from high-density electrocorticography using graph-optimized block term tensor regression

Objective. A novel method is introduced to regress over the sign language finger movements from human electrocorticography (ECoG) recordings. Approach. The proposed graph-optimized block-term tensor regression (Go-BTTR) method consists of two components: a deflation-based regression model that sequentially Tucker-decomposes multiway ECoG data into a series of blocks, and a causal graph process (CGP) that accounts for the complex relationship between finger movements when expressing sign language gestures. Prior to each regression block, CGP is applied to decide which fingers should be kept separate or grouped and should therefore be referred to BTTR or its extended version eBTTR, respectively. Main results. Two ECoG datasets were used, one recorded in five patients expressing four hand gestures of the American sign language alphabet, and another in two patients expressing all gestures of the Flemish sign language alphabet. As Go-BTTR combines fingers in a flexible way, it can better account for the nonlinear relationship ECoG exhibits when expressing hand gestures, including unintentional finger co-activations. This is reflected by the superior joint finger trajectory predictions compared to eBTTR, and predictions that are on par with BTTR in single-finger scenarios. For the American sign language alphabet (Utrecht dataset), the average correlation across all fingers for all subjects was 0.73 for Go-BTTR, 0.719 for eBTTR and 0.70 for BTTR. For the Flemish sign language alphabet (Leuven dataset), the average correlation across all fingers for all subjects was 0.37 for Go-BTTR, 0.34 for eBTTR and 0.33 for BTTR. Significance. Our findings show that Go-BTTR is capable of decoding complex hand gestures taken from the sign language alphabet. Go-BTTR also demonstrates computational efficiency, providing a notable benefit when intracranial electrodes are inserted during a patient’s pre-surgical evaluation. This efficiency helps reduce the time required for developing and testing a brain-computer interface solution.