Pre-CV feature selection inflated SEEG word-recall AUC and produced high AUCs with permuted labels

We tested feature-selection leakage in a weak-signal SEEG word-recall task using archived per-word band-power features from 19 patients in OpenNeuro ds004789 (FR1, session 0; 960–1520 features). We compared top-k t-test feature selection before 5-fold list-grouped cross-validation with feature selection inside each training fold, using shrinkage LDA. The plan was written before computation. At 300 words and k = 10, leaky selection increased mean AUC from 0.615 to 0.774 (+0.159) in all 19 patients (sign-flip p < 1e-4). Inflation was 0.30–0.40 at 60 and 120 words. With permuted labels, leaky selection produced mean AUCs of 0.75–0.96, while proper selection produced 0.46–0.50. The honest decoder effect (+0.06 to +0.11 AUC) was smaller than the leakage error. The acquisition code targets the live OpenNeuro S3 mirror, and the exact dataset release used is uncertain. Limits: one feature type, one decoder family, session 0 only, and 20 permutations per setting.

feature selection · cross-validation · data leakage · SEEG · word recall · AUC · permutation test · shrinkage LDA