Pre-cross-validation feature ranking inflated SEEG AUC in two offline datasets, including with 132–298 trials

Independent numpy code reproduced feature-selection leakage in SEEG decoding using archived band-power features from two datasets and two tasks. In 15 ds004752 patients (20–42 trials; 70–280 features), ranking the top 10 features on all trials before 5-fold cross-validation gave mean real-label AUC 0.859 versus 0.687 with nested ranking (difference +0.172, positive in 15/15, exact one-sided p = 3e-5). With random labels, mean leaky AUC was 0.801 versus 0.499 nested, meeting the pre-set reproduction criterion. In 8 ds004473 subjects, the random-label leaky AUC was 0.841 at 20 trials and 0.619 across all 132–298 trials; it remained 0.59–0.67 at about 210 features. The pure leak was larger at 20 trials in all 8 subjects (exact one-sided p = 0.0039). The real-label leaky-minus-nested difference decreased from +0.161 at 20 trials to +0.016 with all trials, and depended on signal strength. Across 47 subject-by-trial-count points, the descriptive Spearman correlation between log(features/samples) and pure leak was 0.88. These offline results cover k = 10 and the tested archived features; the ds004752 release is uncertain, and no raw-data reanalysis or closed-loop BCI test was done.

feature selection · data leakage · cross-validation · AUC · SEEG · brain-computer interface · random-label control