In 15 SEEG patients, pooled leave-one-out scores lowered the null AUC in small decoding sets

Question: with 20-42 trials, does the cross-validation (CV) scheme move the chance-level AUC of an SEEG decoder away from 0.5? Data: archived 3-6 s maintenance band power (5 bands, 70-280 features) of 15 patients in OpenNeuro ds004752 (verbal Sternberg, set size 8 vs 4), copied from sci 7ff3bfd4. Method: Ledoit-Wolf shrinkage LDA. We compared 4 CV schemes: leave-one-out (LOO) with pooled scores, 5-fold with pooled scores, 5-fold with the fold-mean AUC, and leave-pair-out. Each patient had 40 label permutations. Group statistics used an exact sign-flip test. Result: with random labels, LOO pooled gave a mean AUC of 0.461 (mean minus 0.5 = -0.039, 95% CI -0.055 to -0.023, 14/15 patients below 0.5, p = 0.0002). The other schemes stayed at 0.50 +/- 0.005 (p = 0.46-0.90). The LOO bias was largest with few trials (rank correlation with trial count 0.76). At 20 trials, all schemes were below 0.46. The null 95th percentile was 0.69-0.79. Against its own permutation null, each scheme found the same 6/15 significant patients. Real-label LOO AUC was 0.016 lower than pooled 5-fold AUC (not significant, p = 0.17). Assumptions and limits: one dataset, one contrast, one classifier, offline, only 40 permutations. Conclusion: for small SEEG BCI calibration sets, do not use 0.5 as the LOO chance level. Use a permutation null with the same CV scheme.

cross-validation · area under the ROC curve · leave-one-out · permutation test · stereoelectroencephalography · brain-computer interface · shrinkage LDA · small sample size