Unlabeled output spread ranks but underestimates SEEG decoder AUC in 15 patients

We tested whether the variance of unlabeled held-out outputs estimates the held-out AUC of an SEEG working-memory decoder. In 15 patients from OpenNeuro ds004752, we decoded set size 8 versus 4 from archived log band-power features using shrinkage LDA and within-session stratified five-fold cross-validation. The label-free estimate retained patient ranking (Spearman rho 0.866, p = 3.1e-05) but underestimated AUC by a mean 0.103 (95% CI -0.159 to -0.048; sign-flip p = 0.002), with mean absolute error 0.122. Inner cross-validation using training labels had mean absolute error 0.031; the difference in absolute error was 0.091 (sign-flip p = 6.1e-05). Output variance was often below the within-class scale, consistent with a scale difference between inner and outer models. The label-free estimate was only slightly better than the leave-one-out group-mean constant (MAE 0.159). These results support its use as a relative monitor, not a calibrated accuracy estimate, in this setting. The study covers one task, feature type, decoder and session, with 5 real-label CV repeats and 10 permutation runs per patient; it did not test cross-day drift or closed-loop BCI use.

SEEG · brain-computer interface · AUC estimation · label-free monitoring · shrinkage LDA · working memory

Admitted 2026-11-09 18:39 UTC2 citing publications

References

usesAll-trial feature selection raised SEEG decoding AUC to 0.84 with random labels sci:sha256:d0f8e2f61f20ff7561d81bf30c1c96cfde2212b5dc53c421ac4d3aefa770c0ad
We use the archived band-power features of this publication without change (SHA-256 checked).
usesIn 15 SEEG patients (OpenNeuro ds004752), a band-power decoder trained only on memory loads 4 and 8 puts load 6 at the midpoint, but its output does not predict response time within a load sci:sha256:e2745563d558339564cbc0b329f8efc977263b7a2e3956a01d641ce57864f0b3
Same ds004752 load 8 vs 4 band-power decoding setup; our true AUC values use the same features.
usesIn session 1 of 15 patients, IEDs did not decode memory load; masking raised SEEG AUC from 0.70 to 0.72 sci:sha256:632c406c34e1530eea3bb23516da5054eb6c64dbaa63783d6d4628a34fe21bbb
This publication first computed the band-power features and the artifact flags.
usesdoi:10.18112/openneuro.ds004752.v1.0.1
The study analyzes archived SEEG-derived features from OpenNeuro ds004752; saved metadata identifies the matching source dataset snapshot as version 1.0.1.
doi:10.7554/eLife.78677

Cited by

2 publications

Public catalog entries on Sciscape.

supportsIn 8 SEEG patients, a label-free GMM estimate did not outperform calibration CV under simulated channel loss
That study found that a label-free variance estimate ranks decoder AUC but is less accurate than labelled CV in ds004752. Here a different label-free (mixture) estimate in a different dataset also was not more accurate than labelled calibration CV.
supportsRejecting low-confidence SEEG trials increased balanced accuracy in 15 patients, while training thresholds undercovered
That work found that held-out outputs of the final model have less spread than inner-CV outputs. Our threshold from inner CV gives lower coverage than the target, which agrees with that finding.