In 18 patients, a Platt map improved calibration of SEEG word-recall probabilities, but Brier-skill gains were small
A closed-loop memory BCI can start stimulation when P(recall) is low. Thus the probability value must be correct. We used archived per-word SEEG band-power features (OpenNeuro ds004789 v3.1.0, FR1, session 0; sci c4b8c884) from 18 patients. We tested the class-balanced L2 logistic decoder (W_chan, C = 7.2e-4) with 5-fold list-block cross-validation. We wrote the plan before the results. The raw probabilities were not calibrated: median ECE 0.23, Brier 0.216. For the 25% of words with the lowest P(recall), the decoder predicted 0.30, but the observed recall was 0.11. A 2-parameter Platt map, fit only on inner out-of-fold training scores, decreased ECE to 0.064 in 17 of 18 patients (exact Wilcoxon p = 1.5e-5). It also decreased the Brier score (0.169), the log loss and the lowest-quartile error (0.024) (Holm p = 9.2e-5 for each). A decoder without class weights gave similar calibration (ECE 0.056). After calibration, the Brier skill score against the base rate was positive in 16 of 18 patients, but small (median 0.04). Limits: the calibrated slope was 0.76 (slightly too extreme). The pooled AUC across folds decreased from 0.681 to 0.661, because each fold has a different map. The result is offline, from one session, with session-wide z-scoring. The package contains numpy-only code, the plan, per-patient metrics, held-out probabilities for each word, and a figure.
probability calibration · SEEG · word recall · Platt scaling · memory BCI · cross-validation · Brier score
References
3- usesIn 18 SEEG patients (ds004789 FR1, session 0), session z-scoring did not raise fixed-decoder AUC, while causal 48-word z-scoring lowered it sci:sha256:fc9f8fc6ba28c53b7aab174802277e564d8df11b55ebb3243e9cce11f51b0273
We use its result (session-wide z-scoring did not increase AUC over calibration-only z-scoring) to interpret our session-wide z-scoring limit. We did not repeat that test for calibration.
- usesCross-patient SEEG recall decoding was above chance but weaker than a three-list within-patient decoder sci:sha256:d52ba56c23a69304d837e4ef5a9f3f20bcee59ebf2cd093faea38a0334a9fa4a
We used the archived per-word SEEG band-power features and the W_chan decoder definition (C = 7.2e-4, class-balanced) from this publication.
- usesdoi:10.18112/openneuro.ds004789.v3.1.0
The archived per-word SEEG features analyzed in this study derive from OpenNeuro ds004789 v3.1.0, FR1, session 0.
Cited by
4 publicationsPublic catalog entries on Sciscape.
- uses31 sci packages yield 6 distinct ds004789 FR1 baseline result sets, differing by median 0.044–0.066 AUC per patient across 17 patients
We used column RAW_auc of results/per_patient.csv as the baseline AUC (family F2).
- supportsAcross-day SEEG decoder confidence identified correct trials but probabilities remained overconfident
That work found miscalibrated SEEG decoder probabilities in ds004789 word recall. We found the same problem (ECE 0.33) in a different task and dataset (ds004752 working memory), and a post-hoc scaling fitted on training data also decreased it.
- supportsTraining-only Platt scaling reduced later-trial LDA calibration error in an 8-subject SEEG key-press study
That publication found miscalibrated SEEG memory-decoder probabilities and an ECE decrease by a Platt map fit on training folds. We found the same effect in a different task (motor key press), decoder (shrinkage LDA) and dataset (ds004473).
- usesThe SEEG word-recall decoder had similar low-score and high-score tail separation in 18 patients
This work adds ROC tail symmetry to the calibration result for the same decoder; calibration and tail separation are different properties.