EM prior correction did not improve log loss in an offline simulated key-press study of 8 SEEG subjects

This work reproduces sci publication 996e476b with new code. A self-paced BCI sees movement rarely, but you usually calibrate the detector on balanced trials. We tested if a label-free EM estimate of the movement prior (Saerens et al. 2002) makes the detector probabilities better. We used archived HG and beta features of 8 SEEG subjects (OpenNeuro ds004473, from sci 04683a2a). A shrinkage LDA with Platt scaling was calibrated on balanced trials 1-100. On trials 101+, we simulated a move prevalence of 0.5, 0.2, 0.1 and 0.05 by subsampling move windows (200 draws). The pre-registered primary endpoint was the log-loss difference EM minus unadjusted (U) at prevalence 0.1. EM did not make the log loss better: mean +0.276 nats per window, 3 of 8 subjects better, two-sided sign-flip p = 0.18. With no shift (prevalence 0.5), EM made the log loss worse in 8 of 8 subjects (+0.069, p = 0.0078). Adjustment to the true prior decreased the log loss (-0.08 to -0.27). Thus, the adjustment is correct, but the EM prior estimate is not sufficiently accurate. EM estimated a prior near 0 in 5 of 8 subjects at prevalence 0.1. Adjusted count (ACC) had a lower prevalence error than EM (0.066 vs 0.075). The values agree with 996e476b to within 0.07 nats and 0.01 prevalence error. Limits: offline, one task, fixed windows not a continuous stream, simulated label shift, threshold 0.5 only. We changed the Platt fit after the first runs (smoothed targets, scaled scores) because the plain fit did not converge.

brain-computer interface · prior shift · label shift · expectation-maximization · log loss · intracranial EEG · shrinkage LDA · probability calibration · prevalence estimation · independent reproduction