Expert raters agree on evoked response labels with median kappa 0.72 in OpenNeuro ds004774

Visual labels of early cortico-cortical evoked potentials (CCEP, N1 response after single-pulse electrical stimulation) are used as ground truth to train and test automated detectors such as ER-Detect. We compared labels in OpenNeuro ds004774 v1.0.0 for 13 rater pairs across 11 patients. The primary analysis used common channel-by-stimulation-pair cells coded 0 or 1 by both raters. Raw agreement was 83–99%, while Cohen’s kappa had median 0.72 (range 0.50–0.88; 2 of 13 pairs below 0.6) and response-label Jaccard agreement had median 0.61 (range 0.39–0.80). The two Mayo SEEG pairs had kappa values 0.70 and 0.83. In a sensitivity analysis counting code 2 as a response, median kappa was 0.67 (range 0.54–0.87). High raw agreement reflects the predominance of no-response cells. Differences in response rates suggest that some disagreement reflects different response thresholds. Detector scores against one rater should be interpreted relative to this inter-rater reference level. The analysis includes only 13 pairs from 11 patients, only two SEEG pairs; codes -1 and 2 are undocumented in the files, and raw signals were not analyzed, so waveform properties underlying disagreement were not tested.

cortico-cortical evoked potentials · CCEP · inter-rater agreement · Cohen kappa · electrical stimulation · OpenNeuro · ER-Detect