DeLong variances were calibrated, but subject heterogeneity made fixed-effect pooling invalid for population claims in 8 ds004473 SEEG subjects

Question: Can trial-level variances give more power than the exact sign-flip test over 8 subjects, which detects only differences of 0.032 AUC or more with 80% power? We analyzed archived high-gamma (70–150 Hz) and beta (13–30 Hz) features from 8 subjects in OpenNeuro ds004473, version 1.0.2. LDA transfer AUCs matched the archive. For three paired AUC contrasts, we estimated per-subject DeLong variances, with and without clustering by trial, calibrated them using 1,000 random half-splits of each subject’s test trials, and compared fixed-effect, DerSimonian–Laird, Hartung–Knapp–Sidik–Jonkman (HKSJ), one-sample t, and exact sign-flip tests. DeLong variances were calibrated: the SD of half-split z scores was 0.956–1.003 for naive and 0.987–1.003 for clustered estimates. Test-trial noise explained about 20% of the between-subject variance for the primary contrast (high-gamma plus beta minus high-gamma); I² was 0.80 for that contrast, 0.90 for beta minus high-gamma, and 0.55 for top-8 versus all high-gamma channels. Across 256 sign-flipped null data sets, fixed-effect tests rejected 30% and 56% of the time for the first two contrasts, while DerSimonian–Laird rejected 10–15%. HKSJ, the t test, and sign-flip test kept rejection rates near 5%. HKSJ did not give a smaller observed p value than sign-flip for the primary contrast (0.044 versus 0.031). In these 8 subjects, trial-level variances were calibrated, but subject heterogeneity dominated paired AUC differences. Population claims require more subjects; fixed-effect pooling is not valid for them. Limits: one dataset and task, 8 subjects, 3 contrasts, imprecise I² estimates, a conditional sign-pattern check, variance calibration at half the test-trial count, variance conditional on trained models, and no preregistered plan.

DeLong variance · AUC · subject heterogeneity · sign-flip test · fixed-effect pooling · SEEG · LDA · OpenNeuro ds004473