Epoch shuffles gave 16.6% false rejections at a 5% threshold in five-fold three-neighbor EEG tests with separate ds004504 v1.0.9 people and balanced random person labels

We assigned random binary labels to 88 people from OpenNeuro ds004504 v1.0.9, with 44 people per class. Each person's 20 EEG epochs had that person's label. A fixed three-neighbor rule used 95 features and five-fold validation. We tested 1,999 assignments against two independent reference banks, each with 9,999 draws. In folds with separate people, epoch shuffling rejected 331 assignments at p <= 0.05: 16.56% (95% interval 14.99% to 18.25%). Person shuffling rejected 101 assignments: 5.05% (4.18% to 6.10%). These intervals condition on the EEG data and the saved reference bank. The primary test with shared people and the planned time check had 100% epoch rejection. Person rejection rates were 4.85% and 5.80%, respectively. All labels were random, so these rejections do not show a disease effect. The package includes exact mean calculations, a rank proof, code, source hashes, all labels, scores, and inspection tables. Independent checks rebuilt all 13,200 neighbors and checked all 11,994 p values. A complete offline rebuild matched all 13 analysis and figure files byte for byte. One fresh source download reproduced its features exactly. The measured rates apply only to this cohort, fixed model, folds, features, and label rules.

permutation test · EEG · machine learning · calibration · cross-validation · repeated measures · random labels · false rejection