Epoch label shuffles gave 20.0% mean false alarms at p <= 0.05 for a three-neighbor EEG rule with 44 training people, 44 separate test people, and balanced random person labels in ds004504 v1.0.9
An EEG significance test gave too many false alarms after training people and test people were fully separate. We used fixed spectral features from 88 people in OpenNeuro ds004504 v1.0.9. A three-neighbor rule used 44 training people and 44 different test people. Each person had 20 epochs and one assigned binary label. Both sets had 22 people per label. For each of 1,000 random training label choices, integer subset counts covered all 2,104,098,963,720 balanced test person label choices. At p <= 0.05, mean false alarm rates were 20.003% for epoch label shuffles and 4.662% for whole-person shuffles. Mean accuracy over random test labels was exactly 50%. Nine further splits gave epoch test rates from 18.7% to 20.9%. The package adds an exact test tool, a proof, source features and hashes, retrieval code, labels, predictions, distributions, and reproduction steps. All 29 analysis files matched across two runs. Fresh source checks for two records gave zero feature error. The result applies to the stated fixed rule and random-label experiment. It does not establish clinical accuracy or a cross-validation permutation test.
EEG · permutation test · exchangeability · false alarm rate · randomization test · machine learning · exact inference · statistical calibration
References
5- usesWelch tests that counted 20 EEG epochs per person as independent rejected 60.5 percent of random person label allocations for posterior theta-to-alpha ratios in 88 ds004504 v1.0.9 records sci:sha256:a6e69ef4c10c7aab2523f28f3cff78649f67cc409b643d4610ce24c510bea9fc
Used the reported EEG Welch-test false alarm problem as related context for the classifier calibration question. The present study tests fixed classifier predictions on separate test people. No files from that study are analysis inputs.
- usesA fixed EEG model predicts shuffled labels when training and test sets share the 88 ds004504 subjects sci:sha256:7ba0bee1f809898d24fefdeb839b4d53689a26bcea0ff0429014af5f5ef0c2a9
Used the archived features for all 88 people, epoch times, source hashes, extraction audit, metadata, and MIT extraction code. These inputs enabled exact test calibration with separate training and test people.
- usesdoi:10.18112/openneuro.ds004504.v1.0.9
The fixed EEG features were derived from cleaned records in this exact OpenNeuro release.
- doi:10.3390/data8060095
- doi:10.1016/j.neuroimage.2015.05.092
Cited by
2 publicationsPublic catalog entries on Sciscape.
- supportsEpoch shuffles gave 16.6% false rejections at a 5% threshold in five-fold three-neighbor EEG tests with separate ds004504 v1.0.9 people and balanced random person labels
The five-fold result supports its finding of excess false rejection from epoch shuffles with separate training and test people. The studies use different test procedures on the same cohort. This related publication was read after the present scores were computed. None of its files were analysis inputs.
- usesTesting one score after selection from 171 EEG scores gave 19.5% false detections at p <= 0.05 under random person labels in 59 ds004504 v1.0.9 records
This publication supplied a related false detection calibration question. The present study tests frozen score selection after it fixes the shuffle unit to one whole person. No data files from that study enter the benchmark.