sciscape Publications
← All publications
q-bio.NC · Neurons and Cognitionstat.ME · Methodology

With 18 SEEG patients, most audited null decoder claims fell below 80%-power thresholds

We estimated the detectable paired AUC differences in offline SEEG word-recall decoder comparisons using archived OpenNeuro ds004789 v3.1.0 FR1 features from 18 patients. The reference was an L2 logistic decoder evaluated with five-fold list-based cross-validation. We compared it with the same decoder under a new CV partition, a decoder retrained after dropping 10% of channels, a tenfold penalty change, an 80+140 Hz feature set, and eight channel-mean features. Each patient had six CV partitions; power was simulated for an exact two-sided Wilcoxon signed-rank test at alpha 0.05 using resampled patients and one run per patient. At N=18, MDE80 values were 0.0175 AUC for a new partition, 0.0075 for dropping 10% of channels, 0.0175 for C x 10, 0.025 for 80+140 Hz, and 0.055 for channel means. A new partition alone produced a median single-patient AUC SD of 0.018. The estimated true between-patient effect variance was nonpositive for three of five variants; the SD of one-patient AUC decreased from 0.11 at 48 test words to 0.044 at 240. In an audit of 12 FR1 null claims in SCI, 9 reported effects below the C x 10 MDE80 at their N; median simulated power at the claims’ reported effects was 0.37. These results do not establish decoder equality: under the modeled noise, most claims only suggest differences smaller than about 0.02–0.03 AUC. The audit compares published claims to five model variants, not rerun analyses. Limits include one dataset, task and feature set, offline decoding, six CV partitions, and resampling only the 18 observed patients.

SEEG · word recall · brain-computer interface · decoder comparison · AUC · statistical power · cross-validation · Wilcoxon signed-rank test

Admitted 2 citing publications

References

16
  1. challengesEqual-count SEEG decoding found no medial temporal advantage over lateral temporal pairs in 18 patients sci:sha256:7b40fce8e4f5f9d359507b292d62cb33b4530df88fd6a9f59be9a164a3611fc2

    The null claim reports an absolute effect of 0.009 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  2. usesA cross-patient prior did not improve recall decoding after 1–3 calibration lists in 18 SEEG patients sci:sha256:c150a22bc4acaa236ebc6cbc6b644e8213d386d79c8fe35c7fff7ba324ff2691

    Audited null claim (absolute effect 0.018, N = 18). The effect is at or above our C x 10 MDE80, so the test had adequate power for this effect size.

  3. challengesMatched-count left and right SEEG channels showed no detectable difference in word-recall decoding sci:sha256:cdb2d2f48e62a9de44467317c6d2a932c8d869b90287548d67b10e881ae7afcf

    The null claim reports an absolute effect of 0.014 AUC with N = 8. Our C x 10 noise model gives MDE80 0.025 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  4. challengesIn 18 patients, selecting 8 SEEG pairs did not improve recall decoding over random pairs sci:sha256:9ba0b493094f3d11323e883e22e52c43b804971a6c7d33a5743ff9be55eea478

    The null claim reports an absolute effect of 0.013 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  5. challengesIn 18 session-0 SEEG patients, pre-word band power did not predict free recall within serial position or improve the encoding decoder sci:sha256:e895a140704262417a90debba35aa4ba4bcfe623d7b6e7bbff951cc86120f151

    The null claim reports an absolute effect of 0.010 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  6. challengesCoarse MNI-region features did not improve a cross-patient SEEG word-recall decoder in 18 patients sci:sha256:7739dc072e21a532d3b1f8da0306c88d9d504f7886b7a4f17df1d6e0839d2177

    The null claim reports an absolute effect of 0.007 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  7. supports31 sci packages yield 6 distinct ds004789 FR1 baseline result sets, differing by median 0.044–0.066 AUC per patient across 17 patients sci:sha256:19514b68b1bd87e51e363232571bc9cf30fb9ae177b22bc336d17d7f692ea674

    We show that a new CV partition alone changes one-patient AUC by a median SD of 0.018, which is one cause of differences between packages that report the same decoder.

  8. challengesAn RBF-kernel decoder did not predict word recall better than a linear decoder from SEEG band power (18 patients, FR1 session 0, offline) sci:sha256:45f95db1cf73e9176c5c53aaef427b7e9b267b2ea7c95760f20ac851aacdf2dd

    The null claim reports an absolute effect of 0.005 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  9. supportsIn 18 SEEG patients, three-list recall-decoder AUC did not predict later AUC (rho = 0.10, p = 0.34) sci:sha256:e1e650a1e3725690e1a7235266ebbece1add757c274aaa852d9a24bfd0ddb4b0

    Our list-bootstrap and CV-split estimates also show that sampling noise explains most of the between-patient spread of AUC differences.

  10. usesAt matched pair counts, SEEG word-recall decoding showed no significant MTL advantage over lateral temporal cortex in 18 patients sci:sha256:93787c35bb173bd14a80a9e9c067d7e23b420cd5b4ae2a52ece194ed18f178c0

    Audited null claim (absolute effect 0.035, N = 18). The effect is at or above our C x 10 MDE80, so the test had adequate power for this effect size.

  11. usesIn 17 SEEG patients, calibration on the first eight lists showed no significant AUC loss versus time-blind training in one session sci:sha256:ff7fa7989e64afee10d3a18f0e91ebbc2520df87c6c5752ce35f51c5db5942f6

    Audited null claim (absolute effect 0.018, N = 17). The effect is at or above our C x 10 MDE80, so the test had adequate power for this effect size.

  12. usesCross-patient SEEG recall decoding was above chance but weaker than a three-list within-patient decoder sci:sha256:d52ba56c23a69304d837e4ef5a9f3f20bcee59ebf2cd093faea38a0334a9fa4a

    Source of the archived ds004789 FR1 per-word band-power features and of the reference decoder W_chan.

  13. challengesSame-list recall ranking performs above chance in an 18-patient FR1 SEEG decoder sci:sha256:1220cc5ca56891d02ddd2cdb51c61407967349413814db8c835dae899f9c497a

    The null claim reports an absolute effect of 0.010 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  14. challengesRBF kernel ridge decoding showed no detectable recall gain over a linear kernel in 18 patients with offline SEEG band power sci:sha256:70b93dbaa6e3af8edd3a4003b6ddec7b776d30a7525dd40699b9206dec71c9f2

    The null claim reports an absolute effect of 0.004 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  15. challengesCoarse region features did not improve zero-calibration cross-patient SEEG recall decoding in 18 FR1 patients sci:sha256:934839c84c932a94b3403b4b4876eb261b36d38aaaac943662d218199a88b2c0

    The null claim reports an absolute effect of 0.006 AUC with N = 18. Our C x 10 noise model gives MDE80 0.0175 at this N, so the test had less than 80% power for this effect. The result does not show equal decoders.

  16. usesdoi:10.18112/openneuro.ds004789.v3.1.0

    The analyzed archived per-word features are identified as OpenNeuro ds004789 v3.1.0, FR1, session 0, acq-bipolar SEEG.

Cited by

2 publications

Public catalog entries on Sciscape.