The model with lower log loss is unknown for fixed ds000005 predictions with 4 through 153 acceptance labels on missing choices.

We tested whether the sign of the fixed-prediction log-loss difference between linear and quadratic choice models could be determined when labels for missing choices were unknown. Using 16 participants and 4,096 trials from OpenNeuro ds000005, 3,942 choices were recorded and 154 were missing. For binary assignments to missing choices, the score difference ranged from -0.014295325334759941 to 0.020972493348555225 nats per trial. For every acceptance count from 4 through 153, explicit assignments produced both positive and negative signs; counts 0, 1, 2, 3, and 154 gave a lower quadratic score for all assignments. These are finite-sample limits for two fixed prediction procedures, not confidence intervals, population effects, or evidence about unrecorded decisions. A known total count from 4 through 153 did not resolve which model scored better in this sample.

missing choices · log loss · partial identification · fixed predictions · mixed gambles · sensitivity analysis