How often would one benchmark score stand out?
Check how selection changes the chance of a benchmark alert.
The question
The interviewer gives you a fictional classifier with a predeclared test. Under the no-signal model, it guesses uniformly and independently on each example.
- Model
- One variant declared before scoring.
- Choices
- Each example has 4 options, with one correct.
- Examples
- Score 4 independent examples.
- Alert rule
- At least 3 correct answers trigger an alert.
“What is the chance this benchmark triggers an alert under the no-signal model?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Which scores trigger an alert?
What would you do here? Open the move you’d make.
Count exactly three correct answers
Tempting, but it misses something. A perfect score also triggers an alert.
Include two correct answers
Tempting, but it misses something. Two is below the stated threshold.
Include three and four correct answers
The strong move. Both scores meet the stated threshold.
See how a strong candidate says it
“Three or four correct answers trigger an alert. I’ll include both.”
Why it works: The threshold is inclusive and four is the maximum.
Step 2: What is the chance for one model?
What would you do here? Open the move you’d make.
12/256, about 4.69%
Tempting, but it misses something. That counts only strings with exactly three correct.
13/256, about 5.08%
The strong move. This counts all 13 alert strings among the 256 equally likely strings.
1/64, about 1.56%
Tempting, but it misses something. That is the chance of getting three particular items correct. The miss may occur at any item.
See how a strong candidate says it
“Exactly three correct gives 12/256. Exactly four gives 1/256. The alert chance is 13/256, about 5.08%.”
Why it works: For exactly three, any of four items can be the miss, with three wrong choices for that item. There is one fully correct answer string.
Step 3: What is the chance after selection?
The reported score is the highest of 4 variants. Each had its own 4 independent examples and independent guesses. Under the no-signal model, the four scores are independent.
What would you do here? Open the move you’d make.
About 18.82%, using the chance none alerts
The strong move. Each variant gets its own independent opportunity to cross the threshold.
About 5.08%, because only one score is shown
Tempting, but it misses something. The reported score was selected from four tries.
About 20.31%, adding four single-alert chances
Tempting, but it misses something. That counts sets with more than one alert more than once.
See how a strong candidate says it
“The chance no variant alerts is (243/256)⁴. The chance at least one alerts is 1 minus that, about 18.82%.”
Why it works: The highest score alerts if any variant alerts. Use the complement because multiple variants can alert in the same set of runs.
Step 4: What evidence would you ask for?
What would you do here? Open the move you’d make.
Report the winning score as proof of signal
Tempting, but it misses something. A selected score can cross the threshold without predictive signal.
Call 18.82% the chance the winner has no signal
Tempting, but it misses something. The calculation assumes no signal. It does not estimate the chance that assumption is true.
Freeze a variant and test it on fresh examples
The strong move. This gives the selected variant a new check after the initial search.
See how a strong candidate says it
“I’d freeze one variant and score it on unopened examples. The alert probability tells me how the rule behaves without signal. It does not tell me the chance this winning variant has no signal.”
Why it works: A new test can use a selection rule agreed before seeing the result.
The answer, in short
- Decision
- Use the four-variant alert chance for the selected score and ask for a fresh test.
- Because
- The selection rule changes how often a score can stand out by chance.
- What would change it
- If the variants share examples or guesses, reassess dependence before multiplying the no-alert probabilities.
Common traps
- Treating a selected maximum as a predeclared test
- Confusing an alert rate under a null model with the probability of the null
Check your answer
- What score raises the alert?
- How many variants were tried?
- Why can you multiply the no-alert probabilities here?
- What would make the next test independent of selection?
Last reviewed Nov 11, 2026