Five Trials or One: Why Averaging Beats Your Best Score

Your fastest attempt is mostly a measurement of luck. It is also the number most sites lead with, which is a mistake worth explaining.

The 5-Trial Challenge exists specifically because of this. It records every attempt, reports the average, and gives you the spread as a consistency figure. Here is what that buys you, with arithmetic you can check.

A worked session

Five attempts in one sitting. Four cluster tightly and one is an outlier — an interruption, a notification, or simply a lapse. The values below are illustrative rather than recorded, but the statistics are exactly what the mode computes.

What the three figures say

• Best = 208 ms. Real, but it describes one attempt out of five. Using it to represent "your reaction time" is using a minimum as if it were a typical value.
• Mean = 255 ms. This is the figure that describes the run. Note that the single slow trial pulled it up by about-9 ms — which is correct behaviour, not a flaw.
• Spread = 194 ms range, 74 ms SD. A tight session has a small standard deviation, which is what makes the mean trustworthy. A wide one means the run was disturbed, and the mean is less representative.

Why not just drop the outlier?

Because the research literature treats the removal of slow trials as a biasing practice rather than a refinement. In serial reaction-time work, discarding the fastest and slowest readings to compute a "clean" average is standard in some protocols — but it is also exactly how a genuine performance problem gets hidden from the person who needs to see it.

A session where four attempts land at 210–230 ms and one comes in at 400 ms is information. The 400 ms attempt tells you something happened: an interruption, a lapse of attention, or a genuine dip in alertness. Deleting it produces a nicer number and less truth.

This site keeps every attempt. The consistency figure exists so you can see how much to trust the mean, rather than so you can discard the inconvenient samples.

The run-void rule, and why it is different

The 5-Trial Challenge does discard one thing: if too many trials end too early or unanswered, the whole run is marked void and nothing is recorded.

That is categorically different from removing outliers. Voiding discards the entire run rather than the inconvenient part, so it cannot be used to flatter a result — if anything it costs you the score. It exists because a run of early presses does not measure reaction time at all, and recording an average of it would be misleading.

How many attempts do you need?

Five is the practical minimum, and it is what this mode records. The reason is that the standard error of a mean falls roughly as the square root of the sample count:

The first four rows are why a single attempt is nearly worthless: one reading has a typical error around 20 ms, which is larger than most of the differences people hope to find. Five attempts halves it to about 9 ms. Twenty gets you to roughly 4.5 ms — which is the scale at which genuine effects become detectable.

This is also the honest answer to "how do I know if I have improved?" — not by comparing two single runs, but by accumulating enough attempts that the error is smaller than the change.

A workable protocol

1. Run the 5-Trial Challenge, or any mode, five times.
2. Discard the first attempt as a warm-up — the practice effect is real and front-loaded.
3. Average the remaining attempts. Do not take the best, and do not remove outliers.
4. Check the consistency figure: a wide spread means the run was disturbed, so repeat it.
5. Compare against your own mean from previous weeks on the same setup, not against a published figure.

The mean, best and standard deviation quoted here are this site's own, implemented insrc/engine/stats.ts. The standard error table uses the conventional σ/√n relationship with an assumed 20 ms spread; the attempt values are constructed for illustration, not recorded from a user.