Position
Scores rise with practice or fall with fatigue as a session goes on. Vigilance falls with time on task, faster after lost sleep (Lim and Dinges, 2008).
Test batteries
Does test order change the scores?
Send simulated people through thousands of test orders with effects you set. See what the analysis finds and whether reliability holds.
Research use only.
One order
Order effects cannot be measured
Balanced Latin square
14 orders, balanced by place and by the test before
Random orders
A new order for each person
Order lab
Example dataLanguage-Minimized Screening Battery1,000 simulated people
1,000 people took 903 different orders. Found a position effect of −0.28 SD and a carryover of −0.32 SD after Eriksen Flanker Task. Reliability with the order removed .84, against .84 with no order effects.
- Different orders taken
- 903
- Possible orders of 8 tests
- 40,320
- With Trail Making Test A before B
- 5,040
The first 10 people's orders
| Order | 1st | 2nd | 3rd | 4th | 5th | 6th | 7th | 8th |
|---|---|---|---|---|---|---|---|---|
| 1 | 2 Choice Reaction Time | 3 Eriksen Flanker Task | 1 Simple Reaction Time | 8 Paired Associate Learning | 7 1-Back Task | 4 Go/No-Go Task | 5 Trail Making Test A | 6 Trail Making Test B |
| 2 | 3 Eriksen Flanker Task | 2 Choice Reaction Time | 7 1-Back Task | 1 Simple Reaction Time | 8 Paired Associate Learning | 4 Go/No-Go Task | 5 Trail Making Test A | 6 Trail Making Test B |
| 3 | 7 1-Back Task | 2 Choice Reaction Time | 8 Paired Associate Learning | 4 Go/No-Go Task | 5 Trail Making Test A | 6 Trail Making Test B | 3 Eriksen Flanker Task | 1 Simple Reaction Time |
| 4 | 3 Eriksen Flanker Task | 8 Paired Associate Learning | 7 1-Back Task | 5 Trail Making Test A | 6 Trail Making Test B | 1 Simple Reaction Time | 4 Go/No-Go Task | 2 Choice Reaction Time |
| 5 | 3 Eriksen Flanker Task | 4 Go/No-Go Task | 2 Choice Reaction Time | 1 Simple Reaction Time | 7 1-Back Task | 8 Paired Associate Learning | 5 Trail Making Test A | 6 Trail Making Test B |
| 6 | 4 Go/No-Go Task | 8 Paired Associate Learning | 3 Eriksen Flanker Task | 1 Simple Reaction Time | 5 Trail Making Test A | 6 Trail Making Test B | 2 Choice Reaction Time | 7 1-Back Task |
| 7 | 5 Trail Making Test A | 6 Trail Making Test B | 8 Paired Associate Learning | 7 1-Back Task | 3 Eriksen Flanker Task | 1 Simple Reaction Time | 2 Choice Reaction Time | 4 Go/No-Go Task |
| 8 | 8 Paired Associate Learning | 3 Eriksen Flanker Task | 1 Simple Reaction Time | 5 Trail Making Test A | 6 Trail Making Test B | 7 1-Back Task | 2 Choice Reaction Time | 4 Go/No-Go Task |
| 9 | 2 Choice Reaction Time | 8 Paired Associate Learning | 5 Trail Making Test A | 6 Trail Making Test B | 7 1-Back Task | 1 Simple Reaction Time | 3 Eriksen Flanker Task | 4 Go/No-Go Task |
| 10 | 4 Go/No-Go Task | 3 Eriksen Flanker Task | 7 1-Back Task | 2 Choice Reaction Time | 5 Trail Making Test A | 6 Trail Making Test B | 8 Paired Associate Learning | 1 Simple Reaction Time |
- 1Simple Reaction Time
- 2Choice Reaction Time
- 3Eriksen Flanker Task
- 4Go/No-Go Task
- 5Trail Making Test A
- 6Trail Making Test B
- 71-Back Task
- 8Paired Associate Learning
These orders can measure
- Measured: Battery position effect
- Measured: Each test's position effect
- Measured: Carryover by test
- Measured: Carryover by pair
Position effect, first place to last
−0.28SDMoves
95% interval −0.35 to −0.22. Set −0.28. A median score: 50th percentile in first place, 39th in last.
Carryover after Eriksen Flanker Task
−0.32SDMoves
95% interval −0.40 to −0.24. Set −0.33.
Reliability, order removed
.84
- No order effects
- .84
- As observed
- .84
Standardized alpha, from tests not taken back to back.
Correlation of the same test pairs
.43back to back
.39further apart
Gap +.04, 95% interval +.02 to +.07, over 27 pairs.
- Found, with its 95% interval
- Set
- Steady zone, −0.20 to +0.20 SD
- Charts run from −1 to +1 SD.
Position effect by test
| Test | Set | Found (95% interval) | Chart, in standard deviations | Reading |
|---|---|---|---|---|
| All tests | −0.28 | −0.28−0.35 to −0.22 | Moves | |
| Simple Reaction Time | −0.28 | −0.26−0.45 to −0.08 | Moves | |
| Choice Reaction Time | −0.28 | −0.38−0.58 to −0.18 | Moves | |
| Eriksen Flanker Task | −0.29 | −0.26−0.45 to −0.07 | Moves | |
| Go/No-Go Task | −0.28 | −0.24−0.44 to −0.03 | Moves | |
| Trail Making Test A | −0.28 | −0.20−0.41 to 0.00 | Needs more people | |
| Trail Making Test B | −0.28 | −0.24−0.44 to −0.05 | Moves | |
| 1-Back Task | −0.28 | −0.35−0.53 to −0.16 | Moves | |
| Paired Associate Learning | −0.30 | −0.34−0.53 to −0.15 | Moves |
Carryover by test
Averaged over the tests that came next.
| Test before | Set | Found (95% interval) | Chart, in standard deviations | Reading |
|---|---|---|---|---|
| Simple Reaction Time | 0.00 | −0.01−0.10 to +0.08 | Steady | |
| Choice Reaction Time | 0.00 | −0.06−0.14 to +0.03 | Steady | |
| Eriksen Flanker Task | −0.33 | −0.32−0.40 to −0.24 | Moves | |
| Go/No-Go Task | 0.00 | 0.00−0.08 to +0.08 | Steady | |
| Trail Making Test B | 0.00 | +0.03−0.06 to +0.12 | Steady | |
| 1-Back Task | 0.00 | +0.02−0.06 to +0.11 | Steady | |
| Paired Associate Learning | 0.00 | +0.04−0.04 to +0.13 | Steady |
Carryover by pair
Rows are the test before, columns the test after, by number. Shaded where the 95% interval leaves zero, darker for larger.
| Before, then after | 1 Simple Reaction Time | 2 Choice Reaction Time | 3 Eriksen Flanker Task | 4 Go/No-Go Task | 5 Trail Making Test A | 7 1-Back Task | 8 Paired Associate Learning |
|---|---|---|---|---|---|---|---|
| 1 Simple Reaction Time | · | +0.25 | 0.00 | −0.25 | −0.04 | +0.13 | −0.14 |
| 2 Choice Reaction Time | +0.04 | · | +0.01 | −0.23 | −0.15 | +0.15 | −0.15 |
| 3 Eriksen Flanker Task | −0.36 | −0.05 | · | −0.42 | −0.36 | −0.30 | −0.42 |
| 4 Go/No-Go Task | −0.05 | +0.22 | +0.01 | · | −0.08 | −0.02 | −0.10 |
| 6 Trail Making Test B | −0.03 | +0.25 | +0.02 | −0.09 | · | +0.01 | +0.01 |
| 7 1-Back Task | +0.01 | +0.22 | +0.11 | −0.04 | 0.00 | · | −0.17 |
| 8 Paired Associate Learning | +0.18 | +0.13 | +0.08 | −0.12 | −0.03 | +0.04 | · |
Reliability by test
The battery's line is the standardized alpha of its tests' sum; a test's line, its correlation with the sum of the others. The last column is the same simulated people with no order effects.
| Measure | As observed | Order removed | Not back to back | No order effects |
|---|---|---|---|---|
| Battery | .84 | .84 | .84 | .84 |
| Simple Reaction Time | .61 | .62 | .63 | .63 |
| Choice Reaction Time | .53 | .54 | .53 | .52 |
| Eriksen Flanker Task | .57 | .57 | .57 | .58 |
| Go/No-Go Task | .47 | .48 | .47 | .47 |
| Trail Making Test A | .65 | .66 | .65 | .68 |
| Trail Making Test B | .54 | .55 | .54 | .53 |
| 1-Back Task | .59 | .61 | .60 | .64 |
| Paired Associate Learning | .55 | .57 | .56 | .57 |
What order does to scores
Carryover
One test can change the next. Some expected carryovers are tiny: in 23 labs and 2,141 people, one effortful task changed the next by d = 0.04 (Hagger and colleagues, 2016).
Shared state
Tests taken back to back share alertness and room noise, so they correlate a little more. With one order, that can raise a battery's reliability.
One order
When everyone takes the same order, each test always sits in the same place. Place and test cannot be told apart.
In 587 people, a 12-task battery's measurement models were largely the same whether its order was fixed or random for each person (Robison and others, 2024).
How many orders to run
A battery of n tests has n! possible orders. A balanced Latin square needs only n of them, or 2n when n is odd (Williams, 1949). Trail Making Test A always runs right before Trail Making Test B (Bowie and Harvey, 2006), so the two count as one block.
| Tests | Possible orders | Balanced square | The square can measure |
|---|---|---|---|
| 2 | 2 | 2 | One position effect for the battery |
| 3 | 6 | 6 | Position effects, carryover by pair |
| 4 | 24 | 4 | Position effects, carryover by test |
| 5 | 120 | 10 | Position effects, carryover by pair |
| 6 | 720 | 6 | Position effects, carryover by test |
| 7 | 5,040 | 14 | Position effects, carryover by pair |
| 8 | 40,320 | 8 | Position effects, carryover by test |
| 9 | 362,880 | 18 | Position effects, carryover by pair |
| 10 | 3,628,800 | 10 | Position effects, carryover by test |
| 11 | 39,916,800 | 22 | Position effects, carryover by pair |
| 12 | 479,001,600 | 12 | Position effects, carryover by test |
People to read each effect to ±0.2 SD
Language-Minimized Screening Battery, balanced Latin square, 14 orders
- Battery position effect
- 130
- Each test's position effect
- 1,800
- Carryover by test
- 240
- Carryover by pair
- 2,030
Bars use a log scale. The last column comes from each square's design. Random orders reach every pair once enough people take them. The people figures assume the tests correlate .30 with each other.
How the analysis works
- 1
Put every test on one scale
Each test's scores become standard deviations from that test's mean.
- 2
Compare each person with themselves
Each person's average comes out first, so ability cannot pass for an order effect.
- 3
Fit place and carryover
One model holds each test's position effect and the carryover from the test before, in the most detail the orders allow. Its 95% intervals allow for one person's scores moving together.
- 4
Remove the order
Each score loses its position effect and the carryover from the test before it.
- 5
Check the battery
Reliability and each test's correlation with the rest, read as observed, with the order removed, and from tests not taken back to back.
Readings
- Steady
- The 95% interval stays inside −0.20 to +0.20 SD and includes zero.
- Small shift
- The interval stays inside that zone but leaves out zero.
- Moves
- The interval leaves out zero and reaches past the zone.
- Needs more people
- The interval includes zero and reaches past the zone.
In a study, the builder shows what your order can measure before you publish, and the study report checks your own scores for order effects. How test order works in a study
Sources
Sources
Checked on 11 November 2026.
- Williams EJ (1949). Experimental designs balanced for the estimation of residual effects of treatments. Australian Journal of Scientific Research, Series A, 2(2), 149-168.
- Bowie CR, Harvey PD (2006). Administration and interpretation of the Trail Making Test. Nature Protocols, 1(5), 2277-2281.
- Hagger MS, Chatzisarantis NLD, Alberts H, and colleagues (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546-573.
- Lim J, Dinges DF (2008). Sleep deprivation and vigilant attention. Annals of the New York Academy of Sciences, 1129, 305-322.
- Robison MK, Celaya X, Ball BH, Brewer GA (2024). Task sequencing does not systematically affect the factor structure of cognitive abilities. Psychonomic Bulletin & Review, 31(2), 670-685.
Rotate the order in your own study
Run a section's tests in one order, a balanced Latin square or a new random order for each participant. Research use only.