Study design
Randomize test order and assign study groups
Put tests in sections, choose the order inside each section, and assign participants to study groups. Each participant's group and order are fixed when they agree to the consent form, or when they start the study if the consent screen is off, and both go into your exports.
In the protocol builder
- Sections
- Up to 20, holding tests, instructions and breaks
- Order inside a section
- Fixed order, Random order or Latin square
- Section order
- Fixed, or rotated in a Latin square
- Study groups
- Up to 12. Random, weighted or balanced assignment, with optional caps
- Saved for each participant
- Group, order number and the tests in the order received
See each participant's path
The protocol builder's Flow view draws the path from start to end for each study group, with a text version below it. This is the example study used on this page, drawn by the same component. Practice stays first, the attention tests rotate in a Latin square, and each group gets one more section.
Text version of the diagram
- Consent: Participant agrees to take part.
- Balanced assignment to Group A, Group B.
- Group A: Practice (1 test, fixed order); Attention (3 tests, Latin square); Speed (1 test, fixed order).
- Group B: Practice (1 test, fixed order); Attention (3 tests, Latin square); Memory and speed (2 tests, random order).
- Group A: Attention, Speed rotate across participants in a Latin square with 2 orders.
- Group B: Attention, Memory and speed rotate across participants in a Latin square with 2 orders.
- End: completion page.
Choose the order inside each section
Random order
Latin square
Instructions are plain text of up to 5,000 characters. A break can ask for up to 900 seconds of rest (15 minutes), with or without an early Continue. A protocol can hold up to 20 sections and 128 steps.
Counterbalance whole sections
Section order can rotate in a balanced Latin square too. Keep a section in place, such as practice at the start, and the other sections rotate around it.
When a study counterbalances more than one thing, each participant gets one order number and every part stays balanced. In the example study, the 3 attention tests have 6 orders and the 2 sections after practice have 2, so each group rotates through 6 orders. A group can have up to 1,000 orders.
Latin square counterbalancing, explained
When every participant takes the tests in the same order, practice and fatigue build up in the same tests for everyone. Counterbalancing spreads those order effects across the tests, so they do not pile up on one.
A Latin square is a set of orders in which each test appears once in each position. A balanced Latin square, also called a Williams design, adds one more rule: each test comes directly after every other test equally often. ConductCognition builds balanced Latin squares.
With an even number of tests, a balanced square needs as many orders as tests. With an odd number, it needs twice as many. Each new participant gets the order that the fewest active participants in their group have, so the orders fill evenly. For an exactly balanced design, plan a number of participants per group that is a multiple of the number of orders.
Random order is the other choice. It gives each participant a new shuffle, so positions even out only on average across many participants.
Williams, E. J. (1949). Experimental designs balanced for the estimation of residual effects of treatments. Australian Journal of Scientific Research, Series A, 2(2), 149-168. https://doi.org/10.1071/CH9490149
| Order | 1st | 2nd | 3rd | 4th |
|---|---|---|---|---|
| 1 | A | B | D | C |
| 2 | B | C | A | D |
| 3 | C | D | B | A |
| 4 | D | A | C | B |
Orders a balanced square needs
- 2 tests: 2 orders
- 3 tests: 6 orders
- 4 tests: 4 orders
- 5 tests: 10 orders
- 6 tests: 6 orders
- 7 tests: 14 orders
- 8 tests: 8 orders
Assign participants to study groups
| Assignment | How it works |
|---|---|
| Random | Each participant has an equal chance of joining each group. |
| Weighted | Each group's chance follows its ratio. |
| Balanced | Each participant joins the group furthest below its ratio. Someone who has not started a test 60 minutes after joining their group stops counting, so their place is reused. |
| Caps | A cap limits a group's active participants. A full group takes no one new. Someone who has not started a test 60 minutes after joining their group frees their place. |
A study can have up to 12 groups, each with a ratio from 1 to 100 and an optional cap of up to 100,000 participants. Groups differ by the sections they receive. To give two groups the same test with different settings, put each version in a section that only one group receives.
Before launch, Simulate assignments in the Groups view shows how a number of participants would spread across groups and orders. People join one after another and every one starts. Nothing is saved and no places or credits are used.
Running a trial with arms? See cognitive tests in clinical trials.
Preview as any group and order
Choose a study group and an order number, then run the whole protocol through the screens participants see. The preview opens in a new tab. It creates no participant and uses no places or credits.
What the exports record
| Column | What it holds |
|---|---|
| group_key | The study group's key. |
| group_name | The group's name in the protocol version the participant ran. |
| assignment_method | How groups were assigned: random, weighted or balanced. |
| order_index | The participant's order number, counted from 1 as in the app. |
| section_label | The section this test was in. |
| sequence_position | Where this test came among the participant's tests. |
| realized_sequence | The participant's tests in the order they received them, joined by >. |
Example: the first 8 participants
Simulated participants in the example study, placed by the same assignment code the app runs when a participant joins. Example data, not real participants.
| Participant | group_name | assignment_method | order_index | realized_sequence |
|---|---|---|---|---|
| 1 | Group A | balanced | 1 | srt>flanker>stroop>gonogo>crt |
| 2 | Group B | balanced | 5 | srt>flanker>gonogo>stroop>digit-span>crt |
| 3 | Group B | balanced | 1 | srt>flanker>stroop>gonogo>crt>digit-span |
| 4 | Group A | balanced | 3 | srt>gonogo>flanker>stroop>crt |
| 5 | Group A | balanced | 4 | srt>crt>gonogo>stroop>flanker |
| 6 | Group B | balanced | 2 | srt>crt>digit-span>stroop>gonogo>flanker |
| 7 | Group A | balanced | 6 | srt>crt>stroop>flanker>gonogo |
| 8 | Group B | balanced | 6 | srt>digit-span>crt>stroop>flanker>gonogo |
Every column and file: research data export. The tests in the example: the full test list.
Repeat visits in one study, and what is not built
A study can test the same people again. Each later visit opens a set number of days after the participant finishes the first, stays open for a set number of days and can accept late sessions. Each participant keeps one link, which opens the visit that is due, and uses one place for all visits. The longitudinal export gives each score at two visits and the change between them.
Branching is not built: a score or an answer cannot send a participant down a different path.
Price a design with repeat visits: the study cost calculator.
Questions about study design
What is the difference between random order and a Latin square?
Random order shuffles the tests for each participant, so each test lands in each position only on average. A Latin square rotates the tests through a fixed set of orders, so each test is used in each position equally often, and each new participant gets the order the fewest active participants in their group have.
How many orders does a balanced Latin square need?
As many orders as tests when the number of tests is even, and twice as many when it is odd. 4 tests need 4 orders; 5 tests need 10.
When is a participant assigned to a group and an order?
When they agree to the consent form or, if the consent screen is off, when they start the study. Their group, order number and exact sequence of tests are saved then and do not change, even when you publish a new protocol version.
What happens when a group is full?
A full group takes no one new. When every group has a cap and all are full, the open study link turns new people away before they start, and a participant link shows that the study is full before the person starts. Someone who has not started a test 60 minutes after joining their group stops counting toward caps and balance.
Can a score decide what a participant does next?
No. Branching on a score or an answer is not built. Every participant in a group gets that group's sections.
Can participants come back for a second visit?
Yes. A study can hold repeat visits. Each later visit opens a set number of days after the participant finishes the first, stays open for a set number of days and can accept late sessions. Each participant keeps one link, which opens the visit that is due, and the longitudinal export gives each score at two visits and the change between them.
Do sections and study groups cost extra?
No. They work on every plan, including the free plan. A participant counts once toward your participant limit, or uses one credit, whatever their group or order.
Plans and prices: pricing.
Design your study
Start free, build a protocol with sections and groups, and preview it as each group.