Reasoning
Diagram Rule Task
Also called: Bongard problems, Bongard problem task, Bongard-type puzzles, visual concept learning task, rule induction from examples
Measure visual concept finding: participants find the rule that separates two groups of pictures, then sort new pictures into the groups, in puzzles drawn for each session.
6-10 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 4
- Evidence grade
- D
- Registered clinical studies
- 0
What it measures
Measures finding a visual rule from examples. Each of 8 puzzles shows two groups of 6 pictures drawn for each session: every picture in group A follows a rule that no picture in group B follows, such as a large shape against a small one. Participants study the groups for as long as they like, then sort 6 new pictures into the two groups. The main score is the number of puzzles whose new pictures were all sorted by the rule.
Constructs
- Visual concept learning
- Rule induction
- Inductive reasoning
- Visual abstraction
Research uses
- Reasoning and concept learning research
- Visual cognition research
- Comparison with machine concept learners
- Within-study group comparison
How the paradigm works
The diagram rule task gives puzzles in the format that Bongard (1970) made for the study of pattern recognition, known as Bongard problems: two groups of six pictures, where every picture in one group follows a rule that no picture in the other follows, and the solver finds the rule. Such puzzles remain a test of concept learning for people and machines (Depeweg, Rothkopf and Jäkel 2024). Weitnauer, Carvalho, Goldstone and Ritter (2014) had people study physical versions of these puzzles and then sort six test scenes, one at a time, into the left or the right group before describing the rule in words; with every scene shown at once, 91 people solved 11.5 of 22 puzzles on average, judged from their descriptions.
More about the paradigm
Here, the pictures are drawn by our code for each session from eight rules written for this task: a large shape against a small one, three shapes against two, high in the panel against low, curved outlines against straight ones, two shapes of the same kind against two different, a small shape inside a large one against outside it, three dots on a straight line against a bent one, and a steep line against a flat one. No published puzzle or diagram is used. Every property a rule leaves free varies within both groups and is balanced between them. Each puzzle shows the two groups side by side, group A in a solid frame and group B in a dashed one. The participant studies them for as long as they like, then sorts six new pictures, three from each group, one at a time, into group A or group B, with the groups still on screen. The rules come in an order drawn from the session's seed. One practice puzzle, about filled and outlined shapes, gives feedback and shows its rule; the eight puzzles give none. There is no time limit.
The main score is the number of the eight puzzles whose six new pictures were all sorted into the right group; guessing does that on one puzzle in 64. The scores also give the new pictures sorted rightly, out of 48, the time spent studying each puzzle's groups before sorting, and sort times. Each row saves its puzzle's rule and every picture, so a study can score each rule. Scores are research measures and do not diagnose.
Scores
- Puzzles Solvedcount
- Puzzles whose 6 new pictures were all sorted into the right group, out of 8.
- Higher means more rules found
- Pictures Sorted Rightlycount
- New pictures sorted into the right group, out of 48.
- Higher is better; guessing gives 24 on average
- Median Study Timems
- Median time from a puzzle's groups appearing to the participant choosing to sort its new pictures.
- Describes the time spent finding each rule; the test has no time limit
- Median Sort Timems
- Median time from a new picture appearing to its sort, over sorts at or after 100 ms.
- Describes pace; the test has no time limit
- Sorts Under 100 mscount
- Sorts made less than 100 ms after the new picture appeared. They still count.
- Expected to be 0
- Puzzles Finishedcount
- Puzzles with all 6 new pictures sorted.
- 8 in a full session
Use and evidence
4 papers from 2021 to 2025 name this task in their title or abstract. That ranks it 476 of 495 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 0 |
| 2024 | 2 |
| 2023 | 1 |
| 2022 | 0 |
| 2021 | 1 |
| 2020 | 0 |
| 2019 | 0 |
| 2018 | 1 |
| 2017 | 1 |
| 2016 | 1 |
| 2015 | 0 |
| 2014 | 1 |
| 2013 | 0 |
| 2012 | 1 |
| 2011 | 0 |
| 2010 | 0 |
| 2009 | 0 |
| 2008 | 0 |
| 2007 | 0 |
| 2006 | 0 |
| 2005 | 0 |
| 2004 | 0 |
| 2003 | 0 |
| 2002 | 1 |
| 2001 | 0 |
| 2000 | 1 |
Use grade E
Fewer than 30 papers from 2021 to 2025.
Evidence grade D
Little published measurement evidence found.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 0
- Registered clinical studies using it as an outcome
- 0
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.22.0, counted 3 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Puzzles
- 8, one per rule, in an order drawn from the session's seed: a large shape against a small one, three shapes against two, high against low, curved outlines against straight ones, two shapes alike against two different, inside against outside, three dots on a line against bent, and a steep line against a flat one.
- Pictures
- Drawn by our code for each session, 6 in group A and 6 in group B, side by side. Every property a rule leaves free, such as shape, fill, size or position, varies within both groups and is balanced between them.
- Sorting
- When ready, the participant sorts 6 new pictures, 3 from each group, one at a time into group A or group B, with the groups still on screen. No time limit and no feedback.
- Practice
- 1 puzzle about filled and outlined shapes, with 4 new pictures, feedback after each sort and its rule shown at the end.
- Scores
- Puzzles whose 6 new pictures were all sorted rightly, which guessing achieves on one puzzle in 64; new pictures sorted rightly out of 48; study and sort times.
The puzzles follow a classic format for visual concept learning, but every picture and rule was made for this task; no published puzzle or diagram is used. Whether the rule was found is read from the sorting, not from a rule described in words. The scores support research and do not diagnose.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: the scored run's sorts of new pictures into group A or group B, 6 per puzzle, half belonging to each group. Accuracy significantly below chance invalidates and accuracy not above chance is flagged. One group on 90% of the sorts invalidates unless accuracy is above chance. A run with fewer than half its puzzles finished is invalid and any other shortfall is flagged. Sorts under 200 ms on at least half the pictures invalidate.
Why these rules
Each new picture belongs to group A or group B, so a guess is right half the time, and each puzzle has three of each. Accuracy significantly below chance (one-sided exact test, p < .05) cannot come from guessing; it means the response mapping was reversed or the task was not followed (Slick, Sherman and Iverson 1999 treat below-chance forced-choice performance as definite response bias). Chance-level accuracy can be a real result in impaired participants, so it is only flagged. One option on at least 90% of answered trials, when the correct options are balanced and accuracy is not above chance, means the answers did not depend on the items (the task counterpart of long-string responding in surveys; Curran 2016). An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast.
Running this test
Settings you can change
This test runs with fixed settings.
Need a different version of this test?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Diagram Rule Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Bongard, 1970; Weitnauer, Carvalho, Goldstone & Ritter, 2014). The main score was puzzles solved (puzzles whose 6 new pictures were all sorted into the right group, out of 8).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Bongard, M. M. (1970). Pattern recognition. Spartan Books.
- Weitnauer, E., Carvalho, P. F., Goldstone, R. L., & Ritter, H. (2014). Similarity-based ordering of instances for efficient concept learning. Proceedings of the 36th Annual Meeting of the Cognitive Science Society, 1760-1765. Sorting new scenes into the two groups after studying them.
- Weitnauer, E., Goldstone, R. L., & Ritter, H. (2023). Perception and simulation during concept learning. Psychological Review, 130(5), 1203-1238. https://doi.org/10.1037/rev0000433People and a model on the same puzzles.
- Depeweg, S., Rothkopf, C. A., & Jäkel, F. (2024). Solving Bongard problems with a visual language and pragmatic constraints. Cognitive Science, 48(5), e13432. https://doi.org/10.1111/cogs.13432The puzzles as a test of concept learning for people and machines.
Source papers
- Weitnauer E, Carvalho PF, Goldstone RL, Ritter H (2014). Similarity-based ordering of instances for efficient concept learning. Proceedings of the 36th Annual Meeting of the Cognitive Science Society, 1760-1765.Paradigm source, Adults
Questions
How long does the Diagram Rule Task take?
About 6 to 10 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Puzzles Solved, Pictures Sorted Rightly, Median Study Time, Median Sort Time, Sorts Under 100 ms and Puzzles Finished. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Diagram Rule Task used in research?
4 papers from 2021 to 2025 name it in their title or abstract. That ranks it 476 of 495 test families in the ConductCognition test census (use grade E). Source: ConductCognition test census 0.22.0, counted 3 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Diagram Rule Task have?
In the published research, 0 papers report its test-retest reliability and 0 registered clinical studies use it as an outcome (evidence grade D in the ConductCognition test census).
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.