Reasoning
Cognitive Reflection Test
Also called: cognitive reflection task, CRT
Measure the tendency to check a fast answer and work out the right one, with short typed questions that each invite a tempting wrong answer.
3-5 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Preview any test free after you sign up.
- Papers, 2021 to 2025
- 297
- Evidence grade
- C
- Registered clinical studies
- 7
What it measures
Measures the tendency to stop a fast answer that comes to mind and work out the right one. Participants type answers to 8 short questions, 4 that need simple arithmetic and 4 that need none, each written to invite a tempting wrong answer. The questions are in English.
Constructs
- Cognitive reflection
- Checking an intuitive answer
- Analytic thinking
Research uses
- Judgment and decision research
- Individual differences in thinking style
- Within-study group comparison
How the paradigm works
The Cognitive Reflection Test measures the tendency to check a fast answer that comes to mind and work out the right one. Each question is written so that a quick reading gives a tempting wrong answer, and a little thought shows why it is wrong (Frederick 2005). The 8 questions here were written for this task. Four need simple arithmetic and four need none, since a set of arithmetic problems also reflects numeracy (Thomson and Oppenheimer 2016; Sirota and colleagues 2021). The questions are English; sessions run in other languages are flagged for review.
More about the paradigm
Participants type each answer in an open box. There is no time limit and no feedback, and response time is recorded. One practice question with no trap shows the answer box. Question order is drawn from the session's random seed and recorded, with no more than 2 questions of one kind in a row. A last question asks whether the participant had seen any of these questions before, since having seen such questions raises scores (Haigh 2016).
Each answer is coded as correct, the tempting wrong answer, another wrong answer or blank (Thomson and Oppenheimer 2016; Sirota and colleagues 2021). The reflective score counts correct answers, the standard score. The intuitive score counts tempting wrong answers, so it separates a pull toward the fast answer from a wrong answer of another kind. Scores support research and do not diagnose.
Scores
- Reflective Scoreitems
- Questions answered correctly, of 8.
- Higher means more correct answers
- Intuitive Scoreitems
- Questions answered with the tempting wrong answer.
- Higher means more answers followed the quick reading
- Other Wrong Answersitems
- Answers that are neither correct nor the tempting wrong answer.
- Higher means more errors of another kind
- Blank Answersitems
- Questions submitted with an empty box.
- Lower means more questions were attempted
- Arithmetic Questions Correctitems
- Correct answers among the 4 questions that need simple arithmetic.
- Higher means more correct answers
- Non-Arithmetic Questions Correctitems
- Correct answers among the 4 questions that need no arithmetic.
- Higher means more correct answers
- Median Time per Questionms
- Median time from a question's appearance to its submission, over answered questions, leaving out answers under 100 ms.
- Describes pace; the test has no time limit
- Answers Under 100 mscount
- Answered questions submitted within 100 ms.
- Expected to be 0
- Questions Answereditems
- Questions submitted with an answer.
- A session with fewer than 4 is invalid
- Questions Shownitems
- Questions presented in the scored run.
- 8 in a completed run
- Largest Same Answer Countitems
- The largest number of questions given one identical answer.
- 6 or more is flagged for review
- Seen Before0/1
- 1 if the participant said they had seen any of these questions before, 0 if not.
- A 1 is flagged for review
Use and evidence
297 papers from 2021 to 2025 name this task in their title or abstract, about the same as from 2016 to 2020. That ranks it 86 of 395 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 69 |
| 2024 | 54 |
| 2023 | 53 |
| 2022 | 60 |
| 2021 | 61 |
| 2020 | 61 |
| 2019 | 61 |
| 2018 | 55 |
| 2017 | 56 |
| 2016 | 64 |
| 2015 | 46 |
| 2014 | 31 |
| 2013 | 21 |
| 2012 | 16 |
| 2011 | 10 |
| 2010 | 7 |
| 2009 | 3 |
| 2008 | 3 |
| 2007 | 1 |
| 2006 | 1 |
| 2005 | 2 |
| 2004 | 0 |
| 2003 | 0 |
| 2002 | 0 |
| 2001 | 0 |
| 2000 | 0 |
Use grade C
100 to 299 papers from 2021 to 2025.
Evidence grade C
At least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies.
- Papers on test-retest reliability
- 2
- Papers on reliability or validity
- 107
- Registered clinical studies using it as an outcome
- 7
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.14.0, counted 31 October 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Questions
- 8 short questions written for this task: 4 need simple arithmetic and 4 need none. Each has one correct answer and one tempting wrong answer.
- Answers
- Typed in an open box, with no time limit and no feedback. Response time is recorded, and pasting is blocked.
- Scoring
- Each answer is coded as correct, the tempting wrong answer, another wrong answer or blank. Case and punctuation are ignored, and one wrong, missing, extra or swapped letter is accepted in words of 5 or more letters unless the typed word is another English word.
- Order
- Drawn from the session's seed and saved, with no more than 2 questions of one kind in a row.
- Practice
- 1 question with no trap, followed by its answer.
- Last question
- Whether the participant had seen any of these questions before. A yes is flagged for review and the session is kept.
The questions were written for this task; no published question set is used. The questions are English; sessions run in other languages are flagged for review.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: cognitive_reflection_item from the scored run, one typed answer per question. Typed answers have no chance level, so accuracy is not judged against chance, and questions have no time limit, so none can be missed. Fewer than 4 of the 8 questions answered invalidates. Typed answers under 200 ms on at least half of 4 or more answered questions invalidate. One identical answer on 6 or more questions, a report of having seen the questions before, or a session run in another language is flagged for review. A hidden tab is flagged.
Why these rules
There is no correct option to compare with chance, so accuracy is not judged against chance. A score over fewer than half the questions cannot be compared with a score over all of them, so it invalidates. A low score is the result the test measures and never invalidates. The 8 correct answers all differ, so one answer typed for nearly every question cannot follow the questions; it is flagged. People who have seen cognitive reflection questions score higher (Haigh 2016), and familiarity predicts correct answers (Byrd and colleagues 2023), so a report of having seen these is flagged and the session is kept. An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast. The questions are English and turn on exact wording. A session run in another language shows English questions to a participant who may not read English fluently, so it is flagged for review. Questions stay on screen until they are submitted, so a hidden tab does not change what was shown; it is flagged.
Running this test
Settings you can change
This test runs with fixed settings.
Need a different version of this test?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Instructions and screens: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. Words and sentences: English.
- Available on: Every plan, including the free plan.
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Cognitive Reflection Test in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Frederick, 2005). The main score was reflective score (questions answered correctly, of 8).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25-42. https://doi.org/10.1257/089533005775196732
- Toplak, M. E., West, R. F., & Stanovich, K. E. (2014). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147-168. https://doi.org/10.1080/13546783.2013.844729
- Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99-113. https://doi.org/10.1017/S1930297500007622
- Sirota, M., Dewberry, C., Juanchich, M., Valuš, L., & Marshall, A. C. (2021). Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test. Journal of Behavioral Decision Making, 34(3), 322-343. https://doi.org/10.1002/bdm.2213
- Haigh, M. (2016). Has the standard Cognitive Reflection Test become a victim of its own success? Advances in Cognitive Psychology, 12(3), 145-149. https://doi.org/10.5709/acp-0193-5
Source papers
- Frederick S (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4):25-42.Paradigm source, Adults
- Sirota M, Dewberry C, Juanchich M, Valuš L, Marshall AC (2021). Measuring cognitive reflection without maths: development and validation of the verbal cognitive reflection test. Journal of Behavioral Decision Making, 34(3):322-343.Paradigm source, Adults
Questions
How long does the Cognitive Reflection Test take?
About 3 to 5 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Reflective Score, Intuitive Score, Other Wrong Answers, Blank Answers, Arithmetic Questions Correct, Non-Arithmetic Questions Correct, Median Time per Question, Answers Under 100 ms, Questions Answered, Questions Shown, Largest Same Answer Count and Seen Before. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Cognitive Reflection Test used in research?
297 papers from 2021 to 2025 name it in their title or abstract, about the same as from 2016 to 2020. That ranks it 86 of 395 test families in the ConductCognition test census (use grade C). Source: ConductCognition test census 0.14.0, counted 31 October 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Cognitive Reflection Test have?
In the published research, 2 papers report its test-retest reliability and 7 registered clinical studies use it as an outcome (evidence grade C in the ConductCognition test census).
Which languages can participants use?
The instructions and screens are in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. The words and sentences stay in English, so sessions in other languages are flagged for review. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.