Reasoning
Decision Quality Task
Also called: outcome bias, outcome effect, outcome bias in decision evaluation, judging decisions by their outcomes, evaluating decisions after the fact
Measure the outcome bias: participants read short stories of decisions made with stated odds, learn whether each turned out well or badly, and rate the decision and the person's skill at such decisions.
5-7 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 172
- Evidence grade
- C
- Registered clinical studies
- 1
What it measures
Measures the outcome bias. Participants read 12 short stories in which a person takes a chance with stated odds, such as ordering flour from a cheaper mill that is sometimes late, learn whether it turned out well or badly, and rate the decision and the person's skill at such decisions. The main score is how much better the same kind of decision is rated when it turned out well.
Constructs
- Outcome bias
- Decision evaluation
- Separating luck from judgment
- Judgment under risk
Research uses
- Judgment and decision-making research
- Management and organizational research
- Debiasing and training research
- Within-study group comparison
How the paradigm works
The decision quality task measures the outcome bias: people judge the same decision better when it turned out well than when it turned out badly, even when they know everything the decider knew. Baron and Hershey (1988) gave people descriptions of decisions made under stated risk, such as an operation with an 8% chance of death, together with their outcomes, and asked them to rate the quality of the thinking that went into each decision. In their first experiment, 20 students rated matched cases that differed only in success or failure; cases that succeeded were rated 0.70 points higher on a scale from -3 to 3, and only 9.3% of the pairs were rated higher after failure. The bias held in five experiments, for medical decisions and money gambles, for the decider's predicted competence as well as the decision (the two biases correlated 0.85 across people), and among people who said outcomes should not count. In a preregistered replication with 692 online participants who each saw one version, Aiyer and colleagues (2023) found decisions that succeeded rated 1.78 against 0.68 for those that failed on the same scale (d = 1.10 for a physician's decision and 0.77 for a patient's).
More about the paradigm
Here, 12 stories written for this task, such as a bakery ordering flour from a cheaper mill that is sometimes late, state the odds the decider faced and how the decision turned out; each story has a good and a bad ending that differ only in their last sentence. After each story, people rate how good the decision was and how good the person is at decisions like it, from 1 (very bad) to 7 (very good). As standard, each story comes once and half end well; a study can show each story twice, once with each ending, as in Baron and Hershey's within-person cases. The main score is the mean rating of the decisions that ended well minus that of those that ended badly.
Scores
- Outcome Biasscore
- Mean rating of how good the decision was for stories that ended well minus that for stories that ended badly, from -6 to 6.
- Positive means the same kind of decision was judged better after a good outcome, the outcome bias
- Outcome Bias on Competencescore
- The same difference for the ratings of how good the person is at decisions like it.
- Positive means the person was judged more skilled after a good outcome
- Decision Rating After a Good Outcomescore
- Mean rating of how good the decision was for stories that ended well, from 1 to 7.
- Informational
- Decision Rating After a Bad Outcomescore
- Mean rating of how good the decision was for stories that ended badly, from 1 to 7.
- Informational
- Competence Rating After a Good Outcomescore
- Mean rating of the person's skill at such decisions for stories that ended well, from 1 to 7.
- Informational
- Competence Rating After a Bad Outcomescore
- Mean rating of the person's skill at such decisions for stories that ended badly, from 1 to 7.
- Informational
- Stories Rated Higher With the Good Ending%
- With each story shown twice, the share of stories whose good ending got the higher decision rating; empty with each story once.
- Informational
- Median Reading Timems
- Median time spent on a story screen before pressing Next.
- Informational
- Median Rating Timems
- Median time from a rating screen appearing to a rating, over ratings at or after 100 ms.
- Informational
- Ratings Under 100 mscount
- Ratings made less than 100 ms after their screen appeared.
- 0 expected
- Stories Ratedcount
- Story screens rated on both questions.
- Informational
Use and evidence
172 papers from 2021 to 2025 name this task in their title or abstract, up 59% compared with 2016 to 2020. That ranks it 192 of 711 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 45 |
| 2024 | 33 |
| 2023 | 35 |
| 2022 | 33 |
| 2021 | 26 |
| 2020 | 25 |
| 2019 | 22 |
| 2018 | 24 |
| 2017 | 15 |
| 2016 | 22 |
| 2015 | 21 |
| 2014 | 24 |
| 2013 | 18 |
| 2012 | 10 |
| 2011 | 21 |
| 2010 | 18 |
| 2009 | 10 |
| 2008 | 7 |
| 2007 | 13 |
| 2006 | 10 |
| 2005 | 10 |
| 2004 | 8 |
| 2003 | 5 |
| 2002 | 4 |
| 2001 | 4 |
| 2000 | 3 |
Use grade C
100 to 299 papers from 2021 to 2025.
Evidence grade C
At least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 43
- Registered clinical studies using it as an outcome
- 1
Measured on ConductCognition
Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.
No pooled figure yet.
Note on the counts: includes clinical and management papers that name the bias.
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.58.0, counted 13 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Stories
- 12 written for this task, each a person taking a chance with stated odds, such as a bakery ordering flour from a cheaper mill that is late 1 time in 5, with the safer choice it gives up. Each has a good and a bad ending that differ only in their last sentence; the chances run from 1 in 10 to 2 in 5. None has political, religious, group, health or injury content, and none is a published case. They are shown in the session's language.
- Ratings
- After each story, two screens: how good the decision was and how good the person is at decisions like it, from 1 (very bad) to 7 (very good) on seven buttons. A check with feedback first makes sure people know each person had only the information in the story.
- Order
- 12 stories as standard, or the first 8. Each story comes once, half of them ending well and half badly as drawn from the session's seed, in an order drawn from it too. A study can show each story twice, once with each ending, one ending in each half of the session, as in Baron and Hershey's within-person cases.
- Scores
- The outcome bias: the mean decision rating of stories that ended well minus that of stories that ended badly, from -6 to 6. Also the same difference for the ratings of the person's skill, the four means behind them and, with each story twice, the share of stories rated higher with the good ending.
Baron and Hershey (1988) told people everything the decider knew and still found decisions rated better after good outcomes, and Aiyer and colleagues (2023) replicated it with 692 participants. This task uses their method with stories of its own, none from either source. Scores describe how much outcomes move ratings of decisions and are best compared between groups; they do not show how well a person decides and do not diagnose.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: the scored run's ratings of the stories. No rating is correct, so accuracy is not judged against chance. A run with no rated story that ended well or none that ended badly, or with fewer than half the stories rated on both questions, is invalid, and any other shortfall is flagged; one number chosen for at least 90% of the ratings is flagged. Ratings under 200 ms on at least half the screens invalidate.
Why these rules
There is no correct option to compare with chance, so accuracy is not judged against chance. The scores describe how much an outcome moves the rating of a decision made with the same information, so any rating is a possible answer, and so is rating every decision the same. The outcome bias needs stories with both endings rated. One number for nearly every rating suggests the scale was not used; it is flagged for review because it can also be a real answer. An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast.
No percentile. Scores and quality flags only.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Stories | 12 stories (standard) | 8 stories, 12 stories (standard) |
| Endings | Each story once, with one ending (standard) | Each story once, with one ending (standard), Each story twice, once with each ending |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- No percentile. Scores and quality flags only. How norms work
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Decision Quality Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Baron J, Hershey JC, 1988; Aiyer S, Kam HC, Ng KY, Young NA, Shi J, Feldman G, 2023), with the standard settings (stories: 12 stories; endings: Each story once, with one ending). The main score was outcome bias (mean rating of how good the decision was for stories that ended well minus that for stories that ended badly, from -6 to 6).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Baron, J., & Hershey, J. C. (1988). Outcome bias in decision evaluation. Journal of Personality and Social Psychology, 54(4), 569-579. https://doi.org/10.1037/0022-3514.54.4.569
- Aiyer, S., Kam, H. C., Ng, K. Y., Young, N. A., Shi, J., & Feldman, G. (2023). Outcomes affect evaluations of decision quality: Replication and extensions of Baron and Hershey's (1988) outcome bias Experiment 1. International Review of Social Psychology, 36(1), 12. https://doi.org/10.5334/irsp.751A preregistered replication with 692 participants, each rating one version of one decision.
Source papers
- Aiyer and colleagues (2023): 692 adults in the United States, recruited online, each rated one version of one medical decision from -3 to 3; decisions that succeeded were rated 1.78 (SD 0.81) and those that failed 0.68 (SD 1.47).Paradigm source, Adults
Questions
How long does the Decision Quality Task take?
About 5 to 7 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Outcome Bias, Outcome Bias on Competence, Decision Rating After a Good Outcome, Decision Rating After a Bad Outcome, Competence Rating After a Good Outcome, Competence Rating After a Bad Outcome, Stories Rated Higher With the Good Ending, Median Reading Time, Median Rating Time, Ratings Under 100 ms and Stories Rated. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Decision Quality Task used in research?
172 papers from 2021 to 2025 name it in their title or abstract, up 59% compared with 2016 to 2020. That ranks it 192 of 711 test families in the ConductCognition test census (use grade C). Source: ConductCognition test census 0.58.0, counted 13 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Decision Quality Task have?
In the published research, 0 papers report its test-retest reliability and 1 registered clinical study uses it as an outcome (evidence grade C in the ConductCognition test census).
Does it report a percentile?
No. Scores and quality flags only.
Which settings can a researcher change?
Stories (8 stories, 12 stories (standard)) and endings (Each story once, with one ending (standard), Each story twice, once with each ending). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese, with its words in the same language. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.