Executive Function
Algorithm Advice Task
Also called: algorithm aversion, algorithm reliance, human-algorithm forecasting
Measure use of programmed forecasts after seeing them make errors. Participants forecast daily counts, choose their own answer or the program's, then see both errors.
6-10 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 396
- Evidence grade
- C
- Registered clinical studies
- 0
What it measures
Measures how seeing an algorithm's errors relates to the use of its advice. Participants forecast fictional daily counts, choose their own or a programmed forecast, and see both errors. Scores algorithm use after exact and erroneous forecasts.
Constructs
- Algorithm aversion
- Advice use
- Forecasting
- Decision making
Research uses
- Human factors research
- Judgment and decision making
- Responses to algorithm errors
How the paradigm works
Algorithm aversion describes reluctance to use an algorithm after seeing it make mistakes. This task measures how choices between one's own forecast and a programmed forecast change after error feedback.
More about the paradigm
Participants read three recent daily counts from an original fictional example, type the next count from 0 to 999, then choose their forecast or the program's. Each round shows the actual count and both absolute errors. Two practice forecasts precede 24 scored forecasts as standard, configurable from 16 to 40 in sets of eight. The eight everyday scenarios are written in each language.
The program follows a fixed schedule independent of the participant: half its forecasts are exact, one quarter are four counts away and one quarter are twelve counts away. Thus 75% are within five counts and the mean absolute error is 4. Nothing is learned or adjusted from participant responses. The primary score is program use after an exact forecast minus program use after an error, using the previous round's completed feedback. The first scored forecast is excluded from that comparison. Overall use, switching away after errors and the participant's, program's and chosen forecast errors are also reported. Responses under 100 ms are counted as anticipations and excluded.
Scores
- Drop in Program Use After Errors%
- Program choice share after an exact forecast minus its share after an erroneous forecast. The first trial has no previous feedback.
- Positive means less use after errors
- Program Use After Errors%
- Share choosing the program after seeing its previous forecast err.
- Informational
- Program Use After Exact Forecasts%
- Share choosing the program after seeing its previous forecast be exact.
- Informational
- Program Use%
- Share choosing the program across eligible forecasts.
- Informational
- Switching Away After an Error%
- Share switching to their own forecast after choosing the program and seeing it err.
- Informational
- Own Forecast Errorcount
- Mean absolute error of the participant's forecasts, in counts.
- Lower is closer
- Program Forecast Errorcount
- Mean absolute program error on eligible trials. The complete preset schedule averages 4 counts.
- Lower is closer
- Chosen Forecast Errorcount
- Mean absolute error of the selected forecasts.
- Lower is closer
- Choices After Errorstrials
- Eligible choices following completed error feedback.
- Informational
- Choices After Exact Forecaststrials
- Eligible choices following completed exact feedback.
- Informational
- Median Choice Timems
- Median time to choose a forecast.
- Informational
- Anticipationscount
- Trials with a forecast or choice under 100 ms, left out of behavioral scores.
- 0 expected
- Forecasts Completedtrials
- Forecasts completed through the feedback screen.
- Informational
- Planned Forecaststrials
- Forecasts planned for the run.
- Informational
Use and evidence
396 papers from 2021 to 2025 name this task in their title or abstract, up 620% compared with 2016 to 2020. That ranks it 71 of 582 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 159 |
| 2024 | 100 |
| 2023 | 54 |
| 2022 | 47 |
| 2021 | 36 |
| 2020 | 17 |
| 2019 | 16 |
| 2018 | 11 |
| 2017 | 5 |
| 2016 | 6 |
| 2015 | 0 |
| 2014 | 6 |
| 2013 | 1 |
| 2012 | 0 |
| 2011 | 0 |
| 2010 | 0 |
| 2009 | 0 |
| 2008 | 0 |
| 2007 | 0 |
| 2006 | 0 |
| 2005 | 0 |
| 2004 | 1 |
| 2003 | 0 |
| 2002 | 0 |
| 2001 | 0 |
| 2000 | 0 |
Use grade B
300 to 999 papers from 2021 to 2025.
Evidence grade C
At least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 68
- Registered clinical studies using it as an outcome
- 0
Measured on ConductCognition
Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.
No pooled figure yet.
Note on the counts: many papers are surveys or vignette studies of attitudes to AI.
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.39.0, counted 8 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Forecasts
- 24 scored forecasts as standard, from 16 to 40 in sets of 8. Two practice forecasts show the whole sequence first.
- Items
- Eight original fictional scenarios in each language. Three recent daily counts are followed by a typed forecast from 0 to 999. The next count includes an irregularity, so the trend need not continue exactly.
- Program
- Answers are fixed before responses. Each set has 4 exact forecasts, 2 with an error of 4 and 2 with an error of 12. Accuracy is 50% exact and 75% within five; mean absolute error is 4 counts.
- Choice and feedback
- Choose your forecast or the program's. Then see the actual count, both absolute errors and the chosen forecast. No time limit. The positions of the two forecasts are fixed within a session and counterbalanced across session seeds.
- Score
- Program choice share after an exact forecast minus its share after an erroneous forecast. Only the immediately preceding completed feedback counts; the first scored forecast has no prior exposure. Anticipations under 100 ms are excluded.
This is an adaptation with original fictional counts, a fixed program and repeated choices. The source used real forecasting data and a later choice of forecaster. Our score describes choices after feedback and is not a causal estimate of the effect of seeing an error. Scores support research and do not diagnose.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Completed forecasts from the most complete run. Choosing the program or one's own forecast has no chance threshold. Anticipations under 100 ms are counted and excluded; choices under 200 ms on at least half the trials invalidate. Incomplete runs and a missing feedback contrast are flagged; fewer than half the trials is invalid.
Why these rules
Persistent use or rejection of the program is the behavior being measured, so neither pattern invalidates. The after-error contrast requires a completed preceding feedback screen. These original fictional forecasts have no diagnostic or published norm interpretation.
No percentile. Scores and quality flags only.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Sets of forecasts | 3 | 2 to 5 |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- No percentile. Scores and quality flags only. How norms work
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Algorithm Advice Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Dietvorst BJ, Simmons JP, Massey C, 2015), with the standard settings (sets of forecasts: 3). The main score was drop in program use after errors (program choice share after an exact forecast minus its share after an erroneous forecast).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126. https://doi.org/10.1037/xge0000033
Source papers
- Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126.Paradigm source, Adults
Questions
How long does the Algorithm Advice Task take?
About 6 to 10 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Drop in Program Use After Errors, Program Use After Errors, Program Use After Exact Forecasts, Program Use, Switching Away After an Error, Own Forecast Error, Program Forecast Error, Chosen Forecast Error, Choices After Errors, Choices After Exact Forecasts, Median Choice Time, Anticipations, Forecasts Completed and Planned Forecasts. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Algorithm Advice Task used in research?
396 papers from 2021 to 2025 name it in their title or abstract, up 620% compared with 2016 to 2020. That ranks it 71 of 582 test families in the ConductCognition test census (use grade B). Source: ConductCognition test census 0.39.0, counted 8 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Algorithm Advice Task have?
In the published research, 0 papers report its test-retest reliability and 0 registered clinical studies use it as an outcome (evidence grade C in the ConductCognition test census).
Does it report a percentile?
No. Scores and quality flags only.
Which settings can a researcher change?
Sets of forecasts (2 to 5). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese, with its words in the same language. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.
Related tests