Executive Function
Probability Words Task
Also called: verbal probability expressions, words of estimative probability, probability phrases, verbal probabilities, numerical translation of probability words
Measure what probability words mean in numbers: participants give the chance that 12 words such as likely or rare mean to them, alone and in sentences, and type each word's range, scored against the IPCC likelihood scale.
6-10 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 64
- Evidence grade
- D
- Registered clinical studies
- 1
What it measures
Measures what probability words such as likely, rare and almost certain mean in numbers. Participants give the chance each of 12 words means to them on a slider from 0 to 100, alone and in a short sentence written for this test in each of the eight languages, then type the range each word covers. The main score is the share of answers to the 7 terms of the IPCC's likelihood scale that fall in the IPCC's range.
Constructs
- Verbal probability
- Risk communication
- Uncertainty communication
- Numeracy
- Vagueness
Research uses
- Judgment and decision-making research
- Risk and uncertainty communication research
- Forecasting and climate communication research
- Cross-language comparison of probability words
How the paradigm works
The Probability Words Task measures what verbal probability expressions, such as likely, rare or almost certain, mean in numbers to each person, a question for risk labels, forecasts and climate reports, since people read the same words very differently (Wallsten and colleagues, 1986; Budescu, Broomell and Por, 2009). Twelve words and phrases, written for this task in each language, span the scale from almost impossible to virtually certain; seven are terms of the likelihood scale in the IPCC's guidance for its Fifth Assessment Report, which gives each a range (likely, 66 to 100%; very unlikely, 0 to 10%). Each word comes alone and in one short sentence about an everyday event, in one order drawn from the session's seed, and the participant moves a slider from 0 to 100 to the chance it means to them. Then each word comes again and the participant types the lowest and the highest chance it can mean. A setting shows the IPCC range beside each term in the sentences, the format Budescu and colleagues proposed. There is no time limit and no feedback.
More about the paradigm
The main score is the share of answers to the IPCC terms, alone and in sentences, that fall inside the IPCC range. Beside it come the mean distance outside the range in percentage points and how far answers move toward 50% from each term's midpoint (laypeople read the IPCC's terms as closer to 50% than intended; Budescu and colleagues, 2014). Also reported are the share of pairs of terms answered in the scale's order, the mean difference between a word alone and in its sentence, the median width of the typed ranges and the share of answers inside the person's own range. Every answer names its word, so a study can measure the spread across its participants for each word.
Scores
- Consistency With the IPCC Scale%
- Share of answers to the IPCC terms, alone and in sentences, inside the IPCC range, ends included.
- Higher means closer to the IPCC's ranges
- Distance From the IPCC Scale%
- Mean distance of those answers outside the IPCC range, in percentage points; 0 inside.
- Lower is closer
- Shift Toward 50%%
- Mean of how much closer to 50% an answer is than its term's IPCC midpoint, for terms whose range leaves out 50%.
- Above 0, answers ran toward the middle of the scale
- Order Agreement%
- Share of pairs of IPCC terms, answered alone, in the scale's order.
- 1 keeps the order; 0.2 or less is flagged
- Sentence Shift%
- Mean absolute difference between a word's answer alone and in its sentence, in percentage points.
- Informational
- Median Range Width%
- Median of the highest minus the lowest typed chance.
- Wider means the words cover more chances
- Answers Inside Own Range%
- Share of words whose answer alone falls inside the person's own range for it.
- Informational
- Median Answer Timems
- Median time from a screen appearing to its answer, for answers of at least 100 ms.
- Informational; the task has no time limit
- Answers Under 100 mscount
- Answers given less than 100 ms after their screen appeared.
- 0 expected
- Words Answeredcount
- Words answered alone in the scored run.
- Fewer than half the words and sentences together invalidates
- Sentences Answeredcount
- Sentences answered in the scored run.
- Informational
- Ranges Answeredcount
- Words with a typed range in the scored run.
- Informational
- Planned Wordscount
- Words the run planned: 12, each alone, in a sentence and with a range.
- Informational
Use and evidence
64 papers from 2021 to 2025 name this task in their title or abstract, up 21% compared with 2016 to 2020. That ranks it 402 of 711 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 21 |
| 2024 | 10 |
| 2023 | 9 |
| 2022 | 12 |
| 2021 | 12 |
| 2020 | 13 |
| 2019 | 18 |
| 2018 | 5 |
| 2017 | 9 |
| 2016 | 8 |
| 2015 | 11 |
| 2014 | 8 |
| 2013 | 14 |
| 2012 | 5 |
| 2011 | 11 |
| 2010 | 7 |
| 2009 | 10 |
| 2008 | 7 |
| 2007 | 1 |
| 2006 | 9 |
| 2005 | 2 |
| 2004 | 7 |
| 2003 | 2 |
| 2002 | 7 |
| 2001 | 3 |
| 2000 | 6 |
Use grade D
30 to 99 papers from 2021 to 2025.
Evidence grade D
Little published measurement evidence found.
- Papers on test-retest reliability
- 1
- Papers on reliability or validity
- 11
- Registered clinical studies using it as an outcome
- 1
Measured on ConductCognition
Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.
No pooled figure yet.
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.58.0, counted 13 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Words
- 12 probability words and phrases written for this library in each language, from almost impossible to virtually certain. 7 are terms of the IPCC likelihood scale, which gives each a range, such as likely, 66 to 100%; the other 5, such as rare and doubtful, are everyday words it does not define.
- Answer
- Each word alone and in one short sentence about an everyday event, such as a train arriving on time: a slider from 0 to 100% that starts with no value chosen. Then each word again, with the lowest and the highest chance it can mean typed in two boxes.
- Order
- The 24 slider answers in one order drawn from the session's seed, then the 12 ranges in another. One practice word comes first; there is no feedback and no time limit.
- Setting
- The IPCC ranges in the sentences: hidden (standard) or shown beside each IPCC term, the format Budescu and colleagues proposed. The words alone and the ranges never show them.
- Scores
- The share of answers to the IPCC terms inside the IPCC range (the main score); their mean distance outside it; the shift toward 50%; the share of pairs of terms in the scale's order; the mean difference between a word alone and in its sentence; the median range width; and the share of answers inside the person's own range.
Every answer names its word, so a study can measure how much its participants differ on each word. The words and sentences are this library's own in each language, and the IPCC defined its ranges for its English terms. The scores describe how words are read; they support research and do not judge a person.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: probability_words_item from the run with the most of them, latest on ties, and that run's probability_words_range rows. A chance for a word has no right answer, and answers far from the IPCC's ranges are part of what the task measures, so they never invalidate. Fewer than half the planned words and sentences answered invalidates, and any other incomplete run is flagged. IPCC terms answered alone against the scale's order on at least 80% of their pairs are flagged for review. Answers under 100 ms are counted as anticipations, and answers under 200 ms on at least half the answers invalidate. A hidden tab withholds the percentile.
Why these rules
There is no correct option to compare with chance, so accuracy is not judged against chance. People read the same probability words very differently, so an answer outside a term's IPCC range is a result, not a failed task. Answers that put "very unlikely" above "very likely" on most pairs suggest the slider was read the wrong way round, so they are flagged and never exclude. An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast. A hidden tab could allow a search for published ranges, so the percentile is withheld after one.
No percentile. Scores and quality flags only.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| IPCC ranges in sentences | Hidden (standard) | Hidden (standard), Shown |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- No percentile. Scores and quality flags only. How norms work
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Probability Words Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Wallsten TS, Budescu DV, Rapoport A, Zwick R, Forsyth B, 1986; Budescu DV, Broomell S, Por H-H, 2009), with the standard settings (ipcc ranges in sentences: Hidden). The main score was consistency with the IPCC scale (share of answers to the IPCC terms, alone and in sentences, inside the IPCC range, ends included).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Wallsten, T. S., Budescu, D. V., Rapoport, A., Zwick, R., & Forsyth, B. (1986). Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General, 115(4), 348-365. https://doi.org/10.1037/0096-3445.115.4.348The vague meanings of probability terms such as doubtful, probable and likely as functions over the probability scale, with large but stable differences between people.
- Budescu, D. V., Broomell, S., & Por, H.-H. (2009). Improving communication of uncertainty in the reports of the Intergovernmental Panel on Climate Change. Psychological Science, 20(3), 299-308. https://doi.org/10.1111/j.1467-9280.2009.02284.xReaders gave numbers to the probability terms in sentences from the 2007 IPCC report, and their numbers deviated from the IPCC's guidelines even with the guidelines at hand; the authors proposed another way to communicate uncertainty.
- Budescu, D. V., Por, H.-H., Broomell, S. B., & Smithson, M. (2014). The interpretation of IPCC probabilistic statements around the world. Nature Climate Change, 4(6), 508-512. https://doi.org/10.1038/nclimate2194In 25 samples in 24 countries and 17 languages, laypeople read IPCC statements as closer to 50% than intended, and numerical ranges beside the terms brought readings closer to the guidelines.
- Mastrandrea, M. D., Field, C. B., Stocker, T. F., Edenhofer, O., Ebi, K. L., Frame, D. J., Held, H., Kriegler, E., Mach, K. J., Matschoss, P. R., Plattner, G.-K., Yohe, G. W., & Zwiers, F. W. (2010). Guidance note for lead authors of the IPCC Fifth Assessment Report on consistent treatment of uncertainties. Intergovernmental Panel on Climate Change. The likelihood scale: virtually certain 99 to 100%, very likely 90 to 100%, likely 66 to 100%, about as likely as not 33 to 66%, unlikely 0 to 33%, very unlikely 0 to 10% and exceptionally unlikely 0 to 1%.
Source papers
- Budescu DV, Broomell S, Por H-H (2009). Improving communication of uncertainty in the reports of the Intergovernmental Panel on Climate Change. Psychological Science, 20(3):299-308.Paradigm source
- No normative data for this task.Paradigm source, N/A
Questions
How long does the Probability Words Task take?
About 6 to 10 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Consistency With the IPCC Scale, Distance From the IPCC Scale, Shift Toward 50%, Order Agreement, Sentence Shift, Median Range Width, Answers Inside Own Range, Median Answer Time, Answers Under 100 ms, Words Answered, Sentences Answered, Ranges Answered and Planned Words. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Probability Words Task used in research?
64 papers from 2021 to 2025 name it in their title or abstract, up 21% compared with 2016 to 2020. That ranks it 402 of 711 test families in the ConductCognition test census (use grade D). Source: ConductCognition test census 0.58.0, counted 13 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Probability Words Task have?
In the published research, 1 paper reports its test-retest reliability and 1 registered clinical study uses it as an outcome (evidence grade D in the ConductCognition test census).
Does it report a percentile?
No. Scores and quality flags only.
Which settings can a researcher change?
Ipcc ranges in sentences (Hidden (standard), Shown). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese, with its words in the same language. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.