Episodic Memory
Judgments of Learning Task
Also called: JOL task, metamemory task, delayed-JOL task
Measure how well people predict their own memory: word pairs rated from 0 to 100% for the chance of recall, then a cued-recall test.
8-10 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Preview any test free after you sign up.
- Papers, 2021 to 2025
- 379
- Evidence grade
- C
- Registered clinical studies
- 4
What it measures
Measures how well people predict their own memory. Participants study 30 word pairs, rate the chance of recalling each second word, then recall the pairs after a short break.
Constructs
- Metamemory
- Monitoring of learning
- Episodic memory
Research uses
- Metacognition and memory research
- Education and study-strategy research
- Within-study comparison of memory monitoring
How the paradigm works
The judgments of learning task measures metamemory: how well people can tell which items they will remember. Participants study 30 pairs of English nouns, 5 seconds each. Half the pairs are related, such as author and novel, and half are unrelated. For each pair, participants see the first word again and choose how sure they are that they will recall the second word in the test: 0, 20, 40, 60, 80 or 100%. By default the judgment comes right after each pair. As a setting it comes after the whole list instead, which usually makes judgments far more accurate (the delayed-JOL effect). After a 1 minute break of true-or-false arithmetic, participants see each first word and type the second.
More about the paradigm
The word pairs were written for this task, and the related and unrelated pairs are matched on word length and word frequency. Typed answers ignore case, accents, punctuation and a leading article, and one wrong, missing, extra or swapped letter is accepted in words of 4 or more letters. No judgment or answer is timed.
The main score is resolution: the Goodman-Kruskal gamma between each pair's judgment and whether it was recalled, from -1 to 1. Calibration bias is the mean judgment minus the percent of pairs recalled; positive values mean overconfidence. Recall and mean judgments for all pairs and for related and unrelated pairs, and judgment times, are also reported. Scores support research and do not diagnose.
Scores
- Resolution (Gamma)score
- Goodman-Kruskal gamma between each pair's judgment and whether it was recalled, from -1 to 1.
- Higher means judgments better tell remembered from forgotten pairs
- Calibration Bias%
- Mean judgment minus the percent of pairs recalled, in percentage points.
- Positive is overconfidence, negative is underconfidence
- Pairs Recalled%
- Percent of the 30 pairs recalled in the test.
- Higher is better
- Mean Judgment%
- Mean predicted chance of recalling the second word, across pairs.
- Context for calibration
- Mean Judgment, Related Pairs%
- Mean judgment for the 15 related pairs.
- Usually higher than for unrelated pairs
- Mean Judgment, Unrelated Pairs%
- Mean judgment for the 15 unrelated pairs.
- Context for calibration
- Recall, Related Pairs%
- Percent of the 15 related pairs recalled.
- Higher is better
- Recall, Unrelated Pairs%
- Percent of the 15 unrelated pairs recalled.
- Higher is better
- Median Judgment Timems
- Median time to make a judgment; judgments under 100 ms are left out.
- Context for the judgments
- Pairs Recalledcount
- Pairs recalled in the test.
- Higher is better
- Typo Matchescount
- Answers accepted with one wrong, missing, extra or swapped letter.
- Context for recall scores
- Blank Answerscount
- Answers left blank.
- Context for recall scores
- Break Accuracy%
- Share of true-or-false arithmetic answered correctly during the break.
- Shows the break task was done
Use and evidence
379 papers from 2021 to 2025 name this task in their title or abstract, up 11% compared with 2016 to 2020. That ranks it 66 of 395 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 98 |
| 2024 | 76 |
| 2023 | 87 |
| 2022 | 55 |
| 2021 | 63 |
| 2020 | 62 |
| 2019 | 73 |
| 2018 | 63 |
| 2017 | 67 |
| 2016 | 77 |
| 2015 | 66 |
| 2014 | 75 |
| 2013 | 71 |
| 2012 | 65 |
| 2011 | 75 |
| 2010 | 52 |
| 2009 | 39 |
| 2008 | 36 |
| 2007 | 38 |
| 2006 | 42 |
| 2005 | 36 |
| 2004 | 38 |
| 2003 | 19 |
| 2002 | 18 |
| 2001 | 16 |
| 2000 | 23 |
Use grade B
300 to 999 papers from 2021 to 2025.
Evidence grade C
At least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies.
- Papers on test-retest reliability
- 4
- Papers on reliability or validity
- 50
- Registered clinical studies using it as an outcome
- 4
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.14.0, counted 31 October 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Pairs
- 30 pairs of common English nouns, 15 related and 15 unrelated, matched on word length and frequency. Each pair is shown for 5 seconds (standard; 3 to 10), with no more than 3 related or unrelated pairs in a row, in an order drawn from the session's seed.
- Judgments
- The first word again, with the question how sure the participant is to recall the second word in the test, answered with one of six buttons: 0, 20, 40, 60, 80 or 100%. No time limit.
- Timing
- Right after each pair (standard), or after the whole list, a third of the list at a time, each third in a new random order.
- Practice
- 2 pairs with judgments and a practice test, each answer followed by the correct pair for 3,000 ms.
- Break
- 60 seconds (standard; 30 to 300) of true-or-false arithmetic.
- Test
- Typed cued recall of all 30 pairs, without feedback or a time limit, a sixth of the study list at a time, each sixth in a new random order.
- Answers
- Case, accents, punctuation and a leading article are ignored, and one wrong, missing, extra or swapped letter is accepted in words of 4 or more letters.
The word pairs were written for this task; no published word list is used. The pairs are English.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: judgments_of_learning_recall from the scored run, one typed answer per pair, with the judgments from the same run. Typed answers have no chance level, so accuracy is not judged against chance, and answers are untimed, so none can be missed. Typed answers under 200 ms on at least half of 4 or more answered pairs invalidate, and so do judgments under 200 ms on at least half of 4 or more judgments. Recalling no pairs, one judgment value for every pair, the same answer for 90% of at least 12 answers, no arithmetic answered during the break, or break arithmetic at or below chance over at least 10 answers is flagged for review. A session run in another language is flagged. A hidden tab withholds the percentile.
Why these rules
There is no correct option to compare with chance, so accuracy is not judged against chance. Recalling nothing can be a real result, so it is flagged rather than invalidated, as for verbal learning. A judgment under 200 ms is faster than reading the cue, so judgments that fast on half the pairs cannot reflect the pairs, which is what gamma measures. One judgment value for every pair leaves gamma undefined. One word typed for nearly every cue cannot recall 30 different pairs, but it can be a placeholder typed for forgotten pairs, so it is flagged. The break keeps the last pairs out of short-term memory before the test; no arithmetic answered, or true-or-false answers at or below chance, means the break may not have been used for the arithmetic. An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast. The word lists are English. A session run in another language shows English words to a participant who may not read English fluently, so it is flagged for review. A hidden tab lets the study, delay or response timers run without the participant, so a percentile is withheld after one.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Study time per pair (seconds) | 5 | 3 to 10 |
| When judgments are made | Right after each pair (standard) | Right after each pair (standard), After the whole list |
| Break before the recall test (seconds) | 60 | 30 to 300 |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Instructions and screens: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. Words and sentences: English.
- Available on: Every plan, including the free plan.
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Judgments of Learning Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Nelson & Dunlosky, 1991), with the standard settings (study time per pair (seconds): 5; when judgments are made: Right after each pair; break before the recall test (seconds): 60). The main score was resolution (Gamma; goodman-kruskal gamma between each pair's judgment and whether it was recalled, from -1 to 1).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Nelson, T. O., & Dunlosky, J. (1991). When people's judgments of learning (JOLs) are extremely accurate at predicting subsequent recall: The delayed-JOL effect. Psychological Science, 2(4), 267-271. https://doi.org/10.1111/j.1467-9280.1991.tb00147.x
- Dunlosky, J., & Nelson, T. O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of various study activities depend on when the JOLs occur? Journal of Memory and Language, 33(4), 545-565. https://doi.org/10.1006/jmla.1994.1026
- Koriat, A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4), 349-370. https://doi.org/10.1037/0096-3445.126.4.349
- Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95(1), 109-133. https://doi.org/10.1037/0033-2909.95.1.109
Source papers
- Nelson TO, Dunlosky J (1991). When people's judgments of learning (JOLs) are extremely accurate at predicting subsequent recall: the delayed-JOL effect. Psychological Science, 2(4):267-271.Paradigm source, Young adults
- Dunlosky J, Nelson TO (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of various study activities depend on when the JOLs occur? Journal of Memory and Language, 33(4):545-565.Paradigm source, Young adults
- Koriat A (1997). Monitoring one's own knowledge during study: a cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4):349-370.Paradigm source, Young adults
- Nelson TO (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95(1):109-133.Paradigm source, Mixed
Questions
How long does the Judgments of Learning Task take?
About 8 to 10 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Resolution (Gamma), Calibration Bias, Pairs Recalled, Mean Judgment, Mean Judgment, Related Pairs, Mean Judgment, Unrelated Pairs, Recall, Related Pairs, Recall, Unrelated Pairs, Median Judgment Time, Pairs Recalled, Typo Matches, Blank Answers and Break Accuracy. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Judgments of Learning Task used in research?
379 papers from 2021 to 2025 name it in their title or abstract, up 11% compared with 2016 to 2020. That ranks it 66 of 395 test families in the ConductCognition test census (use grade B). Source: ConductCognition test census 0.14.0, counted 31 October 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Judgments of Learning Task have?
In the published research, 4 papers report its test-retest reliability and 4 registered clinical studies use it as an outcome (evidence grade C in the ConductCognition test census).
Which settings can a researcher change?
Study time per pair (seconds) (3 to 10), when judgments are made (Right after each pair (standard), After the whole list) and break before the recall test (seconds) (30 to 300). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
The instructions and screens are in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. The words and sentences stay in English, so sessions in other languages are flagged for review. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.