Executive Function
Inbox Decisions Task
Also called: phishing detection task, phishing email classification task, email legitimacy judgment task
Measure phishing detection, response bias and confidence with original emails from fictional companies.
4-6 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 13
- Evidence grade
- D
- Registered clinical studies
- 1
Your booking
Company B
Your room booking for Thursday is confirmed. You can change it in your usual app.
What it measures
Measure how well people distinguish phishing from genuine emails, their response bias and confidence. Uses original messages from fictional companies.
Constructs
- Phishing discrimination
- Response bias
- Confidence in decisions
Research uses
- Decision-making research
- Research on email judgments
How the paradigm works
The Inbox Decisions Task measures how well participants distinguish phishing from genuine emails. Each original email shows a fictional company sender, a subject and a short message. Participants choose Genuine or Phishing, then rate confidence from 1 to 4. There are no active links, domains or attachments.
More about the paradigm
The standard session has 12 emails: 6 genuine and 6 phishing, paired across parcel collection, bookings, files, subscriptions, receipts and delivery. The shorter setting selects 3 topic pairs for 6 emails. Seeded ordering limits runs of the same message type to 3. Two separate practice emails give feedback; the scored trials give none. The debrief identifies every presented phishing message and explains its cue, one per page. All items follow the session language. Each person takes the task once because the debrief reveals the answers.
The primary measure is d-prime, treating phishing judgments on phishing emails as hits and phishing judgments on genuine emails as false alarms. The loglinear correction adds 0.5 to all four cells before converting rates to z scores. Criterion c separates response bias from sensitivity: positive values mean fewer phishing judgments. Accuracy, hit and false-alarm rates, and confidence overall and for correct and incorrect answers are also reported. The final row of each classification-and-confidence pair retains classification time separately from confidence time.
Scores
- Phishing sensitivityscore
- Signal-detection d-prime using 0.5 added to each cell. Phishing is the signal.
- Higher means better separation of phishing and genuine messages
- Response biasscore
- Signal-detection criterion c.
- Positive means fewer phishing judgments; negative means more
- Phishing detected%
- Share of phishing messages judged phishing.
- Read alongside false alarms
- False alarms%
- Share of genuine messages judged phishing.
- Lower means fewer false alarms
- Accuracy%
- Share of valid judgments matching message type.
- Higher means more correct judgments
- Mean confidencescore
- Mean confidence from 1 (not sure) to 4 (very sure).
- Confidence in these judgments
- Confidence when correctscore
- Mean confidence for correct judgments.
- Read alongside confidence when incorrect
- Confidence when incorrectscore
- Mean confidence for incorrect judgments.
- Null when there are no rated errors
- Confidence ratingscount
- Number of valid confidence ratings.
- Compare with items answered
- Emails answeredcount
- Valid judgments in completed judgment-and-confidence pairs.
- At least 90% of planned items needed
- Emails plannedcount
- Number of emails planned.
- 6 or 12
- Median judgment timems
- Median classification time, excluding responses under 100 ms; confidence time is separate.
- Descriptive
- Responses under 100 mscount
- Very fast classifications retained in the scores.
- Expected to be 0
- Earlier sessionscount
- Earlier sessions when known.
- Flagged because the debrief reveals answers
Use and evidence
13 papers from 2021 to 2025 name this task in their title or abstract. That ranks it 518 of 582 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 5 |
| 2024 | 2 |
| 2023 | 2 |
| 2022 | 3 |
| 2021 | 1 |
| 2020 | 1 |
| 2019 | 2 |
| 2018 | 1 |
| 2017 | 0 |
| 2016 | 0 |
| 2015 | 0 |
| 2014 | 0 |
| 2013 | 0 |
| 2012 | 0 |
| 2011 | 0 |
| 2010 | 0 |
| 2009 | 0 |
| 2008 | 0 |
| 2007 | 0 |
| 2006 | 0 |
| 2005 | 0 |
| 2004 | 0 |
| 2003 | 0 |
| 2002 | 0 |
| 2001 | 0 |
| 2000 | 0 |
Use grade E
Fewer than 30 papers from 2021 to 2025.
Evidence grade D
Little published measurement evidence found.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 2
- Registered clinical studies using it as an outcome
- 1
Measured on ConductCognition
Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.
No pooled figure yet.
Note on the counts: most phishing studies with people appear in computer science venues, which the census leaves out.
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.39.0, counted 10 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Emails
- 12 original emails, 6 genuine and 6 phishing across 6 topics. The shorter setting selects 3 topic pairs for 6 emails. Text follows the session language.
- Practice
- 2 separate emails, with correctness feedback and an explanation.
- Trial
- Read an email, choose Genuine or Phishing, then rate confidence from 1 to 4 on a separate screen. No deadline. A 500 ms lead-in and 300 ms gap between completed pairs.
- Order
- Seeded selection and order, with at most 3 consecutive emails of the same type.
- Score
- Sensitivity (d-prime), response bias (criterion c), hit and false-alarm rates, accuracy and confidence. Phishing is the signal; a 0.5 correction is applied in each cell.
- Debrief
- Every presented phishing email is identified with an explanation, one per page. The end says all senders and companies were fictional. Each person takes the task once.
All messages and translations were written for this task, with fictional companies named by letter and no real brand, person, domain or working link. Scores describe these judgments. Norms and percentiles are off.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: inbox_decisions_test from the run with most rows, ties going to the last run. Judgments are untimed. Below 90% of planned items answered invalidates. One rating on at least 90% of 12 or more answers is flagged. Answers under 200 ms on at least half of 4 or more items invalidate. An earlier session is flagged because the debrief reveals the phishing messages. A hidden tab withholds the percentile.
Why these rules
Sensitivity and response bias are outcomes, so an inaccurate or uniform judgment does not itself invalidate a run. Fewer than 90% of planned judgments invalidates; answers under 200 ms on at least half of four or more items invalidate. The debrief reveals the phishing messages, so earlier sessions are flagged.
No percentile. Scores and quality flags only.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Emails | 12 emails (standard) | 6 emails, 12 emails (standard) |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- No percentile. Scores and quality flags only. How norms work
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Inbox Decisions Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Jones et al., 2019; Hautus, 1995), with the standard settings (emails: 12 emails). The main score was phishing sensitivity (signal-detection d-prime using 0.5 added to each cell).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Jones, H. S., Towse, J. N., Race, N., & Harrison, T. (2019). Email fraud: The search for psychological predictors of susceptibility. PLOS ONE, 14(1), e0209684. https://doi.org/10.1371/journal.pone.0209684Email legitimacy judgments analyzed with signal detection. Our task uses original fictional messages, separate classification and confidence, and no deadline.
- Hautus, M. J. (1995). Corrections for extreme proportions and their biasing effects on estimated values of d-prime. Behavior Research Methods, Instruments, & Computers, 27(1), 46-51. https://doi.org/10.3758/BF03203619The loglinear 0.5 correction in each signal-detection cell.
Source papers
- Jones, H. S., Towse, J. N., Race, N., & Harrison, T. (2019). Email fraud: The search for psychological predictors of susceptibility. PLOS ONE, 14(1), e0209684.Paradigm source, Adults
Questions
How long does the Inbox Decisions Task take?
About 4 to 6 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Phishing sensitivity, Response bias, Phishing detected, False alarms, Accuracy, Mean confidence, Confidence when correct, Confidence when incorrect, Confidence ratings, Emails answered, Emails planned, Median judgment time, Responses under 100 ms and Earlier sessions. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Inbox Decisions Task used in research?
13 papers from 2021 to 2025 name it in their title or abstract. That ranks it 518 of 582 test families in the ConductCognition test census (use grade E). Source: ConductCognition test census 0.39.0, counted 10 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Inbox Decisions Task have?
In the published research, 0 papers report its test-retest reliability and 1 registered clinical study uses it as an outcome (evidence grade D in the ConductCognition test census).
Does it report a percentile?
No. Scores and quality flags only.
Which settings can a researcher change?
Emails (6 emails, 12 emails (standard)). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese, with its words in the same language. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.