Reasoning
Human or AI Text Task
Also called: AI text detection task, human or machine text task, Turing test for text, machine-generated text detection by people, real or fake text, AI authorship judgment task
Measure how well people tell texts written by people from texts written by an AI system: participants read short answers to everyday questions, say who wrote each and how sure they are, and see at the end which were which.
8-11 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 84
- Evidence grade
- D
- Registered clinical studies
- 0
What it measures
Measures how well people tell texts written by people from texts written by an AI system. Participants read 24 short English answers to everyday questions, half written by people and half by an AI system with the same topics and lengths, choose who wrote each one and rate how sure they are, from 1 to 5. Scores are accuracy, sensitivity and bias toward either answer. The end says which texts were which.
Constructs
- Detection of AI-written text
- Authorship judgment
- Confidence in one's own judgments
- Response bias
Research uses
- Research on how people judge AI-generated text
- Research on media literacy and trust in generated content
- Within-study group comparison
How the paradigm works
The human or AI text task measures how well people tell texts written by people from texts written by an AI system. Clark and colleagues (2021) asked non-expert readers to say whether stories, news articles and recipes were written by a person or by GPT-2 or GPT-3, and found that without training they told GPT-3 text from human text at random chance level, with training lifting accuracy to at most 55%. Köbis and Mossink (2021) ran two incentivized experiments with 830 people on poems written by GPT-2 and found that readers failed to reliably detect the algorithm's poems when a person picked the best of its samples. Jakesch, Hancock and Naaman (2023) found that people could not detect AI-generated self-presentations, because they relied on flawed heuristics about what AI text looks like.
More about the paradigm
This task shows 24 short answers to everyday questions, such as why geese fly in a V. Twelve were written by people, the science librarians of the Library of Congress, and 12 by an AI system, Claude Sonnet 5.5, from the same questions and with the same lengths. Each question appears once in a session, in one of its two versions drawn from the session's seed, so across sessions neither topic nor length can tell the two kinds apart. For each text the participant chooses who wrote it and says how sure they are, from 1 to 5. The scores treat the AI texts as the signal: sensitivity (d prime), the response criterion, which shows a lean toward answering a person or an AI system, accuracy for each kind of writer, and how much more sure the participant was when right than when wrong. The end of the session says which texts were which. The scores describe how a reader handles these texts and say nothing about anyone's writing.
Scores
- d primed'
- Sensitivity to AI-written texts: the corrected rate of AI texts judged AI minus the corrected rate of people's texts judged AI, on the z scale.
- Informational. 0 means the two kinds are not told apart; below 0 means people's texts were judged AI more often than AI texts
- Response criterionz
- Lean toward one answer. Positive values lean toward answering A person; negative values lean toward An AI system.
- Informational
- Accuracy%
- Percent of answered texts judged correctly.
- Informational. 50 is chance with equal numbers of the two kinds
- Accuracy for people's texts%
- Percent of the texts written by people that were judged to be written by a person.
- Informational
- Accuracy for AI texts%
- Percent of the texts written by the AI system that were judged to be written by an AI system.
- Informational
- Texts judged AI-written%
- Percent of all answered texts judged to be written by an AI system, whoever wrote them.
- Informational. 50 means no lean
- Mean confidencescore
- Mean of the confidence ratings, from 1 (just guessing) to 5 (completely sure).
- Informational
- Confidence when rightscore
- Mean confidence on texts judged correctly, from 1 to 5.
- Informational
- Confidence when wrongscore
- Mean confidence on texts judged wrongly, from 1 to 5.
- Informational
- Confidence gapscore
- Confidence when right minus confidence when wrong.
- Positive means more sure when right; 0 or below means confidence did not follow accuracy
- Median answer timems
- Median time from the text appearing to the answer, over answers of 100 ms or more. It includes reading.
- Informational
- Texts answeredcount
- Texts with a recorded answer and a known text.
- Quality check
- Planned textscount
- 12 or 24 test texts.
- Informational
- Answers under 100 mscount
- Answers and confidence ratings made less than 100 ms after their screen appeared.
- 0 expected
- Session language0/1
- 1 when the session ran in another language than English, since the texts are English.
- Quality check; 1 is flagged for review
Use and evidence
84 papers from 2021 to 2025 name this task in their title or abstract. That ranks it 321 of 711 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 48 |
| 2024 | 27 |
| 2023 | 9 |
| 2022 | 0 |
| 2021 | 0 |
| 2020 | 0 |
| 2019 | 0 |
| 2018 | 0 |
| 2017 | 0 |
| 2016 | 0 |
| 2015 | 0 |
| 2014 | 0 |
| 2013 | 0 |
| 2012 | 0 |
| 2011 | 0 |
| 2010 | 0 |
| 2009 | 0 |
| 2008 | 0 |
| 2007 | 0 |
| 2006 | 0 |
| 2005 | 0 |
| 2004 | 0 |
| 2003 | 0 |
| 2002 | 0 |
| 2001 | 0 |
| 2000 | 0 |
Use grade D
30 to 99 papers from 2021 to 2025.
Evidence grade D
Little published measurement evidence found.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 25
- Registered clinical studies using it as an outcome
- 0
Measured on ConductCognition
Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.
No pooled figure yet.
Note on the counts: includes studies that compare texts by software or by their wording.
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.58.0, counted 16 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Texts
- 24 everyday questions, such as why geese fly in a V. Each has two answers of the same length, one written by people and one by an AI system. A session shows each question once, in one of its two versions drawn from the session's seed: 12 written by people and 12 by the AI system, with no more than 3 of one kind in a row. They are English in every session.
- People's answers
- Answers of the Library of Congress's Everyday Mysteries, written by its science librarians and posted from 2018 to 2020. Each was checked against copies of the Library's page made before June 2022, before AI systems wrote text at this level. They are works of the United States Government, in the public domain. They are shown as published, apart from joining paragraphs, stopping at a sentence end and leaving out citations.
- AI answers
- Written by Claude Sonnet 5.5 from the question and a word target equal to the length of the people's answer, under its maker's commercial terms. No wording is taken from the people's answers, and the two lengths agree within 10 percent.
- Trial
- The question, the text in a light box and two buttons, A person and An AI system, with their sides drawn from the seed. Then how sure the participant is, from 1 (just guessing) to 5 (completely sure), on five buttons. No time limit.
- Session
- 2 practice texts with feedback, then 24 with none, or 12 in the short form. The end of the session says which texts were written by people.
- Scores
- Sensitivity (d prime, with the AI texts as the signal), accuracy, accuracy for each kind of writer, the share of texts judged AI-written, the response criterion, mean confidence, and how much more sure people were when right than when wrong.
Clark and colleagues (2021) found non-expert readers at chance in telling GPT-3 text from human text in stories, news articles and recipes, and Jakesch, Hancock and Naaman (2023) found readers guided by flawed heuristics. This task gives every reader the same questions in both versions and reports sensitivity, bias and confidence. The texts are short answers on everyday science, not stories or profiles, so the scores describe how readers handle these texts and are best compared between groups. They say nothing about anyone's writing.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: human_or_ai_text_test from the scored run. Below-chance accuracy, one-sided answers and confident wrong answers are retained, because how people read the texts is the finding. Answers under 1.5 seconds on at least half the texts invalidate. An incomplete run is flagged for review, and so is a session that ran in another language.
Why these rules
Studies of human and machine text find readers near chance (Clark et al., 2021; Jakesch et al., 2023), so chance, bias and misplaced confidence are outcomes, not exclusion rules. An answer under 1.5 seconds cannot follow reading a text of 60 words or more, so the session is invalid only when at least half the answers are that fast (Wise and Kong 2005). The word lists are English. A session run in another language shows English words to a participant who may not read English fluently, so it is flagged for review.
No percentile. Scores and quality flags only.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Texts of each kind | 12 (24 texts, standard) | 12 (24 texts, standard), 6 (12 texts) |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Instructions and screens: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. Words and sentences: English.
- Available on: Every plan, including the free plan.
- No percentile. Scores and quality flags only. How norms work
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Human or AI Text Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Clark et al., 2021; Jakesch, Hancock and Naaman, 2023; Köbis and Mossink, 2021), with the standard settings (texts of each kind: 12 (24 texts, standard)). The main score was d prime (sensitivity to AI-written texts: the corrected rate of AI texts judged AI minus the corrected rate of people's texts judged AI, on the z scale).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021). All that's "human" is not gold: Evaluating human evaluation of generated text. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 7282-7296. https://doi.org/10.18653/v1/2021.acl-long.565The method: non-experts judge whether short texts were written by a person or an AI system; without training they did no better than chance for GPT-3 text, and training raised accuracy to at most 55%.
- Jakesch, M., Hancock, J. T., & Naaman, M. (2023). Human heuristics for AI-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11), e2208839120. https://doi.org/10.1073/pnas.2208839120People could not detect AI-generated self-presentations, because their judgments rested on flawed heuristics about AI language.
- Köbis, N., & Mossink, L. D. (2021). Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry. Computers in Human Behavior, 114, 106553. https://doi.org/10.1016/j.chb.2020.106553An incentivized version of the Turing test, with 830 people in two experiments, in which readers failed to reliably detect poems the algorithm produced when a person chose its best sample.
Source papers
Questions
How long does the Human or AI Text Task take?
About 8 to 11 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
d prime, Response criterion, Accuracy, Accuracy for people's texts, Accuracy for AI texts, Texts judged AI-written, Mean confidence, Confidence when right, Confidence when wrong, Confidence gap, Median answer time, Texts answered, Planned texts, Answers under 100 ms and Session language. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Human or AI Text Task used in research?
84 papers from 2021 to 2025 name it in their title or abstract. That ranks it 321 of 711 test families in the ConductCognition test census (use grade D). Source: ConductCognition test census 0.58.0, counted 16 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Human or AI Text Task have?
In the published research, 0 papers report its test-retest reliability and 0 registered clinical studies use it as an outcome (evidence grade D in the ConductCognition test census).
Does it report a percentile?
No. Scores and quality flags only.
Which settings can a researcher change?
Texts of each kind (12 (24 texts, standard), 6 (12 texts)). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
The instructions and screens are in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. The words and sentences stay in English, so sessions in other languages are flagged for review. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.