ConductCognition
  • Domains

    Tests

    • Processing speed55
    • Attention143
    • Working memory51
    • Executive function191
    • Episodic memory84
    • Response inhibition10
    • Social cognition97
    • Language50

    Most used

    Evidence

    1. Verbal Fluency TestGradeA
    2. Stroop Color-Word TestGradeA
    3. Trail Making TestGradeA
    4. N-Back TaskGradeA
    5. Go/No-Go TaskGradeA
    6. Digit Span TestGradeA
    7. Symbol Coding TestGradeA
    8. Verbal Learning TestGradeA
    REDTry it nowStroop Color-Word Test
    All 681 tests

    Browse by

    • Phone380
    • Tablet415
    • Untimed419
    • Up to 5 minutes81
    • Evidence grade A48
  • First studies

    Grade, length

    • Reaction Time StudyGradeA7 min
    • Pilot ScreeningGradeA15 min
    • Medication StudiesGradeA−26 min
    • Language-Minimized ScreeningGradeA−30 min

    Age groups

    • Older Adult Executive SpeedGradeA20 min
    • School-AgeGradeA−33 min

    Memory

    • At-Home MemoryGradeA26 min
    • Working Memory and LearningGradeA−19 min
    • Associative MemoryGradeB32 min
    • Working Memory CapacityGradeB34 min

    Language and perception

    • LanguageGradeA−33 min
    • Visual PerceptionGradeB27 min
    • Listening and Auditory ProcessingGradeB31 min

    Executive function and attention

    • Executive ControlGradeA18 min
    • Attention and ImpulsivityGradeA−26 min
    • Visual AttentionGradeB27 min
    • Creativity and Flexible ThinkingGradeB30 min
    • Reasoning and SpatialGradeB32 min

    Movement

    • Motor and AttentionGradeA−29 min
    • Motor LearningGradeB34 min

    Behavior, mood and choice

    • Food Choice and Self-ControlGradeA−29 min
    • Sleep and FatigueGradeB23 min
    • Social CognitionGradeB23 min
    • Impulsivity and Substance UseGradeB26 min
    • Emotion and MoodGradeB28 min
    • Decisions and RewardGradeB29 min
    All 26 batteries

    Find one

    Up to 15 minutes2Up to 30 minutes19Phone8
    Test order labBattery gradesHow batteries work
  • Use and evidence

    • Overview
    • Compare all 681 tests
    • Test-retest
    • Reliability and validity
    • Battery grades26
    • Download and cite

    Clinical trials

    • Overview
    • By condition25
    • By test103
    • Trial design benchmark
    • Cognitive outcomes, 2026

    Conditions

    Registered studies

    • Dementia and Alzheimer's1,673
    • Aging, frailty and exercise1,400
    • Depression and bipolar disorder1,249
    • Stroke1,237
    • Substance use and addiction904
    • Parkinson's disease894
    • Cancer762
    All 25 conditions
    16,526Registered studies name a cognitive test in an outcome measure11,848 name a test you can run hereExplore the studies
  • 1

    Build

    • Study design
    • Customize tests
    • Repeat visits
    • Study cost calculator
    2

    Recruit

    • Recruitment setup
    • Prolific
    • Sona Systems
    • CloudResearch Connect
    • Qualtrics
    3

    Run

    • Remote testing
    • Participant devices
    • Phone timing check
    • Offline edition
    4

    Results

    • Automated scoring
    • Test norms
    • Study Quality Monitor
    • Data export
    Platform overview

    Compliance

    • HIPAA-ready testing
    • US clinical research
    • Data Policy
    • Privacy Policy
    • Terms of Use
  • Research areas

    Tests used in each

    • Aging and dementia202
    • Decisions and reward136
    • Child development and education114
    • Social behavior87
    • Language and bilingualism80
    • Psychiatric groups35
    • Training and intervention31
    • Mood, anxiety and trauma28
    • Substance use and impulsivity26
    • Neurological and movement research21
    • Sleep and fatigue17
    • Brain injury and concussion14
    • ADHD11
    • Eating and obesity11
    • Medication and drug trials8

    Research settings

    • Student participant pools
    • Teaching a class
    • Clinical trials
    • US clinical research
    • Sites without internet
    Planning a study?
  • Methods

    • How evidence grades work
    • How test batteries work
    • How test order works
    Methods and questions

    Help

    • FAQ
    • Recruitment guides
    • Compare platforms

    Updates

    • Blog
    • New tests by email
    Latest post10 November 2026What you can do in ConductCognition todayA short tour of ConductCognition as it stands on 10 November. After this, we'll post when something changes how you run a study.Read the post
  • Pricing
Start Free
Log In
Log InStart Free
  1. Home
  2. /Tests
  3. /Second Estimate Task

Executive Function

Second Estimate Task

Also called: wisdom of the inner crowd task, crowd within task, dialectical bootstrapping task, repeated estimation task, within-person averaging

Measure the wisdom of the inner crowd: participants estimate the year of 20 events, then estimate again after supposing their first answer was wrong, scored as how much the average of the two estimates beats the first.

8-12 minUntimed

Create a study with this test

Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.

Papers, 2021 to 2025
30
Evidence grade
D
Registered clinical studies
0
How these are counted
In what year did Rembrandt complete The Night Watch?
1690
Your first estimate: 1690
Suppose it is wrong. Too early or too late?
1620
Average against first estimateBracketing rate
Estimate the year of each event, then give a second estimate after supposing the first was wrong
  • Overview
  • Evidence
  • Task design
  • Quality checks
  • Settings
  • Cite
  • Questions

What it measures

Measures the wisdom of the inner crowd: how much the average of a person's two estimates beats their first. Participants estimate the year of 20 events, written for this test in each of the eight languages, then give a second estimate for each after assuming their first was off the mark. The main score is how much smaller the error of the averages is.

Constructs

  • Wisdom of the inner crowd
  • Dialectical bootstrapping
  • Quantitative estimation
  • Considering the opposite

Research uses

  • Judgment and decision-making research
  • Research on aggregation and the wisdom of crowds
  • Debiasing and forecasting research
  • Within-study comparison of how second estimates are asked

How the paradigm works

The Second Estimate Task measures the wisdom of the inner crowd: whether the average of two estimates from the same person is more accurate than their first (Vul and Pashler, 2008; Herzog and Hertwig, 2009). Each of 20 questions, written for this task in each language, asks for the year of an event from 1500 to 1899, five from each century. In the first round the participant types an estimate for each, not told that a second round will follow. Then every question comes again in a new order. In the standard second round, dialectical bootstrapping, the first estimate is on screen and the participant supposes it is wrong, thinks of reasons why, judges whether it was too early or too late and gives a second, different estimate (Herzog and Hertwig, 2009). A setting asks only for a second, different estimate, without showing the first, as Vul and Pashler did, and another adds up to 600 seconds of true-or-false arithmetic between the rounds. There is no time limit and no feedback; one practice question with its answer comes first.

More about the paradigm

The main score is the averaging gain: how much smaller the median error of the averages of the two estimates is than the median error of the first estimates, in percent; averages are rounded to whole years, as Herzog and Hertwig rounded theirs. The same comparison with mean squared errors, the median errors of the first, second and averaged estimates, the share of questions where the average beat the first estimate, the bracketing rate (the two estimates on opposite sides of the right year), the share of unchanged estimates and the median change are reported too. In Herzog and Hertwig's study, averaging with a dialectical estimate improved accuracy by 4.1 percentage points against 0.3 for a plain second estimate, and the two estimates bracketed the right date on 13.6% of questions against 7.9%. Asking people to state outright whether their first guess was too high or too low can make second guesses more extreme and remove the gain (Gaertig and Simmons, 2021), so no such answer is asked here. Scores describe two estimates for these questions; they are research measures and do not judge a person's knowledge or judgment.

Scores

Averaging Gain%
How much smaller the median error of the rounded averages of the two estimates is than the median error of the first estimates, as a percent of the latter.
Above 0, averaging the two estimates beat the first; below 0, it did worse
Squared Error Gain%
The same comparison with mean squared errors, the measure of Vul and Pashler (2008).
Above 0, averaging beat the first estimate; large errors weigh more
First Estimate Erroryears
Median distance of the first estimates from the right years, over the questions answered twice.
Lower is more accurate
Second Estimate Erroryears
Median distance of the second estimates from the right years.
Lower is more accurate
Average Estimate Erroryears
Median distance of the rounded averages of the two estimates from the right years.
Lower is more accurate
Questions Improved%
Share of questions where the average was closer to the right year than the first estimate.
Informational
Bracketing Rate%
Share of questions where the two estimates fell on opposite sides of the right year.
Higher means averaging can help more
Unchanged Estimates%
Share of questions where the second estimate was the same year as the first.
0.8 or more is flagged
Median Changeyears
Median distance between the first and second estimates.
Informational
Median First Estimate Timems
Median time from a question appearing to its first estimate, for answers of at least 100 ms.
Informational; the task has no time limit
Median Second Estimate Timems
Median time from a question appearing to its second estimate, for answers of at least 100 ms.
Informational
Answers Under 100 mscount
Estimates submitted less than 100 ms after their question appeared.
0 expected
Questions Answered Twicecount
Questions with a first and a second estimate in the scored run.
Fewer than half the planned questions invalidates
Planned Questionscount
Questions the run planned: 20.
Informational
Earlier Sessionscount
This participant's sessions of this test begun before this one.
More than 0 is flagged: the second round is a surprise only the first time

Use and evidence

30 papers from 2021 to 2025 name this task in their title or abstract. That ranks it 567 of 711 test families in our census. See the counts for every test.

2021 to 2025, the years gradedEarlier years
Papers per year naming the Second Estimate Task, 2000 to 2025Bar chart. 2021: 1; 2022: 2; 2023: 4; 2024: 14; 2025: 9 papers. Peak year 2024.0510152000: 0 papers2001: 0 papers2002: 0 papers2003: 0 papers2004: 0 papers2005: 0 papers2006: 0 papers2007: 0 papers2008: 1 papers2009: 1 papers2010: 0 papers2011: 1 papers2012: 3 papers2013: 1 papers2014: 3 papers2015: 1 papers2016: 0 papers2017: 2 papers2018: 0 papers2019: 2 papers2020: 5 papers2021: 1 papers2022: 2 papers2023: 4 papers2024: 14 papers2025: 9 papers200020052010201520202025
Papers per year naming the Second Estimate Task, 2000 to 2025Bar chart. 2021: 1; 2022: 2; 2023: 4; 2024: 14; 2025: 9 papers. Peak year 2024.0510152000: 0 papers2001: 0 papers2002: 0 papers2003: 0 papers2004: 0 papers2005: 0 papers2006: 0 papers2007: 0 papers2008: 1 papers2009: 1 papers2010: 0 papers2011: 1 papers2012: 3 papers2013: 1 papers2014: 3 papers2015: 1 papers2016: 0 papers2017: 2 papers2018: 0 papers2019: 2 papers2020: 5 papers2021: 1 papers2022: 2 papers2023: 4 papers2024: 14 papers2025: 9 papers200020052010201520202025
Papers per year whose title or abstract names this task (OpenAlex).
Show the numbers
YearPapers
20259
202414
20234
20222
20211
20205
20192
20180
20172
20160
20151
20143
20131
20123
20111
20100
20091
20081
20070
20060
20050
20040
20030
20020
20010
20000
D

Use grade D

30 to 99 papers from 2021 to 2025.

D

Evidence grade D

Little published measurement evidence found.

Papers on test-retest reliability
0
Papers on reliability or validity
2
Registered clinical studies using it as an outcome
0

Measured on ConductCognition

Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.

No pooled figure yet.

How the grades work

Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.

Source: ConductCognition test census 0.58.0, counted 12 November 2026 from OpenAlex and ClinicalTrials.gov.

Task design

The trial sequence as the test runs in ConductCognition.

Questions
20 questions about the year of an event from 1500 to 1899, 5 from each century, written for this library in each language, each year checked against a named public source. No war, politics, religion or illness.
First round
One typed year per question, in an order drawn from the session's seed, with no feedback and no time limit. Participants are not told that a second round follows. One practice question with its answer comes first.
Second round
Every question again, in a new order. In the standard round the first estimate is on screen, and the participant supposes it is wrong, thinks of reasons why, judges whether it was too early or too late and types a second, different estimate (dialectical bootstrapping). A setting asks only for a second, different estimate, without showing the first.
Break
A setting adds up to 600 seconds of true-or-false arithmetic between the rounds; 0 as standard.
Scores
The averaging gain, how much smaller the median error of the averages of the two estimates is than that of the first estimates, in percent (the main score); the same with squared errors; the median errors of the first, second and averaged estimates in years; the shares of questions where the average beat the first estimate and where the two estimates fell on opposite sides of the right year; and how often and how far the second estimate moved.

The questions are this library's own, and the Confidence Interval Task asks 20 others from the same bank. The second round works only as a surprise, so a person's later sessions are flagged. The scores describe two estimates for these questions; they support research and do not judge a person's knowledge or judgment.

Quality checks

Each session is marked valid, needs review or invalid, with the reason, before you see its scores.

Scored rows: second_estimate_item from the run with the most rows, latest on ties, paired by question across the two rounds. Typed years have no chance level, and an average that does not beat the first estimate is a result, not a failed task. Fewer than half the planned questions with both estimates invalidates, and any other incomplete run is flagged. Second estimates that repeat the first on at least 80% of the questions, and an earlier session of the test, are flagged for review; an earlier session withholds the percentile. Answers under 100 ms are counted as anticipations, and answers under 200 ms on at least half the questions invalidate. A hidden tab withholds the percentile.

Why these rules

There is no correct option to compare with chance, so accuracy is not judged against chance. Whether a person's two estimates bracket the right year, and whether their average beats the first, is what the task measures (Herzog and Hertwig, 2009), so neither invalidates. A second estimate that only repeats the first leaves nothing to average and may mean the second round was not read, so it is flagged and never excludes. The second round works only when it is a surprise (Vul and Pashler, 2008), so a repeat session is flagged. An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast. A hidden tab could allow a search for the answers, so the percentile is withheld after one.

No percentile. Scores and quality flags only.

Running this test

Settings you can change

SettingStandardAllowed
Second estimatesAssume the first is wrong (standard)Assume the first is wrong (standard), A second, different estimate
Break before the second round (seconds)00 to 600

Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.

How test settings work

Need a setting this test does not have?

Devices and languages

  • Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
  • Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
  • Available on: Every plan, including the free plan.
  • No percentile. Scores and quality flags only. How norms work
  • Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.

Cite and describe

Methods text, with the standard settings

Participants completed the Second Estimate Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Vul E, Pashler H, 2008; Herzog SM, Hertwig R, 2009), with the standard settings (second estimates: Assume the first is wrong; break before the second round (seconds): 0). The main score was averaging gain (how much smaller the median error of the rounded averages of the two estimates is than the median error of the first estimates, as a percent of the latter).

ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.

  • Vul, E., & Pashler, H. (2008). Measuring the crowd within: Probabilistic representations within individuals. Psychological Science, 19(7), 645-647. https://doi.org/10.1111/j.1467-9280.2008.02136.xThe average of two guesses from one person beat either guess, more so when the second came 3 weeks later; the second guess was asked for without advance notice.
  • Herzog, S. M., & Hertwig, R. (2009). The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science, 20(2), 231-237. https://doi.org/10.1111/j.1467-9280.2009.02271.xDialectical bootstrapping on dates from the 16th to 19th centuries: a second estimate after assuming the first is off the mark, its four steps and the accuracy gain.
  • Steegen, S., Dewitte, L., Tuerlinckx, F., & Vanpaemel, W. (2014). Measuring the crowd within again: A pre-registered replication study. Frontiers in Psychology, 5, 786. https://doi.org/10.3389/fpsyg.2014.00786A pre-registered replication of the crowd within, with both rounds in random order and the second unannounced.
  • Gaertig, C., & Simmons, J. P. (2021). The psychology of second guesses: Implications for the wisdom of the inner crowd. Management Science, 67(9), 5921-5942. https://doi.org/10.1287/mnsc.2020.3781Stating outright whether the first guess was too high or too low made second guesses more extreme and could remove the gain, so no such answer is asked.
  • van Dolder, D., & van den Assem, M. J. (2018). The wisdom of the inner crowd in three large natural experiments. Nature Human Behaviour, 2(1), 21-26. https://doi.org/10.1038/s41562-017-0247-6Within-person averaging improved accuracy in 1.2 million estimates, more with time between estimates, and far less than averaging across people.

Source papers

  • Herzog SM, Hertwig R (2009). The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science, 20(2):231-237.N 101, University students
  • No normative data for this task.Paradigm source, N/A

Questions

How long does the Second Estimate Task take?

About 8 to 12 minutes, as listed in the test library.

What device does a participant need?

Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.

What scores does it produce?

Averaging Gain, Squared Error Gain, First Estimate Error, Second Estimate Error, Average Estimate Error, Questions Improved, Bracketing Rate, Unchanged Estimates, Median Change, Median First Estimate Time, Median Second Estimate Time, Answers Under 100 ms, Questions Answered Twice, Planned Questions and Earlier Sessions. Each session is also marked valid, needs review or invalid, with the reason.

How widely is the Second Estimate Task used in research?

30 papers from 2021 to 2025 name it in their title or abstract. That ranks it 567 of 711 test families in the ConductCognition test census (use grade D). Source: ConductCognition test census 0.58.0, counted 12 November 2026 from OpenAlex and ClinicalTrials.gov.

How much reliability evidence does the Second Estimate Task have?

In the published research, 0 papers report its test-retest reliability and 0 registered clinical studies use it as an outcome (evidence grade D in the ConductCognition test census).

Does it report a percentile?

No. Scores and quality flags only.

Which settings can a researcher change?

Second estimates (Assume the first is wrong (standard), A second, different estimate) and break before the second round (seconds) (0 to 600). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.

Which languages can participants use?

Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese, with its words in the same language. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.

Is this a clinical test?

No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.

Every test: the full test list.

New tests and features, about once a month. Unsubscribe at any time.

Run this test in your next study

Start free with 10 free participants on any test, then use participant credits or a yearly plan.

Start free

Related tests

Confidence Interval TaskAdvice Taking TaskAnchoring Task
ConductCognition

Hosted browser-native cognitive testing for academic and commercial research workflows.

A product of ConductScience, Inc.

Tests

  • All cognitive tests
  • Test batteries
  • Use and evidence
  • Test norms
  • Trail Making Test
  • N-Back Task
  • Big Five personality test
  • Automated scoring
  • Study design
  • New tests by email

Guides

  • Methods and questions
  • Blog
  • Recruitment setup
  • Participant devices
  • Prolific
  • Sona Systems
  • CloudResearch Connect
  • Qualtrics

Use cases

  • Student participant pools
  • Aging and dementia research
  • Clinical trials
  • US clinical research
  • Remote testing

Compare and buy

  • Pricing
  • Compare platforms
  • Offline edition
  • FAQ

Trust

  • Study Quality Monitor
  • HIPAA-ready testing
  • Data export
  • Privacy Policy
  • Terms of Use
  • Data Policy

Company

  • ConductScience
  • ConductScience Fellows
© 2026 ConductScience, Inc. Research use only.