Attention
Ensemble Averaging Task
Also called: ensemble perception, ensemble coding, mean size judgment, average size perception, summary statistical representation, set mean discrimination
Measure how precisely people judge the average size of a briefly seen set of circles, beside their memory for its single members.
6-8 minTimed: desktop or laptop with a mouse or trackpad
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 280
- Evidence grade
- D
- Registered clinical studies
- 0
What it measures
A measure of seeing the gist of a set. Eight circles of four sizes flash for 500 ms, and one circle then appears in the middle. In some blocks participants press F or J for whether it is smaller or larger than the set's average size; in others, for whether the set held a circle of that size. The main score is the size difference, in percent, at which 75 percent of average judgments are right, with memory for single members beside it.
Constructs
- Ensemble perception
- Summary statistics
- Size perception
- Visual memory for single items
Research uses
- Perception and attention research
- Individual differences in ensemble perception
- Within-study comparison of averaging and memory for members
How the paradigm works
The Ensemble Averaging Task measures how precisely people judge the average size of a set of circles seen at a glance, beside how well they remember the sizes of its single members. Ariely (2001) showed sets of spots of four sizes for 500 ms, then one test spot: observers judged whether it was larger or smaller than the set's mean size with thresholds of 4 to 12 percent of the spot size, yet hardly told the set's members from sizes it did not hold, even when they differed by 18 percent. Chong and Treisman (2003) found mean size judged about as precisely as the size of a single circle, with 75 percent thresholds of 6 to 8 percent of diameter, and little loss down to a 50 ms display. Myczek and Simons (2008) showed that judging from one or two of the circles can come close to these levels, so a low threshold does not by itself show that every circle was averaged.
More about the paradigm
Each trial shows a cross for 500 ms, then 8 light circles on a dark panel for 500 ms (30 frames of a 60 Hz screen), two of each of four sizes 1.2 apart on a log scale, the whole set scaled at random from trial to trial. One circle then appears in the middle at once and stays until the answer, with no time limit. In average blocks the participant presses F if it is smaller than the set's average size or J if it is larger; it is 6, 12, 18 or 24 percent smaller or larger than the set's mean diameter, differences at which the mean diameter, the diameter of the mean area and the geometric mean all give the same answer. In member blocks the participant presses F if the set held a circle of that size or J if it did not; half the probes are members and half are sizes between them, among them the set's center size, which the set never holds. Four blocks alternate, average first, and 6 rounds (standard; 4 to 10) give 96 trials after 8 practice trials with feedback. Every display time is set before the session, and no answer changes a later trial. Displays switch on screen frames, the measured durations are saved on every trial, and trials more than one frame off plan leave the score. Desktop and laptop only; about 6 to 8 minutes.
The main score is the average threshold: the size difference, in percent of the set's mean diameter, at which 75 percent of average judgments are right, as Chong and Treisman (2003) defined their threshold. Memory for single members is reported beside it as d', members against sizes the set did not hold, with the rate of yes answers to the set's center size. Accuracy at each difference and the lean toward answering larger are also reported. Scores support research and do not diagnose.
Scores
- Average Threshold%
- The size difference from the set's mean diameter, in percent, at which 75 percent of average judgments are right, from a monotone fit to the share right at each difference, interpolated on log difference.
- Lower is finer
- Threshold at the Smallest Differenceflag
- 1 when the 6 percent difference already gave 75 percent right or more, so the score is 6 and an upper bound.
- Informational
- Threshold at the Largest Differenceflag
- 1 when even the 24 percent difference stayed under 75 percent right, so the score is 24 and only a lower bound.
- 1 is flagged for review
- Average Judgments Right%
- Share of average judgments right over all differences.
- Higher is better; chance is 0.5
- Average Judgments Right by Difference%
- Share right at each difference, keyed by its signed percent.
- Rises with the difference
- Average Judgments Right at 18 and 24 Percent%
- Share right at the two largest differences, where the answer is plain.
- Under 0.75 is flagged; under 0.5 invalidates
- Answers at 18 and 24 Percenttrials
- Usable average judgments at the two largest differences.
- Informational
- Larger Answers%
- Share of average judgments answered larger; larger and smaller probes are equally common.
- 0.5 means no lean either way
- Yes to Members%
- Share of yes answers when the set held a circle of the probe's size.
- Informational
- Yes to Sizes Not in the Set%
- Share of yes answers when the set held no circle of the probe's size.
- Informational
- Member Memory (d')d'
- z of the yes rate to members minus z of the yes rate to sizes not in the set, both log-linear corrected.
- Higher means members were told apart better; 0 is no difference
- Member Criterion (c)z
- Minus the mean of the two z scores on member trials.
- Below 0 leans toward answering yes
- Yes to the Center Size%
- Share of yes answers to the set's center size, its geometric mean, which the set never held.
- Informational; often high when the average is what is kept
- Median Answer Timems
- Median time from the probe to the answer.
- Informational; the answer has no time limit
- Anticipationscount
- Answers under 100 ms from the probe, which are not scored.
- Lower is better
- Early Key Pressescount
- F or J pressed before the probe appeared, which are not taken as answers.
- Informational
- Frame Intervalms
- Median time between screen frames during the trials.
- About 16.7 on a 60 Hz screen
- Refresh RateHz
- Screen refresh rate from the frame interval.
- Informational
- Sets on Time%
- Share of answered trials whose set showed within one frame of plan; only these are scored.
- Under 0.9 is flagged for review
- Completed Trialstrials
- Trials answered in the scored run.
- Fewer than planned is flagged for review
- Planned Trialstrials
- Trials the run set out to show.
- Informational
Use and evidence
280 papers from 2021 to 2025 name this task in their title or abstract, up 19% compared with 2016 to 2020. That ranks it 98 of 495 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 64 |
| 2024 | 57 |
| 2023 | 58 |
| 2022 | 48 |
| 2021 | 53 |
| 2020 | 67 |
| 2019 | 48 |
| 2018 | 49 |
| 2017 | 40 |
| 2016 | 31 |
| 2015 | 19 |
| 2014 | 17 |
| 2013 | 17 |
| 2012 | 14 |
| 2011 | 14 |
| 2010 | 17 |
| 2009 | 9 |
| 2008 | 15 |
| 2007 | 4 |
| 2006 | 4 |
| 2005 | 4 |
| 2004 | 9 |
| 2003 | 3 |
| 2002 | 3 |
| 2001 | 2 |
| 2000 | 2 |
Use grade C
100 to 299 papers from 2021 to 2025.
Evidence grade D
Little published measurement evidence found.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 22
- Registered clinical studies using it as an outcome
- 0
Note on the counts: a few counted papers use ensemble coding for groups of neurons.
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.22.0, counted 3 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Set
- On a dark panel: a cross for 500 ms, then 8 light circles, two of each of four sizes 1.2 apart on a log scale around a center size of 34 to 46 px, drawn at random for each set, 80 to 200 px from the middle.
- Duration
- The set shows for 30 frames of a 60 Hz screen, 500 ms, in whole frames of the screen's own rate. One circle then appears in the middle at once and stays until the answer.
- Average blocks
- Press F if the circle is smaller than the set's average size or J if it is larger. It is 6, 12, 18 or 24 percent smaller or larger than the set's mean diameter; at these differences the mean diameter, the diameter of the mean area and the geometric mean give the same answer.
- Member blocks
- Press F if the set held a circle of that size or J if it did not. Half the circles are the set's sizes and half are sizes between them, about 9.5 percent from the nearest, among them the set's center size.
- Trials
- 6 rounds (standard; 4 to 10), each with one average trial at each of the 8 differences and 8 member trials: 96 trials in four blocks, average and member in turn.
- Practice
- 4 average trials at 18 and 24 percent and 4 member trials, each with feedback for 1,500 ms.
- Fixed times
- Every display time and every probe is set before the session, and no answer changes them: there is no staircase and no training mode.
- Score
- The average threshold: the size difference at which 75 percent of average judgments are right. Member memory as d' beside it. Trials whose set ran more than one frame off plan, and answers under 100 ms, leave the score.
The circles are drawn in screen pixels with no calibration, so sizes are not visual angles. Every set has the same four relative sizes, as in Ariely's sets, so judging from one or two circles can come close to judging the whole set (Myczek and Simons, 2008). The scores support research and do not diagnose.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: ensemble_averaging_test from the scored run, F or J after the probe, with no time limit, graded from the saved probe. Average trials (smaller or larger than the set's mean diameter, chance 50%, each direction equally often) are judged against chance: below chance invalidates and not above chance is flagged; one key on 90% of all answers invalidates when accuracy is not above chance. Member trials are not judged against chance, since telling members from sizes the set did not hold is often near chance in the literature. Responses under 100 ms follow the timed-test rule. Under 50% right at the 18 and 24 percent differences invalidates and under 75% is flagged; no threshold, a threshold at the ceiling, a run short of its planned trials, or fewer than 90% of trials with the set within one frame of plan is flagged for review.
Why these rules
Accuracy significantly below chance (one-sided exact test, p < .05) cannot come from guessing; it means the response mapping was reversed or the task was not followed (Slick, Sherman and Iverson 1999 treat below-chance forced-choice performance as definite response bias). Chance-level accuracy can be a real result in impaired participants, so it is only flagged. One option on at least 90% of answered trials, when the correct options are balanced and accuracy is not above chance, means the answers did not depend on the items (the task counterpart of long-string responding in surveys; Curran 2016). Ariely (2001) found members told from sizes the set did not hold barely above chance, so chance-level member answers are a result, not a sign of disengagement. Differences of 18 and 24 percent are two to four times the thresholds Ariely and Chong and Treisman (2003) report, so most of those judgments should be right. The set's 500 ms carries the measure, so trials whose set ran late leave it.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Rounds of trials | 6 | 4 to 10 |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Timed test: participants need a desktop or laptop with a mouse or trackpad. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Ensemble Averaging Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Ariely, 2001; Chong and Treisman, 2003), with the standard settings (rounds of trials: 6). The main score was average threshold (the size difference from the set's mean diameter, in percent, at which 75 percent of average judgments are right, from a monotone fit to the share right at each difference, interpolated on log difference).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Ariely, D. (2001). Seeing sets: Representation by statistical properties. Psychological Science, 12(2), 157-162. https://doi.org/10.1111/1467-9280.00327The task: sets of four sizes on a log scale shown for 500 ms, then one probe judged larger or smaller than the mean, or as a member or not; mean thresholds of 4 to 12 percent while members were hardly told from sizes the set did not hold.
- Chong, S. C., & Treisman, A. (2003). Representation of statistical properties. Vision Research, 43(4), 393-404. https://doi.org/10.1016/S0042-6989(02)00596-5The threshold as the size difference giving 75 percent right by the method of constant stimuli, 6 to 8 percent for mean size, with little loss down to a 50 ms display, and people's means between the mean diameter and the mean area.
- Myczek, K., & Simons, D. J. (2008). Better than average: Alternatives to statistical summary representations for rapid judgments of average size. Perception & Psychophysics, 70(5), 772-788. https://doi.org/10.3758/PP.70.5.772Eight circles for 500 ms, then a probe until the answer, 75 percent right from a 13 percent difference; judging from one or two circles comes close.
- Cha, O., Blake, R., & Gauthier, I. (2022). Contribution of a common ability in average and variability judgments. Psychonomic Bulletin & Review, 29(1), 108-115. https://doi.org/10.3758/s13423-021-01982-1Individual differences in average size judgments, with a split-half reliability of .69 for accuracy.
- Whitney, D., & Yamanashi Leib, A. (2018). Ensemble perception. Annual Review of Psychology, 69, 105-129. https://doi.org/10.1146/annurev-psych-010416-044232
Source papers
- Chong SC, Treisman A (2003). Representation of statistical properties. Vision Research, 43(4):393-404.Paradigm source, Adults
Questions
How long does the Ensemble Averaging Task take?
About 6 to 8 minutes, as listed in the test library.
What device does a participant need?
Timed test: participants need a desktop or laptop with a mouse or trackpad. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Average Threshold, Threshold at the Smallest Difference, Threshold at the Largest Difference, Average Judgments Right, Average Judgments Right by Difference, Average Judgments Right at 18 and 24 Percent, Answers at 18 and 24 Percent, Larger Answers, Yes to Members, Yes to Sizes Not in the Set, Member Memory (d'), Member Criterion (c), Yes to the Center Size, Median Answer Time, Anticipations, Early Key Presses, Frame Interval, Refresh Rate, Sets on Time, Completed Trials and Planned Trials. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Ensemble Averaging Task used in research?
280 papers from 2021 to 2025 name it in their title or abstract, up 19% compared with 2016 to 2020. That ranks it 98 of 495 test families in the ConductCognition test census (use grade C). Source: ConductCognition test census 0.22.0, counted 3 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Ensemble Averaging Task have?
In the published research, 0 papers report its test-retest reliability and 0 registered clinical studies use it as an outcome (evidence grade D in the ConductCognition test census).
Which settings can a researcher change?
Rounds of trials (4 to 10). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.
Related tests