Social Cognition
Norm Ratings Task
Also called: norm elicitation, social appropriateness ratings, coordination-based norm ratings
Measure shared judgments of everyday actions with twelve social appropriateness ratings and matching points calculated after your study closes.
4-6 minUntimed
Free to start: 10 free participants per account, then $0.99 per participant for academic research, or any yearly plan. Try it here first, with no account.
- Papers, 2021 to 2025
- 53
- Evidence grade
- D
- Registered clinical studies
- 0
What it measures
Measures shared judgments of everyday actions. Participants rate how socially appropriate each action is, trying to match the most common rating in their study.
Constructs
- Social norms
- Social appropriateness
- Shared expectations
Research uses
- Social norm research
- Behavioral economics research
- Within-study comparison of action ratings
How the paradigm works
The Norm Ratings Task elicits shared judgments of everyday actions. Four original situations cover dividing 12 points, returning a borrowed tool, finishing a shared-room booking and waiting in a line. Each has three actions, giving twelve ratings. The session seed shuffles situations and the actions within each. A situation is read before its three actions. Participants choose from four ordered social appropriateness ratings, aiming to match the most common rating of other participants in their study. The rule check gives feedback; action ratings have no correctness feedback. Instructions and items are available in eight languages.
More about the paradigm
Ratings are coded -1, -1/3, +1/3 and +1. Each action has its own score, alongside the mean across actions and the share rated appropriate. These are judgments of the stated actions, not a moral assessment of the participant. Under-100 ms responses remain in ratings but leave the time median. The scorer selects one run and one answer per action.
Matching scores stay blank while the study is open. Completing or archiving it computes points from its own completed sessions. Comparisons use the same protocol step, visit number, language, task version and points setting. Each other participant contributes their latest complete session; the person being scored is excluded. All tied modal ratings match. Each match earns 10 points, adjustable from 1 to 100. There is no cash payment. A complete twelve-rating run and at least one complete peer are required. Reopening the study clears matching scores; closing it again recomputes them. Late completed sessions also refresh the closed-study comparison.
The source used coordination incentives to elicit social appropriateness judgments. This adaptation uses original items and study-local points for every matched action, rather than payment for a randomly selected action. There are no population norms or diagnostic thresholds.
Scores
- Mean Appropriatenessscore
- Mean rating across the twelve actions on a scale from -1 to +1.
- Describes this item set; interpret individual actions first.
- Keep all 12 pointsscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Split 6 and 6score
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Give all 12 pointsscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Keep tool without noticescore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Ask to keep tool longerscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Return tool as agreedscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Stay without askingscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Ask for more room timescore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Leave at booking endscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Go to front without askingscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Ask to go firstscore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Wait at end of linescore
- Rating coded -1, -1/3, +1/3 or +1; null when this action was not rated.
- Higher values mean this action was rated more socially appropriate.
- Ratingscount
- Unique ratings in the selected run.
- Twelve in a complete run.
- Planned Ratingscount
- Number of planned action ratings.
- Twelve original actions.
- Appropriate Share%
- Share rated somewhat or very appropriate.
- Describes this set of actions.
- Matching Rule Checkflag
- 1 for a correct check answer, 0 for an incorrect answer.
- An incorrect check is flagged for review.
- Median Rating Timems
- Median response time excluding responses below 100 ms.
- Time is descriptive, not the primary score.
- Anticipationscount
- Ratings made in less than 100 ms.
- These ratings remain in action scores.
- Matched Actionscount
- Actions matching a most common rating of other study participants after closure.
- Null while open or without a complete peer.
- Compared Actionscount
- Actions with a closed-study comparison; twelve when available.
- Only complete twelve-rating sessions enter matching.
- Matching Share%
- Matched actions divided by compared actions.
- Study-local agreement, not moral accuracy.
- Matching Pointspoints
- Matched actions times the configured points per match.
- Points only; no money is paid.
Use and evidence
53 papers from 2021 to 2025 name this task in their title or abstract, up 89% compared with 2016 to 2020. That ranks it 352 of 582 test families in our census. See the counts for every test.
Show the numbers
| Year | Papers |
|---|---|
| 2025 | 20 |
| 2024 | 15 |
| 2023 | 6 |
| 2022 | 8 |
| 2021 | 4 |
| 2020 | 7 |
| 2019 | 4 |
| 2018 | 7 |
| 2017 | 3 |
| 2016 | 7 |
| 2015 | 6 |
| 2014 | 2 |
| 2013 | 2 |
| 2012 | 3 |
| 2011 | 1 |
| 2010 | 1 |
| 2009 | 0 |
| 2008 | 0 |
| 2007 | 1 |
| 2006 | 1 |
| 2005 | 1 |
| 2004 | 0 |
| 2003 | 0 |
| 2002 | 0 |
| 2001 | 0 |
| 2000 | 0 |
Use grade D
30 to 99 papers from 2021 to 2025.
Evidence grade D
Little published measurement evidence found.
- Papers on test-retest reliability
- 0
- Papers on reliability or validity
- 8
- Registered clinical studies using it as an outcome
- 0
Measured on ConductCognition
Pooled across the studies that feed our platform norms, by device group. A figure needs 50 or more people.
No pooled figure yet.
Note on the counts: Original scenarios; study-local matching points only after closure; no cash incentive or population norms..
How the grades work
Use: A is 1,000 or more papers from 2021 to 2025, B 300 to 999, C 100 to 299, D 30 to 99 and E fewer than 30. Evidence: A means at least 25 papers report its test-retest reliability and at least 25 registered clinical studies use it as an outcome; B, at least 8 test-retest papers, and either 5 registered clinical studies or 150 reliability or validity papers; C, at least 2 test-retest papers, 30 reliability or validity papers or 3 registered clinical studies; D, little published measurement evidence found. The counts measure use. They include every paper that names the task in its title or abstract. Grades describe the published research on this task, not ConductCognition's version of it.
Source: ConductCognition test census 0.39.0, counted 10 November 2026 from OpenAlex and ClinicalTrials.gov.
Task design
The trial sequence as the test runs in ConductCognition.
- Situations
- Four original situations: sharing 12 points, returning a tool, using a shared room and waiting in line. Three actions per situation, twelve ratings in all.
- Order
- Seeded situation order and action order within each situation. Read the situation, then rate its actions at your own pace.
- Scale
- Four ordered ratings, from very socially inappropriate to very socially appropriate. Choose the rating you think most other participants will choose.
- Practice
- One matching-rule question with feedback. The twelve action ratings have no correct-answer feedback.
- Matching points
- 10 points per match, adjustable from 1 to 100. Points stay blank until the study is completed or archived. No money is paid.
- Reference participants
- Only other participants in this study, with the same step, visit, language, task version and points setting. One latest complete session per person; tied most common ratings all match.
- Scores
- Each action rating, mean appropriateness, appropriate-rating share and completion measures. Matching counts, share and points are added after closure when a complete peer is available.
- Languages
- Instructions, situations and actions are available in English, Italian, German, Spanish, French, Japanese, Portuguese and Chinese.
Ratings describe the stated actions. Matching describes agreement within this study, not moral correctness. Reopening clears matching scores; closure recomputes them. There are no population percentiles.
Quality checks
Each session is marked valid, needs review or invalid, with the reason, before you see its scores.
Scored rows: one unique rating per action in the selected run. No rating is morally correct, and repeated ratings are allowed. Missing ratings or a wrong rule check are flagged. Ratings under 200 ms on at least half the actions invalidate.
Why these rules
There is no correct option to compare with chance, so accuracy is not judged against chance. Matching measures agreement within a study, not moral correctness. An answer under 200 ms is faster than a simple visual reaction, so it cannot reflect the item. Following response-time effort screening for untimed tests (Wise and Kong 2005), the session is invalid only when at least half the answers are that fast.
No percentile. Scores and quality flags only.
Running this test
Settings you can change
| Setting | Standard | Allowed |
|---|---|---|
| Points per match | 10 | 1 to 100 |
Yearly plans can change settings within these limits. Free and pay-as-you-go studies use the standard settings.
Need a setting this test does not have?
Devices and languages
- Untimed test: runs on desktops, laptops, tablets and phones. Participant devices
- Languages: English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese.
- Available on: Every plan, including the free plan.
- No percentile. Scores and quality flags only. How norms work
- Recruiting online? See the setup guides for Prolific, Sona Systems, CloudResearch Connect and Qualtrics.
Cite and describe
Participants completed the Norm Ratings Task in a web browser (ConductCognition task version 1.0.0, scoring version 1.0.0; Krupka & Weber, 2013), with the standard settings (points per match: 10). The main score was mean appropriateness (mean rating across the twelve actions on a scale from -1 to +1).
ConductCognition versions: task version 1.0.0, scoring version 1.0.0. Every export records the versions each session used, for a methods section.
- Krupka, E. L., & Weber, R. A. (2013). Identifying social norms using coordination games: Why does dictator game sharing vary? Journal of the European Economic Association, 11(3), 495-524. https://doi.org/10.1111/jeea.12006Four-point social appropriateness ratings with coordination incentives; full methods read. This version uses original items and points for every match after study closure.
Source papers
Questions
How long does the Norm Ratings Task take?
About 4 to 6 minutes, as listed in the test library.
What device does a participant need?
Untimed test: runs on desktops, laptops, tablets and phones. The browser must be Chrome, Edge, Firefox or Safari.
What scores does it produce?
Mean Appropriateness, Keep all 12 points, Split 6 and 6, Give all 12 points, Keep tool without notice, Ask to keep tool longer, Return tool as agreed, Stay without asking, Ask for more room time, Leave at booking end, Go to front without asking, Ask to go first, Wait at end of line, Ratings, Planned Ratings, Appropriate Share, Matching Rule Check, Median Rating Time, Anticipations, Matched Actions, Compared Actions, Matching Share and Matching Points. Each session is also marked valid, needs review or invalid, with the reason.
How widely is the Norm Ratings Task used in research?
53 papers from 2021 to 2025 name it in their title or abstract, up 89% compared with 2016 to 2020. That ranks it 352 of 582 test families in the ConductCognition test census (use grade D). Source: ConductCognition test census 0.39.0, counted 10 November 2026 from OpenAlex and ClinicalTrials.gov.
How much reliability evidence does the Norm Ratings Task have?
In the published research, 0 papers report its test-retest reliability and 0 registered clinical studies use it as an outcome (evidence grade D in the ConductCognition test census).
Does it report a percentile?
No. Scores and quality flags only.
Which settings can a researcher change?
Points per match (1 to 100). Settings can be changed on yearly plans; free and pay-as-you-go studies use the standard settings.
Which languages can participants use?
Participants can take this test in English, Spanish, French, German, Italian, Portuguese (Brazil), Chinese (Simplified) and Japanese, with its words in the same language. English is on every plan, and the other languages come with yearly plans. Have a native speaker read the machine-translated text before launch.
Is this a clinical test?
No. ConductCognition is for research use only. It runs, scores and exports the test; clinical interpretation is outside the platform.
Every test: the full test list.
Run this test in your next study
Start free with 10 free participants on any test, then use participant credits or a yearly plan.
Related tests