How to Calibrate a Scorecard When Everyone Is in One Level

How to calibrate a scorecard: read the distribution of results, move the boundaries until every level has people in it, check the levels still tile from zero to one hundred with no gap or overlap, then look again. If eighty per cent land in one level, the cut points are wrong.

How to calibrate a scorecard against real people

You cannot design a discriminating scorecard on paper. You can design a plausible one. Then you fix it against real people.

Most builders never calibrate a scorecard. They publish, collect completions, and never look. That's the whole difference between a mediocre scorecard and a good one, and it costs an afternoon. Calibration is mostly arithmetic, not writing. For someone with limited hours that's the best kind of fix: no new copy, no new questions, just numbers you can move.

Wait until there is something to read

Don't calibrate on ten people. The guide's rhythm is after the first fifty or so completions, then monthly. Before that you're reading noise, and noise tempts you to fix problems that aren't there. After that, patterns emerge that are real.

One rule before you start: tune before you drive traffic. Every change made after launch muddies your history, as the last section explains. So the best time to calibrate is the quiet stretch after your first small batch of real respondents and before you push the scorecard anywhere loud.

Free guide · 28 pagesBuild a scorecard that tells the truth.The Scorecard Playbook: an assessment that separates people and gives you a reason to talk.Get the free guide →

Four reads, four verdicts

There are four things to read. Each has a verdict and each points at a different kind of fix.

The fourth is covered on its own in how to find dead questions in a scorecard. This post is about the other three, because they work at the level of the whole instrument.

The most common finding is also the easiest fix

Say you designed five levels and eighty per cent of people land in one. What went wrong? Not the people. The guide is blunt about this: the cut points are wrong, not the respondents. People are what they are. Your boundaries sit in the wrong place for the scores they actually produce. It's the most common finding by a distance. And it's the easiest to fix, because it's arithmetic rather than writing. You're not rewriting a single question. You're moving a line.

The instinct is to blame the audience. "My people are all just at the same stage." Maybe. But a scorecard that sorts nobody isn't telling you your audience is uniform. It's telling you it can't tell them apart. Fix the instrument first and ask the audience question second.

The wider question of where the lines should go in the first place is in how to set scorecard band boundaries. Calibration is what you do when your first guess met reality.

The loop to calibrate a scorecard

The guide gives calibration a name, and it's a loop of four steps.

1. Read the score distribution. Where do people actually land?
2. Move the boundaries. Shift them until every level is populated.
3. Check the levels still tile. From zero to one hundred, no gap, no overlap.
4. Look at the distribution again. Did it spread the way you wanted?

Step three is the one people forget, and it's the one that hurts. When you drag a boundary, you can leave a gap behind it or let two bands touch. A touching boundary matches twice and the winner is decided by sort order, meaning by accident. A gap is worse. The guide describes a ladder with a hole in it that showed no result in the headline and the best result in the recommendations, on the same page.

The fix is a habit. Write each band so it begins a hundredth of a point above where the last one ends. Then check the whole run, every time you edit. A real example from the guide: someone who scored exactly forty was told their level was "Good" for months because a row began one hundredth of a point early. Nothing errored. It simply matched first.

When your numbers disagree with each other

There's one thing that will confuse you after your first recalibration, so let me say it now.

Recalibrating usually does not go back and change past respondents. They keep the result they were given. So a count in your editor can legitimately disagree with a chart that recalculates live. Neither is broken.

The editor reports what people were told. The chart reports what the new boundaries would say. Both are true. They answer different questions.

Decide which one you're reading before you act on it. If you want to know what your audience looks like against the new lines, use the live chart. If you want to know what was actually promised to a real person who might reply to your email, use the stored result.

Free assessmentHow trapped are you, really?Take the Corporate Suffocation Index, the free solopreneur readiness assessment.Take the assessment →

Change once, write it down

Nothing recomputes, ever. Scores are written once, at completion. There is no rescore. Every change to an option value, a weight or a level boundary forks the instrument, and nothing records which version somebody took.

Your analytics will quietly reinterpret history against the new scale. A shift in your numbers the week after a recalibration may be the recalibration, not your audience.

So build three habits:

• Batch your changes. One sitting, all the edits, then leave it.
• Date every change in your own notes. The tool won't.
• Never read a trend across an edit. Draw a line on the calendar and compare only within each side.

One more thing to check after you finish. Every result should still have a follow-up listening for it. A result with nobody on the other end is the most common silent failure there is. Nothing errors. It just looks like a quiet week. The reasoning behind that, and how to write a follow-up worth having, is in how to write a scorecard action plan.

A calibration session, start to finish

1. Pull the results. Count how many people sit in each level.
2. Name the problem. One level overloaded, an empty level, a narrow dimension, or one dimension always weakest.
3. Edit once. Move the lines. Merge or harden a dimension if needed.
4. Check the tiling. Zero to one hundred, nothing missing, nothing doubled.
5. Write the date and what you changed.
6. Check the follow-ups. Make sure every result still has one listening.

Then close the editor for a month. To calibrate a scorecard well, come back with fresh respondents, not a fresh urge to fiddle.

Free guide · 28 pagesBuild a scorecard that tells the truth.The Scorecard Playbook: an assessment that separates people and gives you a reason to talk.Get the free guide →

The rhythm of the asking

Calibration isn't only about scores. The biggest lever on completion is the rhythm of the questions, and you control it.

One question appears per screen. Single-choice questions move on by themselves after a beat. Multiple-choice, sliders and text boxes all need a press. So changing a question's type changes the physical rhythm of the whole assessment, and almost nobody treats that as a design choice. Keep the spine single-choice. Spend the flow-stoppers deliberately, near the end.

Order matters more than it looks. Question order is usually dimension order, with the dimension name printed above each question. So you are not ordering questions. You are ordering chapters. Open with the dimension that is easiest and most flattering. Put the hardest third or fourth, past the point where people have invested and before they get tired.

Progress is persuasion too. It's usually worked out from position, so question one of twenty-five already reads as a small fraction done. Dots and steppers are honest, but they become daunting past a dozen questions. Over twenty, use a continuous bar. If completions are low even when the results look healthy, look here before you touch the boundaries.

Free assessmentHow trapped are you, really?Take the Corporate Suffocation Index, the free solopreneur readiness assessment.Take the assessment →

Frequently asked questions

How many responses do I need before calibrating?

Around fifty completions is the guide's starting point, then a review monthly. Below that the patterns are mostly noise and you risk fixing things that aren't broken.

What if most people land in one level when I calibrate a scorecard?

Then the cut points are wrong, not the respondents. This is the most common finding and the easiest fix, because you are moving boundaries, not rewriting questions. Move them until every level is populated.

Will recalibrating change old results?

Usually not. Past respondents keep the result they were given, so an editor count can differ from a live chart. Both can be correct. Note the date of the change so you never compare across it.

What if the same dimension is weakest for nearly everyone?

Your scorecard may be one topic wearing several hats. That usually means several dimensions measure the same underlying thing, so check whether they can be merged.

The full method, with the checklists, is in The Scorecard Playbook, a free 28-page playbook.