Quality and Call QA: Calibration
Objective. Open Calibration and read reviewer agreement as the trust metric for the whole programme - the thing that makes scores comparable - not as the same as accuracy or a one-time event.
Calibration measures whether reviewers score the same interaction the same way. This screen answers 'can we trust that a score means the same thing whoever gave it?' - the single metric that turns coverage into credibility.
See it in the demo
What this screen answers
Calibration has several reviewers score the same interactions and measures how closely they agree. It answers 'is our scoring consistent across reviewers, and across AI versus human?' A CSM uses it to validate that aggregates are even comparable before trusting them; Sales uses it to position AI scoring honestly - calibration is the control that keeps the AI first pass and human reviewers aligned.
The task
Scope to the reviewer group and period, run or read a calibration exercise on shared interactions, and look at the spread of scores on the same call. The task is to judge agreement - where reviewers converge and where they diverge on specific rubric items - and to feed divergence back into rubric clarity. Treat calibration as a recurring routine, not a box ticked once.
How to read it
Calibration agreement is the trust metric; coverage without it is noise. The classic misread is confusing calibration with accuracy, or assuming one calibration session fixes alignment permanently. Agreement means scores are consistent and therefore comparable - it does not by itself prove the rubric is right, and reviewers drift over time, so it must repeat. Low agreement is the most important red flag in the app: it means two reviewers reading the same call reach different scores, so every aggregate built from mixed reviewers is mixing incompatible judgements. Read divergence on specific items as a pointer back to ambiguous rubric wording.
Honest limits
High agreement proves consistency, not correctness - a group can be calibrated to the wrong standard. Calibration also does not fix a sample that is unrepresentative. Confirm with product how calibration sessions, agreement thresholds and AI-versus-human comparison are configured and entitled; the demo shows the mechanism on seeded interactions.
Interpretation check
Calibration agreement is the trust metric that makes scores comparable - but it proves consistency, not correctness, and it must repeat because reviewers drift.
In the field
OVO (contact centre): before a QBR, a CSM checks reviewer agreement so the quality number presented is defensible - two leads scoring the same call differently would quietly undermine every figure in the review.
Practical task
Open the Calibration screen, read one exercise, and write a finding: where reviewers agreed, the one item they diverged on, and what you would change in the rubric to close it - then explain why low agreement invalidates a mixed-reviewer aggregate.
Self-check
Why is calibration agreement the metric that matters most?
Because it measures whether a score means the same thing whoever gave it. Without agreement, more evaluations just multiply inconsistent, disputable scores.
Reviewers agree closely. Does that prove the scores are correct?
No - it proves consistency, not correctness. A group can be calibrated to the wrong standard; rubric fit is a separate question.
A team calibrated once in January. Is that enough?
No. Reviewers drift, rubrics change, new reviewers join - calibration is a recurring routine, not a one-time event.
Agreement is low on one checklist item. What does that point to?
Usually ambiguous rubric wording on that item. Feed the divergence back into clarifying the scenario checklist so reviewers can converge.
Verified vs demo. Screen is live in the demo at /#/review/calibration; exact controls and figures are SME-validation-pending.