Quality and Call QA: Analytics
Objective. Open Analytics and read quality trends with every upstream caveat attached - a moving aggregate can reflect sampling, rubric or reviewer drift as easily as real change.
Analytics shows quality trends and breakdowns over time. It answers 'how is quality moving and where?' - but an aggregate is only as trustworthy as the sample, rubric and calibration beneath it.
See it in the demo
What this screen answers
Analytics aggregates scores into trends, distributions and breakdowns by team, period and category. It answers 'is quality moving, and where is the variation?' A CSM uses it to spot trends worth investigating and to frame a QBR; Sales uses it to show the loop produces management-level insight, while being careful not to present an aggregate as settled truth.
The task
Scope to the team and period, read the trend and one or two breakdowns, and before concluding anything, check what moved underneath: did the sampling rule, rubric version, reviewer mix or volume change over the same window. The task is to decide whether a movement is a real quality change or an artefact of the measurement, and to state which, with the evidence.
How to read it
The classic misread is reading a rising quality line as 'quality improved'. A trend can move because quality genuinely changed, or because the sample shifted, the rubric was edited, reviewers drifted more lenient, or calibration slipped - the aggregate cannot tell these apart on its own. Analytics inherits every upstream caveat: sample representativeness, rubric fit and calibration agreement. Read a trend as a question to investigate, not an answer, and reconcile any surprising movement against changes in sampling, rubric and reviewer calibration before you attribute it to the operation.
Honest limits
An aggregate proves neither accuracy nor improvement by itself - it is a roll-up of sampled, reviewer-scored interactions. Comparability over time breaks if the rubric or sampling changed mid-window. Confirm with product which analytics, breakdowns and date handling are entitled; the demo shows the views on seeded data. Never quote a trend to a customer without its sample and calibration basis.
Interpretation check
A moving quality trend can reflect a sampling, rubric or reviewer-calibration change as easily as a real quality change - the aggregate inherits every upstream caveat and cannot tell them apart alone.
In the field
KnackGlobal (RCM): a quality uptick that actually came from sampling easier claims would be a dangerous story in a QBR - reconciling the trend against the sampling rule is what keeps the number defensible.
Practical task
Open the Analytics screen, pick one trend that moved, and write a reconciliation note: the movement, two measurement changes you would rule out first (sampling, rubric, reviewer mix, calibration), and your provisional read - then the caveat you would attach before showing it to a customer.
Self-check
The quality line rose 6 points this quarter. Did quality improve?
Not necessarily. Check whether sampling, rubric version, reviewer mix or calibration changed over the window - any of those can move the aggregate without quality changing.
Why can a trend break comparability over time?
If the rubric or sampling rule changed mid-window, earlier and later points measure different things, so the line is not a like-for-like comparison.
What are the upstream caveats an analytics aggregate inherits?
Sample representativeness, rubric fit and reviewer calibration agreement. A trend built on a biased sample or uncalibrated reviewers is a confident but unreliable line.
How should you use a surprising movement on this screen?
As a question to investigate - reconcile it against sampling, rubric and calibration changes before attributing it to the operation or putting it in a QBR.
Verified vs demo. Screen is live in the demo at /#/analytics; exact controls and figures are SME-validation-pending.