Quality and Call QA troubleshooting - numbers look wrong, coverage gaps, disputed figures
Objective. Give a CSM a symptom-first diagnosis path for the common ways Quality and Call QA goes wrong, with the first checks to run and the point at which it escalates - at app-logic level, without invented error messages or UI paths.
Most QA problems a CSM meets are not product faults - they are trust breaking somewhere in the loop: a rubric that does not fit, a sample that is skewed, reviewers who disagree, an AI score nobody reviewed, or coaching that never closes. This guide is organised by symptom. For each, it gives the likely cause in product-logic terms, the first checks to run upstream, and when to escalate to the implementation specialist or product rather than keep digging.
See it in the demo
Symptom: the numbers look wrong or inconsistent
Likely cause. Scores that disagree, swing between reviewers, or do not match what the customer expects almost always trace to the rubric or calibration, not a calculation fault. A score means only what the Scenarios & Checklists encode, and a mixed-reviewer aggregate is only coherent if reviewers agree on the same call.
First checks (upstream, in order):
- Open Scenarios & Checklists - does the rubric match how this customer defines quality, and are any items ambiguous enough to split two reviewers.
- Open Calibration - do reviewers (and AI versus human) agree on shared calls, and is calibration recurring. Low agreement is the single most common root of 'the numbers look wrong'.
- Open a few individual scores and read the evidence against the rubric - is each defensible, or is a tile hiding an indefensible score.
- Check whether an open dispute is being counted as final, which would make a provisional score look settled.
Escalate when: the rubric fits, calibration is healthy and evidence is defensible but figures still do not reconcile - that points to a configuration or data question (how a metric is computed, a source feed) for the implementation specialist or product.
Symptom: coverage gaps or missing interactions
Likely cause. Interactions that should be reviewable are absent, or whole agents, shifts or channels are under-represented. This is usually upstream of QA - a capture or recording boundary, or a sampling rule that over-weights easy or recent interactions - rather than QA dropping data.
First checks:
- Open Sampling - read the rule and the resulting coverage by agent, shift, channel and work type. A gap may be the sampling design working as set, not a fault.
- Ask what feeds interactions and evidence - recording, telephony and transcription sources - and whether coverage is partial for a channel. If a channel is not captured, no sampling rule can represent it.
- Check capture policy and agent coverage in the monitored-work context (Employee Monitoring / RCM capture) - a capture-policy exclusion bounds the sample before sampling rules apply.
- Separate 'not sampled' from 'not captured' - the first is a sampling-design fix, the second is a source or capture-policy issue.
Escalate when: the gap is a source-coverage, recording or capture-policy question. Supported sources and their coverage are entitlement-dependent and must not be guessed - route to the implementation specialist or product.
Symptom: a customer disputes a figure
Likely cause. The customer challenges an aggregate (a team score, an agent ranking, a quality headline) or an individual score. The usual root is a missing caveat - a figure quoted without its sample, period and calibration basis - or a genuine rubric/calibration gap the dispute has surfaced.
First checks:
- Re-state the figure with its basis: sample, period and calibration. A coverage or activity number read as an accuracy claim is the most common source of a dispute you will actually win by reframing.
- For an individual score, use the Disputes path - trace the original score, the challenge, the evidence re-examined and the resolution. A resolved dispute is where the score becomes defensible.
- Check calibration for the reviewers involved - two leads scoring the same call differently will undermine any figure, and a rising dispute rate is a rubric-clarity or calibration signal, not blame.
- For an agent-level dispute, confirm the comparison aligned call mix, complexity, sample size and calibration before any ranking was shown.
Escalate when: the dispute touches contractual commitments, an SLA figure, or a compliance consequence - those are never settled on a CSM's own authority; involve the implementation specialist, product and the commercial owner.
Symptom: AI scores or coaching are not trusted or not working
Likely cause. Either Ask Quality AI output is being treated as final and resented, or coaching is marked done without quality moving. Both are loop-trust problems, not calculation faults.
First checks:
- For AI distrust - confirm consequential AI scores are passing human review, calibration and evidence, not standing alone. An AI score without human review is a confident-looking number no one validated; route it through the same controls as a human reviewer's score.
- Check AI-versus-human agreement in Calibration - if AI diverges systematically on an item, that is a calibration and rubric-clarity signal.
- For coaching that is not working - open Coaching and separate assigned, completed and improved. High completion with no movement in recurring errors means the loop is busy, not effective.
- Trace one coaching action back to its Root Cause pattern and forward to the next-cycle score on that error - improvement needs the trend and a baseline, and attribution is never clean.
Escalate when: the question is AI model behaviour or limits, or how calibration thresholds and coaching workflows are configured - entitlement- and configuration-dependent, so confirm with product rather than guess.
Interpretation check
Most QA problems are trust breaking somewhere in the loop, not a calculation fault - so diagnose upstream (rubric, sample, calibration, evidence) before suspecting the product, and escalate source-coverage, AI-model, security and contractual questions rather than guessing.
In the field
KnackGlobal (RCM): a 'coverage gap' complaint resolved not by changing sampling but by finding capture policy had excluded a compliance-heavy interaction type - a source/policy escalation, not a QA fault. OVO (contact centre): a disputed agent ranking traced to an uncalibrated reviewer pair, fixed by a calibration session and re-stating the figure with its basis.
Practical task
Take one seeded or real complaint ('our QA score dropped and we don't believe it'). Write the diagnosis: the symptom bucket, the likely cause, the first three upstream checks you would run in order (rubric, calibration, sample/evidence), what you expect to find, and the one thing in this case you would escalate rather than resolve yourself - naming who you escalate to.
Self-check
A customer says 'these scores look wrong.' What do you check first, and why not the calculation?
The rubric fit and calibration - a score means only what the checklist encodes, and a mixed-reviewer aggregate is only coherent if reviewers agree. Inconsistent scores almost always trace to those, not to a calculation fault.
How do you tell a sampling gap from a capture gap?
Read Sampling for 'not sampled' (a design choice you can change) versus confirming source/capture coverage for 'not captured' (a recording or capture-policy boundary upstream of QA). The fix differs: sampling rule versus source/policy escalation.
A customer disputes a team quality figure. What is the most common winnable root cause?
A missing caveat - the figure was quoted without its sample, period and calibration basis, so an activity or coverage number was read as an accuracy claim. Re-state the basis before anything else.
Coaching completion is high but recurring errors have not moved. What does that mean and what do you check?
The loop is busy, not working. Separate assigned/completed/improved, trace an action to its root-cause pattern and to the next-cycle score on that error, and check against a baseline - completion is a step, not a result.
Which symptoms always escalate rather than being resolved by the CSM?
Anything touching supported source coverage, AI model behaviour, how thresholds/workflows are configured, and any contractual, SLA or compliance consequence. These are entitlement-, configuration- or commercially-owned and must never be guessed.
Verified vs demo. The diagnostic screens (Scenarios & Checklists, Sampling, Calibration, Disputes, Coaching, Root Cause, Ask Quality AI) are demo-shown on seeded data; this guide describes causes and checks at app-logic level with no invented error messages or UI paths. Source coverage, AI model behaviour, thresholds and workflow configuration are entitlement-dependent - confirm with product.