Quality and Call QA assessment - prove you can read and position the quality loop
Objective. Test mastery of Quality and Call QA for the certification gate: can the learner read QA numbers with the right caveats, position AI scoring honestly, and work a realistic scenario without a critical error that would mislead a customer.
This assessment is pass-or-fail on honesty as much as knowledge. A learner who can quote every screen but presents coverage as quality, or an AI score as a verdict, has made a critical error and does not pass. Two scenarios test the real skill - reading a figure with its sample and calibration basis, and positioning the loop to a sceptical buyer. The gate is a practical score of at least 80, no critical error, and manager review.
See it in the demo
Scenario A - the suspicious QBR number (CSM lens)
A customer's QA lead opens a business review with: 'Great quarter - we AI-scored 98 percent of calls and our team average is 93, up from 88.' On inspection you find the sample over-weighted one shift and the day channel, two reviewers who last calibrated eight months ago score the same call 84 and 95, there are eleven open disputes rolled into the 93 as settled, and coaching completion is 96 percent while the top recurring error is flat.
The learner must write what they would say to the QA lead and what they would do before this number is used anywhere. A correct response reframes the 93 as not yet trustworthy (not 'wrong' - unproven), names the specific defects (unrepresentative sample, stale calibration, provisional disputes counted as final, busy-but-not-working coaching), states what to validate and fix, and does not accept or repeat the number as a quality result. It should separate coverage (98 percent reach) from accuracy (needs calibration) and improvement (needs the recurring error to move).
Scenario B - the AI-distrust buyer (Sales lens)
A prospect's operations director says in a demo: 'We are not letting an AI decide our agents' scores. We have seen vendors promise auto-scoring and it was a disaster - agents revolted.' The learner must respond and then show the right two or three screens in the right order.
A correct response agrees with the instinct, positions Ask Quality AI as acceleration and first-pass coverage with calibration, four-eye checks and disputes as the trust controls, and states plainly that a human stands behind any score with a consequence - human-in-the-loop by design, not autonomous scoring. It then shows Disputes (a fair challenge path), Calibration (AI versus human agreement as the trust metric) and the rubric (Scenarios & Checklists) as the shared standard, and volunteers the limitation that coverage is not accuracy. It does not oversell AI, does not promise autonomous scoring, and does not dodge the concern.
Scored checklist - what a correct answer contains
Score each scenario out of 100; the learner must average at least 80 across both with no critical error.
- Separates coverage from accuracy and improvement (20) - explicitly says coverage is reach, accuracy is calibration agreement, improvement is recurring-error reduction.
- Treats a QA score as a sample, not a census (15) - checks representativeness before generalising.
- Checks calibration before trusting a mixed-reviewer aggregate (15) - identifies stale or low agreement as invalidating.
- Handles disputes correctly (10) - open disputes are provisional, not final; dispute rate is a rubric/calibration signal.
- Positions AI honestly (15) - first-pass under human review, trust controls named, no autonomous-scoring promise.
- Validates the rubric fit (10) - scoring means only what the checklist encodes; needs customer sign-off.
- Separates assigned/completed/improved coaching (10) - completion is not improvement without the next-cycle trend.
- States the honest limitation and escalation boundary (5) - names what to confirm with product and what never to assert.
Manager review confirms the learner would run or say this live, not just write it.
Critical errors that fail the assessment
Any one of these fails the attempt regardless of the numeric score, because each would mislead a customer:
- Presenting coverage as a quality result - e.g. accepting or repeating '98 percent scored, so quality is strong' without the accuracy and improvement caveats.
- Treating an AI-suggested score as the final word - positioning autonomous scoring, or letting a consequential AI score stand without human review.
- Quoting an aggregate as if it were a census - using a figure without its sample, period and calibration basis.
- Counting an open dispute as settled - rolling a provisional score into an aggregate as final.
- Reading a root-cause pattern as proven causality or summing overlapping patterns into an additive savings claim.
- Presenting coaching completion as improvement without the recurring-error trend and a baseline.
- Asserting a production specific (supported sources, AI model behaviour, security, contractual or SLA points) instead of escalating / confirming with product.
The tie to the gate: practical score at least 80, zero critical errors, and manager review - the same gate as every certification module.
Interpretation check
Passing means reading every QA figure for what it actually evidences - coverage is not accuracy, an aggregate is not a census, an AI score is not a verdict, completion is not improvement - and positioning the loop honestly. Any reading that would mislead a customer is a critical error that fails the attempt.
In the field
Scenario A mirrors a KnackGlobal-style RCM review where a healthy headline can sit over a sample that under-weighted compliance-heavy interactions. Scenario B mirrors an OVO-style voice operation where agents will reject QA they see as AI-decided and unfair, so honest AI positioning and a working dispute path are the whole game.
Practical task
Work both scenarios in writing against the scored checklist, then have a manager review your Scenario B response delivered live on the demo. Self-mark each scenario out of 100, flag any critical error honestly, and for any item below full marks write the one correction that would close the gap. You pass at an 80 average across both scenarios with zero critical errors and manager sign-off.
Self-check
Assessment question 1: In Scenario A, the QA lead presents a 93 team average, 98 percent scored. State the three things wrong with trusting it and what you would say.
Model answer: (1) the sample is unrepresentative (one shift, day channel), so it does not describe the operation; (2) calibration is stale and reviewers disagree by eleven points on the same call, so the aggregate mixes incompatible judgements; (3) open disputes are counted as settled, overstating certainty. Say: the 93 is not yet trustworthy - not wrong, unproven - and I would validate the sample, re-calibrate, exclude provisional disputes and re-state the figure with its basis before it is used. Coverage (98 percent) is reach, not quality.
Assessment question 2: In Scenario B, respond to 'we won't let AI decide our agents' scores' in two sentences, then name the screens you show.
Model answer: 'I agree with the instinct - so Ask Quality AI does the first pass and the coverage, while calibration, four-eye checks and disputes keep a human behind any score that carries a consequence. It is human-in-the-loop by design, not autonomous scoring.' Then show Disputes (fair challenge path), Calibration (AI-versus-human agreement as the trust metric) and Scenarios & Checklists (the shared rubric), and volunteer that coverage is not accuracy.
Assessment question 3: A customer sums the top three root-cause patterns for total savings. Pass or fail that reasoning, with the correction.
Model answer: fail it. Overlapping patterns share interactions so summing double-counts, and each pattern is a hypothesis of co-occurrence, not proven causality. Correct: treat each as a candidate to validate against evidence before coaching or quoting any impact - never an additive total.
Assessment question 4: Which metric do you baseline first to prove the programme works, and why not coverage?
Model answer: calibration agreement - it is the trust metric that makes scores comparable; without it every other number is a blend of inconsistent judgements. Coverage only counts how many interactions were touched and proves nothing about accuracy or improvement.
Assessment question 5: Name three statements that must be escalated or marked 'confirm with product' rather than asserted.
Model answer: which recording/transcription sources are supported and their coverage; how the AI scoring model behaves and its limits; and any security, data-residency, contractual or SLA specific. Asserting any of these is a critical error; they are entitlement-, configuration- or commercially-owned.
Verified vs demo. Scenarios use seeded demo data and the live screens at quality.demo.timechamp.io. The assessment tests reading and positioning, not production specifics - any question about supported sources, AI model behaviour, security or contractual terms is answered 'confirm with product', and asserting them is itself a critical error.