Quality and Call QA data and setup - what the app needs to produce scores you can trust
Objective. Explain, at app-logic level, the data, configuration and dependencies Quality and Call QA relies on, and give a CSM a concrete validation list - what to check before trusting a number and what to escalate - without inventing integration or UI specifics.
A QA score is only as good as four things beneath it: the interactions and evidence it was scored from, the rubric it was scored against, the sample it was drawn from, and the reviewer consistency behind it. This reference describes those inputs in product-logic terms, names the dependencies on other apps, and sets out what a CSM validates before a figure is trusted. Specifics of recording and transcription sources, AI scoring models and dispute workflows are entitlement- and configuration-dependent - confirm with product, never assert.
See it in the demo
The data and signals it depends on
Four input layers decide whether a score is trustworthy:
- The interactions and their evidence. Calls, and the recording or transcript and metadata behind each, are what a reviewer actually scores. The available evidence per interaction sets the ceiling on how defensible a score can be - and on what Ask Quality AI can read for a first pass.
- The rubric (Scenarios & Checklists). The definition of 'good' that every score inherits. A score means only what the checklist encodes; if the rubric is wrong or ambiguous, correct data still produces the wrong answer.
- The sample. Which interactions were selected for review, by what rule, and at what coverage. A score is a sample, not a census, so representativeness of the sample is an input to trust, not an afterthought.
- Reviewer consistency. Who scored, and whether reviewers (and AI versus human) agree on the same call. Calibration is the signal that scores from different reviewers mean the same thing.
Confirm with product which recording, transcription and metadata sources are supported and entitled for a given customer - the demo shows the loop on seeded data and does not establish production source coverage.
Setup and configuration required
Described at app-logic level (specifics are 'confirm with product'):
- Rubric configuration - the scenarios, checklist items, any weightings and auto-fail rules that encode the customer's definition of quality. This needs the customer's explicit sign-off, not a demo default.
- Sampling rules - volume, basis (random, risk-weighted, agent-balanced) and resulting coverage, set so the sample represents all agents, shifts, channels and work types.
- Reviewer roles and calibration - who reviews, the calibration cadence, and how AI-versus-human comparison is handled.
- Four-eye (maker-checker) routing - which items (high-stakes, auto-fail, disputed) go to a second reviewer, and the thresholds.
- Dispute workflow - the challenge path, roles and resolution SLAs.
- Coaching workflow - how findings become assigned, tracked and completed actions.
- Ask Quality AI - which scoring models and first-pass assistance are enabled, always configured to keep a human behind any consequential score.
Which of these are configurable, and with what options, is entitlement-dependent. Never present a demo configuration as the customer's agreed setup.
Dependencies on other apps and sources
Quality and Call QA sits in the Business Outcomes suite and connects outward:
- Recording, telephony and transcription sources feed the interactions and evidence. The demo does not establish which are supported in production - confirm with product.
- Employee Monitoring / RCM capture is where capture policy, agent coverage and classifications live for monitored-work contexts; the capture boundaries set there govern what evidence is available and permissible.
- RCM Operations is the outcome context - quality risk in unreviewed interactions (compliance, documentation) is where QA findings connect to revenue-cycle exposure.
- RCM Insights can question across the quality data, inheriting QA's sample, rubric and calibration caveats.
A QA figure therefore carries the trust state of its upstream sources. If recording coverage is partial, or capture policy excludes certain interactions, the sample is bounded before sampling rules are even applied - a point to surface early, not discover in a QBR.
What a CSM validates, and what to escalate
Validate before trusting any aggregate:
- Rubric fit and sign-off - do the Scenarios & Checklists items match how this customer defines quality, and are ambiguous items fixed.
- Sample representativeness - does the sampling rule fairly cover all agents, shifts, channels and work types, or over-weight easy or recent interactions.
- Calibration agreement - do reviewers (and AI versus human) agree closely on shared calls, and is calibration recurring rather than one-off.
- Dispute path works - are challenges raised and resolved rather than festering.
- Evidence behind a few scores - open individual interactions and confirm the score is defensible against the rubric, not just a number in a tile.
- Coaching closure - do findings turn into completed coaching that moves recurring errors.
Escalate to the implementation specialist / product: supported recording and transcription sources and their coverage; AI scoring model behaviour and limits; how calibration thresholds, four-eye routing and dispute SLAs are configured; any capture-policy or data-residency question. Never guess on source coverage, AI model specifics, security or contractual points - these always escalate.
Interpretation check
A QA score inherits the trust state of four inputs - interactions and evidence, the rubric, the sample and reviewer calibration. Correct data scored on a wrong rubric, or a biased sample scored consistently, still produces a number that does not describe the operation.
In the field
KnackGlobal (RCM): the validation that matters most is that the checklist encodes the documentation and compliance points carrying exposure, and that capture policy has not quietly excluded the interaction types where that risk lives. OVO (contact centre): recording and transcription coverage across a high voice volume sets the ceiling on how representative any sample can be.
Practical task
For one account, write a one-page data-and-setup validation note: list the four input layers and the current state of each, name one upstream source dependency that bounds the sample, and mark three items you will escalate to the implementation specialist. Finish with the single sentence you would use to caveat the customer's current headline QA number given what you found.
Self-check
Name the four input layers that determine whether a QA score is trustworthy.
The interactions and their evidence; the rubric (Scenarios & Checklists); the sample and its representativeness; and reviewer consistency (calibration). A weakness in any one undermines the score no matter how strong the others look.
Recording coverage is only partial for one channel. Why does that matter before sampling rules are even discussed?
Because the sample is bounded by what was captured. If a channel is not recorded, no sampling rule can represent it - the gap caps representativeness upstream, and should be surfaced early, not found in a QBR.
A CSM is asked which transcription sources are supported in production. What is the right move?
Escalate to the implementation specialist or product. Supported recording and transcription sources, their coverage and AI model behaviour are entitlement- and configuration-dependent - the demo does not establish them and they must never be guessed.
Which two validations most directly protect against a confident-but-wrong aggregate?
Rubric fit and sign-off (is 'good' defined correctly) and calibration agreement (is it applied consistently). Sampling protects representativeness, but these two decide whether the score means what the customer thinks.
Why inspect the evidence behind a few individual scores rather than only the aggregate?
Because an aggregate hides whether any single score is defensible against the rubric. Opening interactions confirms the number rests on real, rubric-consistent evidence rather than tile-level activity.
Verified vs demo. The rubric, sampling, calibration, four-eye, dispute, coaching and Ask Quality AI mechanisms are demo-shown on seeded data. Supported recording/transcription/telephony sources and their coverage, AI scoring model specifics, calibration thresholds, four-eye routing, dispute SLAs and capture-policy or data-residency details are entitlement- and configuration-dependent - confirm with product.