KPIs for an outsourced QA vendor you can verify yourself

What a QA KPI should do

A KPI in a vendor contract does two jobs. It tells the vendor what you value, and it gives you an early signal when the service is slipping. It does both only if it measures an outcome you care about and you can check it without relying on the vendor's own numbers.

US federal performance-based acquisition rules describe the approach in general terms: state the work as required results, set measurable performance standards, and use a plan to check them. For a QA contract, that means a short list of measures, each with a written definition, calculated from data in your tools.

A short KPI set you can verify

KPI	What it tells you	How to verify it
Escaped defects in tested areas	Whether testing catches what matters before release	Production defects in your tracker, tagged by feature and linked to the test scope for that release
Turnaround to test result	Whether testing keeps pace with delivery	Time from release candidate in your pipeline to the vendor's result, from pipeline and tracker timestamps
Critical defect triage time	Whether blocking problems get attention quickly	Time from a critical defect being raised to first triage, from tracker history
Defect report quality	Whether developers can act on reports without asking	A monthly sample of reports scored against your format: steps, expected and actual result, build, evidence
Flaky test rate	Whether the automation suite can be trusted	Tests that pass and fail on the same code, from CI history
Report delivered on schedule	Whether you get risk information when you need it	Dates of reports received against the agreed schedule

Pick the ones that match your engagement. A manual exploratory engagement may not need the flaky test rate; an automation build-out needs it most.

Escaped defects, defined carefully

Escaped defects, sometimes called defect leakage, is the most direct measure of whether testing works, and the easiest to argue about. Define in advance which production defects count: only those in features the vendor was asked to test, above an agreed severity, found within an agreed period after release. Exclude defects in areas that were explicitly out of scope. See defect leakage as a vendor metric for the details.

Flaky tests

A flaky test passes and fails on the same code. A post by Google's testing team describes how flaky tests cost developers time and blur the signal a test suite gives, and how flaky tests are identified and marked so that they can be told apart from real failures. Track the flaky rate from your CI history and agree how quickly a flaky test is quarantined and fixed. A vendor that handles flakiness by rerunning until green hides real failures.

Vanity metrics to keep out of the contract

Some numbers look like performance but reward the wrong behaviour.

• Test cases written or executed. Encourages many small, low-value cases and repeated execution of the same checks.
• Raw defect counts. Depends on code quality, rewards splitting issues and logging trivia, and penalises the vendor when your developers improve.
• Automation percentage. Says nothing about which tests are automated or whether they are reliable.
• Hours logged. Measures effort, not results.
• Pass rate. Can be raised by removing hard tests.

These can still be useful as context in a weekly report. They should not carry targets or service credits. The framework includes a red flag for proposals that use defect counts as the main success measure.

Writing KPIs into the contract

The framework's service level clause asks that each measure state what is measured, the target, the data source, the measurement period and who reports it. For example:

Critical defects triaged within [4] business hours of being raised, measured from [Customer]'s defect tracker, reported monthly by [Supplier] and verified by [Customer].

Bracketed values are placeholders to agree with your vendor. Start with a baseline period where you measure without penalties, then set targets from what you observed. Link persistent misses to the service credit and remediation clause rather than to termination on the first miss. The contract clause builder produces both clauses.

Connecting to delivery performance

QA KPIs sit inside your wider delivery measures. The DORA research programme tracks software delivery performance through measures such as change failure rate and deployment frequency. If your change failure rate rises while the vendor's KPIs stay green, the KPIs are measuring the wrong things. Review both together in the service review; managing an outsourced QA team covers how to run it.

Using KPIs during selection

Before you sign, ask each vendor which outcomes it would agree to be held to, and how it defines its metrics. Vendors that answer with activity counts, or will not commit to any measure, score lower on measurable service levels in the scorecard. For how targets and remedies fit together, see SLA metrics that matter.