Defect leakage as a vendor metric: define it fairly

What defect leakage measures

Defect leakage asks a simple question: of the defects that existed in what you shipped, how many did testing miss? A common formula is:

Defect leakage = defects found after the test stage ÷ (defects found during the test stage + defects found after it), for the same release or period.

Written as words for a contract: "the number of defects found in production within [the window] after a release, divided by the total number of defects found in testing and production for that release, reported by severity."

The idea is sound. Escaped defects are what your users feel, and a testing vendor exists to reduce them. DORA research tracks a related delivery measure, the change failure rate, which counts how often a deployment causes a failure in production. Defect leakage looks at the same concern from the testing side.

The trouble is that each part of the formula hides a choice. Unless you settle those choices in writing, the vendor and you will count differently and argue about the result.

The choices you have to write down

Choice	Options	What to decide
What counts as a defect	Any ticket, or only confirmed defects	Exclude duplicates, feature requests, "works as designed"
What counts as leaked	Found in production, or in any later stage (UAT, staging)	Name the stage boundary
Time window	Days or weeks after release	Long enough to catch real issues, short enough to link to a release
Severity	Your scale or the vendor's	One scale, with examples, owned by one person
Scope	All defects, or only those in areas the vendor tested	Exclude areas you removed from scope or did not give environments for
Root cause	Testing miss, requirement gap, environment difference, data issue	Classify each escaped critical defect
Data source	Your issue tracker, their tool	Use your tracker as the single source

ISO/IEC/IEEE 29119-3 sets out what incident and completion reports should record. Using its structure for defect reports makes counting easier because every report carries the same fields.

Making it fair to the vendor

A metric that punishes the vendor for things it did not control will produce defensive behaviour: more low-value defects logged, more time spent disputing counts, less candour about risk. To keep the metric fair:

• Exclude what was out of scope. If a feature was not in the test plan, or you did not provide the environment or data needed to test it, a production defect there is not a testing escape.
• Separate requirement gaps. If the software did what the requirement said and the requirement was wrong, that is a different problem. Track it, but not against the vendor.
• Account for environment differences. Defects that only occur in production configuration you did not replicate for testing should be classified separately.
• Use your own tracker. One data source, visible to both sides, removes arguments about the numbers.

Making it useful to you

A single leakage ratio can look healthy while serious defects slip through. Make it useful by reporting:

• The count of escaped critical and major defects per release, alongside the ratio.
• The trend over several releases, not a single month.
• A short root-cause note for each escaped critical defect: why it was missed and what changes.
• Leakage by product area, which shows where test coverage is thin.

Pair leakage with a measure of effort so you can see trade-offs. A vendor can reduce leakage by testing much longer before each release. If releases slow down, that is a cost too. The SLA metrics guide lists turnaround measures to pair with leakage.

Leakage in the contract

Performance-based contracting guidance such as FAR Subpart 37.6 asks for results that are measurable, with a stated method of assessing performance against them and, where appropriate, incentives tied to them. For defect leakage that means:

1. The definition and formula, in a schedule to the contract.
2. The data source and the reporting cycle.
3. A baseline period with no credits, for example your first quarter together.
4. A target set from the baseline, by severity.
5. A remedy for repeated misses, usually a remediation plan first and service credits after.

The contract clause builder has service level and service credit clauses you can adapt. Leave targets blank until you have baseline data.

Leakage when comparing vendors

During selection, leakage numbers from vendors' other clients are hard to compare, because each client counts differently. Ask instead how they define and count it, and ask for an anonymised report that shows the definition in use. Score the answer in the scorecard under measurable service levels. The KPIs for a QA vendor guide covers the other measures to pair with leakage. The selection path shows when to settle metric definitions, which is before the contract, not after.