SLA metrics that matter for a QA testing vendor

Start from what the vendor controls

A service level only works if the vendor can meet it through its own effort. US federal rules on performance-based acquisition ask buyers to describe the required results, set measurable performance standards, and state how performance will be assessed. That discipline is useful for any buyer: if you cannot say how a number will be measured, or the vendor cannot move it, it is not a service level.

EU financial entities under the Digital Operational Resilience Act (DORA) have a specific duty here. Contracts for ICT services that support critical or important functions must include full service level descriptions with precise quantitative and qualitative performance targets. Even if DORA does not apply to you, it is a reasonable standard to aim for.

Metrics the vendor can control

These are good candidates for service levels with consequences. Use the contract clause builder to turn them into a schedule.

Service level	Definition to agree	Data source
Regression turnaround	Time from release candidate available to regression results reported	CI pipeline timestamps, report timestamp
Blocking defect notice	Time from finding a blocking defect to notifying your named contact	Issue tracker created time, chat or email record
Report delivery	Status and completion reports delivered on the agreed day with agreed contents	Report dates, checklist of required sections
Defect report quality	Share of defects returned by your developers as not reproducible or incomplete	Issue tracker status history
Staffing continuity	Unplanned changes to named team members per period	Team register, timesheets
Suite stability	Share of automated test failures caused by the tests rather than the product, and time to fix a flaky test	CI results, flaky test log
Response to requests	Time to acknowledge and estimate a change request	Change request log

Fill in targets with your own numbers in brackets during negotiation. The targets depend on your release cadence, team size and the model you are buying, so do not copy them from another contract.

Metrics that look good on paper

Some measures appear often in proposals but make poor service levels.

Number of test cases executed or defects found. These reward volume. A vendor can raise both without reducing your risk.

Automation coverage percentage. Meaningless without a stated base: which features, which test levels, which environments. Even with a base, a high percentage of brittle tests is worse than a smaller stable suite.

Defect leakage or escaped defects. Useful to track, but the result depends on what your team built, what was in scope, and what you chose to release with known issues. Credits attached to it lead to arguments about causes.

Pass rate. A high pass rate can mean a good product or weak tests. It says nothing about the vendor's performance on its own.

Delivery performance metrics. The DORA research programme's measures of software delivery performance, such as change failure rate, describe your whole delivery system. They are good shared indicators for a service review, not measures of the testing vendor alone.

What each service level needs

Every service level in the contract should state:

1. what is measured, in one sentence;
2. the target;
3. the data source, preferably your own systems;
4. the measurement period, such as monthly;
5. who reports it, and when;
6. exclusions, such as delays caused by your environments being unavailable;
7. the consequence when it is missed.

Weak and stronger wording side by side:

Weak: "Supplier will provide timely regression testing for each release."

Stronger: "Supplier will report regression results for [the agreed suite] within [X] working hours of a release candidate being available in the staging environment, measured from the pipeline timestamp. Target: met for [Y] of releases in each calendar month. Releases where the staging environment was unavailable for more than [Z] hours are excluded."

Remedies that give service levels weight

Without a consequence, a missed service level becomes a discussion. The UK Model Services Contract ties service levels to service credits and to a termination right for serious or repeated failure. For a testing contract, agree:

• a service credit per missed service level, deducted from the next invoice;
• a statement that credits are a price adjustment and do not limit other rights;
• a remediation plan after a missed period;
• a right to terminate for cause after repeated misses in a set window.

Make the data yours

Measure from your systems where you can: your pipeline, your issue tracker, your repository. Ask for read access to the vendor's test management data at all times. See how to read a vendor's sample report for what the reports behind the numbers should contain, and put your service levels into the RFP so every vendor prices against the same targets. The staffing continuity measure links directly to how QA vendors staff projects.