Running a quarterly business review with a QA vendor

What a QBR is for

A quarterly business review is where you step back from daily test work and ask whether the service is doing what you contracted for. Stand-ups and sprint reviews deal with this week's releases. The QBR deals with trends: are service levels holding, is the team stable, are risks growing, and does the scope still match your needs?

ISO 37500, the guidance standard on outsourcing, treats governance as a continuing part of the arrangement, with regular reviews of performance and of the relationship. The UK Model Services Contract and the Sourcing Playbook both expect performance to be monitored against defined service levels, with regular review meetings. The US Government Accountability Office report on HealthCare.gov shows what happens when contract oversight is weak: the agency did not act in time on contractor performance problems, and costs and delays grew.

A QBR that is only a vendor presentation does not do this job. You need data you trust, a fixed agenda, and written decisions.

Before the meeting

Agree in the contract what the vendor sends and when. A useful data pack contains:

• Results against each service level for the quarter, with the underlying data.
• Escaped defects by severity and root cause, counted using the agreed definition. See defect leakage as a vendor metric.
• Automation health: suite size, run time in your pipeline, flaky tests found and fixed.
• Staffing: joiners, leavers, replacements, and the seniority mix against the contract.
• Hours or charges for the quarter against budget, and any change requests.
• Risks the vendor sees in your product, process or environments.

Where you can, take the numbers from your own issue tracker and CI system rather than only from the vendor's slides. Ask for the pack a few working days before the meeting so you can read it first.

A standard agenda

Use the same agenda each quarter so trends are easy to compare.

Item	Led by	Questions to settle
Actions from last QBR	You	Which were done? Which were not, and why?
Service levels	Vendor, checked by you	Which targets were missed? What was the cause? What changes?
Escaped defects	Both	Which critical defects reached production? Was each a testing miss?
Staffing and continuity	Vendor	Who changed? Were key personnel terms followed? How was knowledge handed over?
Risk log	Both	Which risks are open, growing or closed? Who owns each?
Cost and value	You	Did spend match budget? Is the value in line with the cost?
Next quarter scope	Both	What changes in releases, platforms or priorities? What team size fits?
Improvements	Both	What should each side change?

Leave the vendor room to raise problems on your side too. Slow environment fixes, unclear requirements or late builds all affect their results. A review that only looks at the vendor misses half the causes.

Scoring the quarter

Some buyers rescore the vendor each quarter against the same criteria used at selection. That makes drift visible: a vendor that scored well on staff continuity during selection but lost key people this quarter should score lower now. The scorecard can be reused for this, with a subset of criteria that apply to delivery rather than selection.

Keep the scores and the notes together. Over several quarters they form the record you need for renewal or exit decisions.

Common ways a QBR goes wrong

• The vendor runs the meeting. A polished deck replaces the data pack, and nobody checks the numbers against your own tools.
• No actions carry over. The same issues are raised each quarter with no owner and no date.
• Only the vendor is reviewed. Causes on your side, such as late builds or unstable environments, stay unfixed.
• Scope drifts silently. The team is doing different work from the contract, and nobody updates the statement of work or the price.
• Staffing changes are discovered late. Key people leave or are replaced without the notice the contract requires.

After the meeting

Send written decisions within a few days: what was agreed, who owns it, and by when. If a service level was missed repeatedly, follow the remedy in the contract, such as a remediation plan or service credits. The contract clause builder has service level and service credit clauses that make this step routine rather than a negotiation.

If the same problems return quarter after quarter, the QBR record is also the evidence you need to decide whether to renew, renegotiate or plan an exit. See managing an outsourced QA team for the day-to-day side, and the selection path for where reviews fit after signature.