QA vendor evaluation criteria and the evidence for each

Updated 2026-09-24

Evaluate a QA vendor on eight groups: team and staffing, process and methodology, automation and code ownership, tooling, security and data, reporting and communication, commercials, and exit and continuity. Score each criterion only on evidence you can see or verify, such as named CVs, a sample test plan, a real status report or a certificate with its scope. Buyers tend to overweight polish and price and underweight code ownership, staff continuity and exit terms.

What to check

  1. Write the criteria and their weights down before the first vendor presentation, and share the criteria in the RFP.
  2. For every criterion, name the evidence that earns a high score, such as a CV, an anonymised test plan or a certificate number.
  3. Ask each vendor the same questions in the same order so that answers can be compared line by line.
  4. Score a criterion low when the vendor offers a claim without evidence, however confident the claim sounds.
  5. Check that code ownership, staff continuity and exit terms each carry real weight, not a token score.
  6. Keep a note of the evidence behind every score so the decision can be explained later.

Ask the vendor

  • Who, by name and role, will work on our account in the first three months, and can we see their CVs?
  • Please share an anonymised test strategy or test plan you have written for another client.
  • Where will the test automation code live, and will it be in a repository we own from the first commit?
  • Can you show us a real, anonymised weekly status report from a current engagement?
  • What exactly will you hand over when the contract ends?

What evaluation criteria are for

Evaluation criteria turn a set of proposals that all sound competent into something you can compare. They work only when three things are true: you set them before you read the proposals, you define what evidence earns a high score, and you ask every vendor the same questions. US federal source selection rules make the same point for public buyers: the factors that will affect the award, and their relative importance, are stated in the solicitation. You are not bound by those rules, but the discipline protects you from choosing the most persuasive presenter.

The framework behind the scorecard groups criteria into eight areas. Each area below lists what it covers and the evidence that should move a score.

The eight criteria groups

Group What you are judging Evidence that earns a high score
Team and staffing Who does the work, how senior they are, what happens when they leave Named CVs, allocation per person, replacement notice and overlap terms
Process and methodology How the vendor decides what to test and documents it An anonymised test strategy or plan, entry and exit criteria, a risk assessment walkthrough
Automation and code ownership Who owns the test code and whether you can maintain it Code in your repository, an open framework, written assignment of rights
Tooling Whether the work runs inside your tools and pipeline Tests running in your CI, test management in your accounts, open export formats
Security and data Security posture and handling of test and personal data Certificate with scope, SOC 2 Type II report, data processing terms, masked data approach
Reporting and communication What you see each week and how problems surface A real anonymised status report, defined metrics, overlap hours
Commercials How you pay and what happens when service levels are missed Price build-up by role, rate lock terms, service levels with remedies
Exit and continuity How you leave and who you contract with A draft exit plan, knowledge transfer terms, a verifiable contracting entity

Team and staffing

The people in the proposal are not always the people who start. Score this group on names, CVs and a willingness to list key people in the contract. "Sample profiles" or "we will assign the right people after signature" should score low. Ask to interview the lead before you sign.

Process and methodology

A test strategy is the clearest window into how a vendor thinks. ISO/IEC/IEEE 29119-3 describes the documents a testing process produces, such as test plans and completion reports. You do not need a vendor to follow the standard word for word, but a strong answer shows risk-based scope: what they would test first on your product, what they would leave out, and why.

Automation and code ownership

This group decides whether you can walk away. Look for a clear statement that code lives in your repository from the first commit, uses an open framework your engineers can read, and is assigned to you in writing. See who owns the test automation code for contract wording.

Tooling

The vendor's work should run in your pipeline, your issue tracker and your test management accounts. Tools licensed by the vendor and billed inside the rate create quiet lock-in.

Security and data

Score certificates on their scope, not their existence. A certificate that covers another entity or another site does not cover your work. Check how test data is handled; a vendor that asks for production copies without discussing masking should score low.

Reporting and communication

Ask for a real weekly report from a current engagement, anonymised, with nothing added for the sales process. Strong reports show risk and trend; weak ones list test case counts.

Commercials

Score transparency rather than the lowest rate. A vendor that shows roles, seniority mix, management overhead and tooling in its price lets you check the quote against your own estimate on the cost model.

Exit and continuity

Exit terms are cheapest to agree before signature. Look for a draft exit plan early in the contract, a knowledge transfer period, and a contracting entity you can find in a public company register.

Criteria buyers overweight

A few signals get more weight than they deserve in most evaluations:

  • Presentation quality. A polished deck shows the sales team, not the delivery team.
  • Day rate alone. A low rate with a junior mix or pooled staff can cost more once rework and your own management time are counted.
  • Tester headcount. More people is not more coverage if nobody owns the strategy.
  • Certificates of individual testers. Qualifications show training, not seniority on products like yours.
  • Automation percentages. A promised percentage means little without a defined scope.

Criteria buyers underweight

The criteria that decide whether an engagement can be fixed or ended tend to get token weight: staff continuity, code ownership, knowledge transfer and exit. ISO 37500 treats outsourcing as a life cycle you govern from strategy through transition and delivery, not a single purchase, and exit belongs in that plan from the start. If one of these would make the engagement fail when it goes wrong, give it weight accordingly. How to weight a vendor scorecard covers how to set those weights.

Turning criteria into questions

Every criterion needs at least one question that forces evidence. The questionnaire generates a question set from the framework, with the reason each question matters and what a strong and a weak answer look like. Send it with the RFP or use it in the vendor interview, then score the answers on the scorecard. The selection path shows where each step sits in the process.

Tools for this

Questions buyers ask

How many criteria should a QA vendor evaluation use?
Enough to cover all eight groups without duplicating each other. The framework on this site has 25 criteria; most buyers switch off the ones that do not apply, such as working under the buyer's name, and keep the rest.
Should price be one of the criteria?
Yes, but evaluate it as pricing transparency and total cost, not as the lowest day rate. A clear build-up of roles and seniority tells you more than a single number, and the cost model on /cost/ lets you compare quotes on your own figures.
Can we add criteria specific to our product?
Yes. Add a domain criterion, such as experience with payment flows or medical device software, and define what a strong and a weak answer look like before you read proposals.

Sources

  1. FAR Part 15: Contracting by Negotiation (source selection), US Federal Acquisition Regulation
  2. ISO 37500:2014 Guidance on outsourcing, ISO
  3. The Sourcing and Consultancy Playbooks, UK Cabinet Office
  4. ISO/IEC/IEEE 29119-3:2021 Software testing, Part 3: Test documentation, ISO/IEC/IEEE
  5. Model Services Contract, UK Cabinet Office