How to weight a vendor scorecard for QA selection

How weighted scoring works

Each criterion gets a score for each vendor and a weight that is the same for every vendor. Multiply score by weight, add up the results, and you have a weighted total. The arithmetic is trivial. The hard parts are choosing weights that reflect your real risks and making sure your evaluators mean the same thing when they give a score.

Public procurement rules put a lot of care into this step, and the reasons carry over to a private selection. US federal rules require the solicitation to state all factors that will affect the award and their relative importance, and to say whether the non-price factors together are significantly more important than price, approximately equal to it, or significantly less important. EU and UK public procurement law similarly expects award criteria and their weighting, or at least their order of importance, to be set out in advance. The point is to fix the rules before you see the answers.

Start from what would make the engagement fail

A useful way to set weights is to write down how this particular engagement fails. For example:

• "The testers who start are junior and learn on our product."
• "The vendor builds automation we cannot run without them."
• "We only find out a release is risky after it ships."
• "We cannot leave without losing a year of test assets."

Each failure mode points at criteria: named team and seniority, code ownership and maintainability, reporting that shows risk, exit plan and knowledge transfer. Those criteria get the highest weights. Criteria that would be nice but would not sink the engagement get middle weights. Criteria that do not apply get zero.

The scorecard starts from weights set for your situation, such as a startup's first vendor or the replacement of an underperforming vendor, and lets you change each one. The situation weights are a starting point, not a verdict.

Keep price from dominating

Price has a way of taking over late in a selection. Two habits prevent it.

First, decide the price position explicitly, in the same terms public buyers use: is price more, about as, or less important than everything else combined? Write the answer down and size the commercial criteria to match.

Second, score commercials on transparency and total cost rather than headline rate. A vendor whose quote you can rebuild from roles, seniority and hours is easier to hold to account than one with a lower single figure. Put each quote into the cost model with your own numbers before you score it.

Price position	What it means in practice
Price more important than the rest combined	Suits well-defined, repeatable work where quality differences between vendors are small
Price about as important	Suits most QA engagements with a mix of manual and automation work
Price less important	Suits regulated products, automation build-outs and vendor replacements where failure is expensive

Write anchors for every score

A 1 to 5 scale works when each number has a written meaning. For the named team criterion, the framework uses anchors like these:

• 1: generic roles or "resources" with no names or CVs.
• 3: a named lead, the rest described by role and level.
• 5: every team member named with CV, allocation and a commitment that they start on day one.

Without anchors, one evaluator's 4 is another's 2, and the weighted total measures who was on the panel rather than which vendor is stronger.

Calibrate your evaluators

Before scoring for real, pick one proposal and have every evaluator score it alone. Then compare. Where scores differ by more than a point, discuss the evidence each person used and agree what the anchor means. Record the agreement in a line next to the criterion so that later scoring follows it.

Calibration also surfaces missing evidence. If evaluators disagree because the proposal does not say who the testers are, the fix is a clarification question to the vendor, not an average.

Sense check the result

When the totals are in, ask two questions. Does the vendor that wins have strong evidence on the criteria you marked as failure modes? Would you hire this vendor if you had never seen the numbers? If either answer is no, look for a weight that does not reflect your real priorities, or a score based on a claim rather than evidence. Changing weights after scoring should be rare and recorded with a reason.

A narrow gap between two vendors is a tie. Resolve it with a paid pilot, reference calls on the references tool, or due diligence, not with a decimal point. For the criteria themselves, see QA vendor evaluation criteria.