How should we set the weights?

Start from the preset for your situation, then agree changes with everyone who will score before any proposal arrives. Changing weights after reading proposals lets preference creep into the numbers.

Each evaluator scores alone first, using the scoring guide, then the group discusses differences larger than one point. Record the reason for each final score.

What does the CSV contain?

One row per criterion with its weight and each vendor's score, and formulas that compute weighted totals and the percentage of the maximum. It opens in any spreadsheet.

Is price a criterion?

Price is compared separately as total cost over 12 and 24 months with the cost model, so a low rate does not hide weak answers. Framework v1.0 scores pricing transparency, not price.

QA vendor scoring template

Weighted criteria by situation, editable in the browser, exported as CSV with formulas.

What this does

Every shortlisted vendor is scored 0 to 5 on the same 25 criteria, grouped into 8 themes. Each criterion has a weight from 0 to 5 that you set, starting from a preset for your situation. The table shows weighted totals as you type.

The scoring guide describes what a 1, 3 and 5 look like for every criterion, so different evaluators score the same answer the same way. Export the result as CSV with formulas, or as Markdown for the decision record.

See a sample output

QA vendor scorecard

ChooseQAVendor framework v1.0. Situation: Replacing an underperforming vendor. Nothing you enter in the toolkit is sent anywhere.

Score each vendor from 0 to 5 on each criterion using the scoring guide. The weighted total is the sum of weight times score; the percentage is that total divided by the maximum possible.

Criterion	Weight	Vendor A	Vendor B	Vendor C
Named team and seniority mix	3	4	3	5
Staff continuity and replacement terms	5	3	4	2
Relevant past performance	3	5	2	3
Capacity and speed to start	3	2	5	3
Test strategy and documentation	3	3	4	4
Risk-based scope	3	4	3	5
Onboarding plan and turnaround	4	3	4	2
Ownership of test code and assets	5	5	2	3
Open framework choice	3	2	5	3
Maintainability and flaky test control	3	3	4	4
CI pipeline integration	3	4	3	5
Works in your tools	3	3	4	2
Security posture and certifications	3	5	2	3
Test data and personal data handling	3	2	5	3
Subcontracting transparency	3	3	4	4
Reporting that shows risk	5	4	3	5
Communication and working-hours overlap	3	3	4	2
Pricing transparency	3	2	5	3
Measurable service levels	3	3	4	4
Insurance and liability	3	4	3	5
Audit and information rights	3	3	4	2
Exit plan and assistance	4	5	2	3
Knowledge transfer	5	2	5	3
Contracting entity and jurisdiction	3	3	4	4
Weighted total	max 410	276 (67.3%)	298 (72.7%)	277 (67.6%)

Scoring guide

Criterion	1	3	5
Named team and seniority mix	Generic roles or "resources" with no names or CVs.	Named lead, the rest described by role and level.	Every team member named with CV, allocation and a commitment that they start on day one.
Staff continuity and replacement terms	No commitment; replacements at the vendor's discretion.	Notice period for changes, overlap not specified.	Written notice period, handover overlap at the vendor's cost, and your right to approve replacements.
Relevant past performance	Claims of experience with no examples or references.	Comparable examples, references offered but not in your domain.	Comparable engagements in your domain with references who will speak to them.
Capacity and speed to start	Start date and ramp-up undefined.	Start date committed, scaling terms vague.	Named start date, written ramp-up plan and scaling notice periods in both directions.
Test strategy and documentation	No sample documentation; process described in slogans.	Sample documents exist but are generic.	Sample strategy and completion report that cover scope, risks, approach, environments, entry and exit criteria.
Risk-based scope	Promises to test everything.	Mentions risk but scope is a feature list.	Scope derived from a written risk assessment with explicit exclusions you can challenge.
Onboarding plan and turnaround	No onboarding plan; turnaround unstated.	Outline plan, turnaround left open.	Dated onboarding plan with milestones and a committed turnaround per build.
Ownership of test code and assets	Vendor keeps code in its own repository or licenses it to you.	Assigned to you on payment or on exit.	Written assignment of rights, code in your repository from the first commit.
Open framework choice	Proprietary framework or accelerator required.	Open framework with a vendor-specific wrapper.	Open framework in a language your team uses, chosen with your engineers.
Maintainability and flaky test control	Success measured by number of tests written.	Flaky tests fixed when noticed.	Flake rate tracked and reported, quarantine policy, code review on all test code.
CI pipeline integration	Tests run on the vendor's machines and results are emailed.	Tests run in CI on a schedule.	Tests run in your pipeline on every change with results visible to your engineers.
Works in your tools	Requires its own portal or tools to see results.	Uses your tools with duplicate reporting in theirs.	Works fully in your tools; nothing to license from the vendor.
Security posture and certifications	Certifications claimed without certificate, scope or report.	Valid certificate, scope does not clearly cover the delivery team.	Certificate verified with the issuing body or a current SOC 2 Type II report covering the delivery team, with exceptions explained.
Test data and personal data handling	Expects copies of production data; no processing terms.	Processing agreement offered, data locations vague.	Synthetic or masked data by default, listed processing locations and sub-processors, Article 28 terms.
Subcontracting transparency	Subcontracting not mentioned or reserved without limit.	Disclosed on request.	All subcontractors listed, flow-down of obligations, your prior approval required.
Reporting that shows risk	Reports count test cases executed and bugs logged.	Coverage and defect trends, little on risk.	Sample report states scope covered, gaps, open risks and a clear release recommendation.
Communication and working-hours overlap	Communication through an account manager only; little overlap.	Some overlap, team joins selected meetings.	Committed overlap hours, direct access to the team, named lead with authority.
Pricing transparency	Single lump sum or blended rate with no breakdown.	Rates by role, extras unclear.	Rates by role, included and excluded items, change pricing and rate review terms stated.
Measurable service levels	No service levels; payment for hours only.	Service levels listed without remedies.	Measurable service levels with reporting and remedies such as service credits.
Insurance and liability	No evidence of insurance; liability capped very low.	Insurance evidenced, standard caps.	Certificates of insurance provided, carve-outs for data breach and confidentiality agreed.
Audit and information rights	No audit or information rights.	Information on request, no audit.	Contractual audit and information rights, including for regulators where applicable.
Exit plan and assistance	No exit terms.	Termination terms only.	Written exit plan, assistance period, handover list and fees for exit agreed up front.
Knowledge transfer	Knowledge lives in people's heads.	Documentation on request at exit.	Living documentation in your tools and a defined transfer process with successor shadowing.
Contracting entity and jurisdiction	Contracting entity unclear or differs from the proposal.	Entity known, cross-border data terms missing.	Registered entity verified, governing law agreed, transfer mechanism such as standard contractual clauses in place.

Use 0 when the vendor did not answer, 2 and 4 for answers between the described levels.

Sample files:Sample Markdown 6.9 KB Sample CSV 5.1 KB Blank Markdown 6.8 KB Blank CSV 5.0 KB

Build your scorecard

Weighted scoring table. Weights 0 to 5, scores 0 to 5.
Criterion	Weight (0-5)	Vendor 1 name	Vendor 2 name	Vendor 3 name
Team and staffing
Named team and seniority mix
Staff continuity and replacement terms
Relevant past performance
Capacity and speed to start
Process and methodology
Test strategy and documentation
Risk-based scope
Onboarding plan and turnaround
Automation and code ownership
Ownership of test code and assets
Open framework choice
Maintainability and flaky test control
Tooling
CI pipeline integration
Works in your tools
Security and data
Security posture and certifications
Test data and personal data handling
Subcontracting transparency
Reporting and communication
Reporting that shows risk
Communication and working-hours overlap
Works under your name
Commercials
Pricing transparency
Measurable service levels
Insurance and liability
Audit and information rights
Exit and continuity
Exit plan and assistance
Knowledge transfer
Contracting entity and jurisdiction
Weighted total	max 360	0 (0%)	0 (0%)	0 (0%)

QA vendor scorecard

ChooseQAVendor framework v1.0. Situation: General. Nothing you enter in the toolkit is sent anywhere.

Score each vendor from 0 to 5 on each criterion using the scoring guide. The weighted total is the sum of weight times score; the percentage is that total divided by the maximum possible.

Criterion	Weight	Vendor A	Vendor B	Vendor C
Named team and seniority mix	3
Staff continuity and replacement terms	3
Relevant past performance	3
Capacity and speed to start	3
Test strategy and documentation	3
Risk-based scope	3
Onboarding plan and turnaround	3
Ownership of test code and assets	3
Open framework choice	3
Maintainability and flaky test control	3
CI pipeline integration	3
Works in your tools	3
Security posture and certifications	3
Test data and personal data handling	3
Subcontracting transparency	3
Reporting that shows risk	3
Communication and working-hours overlap	3
Pricing transparency	3
Measurable service levels	3
Insurance and liability	3
Audit and information rights	3
Exit plan and assistance	3
Knowledge transfer	3
Contracting entity and jurisdiction	3
Weighted total	max 360	0 (0%)	0 (0%)	0 (0%)

Scoring guide

Criterion	1	3	5
Named team and seniority mix	Generic roles or "resources" with no names or CVs.	Named lead, the rest described by role and level.	Every team member named with CV, allocation and a commitment that they start on day one.
Staff continuity and replacement terms	No commitment; replacements at the vendor's discretion.	Notice period for changes, overlap not specified.	Written notice period, handover overlap at the vendor's cost, and your right to approve replacements.
Relevant past performance	Claims of experience with no examples or references.	Comparable examples, references offered but not in your domain.	Comparable engagements in your domain with references who will speak to them.
Capacity and speed to start	Start date and ramp-up undefined.	Start date committed, scaling terms vague.	Named start date, written ramp-up plan and scaling notice periods in both directions.
Test strategy and documentation	No sample documentation; process described in slogans.	Sample documents exist but are generic.	Sample strategy and completion report that cover scope, risks, approach, environments, entry and exit criteria.
Risk-based scope	Promises to test everything.	Mentions risk but scope is a feature list.	Scope derived from a written risk assessment with explicit exclusions you can challenge.
Onboarding plan and turnaround	No onboarding plan; turnaround unstated.	Outline plan, turnaround left open.	Dated onboarding plan with milestones and a committed turnaround per build.
Ownership of test code and assets	Vendor keeps code in its own repository or licenses it to you.	Assigned to you on payment or on exit.	Written assignment of rights, code in your repository from the first commit.
Open framework choice	Proprietary framework or accelerator required.	Open framework with a vendor-specific wrapper.	Open framework in a language your team uses, chosen with your engineers.
Maintainability and flaky test control	Success measured by number of tests written.	Flaky tests fixed when noticed.	Flake rate tracked and reported, quarantine policy, code review on all test code.
CI pipeline integration	Tests run on the vendor's machines and results are emailed.	Tests run in CI on a schedule.	Tests run in your pipeline on every change with results visible to your engineers.
Works in your tools	Requires its own portal or tools to see results.	Uses your tools with duplicate reporting in theirs.	Works fully in your tools; nothing to license from the vendor.
Security posture and certifications	Certifications claimed without certificate, scope or report.	Valid certificate, scope does not clearly cover the delivery team.	Certificate verified with the issuing body or a current SOC 2 Type II report covering the delivery team, with exceptions explained.
Test data and personal data handling	Expects copies of production data; no processing terms.	Processing agreement offered, data locations vague.	Synthetic or masked data by default, listed processing locations and sub-processors, Article 28 terms.
Subcontracting transparency	Subcontracting not mentioned or reserved without limit.	Disclosed on request.	All subcontractors listed, flow-down of obligations, your prior approval required.
Reporting that shows risk	Reports count test cases executed and bugs logged.	Coverage and defect trends, little on risk.	Sample report states scope covered, gaps, open risks and a clear release recommendation.
Communication and working-hours overlap	Communication through an account manager only; little overlap.	Some overlap, team joins selected meetings.	Committed overlap hours, direct access to the team, named lead with authority.
Pricing transparency	Single lump sum or blended rate with no breakdown.	Rates by role, extras unclear.	Rates by role, included and excluded items, change pricing and rate review terms stated.
Measurable service levels	No service levels; payment for hours only.	Service levels listed without remedies.	Measurable service levels with reporting and remedies such as service credits.
Insurance and liability	No evidence of insurance; liability capped very low.	Insurance evidenced, standard caps.	Certificates of insurance provided, carve-outs for data breach and confidentiality agreed.
Audit and information rights	No audit or information rights.	Information on request, no audit.	Contractual audit and information rights, including for regulators where applicable.
Exit plan and assistance	No exit terms.	Termination terms only.	Written exit plan, assistance period, handover list and fees for exit agreed up front.
Knowledge transfer	Knowledge lives in people's heads.	Documentation on request at exit.	Living documentation in your tools and a defined transfer process with successor shadowing.
Contracting entity and jurisdiction	Contracting entity unclear or differs from the proposal.	Entity known, cross-border data terms missing.	Registered entity verified, governing law agreed, transfer mechanism such as standard contractual clauses in place.

Use 0 when the vendor did not answer, 2 and 4 for answers between the described levels.

Download your document:

Everything on this page runs in your browser. Nothing you type is sent to us or to anyone else, and nothing is stored after you close the tab.

Framework v1.0 · released 2026-09-24 · changelog

Questions buyers ask

How should we set the weights?: Start from the preset for your situation, then agree changes with everyone who will score before any proposal arrives. Changing weights after reading proposals lets preference creep into the numbers.
Who should score?: Each evaluator scores alone first, using the scoring guide, then the group discusses differences larger than one point. Record the reason for each final score.
What does the CSV contain?: One row per criterion with its weight and each vendor's score, and formulas that compute weighted totals and the percentage of the maximum. It opens in any spreadsheet.
Is price a criterion?: Price is compared separately as total cost over 12 and 24 months with the cost model, so a low rate does not hide weak answers. Framework v1.0 scores pricing transparency, not price.