# QA vendor scorecard

_ChooseQAVendor framework v1.0. Situation: Replacing an underperforming vendor. Nothing you enter in the toolkit is sent anywhere._

Score each vendor from 0 to 5 on each criterion using the scoring guide. The weighted total is the sum of weight times score; the percentage is that total divided by the maximum possible.

| Criterion | Weight | Vendor A | Vendor B | Vendor C |
| --- | --- | --- | --- | --- |
| Named team and seniority mix | 3 | 4 | 3 | 5 |
| Staff continuity and replacement terms | 5 | 3 | 4 | 2 |
| Relevant past performance | 3 | 5 | 2 | 3 |
| Capacity and speed to start | 3 | 2 | 5 | 3 |
| Test strategy and documentation | 3 | 3 | 4 | 4 |
| Risk-based scope | 3 | 4 | 3 | 5 |
| Onboarding plan and turnaround | 4 | 3 | 4 | 2 |
| Ownership of test code and assets | 5 | 5 | 2 | 3 |
| Open framework choice | 3 | 2 | 5 | 3 |
| Maintainability and flaky test control | 3 | 3 | 4 | 4 |
| CI pipeline integration | 3 | 4 | 3 | 5 |
| Works in your tools | 3 | 3 | 4 | 2 |
| Security posture and certifications | 3 | 5 | 2 | 3 |
| Test data and personal data handling | 3 | 2 | 5 | 3 |
| Subcontracting transparency | 3 | 3 | 4 | 4 |
| Reporting that shows risk | 5 | 4 | 3 | 5 |
| Communication and working-hours overlap | 3 | 3 | 4 | 2 |
| Pricing transparency | 3 | 2 | 5 | 3 |
| Measurable service levels | 3 | 3 | 4 | 4 |
| Insurance and liability | 3 | 4 | 3 | 5 |
| Audit and information rights | 3 | 3 | 4 | 2 |
| Exit plan and assistance | 4 | 5 | 2 | 3 |
| Knowledge transfer | 5 | 2 | 5 | 3 |
| Contracting entity and jurisdiction | 3 | 3 | 4 | 4 |
| Weighted total | max 410 | 276 (67.3%) | 298 (72.7%) | 277 (67.6%) |

---

## Scoring guide

| Criterion | 1 | 3 | 5 |
| --- | --- | --- | --- |
| Named team and seniority mix | Generic roles or "resources" with no names or CVs. | Named lead, the rest described by role and level. | Every team member named with CV, allocation and a commitment that they start on day one. |
| Staff continuity and replacement terms | No commitment; replacements at the vendor's discretion. | Notice period for changes, overlap not specified. | Written notice period, handover overlap at the vendor's cost, and your right to approve replacements. |
| Relevant past performance | Claims of experience with no examples or references. | Comparable examples, references offered but not in your domain. | Comparable engagements in your domain with references who will speak to them. |
| Capacity and speed to start | Start date and ramp-up undefined. | Start date committed, scaling terms vague. | Named start date, written ramp-up plan and scaling notice periods in both directions. |
| Test strategy and documentation | No sample documentation; process described in slogans. | Sample documents exist but are generic. | Sample strategy and completion report that cover scope, risks, approach, environments, entry and exit criteria. |
| Risk-based scope | Promises to test everything. | Mentions risk but scope is a feature list. | Scope derived from a written risk assessment with explicit exclusions you can challenge. |
| Onboarding plan and turnaround | No onboarding plan; turnaround unstated. | Outline plan, turnaround left open. | Dated onboarding plan with milestones and a committed turnaround per build. |
| Ownership of test code and assets | Vendor keeps code in its own repository or licenses it to you. | Assigned to you on payment or on exit. | Written assignment of rights, code in your repository from the first commit. |
| Open framework choice | Proprietary framework or accelerator required. | Open framework with a vendor-specific wrapper. | Open framework in a language your team uses, chosen with your engineers. |
| Maintainability and flaky test control | Success measured by number of tests written. | Flaky tests fixed when noticed. | Flake rate tracked and reported, quarantine policy, code review on all test code. |
| CI pipeline integration | Tests run on the vendor's machines and results are emailed. | Tests run in CI on a schedule. | Tests run in your pipeline on every change with results visible to your engineers. |
| Works in your tools | Requires its own portal or tools to see results. | Uses your tools with duplicate reporting in theirs. | Works fully in your tools; nothing to license from the vendor. |
| Security posture and certifications | Certifications claimed without certificate, scope or report. | Valid certificate, scope does not clearly cover the delivery team. | Certificate verified with the issuing body or a current SOC 2 Type II report covering the delivery team, with exceptions explained. |
| Test data and personal data handling | Expects copies of production data; no processing terms. | Processing agreement offered, data locations vague. | Synthetic or masked data by default, listed processing locations and sub-processors, Article 28 terms. |
| Subcontracting transparency | Subcontracting not mentioned or reserved without limit. | Disclosed on request. | All subcontractors listed, flow-down of obligations, your prior approval required. |
| Reporting that shows risk | Reports count test cases executed and bugs logged. | Coverage and defect trends, little on risk. | Sample report states scope covered, gaps, open risks and a clear release recommendation. |
| Communication and working-hours overlap | Communication through an account manager only; little overlap. | Some overlap, team joins selected meetings. | Committed overlap hours, direct access to the team, named lead with authority. |
| Pricing transparency | Single lump sum or blended rate with no breakdown. | Rates by role, extras unclear. | Rates by role, included and excluded items, change pricing and rate review terms stated. |
| Measurable service levels | No service levels; payment for hours only. | Service levels listed without remedies. | Measurable service levels with reporting and remedies such as service credits. |
| Insurance and liability | No evidence of insurance; liability capped very low. | Insurance evidenced, standard caps. | Certificates of insurance provided, carve-outs for data breach and confidentiality agreed. |
| Audit and information rights | No audit or information rights. | Information on request, no audit. | Contractual audit and information rights, including for regulators where applicable. |
| Exit plan and assistance | No exit terms. | Termination terms only. | Written exit plan, assistance period, handover list and fees for exit agreed up front. |
| Knowledge transfer | Knowledge lives in people's heads. | Documentation on request at exit. | Living documentation in your tools and a defined transfer process with successor shadowing. |
| Contracting entity and jurisdiction | Contracting entity unclear or differs from the proposal. | Entity known, cross-border data terms missing. | Registered entity verified, governing law agreed, transfer mechanism such as standard contractual clauses in place. |

Use 0 when the vendor did not answer, 2 and 4 for answers between the described levels.