How to run a paid pilot with a QA vendor

Why a pilot beats another round of presentations

Proposals and presentations show how a vendor sells. A pilot shows how it works: the questions its testers ask, the defects they find, how they write them up, and how they behave when an environment is down. Public digital procurement guidance, such as the US TechFAR Hub, encourages buyers to evaluate suppliers on demonstrated work rather than written claims alone. A short paid pilot is the private-sector version of that.

It also answers the question most buyers worry about: whether the people you met will be the people who do the work. See how QA vendors staff projects.

Choose scope with a known answer

Pick work you can judge. Good pilot scopes:

• regression testing of a release you have already shipped, where you know which defects were found later;
• exploratory testing of a feature with known, unpublished defects, to see how many the vendor finds and how it reports them;
• automating a small set of stable user journeys in your framework and pipeline;
• writing a test strategy for one product area.

Avoid scopes where success depends on things outside the vendor's control, such as an environment that is often unavailable, or where you have no way to tell good work from poor.

Write the scoring sheet before you start

Decide how you will judge the pilot before it begins, so that the result is not argued after the fact. Use the scorecard with a small set of pilot criteria, for example:

Criterion	What you look at
Understanding of the product	Quality of questions in the first week
Defect quality	Reproducible, clear severity, clear steps, no duplicates
Risk judgement	Whether serious issues were found early and flagged quickly
Automation quality	Runs in your pipeline, readable, stable across runs
Communication	Response time, clarity, overlap with your working hours
Reporting	A final report you could make a release decision from
Team match	Same people as proposed, working the agreed hours

Set pass conditions too, such as "all automated tests run green in our pipeline on three consecutive runs without changes".

Agree a short pilot contract

The pilot needs its own short agreement. Use the contract clause builder for the relevant clauses. At minimum it should state:

• the fixed price and what it covers;
• the named people, and that substitutions need your approval;
• the dates and the scope;
• confidentiality, and data rules: no production data, synthetic or masked data only;
• that you own all test code, cases, data and reports produced, from creation;
• that there is no obligation to continue, and no automatic conversion into a longer contract.

If testers may see personal data, a data processing agreement is needed even for a short pilot, because GDPR Article 28 applies whenever a processor handles personal data for you.

Set up access in your tools

Run the pilot in your repository, CI, issue tracker and test management tool. This tells you whether the vendor can work in your setup, and it means the work is already yours if you stop. Give named accounts, with least privilege, and remove them at the end.

Run it and watch the right things

Assign one person on your side to answer questions within a working day. Slow answers from you will show up as slow work from the vendor. Hold a short weekly check-in and keep notes: questions asked, blockers raised, turnaround on each task.

Watch for the things that are hard to see in a proposal:

• whether the team asks about risk and users, or only about test case format;
• whether serious issues are raised as soon as they are found;
• whether the lead from the proposal is actually present;
• whether the vendor pushes to move work into its own tools.

Score, compare and decide

Score with at least two people from your side, separately first, then together. Compare the pilot with the written proposal: where the pilot shows something the proposal did not, trust the pilot. Then decide to proceed, negotiate specific terms, or stop.

If you proceed, carry the pilot terms into the main contract: the named team, repository ownership and the scoring criteria as a basis for service levels. The selection path shows where the pilot sits between scoring and contract.