Situation

Outsourcing test automation you can keep and maintain

You are paying a vendor to build an automated test suite. The value is not the number of scripts delivered but a suite your own engineers can run, read and extend after the engagement ends. That means your repository, your pipeline, an open framework and tests that fail only when something is actually broken.

What changes in criteria and weights

Ownership of test code, open framework choice, maintainability and flaky test control, and CI pipeline integration rise to 5; working in your tools and knowledge transfer rise to 4. Relevant past performance drops to 2, because the quality of the code and how it fits your stack tell you more than a list of past clients.

Criteria whose weight differs from the default
CriterionDefault weightWeight here
Ownership of test code and assets35
Open framework choice35
Maintainability and flaky test control35
CI pipeline integration35
Works in your tools34
Knowledge transfer34
Relevant past performance32

Weights run from 0 (not scored) to 5. Every other criterion keeps its default weight of 3. The scoring template loads these weights when you choose this situation.

Which tools, in which order

  1. Cost model: Model the build and the years of maintenance after it, so you compare the cost of owning the suite and not only the cost of writing it.
  2. RFP generator: State your stack, repository, pipeline and the framework constraints in the RFP so every proposal is built on your terms.
  3. Vendor questionnaire: Filter by this situation to ask about framework choice, code review, flaky test handling and how the suite is handed over.
  4. Proposal red-flag checker: Catch proprietary frameworks, automation percentages with no scope, full automation promised in weeks and flaky tests handled by reruns.
  5. Scoring template: Score with the automation weights and ask for a code sample or a short paid pilot to ground the maintainability score.
  6. Reference-call script: Ask references whether their own engineers still run and extend the suite the vendor built.
  7. Contract clauses: Write down that you own the test code, that repositories and tool accounts are yours, and how knowledge transfer and acceptance work.

Typical mistakes

  • Accepting a vendor's proprietary framework or licensed tool that your team cannot run without the vendor.
  • Letting the suite live in the vendor's repository and pipeline until the end of the contract.
  • Setting an automation percentage as the goal without saying which user journeys and risks it has to cover.
  • Tolerating flaky tests that are rerun until they pass, so the suite stops being trusted.
  • Never having your own engineers review the vendor's code, so style and structure drift from your codebase.
  • Accepting the build as done when the scripts exist, rather than when they run green in your pipeline and your team can maintain them.

Your repository from the first commit

Create the repository in your own organisation and give the vendor access. Every commit then belongs to your history, code review happens in your workflow, and there is nothing to transfer later. The contract should still assign ownership of the test code and any test assets in writing.

An open framework that fits your stack

Ask each vendor which framework it proposes and why. A suite written in the language your product engineers use, on an open-source framework, is one they can pick up. Watch for vendor-owned accelerators or licensed tools that sit underneath the suite: ask what happens to your tests if that licence ends.

Tests that are trusted

A test that fails at random teaches your team to ignore failures. Agree how flaky tests are detected, quarantined and fixed, and ask for that list in regular reports. Reruns hide the problem; they do not solve it.

Scope by risk, not by percentage

Ask the vendor to propose which journeys to automate first, based on business risk and how often each area changes. Unstable features may be better covered by manual testing until they settle. A percentage without a scope says nothing about what is actually protected.

Plan the handover from the start

Schedule pairing sessions in which your engineers add and fix tests while the vendor is still engaged. The handover is complete when your team can extend the suite on its own.

Questions buyers ask

Which framework should the vendor use?
One your engineers can read and maintain, usually an open-source framework that matches your stack, such as Playwright, Selenium or Cypress for web products. Ask the vendor to justify its choice against your stack and team skills, not its own habits.
How do we judge maintainability before signing?
Ask for a code sample from a comparable project or run a short paid pilot on one of your user journeys. Have one of your engineers review the code for structure, naming, test data handling and how failures are reported.
What should acceptance of the suite look like?
The tests run in your pipeline on every agreed trigger, failures point to a clear cause, flaky tests are tracked and fixed, and your engineers have run and changed the suite themselves during a handover period.

Sources

  1. The Digital, Data and Technology Playbook, UK Cabinet Office
  2. The Technology Code of Practice, UK Central Digital and Data Office
  3. Circular 30: Works Made for Hire, US Copyright Office
  4. Flaky Tests at Google and How We Mitigate Them, Google Testing Blog
  5. DORA research program: software delivery performance metrics, DORA
  6. Quality assurance: testing your service regularly, UK Government Digital Service
  7. Model Services Contract, UK Cabinet Office