Test automation ROI when a vendor builds the suite
Updated 2026-09-24
Test automation ROI is the value the suite returns over its life minus what it costs to build, run, maintain and hand over. When a vendor builds it, include maintenance after every product change, time lost to flaky tests, pipeline and environment costs, and the cost of taking the suite over at exit. Estimate each part with your own numbers rather than accepting a vendor's saved-hours figure.
What to check
- List the regression checks you run today, how often, and how long each takes by hand.
- Ask the vendor for a build estimate per journey or test area, not a single lump sum.
- Ask for their maintenance assumption: hours per sprint to keep the suite green after product changes.
- Agree how flaky tests are defined, tracked and fixed, and include the time in the model.
- Confirm the suite lives in your repository and runs in your pipeline, so handover cost stays low.
- Enter build, run, maintenance and handover costs in the cost tool and set a review date.
Ask the vendor
- How do you decide which tests to automate and which to keep manual?
- What maintenance effort do you assume per sprint, and what happens when our UI or APIs change?
- How do you define, detect and deal with flaky tests?
- Could our own engineers run, read and extend the suite without you if the contract ended?
- How long should a full run take in our pipeline, and who pays for the compute?
Why vendor ROI figures need rework
Vendors that sell automation often present ROI as saved manual hours: a regression that takes testers days by hand will run in a fraction of the time once automated, so the build pays for itself after some number of releases. The arithmetic is simple and the conclusion is usually correct in direction. What it leaves out is the cost of keeping the suite useful, and the cost of owning it when the vendor leaves.
A buyer-side model includes both sides of the ledger over the life of the suite, not only the first year.
The value side
Automation returns value in three ways. Estimate each with your own data.
- Manual effort you no longer spend. Count the regression checks you actually run, how often, and how long they take by hand. Only count checks you would really run at that frequency. Automating a check you ran once a quarter saves little.
- Faster feedback. Tests that run on every change in the pipeline find defects while the developer still has the change in mind. DORA research links test automation to software delivery performance: teams that can release frequently with confidence depend on fast, reliable automated tests. Value this as shorter release cycles or fewer late surprises.
- Fewer escaped defects. If automated checks catch regressions that previously reached production, you avoid incidents and emergency fixes. Use your own incident history, not general figures.
The cost side
| Cost line | What drives it | How to estimate it |
|---|---|---|
| Build | Number of journeys, test data needs, environment readiness | Vendor estimate per area, plus your time to review and answer questions |
| Framework and tooling | Open-source or licensed, device services | Licence and subscription costs, if any |
| Pipeline and compute | Run frequency, parallel runs, device clouds | Your CI and cloud costs per run, times runs per month |
| Maintenance | Change rate of the product, test design quality | Hours per sprint to update tests after product changes |
| Flaky test handling | Test design, environment stability, timing | Hours spent rerunning and investigating tests that fail without a real defect |
| Reporting and triage | Volume of failures, clarity of reports | Hours per run to review results |
| Handover | Where the code lives, documentation, framework choice | Days of knowledge transfer at exit, and your team's ramp-up time |
Maintenance is the line most often missing from vendor figures. Every time a screen, API or flow changes, some tests break. A suite for a product that changes every sprint needs continuous upkeep. Ask the vendor for their assumption and check it during the pilot.
Flaky tests are a cost, not a nuisance
A flaky test passes and fails on the same code. Google's testing team has written publicly about how flaky tests affect a large test estate and how they had to build processes to find and quarantine them. The lesson for a buyer is that flakiness does not fix itself. It costs time every run, and worse, it teaches people to ignore red builds, which removes the value of the suite.
Put three things in the contract or statement of work: a written definition of a flaky test, a report of flaky tests each period, and an obligation to fix or remove them within an agreed time. The SLA metrics guide has wording you can adapt.
Handover cost decides long-term ROI
If the suite is built on the vendor's own framework, lives in their repository, or runs on their machines, the value stays with the vendor. At exit you either rebuild or keep paying. Either way, the ROI you calculated at the start no longer holds.
Keep handover cost low from day one:
- The code lives in your repository from the first commit.
- The framework is open-source and widely used, such as Playwright, Selenium, Cypress or Appium, and your engineers can run it.
- Tests run in your CI pipeline, not on a schedule the vendor controls.
- The contract assigns ownership of the test code and documentation to you.
The guide on test automation code ownership covers the contract terms.
Putting numbers on it
Build the model in the cost tool with your own figures:
- Monthly value: manual hours no longer spent, times your internal or vendor rate, plus any incident cost you expect to avoid.
- One-off cost: build estimate from the vendor, plus your review time.
- Monthly cost: maintenance, flaky test handling, triage, pipeline and tooling.
- Exit cost: handover days and ramp-up.
Payback is the point where accumulated value exceeds build cost plus accumulated running cost. If payback depends on the suite surviving longer than your contract term, the handover line matters most.
Check the model during the pilot
A short paid pilot is the cheapest way to test a vendor's assumptions. Ask them to automate a small set of real journeys, then measure build time, how many tests break after your next product change, and how often tests fail without a real defect. Replace the vendor's estimates with those measurements. The paid pilot guide explains how to set one up, and the selection path shows where it fits.
Tools for this
Questions buyers ask
- A vendor showed a saved-hours calculation. Is that ROI?
- It is one side of it. Saved manual hours are real only if you would have run those checks by hand at that frequency. The cost side must include maintenance, flaky test handling, pipeline time and handover.
- When does automation fail to pay back?
- Common causes are automating unstable features that change every sprint, building on a framework only the vendor can maintain, and letting flaky tests erode trust so people stop reading the results.
- Should I measure ROI in money or in release speed?
- Both, if you can. Faster, more frequent releases with confidence are often the main value, and they matter even when the saved hours alone would not justify the build.
Sources
- Flaky Tests at Google and How We Mitigate Them, Google Testing Blog
- DORA research program: software delivery performance metrics, DORA
- Quality assurance: testing your service regularly, UK Government Digital Service
- ISO/IEC/IEEE 29119 series overview, ISO/IEC JTC 1/SC 7