How to choose a performance testing vendor

Script runners and performance engineers

Two kinds of service are sold as performance testing.

A script runner records a few journeys, replays them with many virtual users, and reports response times and error rates. That can be useful, but on its own it rarely tells you why the system slowed down or what to change.

A performance engineer starts from a question, builds a workload that resembles real use, monitors the system under test, finds the bottleneck and explains it. The questions below help you tell them apart before you sign.

ISO/IEC 25010 describes performance efficiency as time behaviour, resource utilisation and capacity. A good proposal addresses all three, not only response time.

Start with your question

Vague goals produce vague tests. Write a question the test must answer:

• "Can checkout handle our expected peak with response times inside our targets?"
• "Where does the search service fail as traffic grows, and what fails first?"
• "Does the system degrade over a long period of steady use?"

Put the question, your response time targets and your expected traffic into the RFP. Ask each bidder how it would answer it.

Workload modelling

The workload model decides whether results mean anything. Ask how the vendor will build it:

• Which journeys, and in what mix.
• Request rates at normal and peak times, taken from your monitoring, not guessed.
• Think time and pacing between steps.
• Data variety, so caches do not make results look better than reality.
• Growth assumptions, if you are testing for future traffic.

A strong answer asks for your production data. A weak answer offers a number of virtual users before asking anything.

Environment parity

Test environments differ from production in size, data volume, network, caches and connected services. Results only transfer if those differences are understood.

Give the vendor a list of differences, and ask how it will account for them. Useful answers include scaling the environment for the test window, loading production-like data volumes, stubbing third-party services with realistic delays, and stating clearly which conclusions do not transfer to production.

Difference	Effect on results	Question to ask
Smaller servers	Limits reached earlier	Will you scale results or scale the environment?
Less data	Queries look faster	How will you load realistic data volumes?
Stubbed third parties	Missing real latency	What delays will stubs simulate?
No CDN or firewall	Different traffic path	Will load pass through the same layers as users?

Load origin

Where load comes from matters. Traffic from one data centre looks different from traffic spread across regions, and it may be blocked or throttled by your CDN or security layers. Ask:

• Which locations the load will come from.
• Whether it passes through the same CDN, firewall and load balancers as real users.
• Which source addresses it will use, so your operations and security teams can recognise it.

Check your hosting, cloud and CDN providers' terms and notify them before tests. Agree a stop procedure with named contacts on both sides.

Data flow and ownership

Performance tests need large volumes of test data, and they produce logs and results. Ask:

• What data the vendor needs, and whether synthetic data can be used. Production personal data needs your approval and, where GDPR applies, processor terms.
• Where scripts, data and results are stored, and for how long.
• Whether the load generation service runs under your account or the vendor's.
• Who owns scripts and results, and whether scripts run without the vendor's tools. See test automation vendor lock-in.

Analysis and reporting

Ask for a redacted sample report. Look for:

• The question and workload model stated up front.
• Results against your targets, not only raw numbers.
• System metrics alongside response times, so causes can be traced.
• Named bottlenecks with evidence and recommendations.
• Limits of the test, including environment differences.

How to read a vendor sample report explains what separates useful reports from volume.

Using the toolkit

The questionnaire includes a question on how a vendor covers non-functional quality, and whether it does performance work itself or uses a specialist. Add the questions on this page to your RFP when performance is a separate lot, and score them with the scorecard.

The selection path shows where a separate performance lot fits in the overall timeline.