# QA vendor questionnaire: buyer's guide _ChooseQAVendor framework v1.0. Situation: Automation build-out. 27 questions. Nothing you enter in the toolkit is sent anywhere._ Issued by Example Product Ltd. Please answer every question in writing by Friday of week 3. Answer with specifics: names, documents, numbers and examples. Where you cannot answer, say so; a blank or generic answer scores zero. --- ## Team and staffing _Who actually does the work, how senior they are, and what happens when people leave._ ### 1. Who, by name and role, will work on our account in the first three months? Please share their CVs and confirm these are the people who will start, not examples. Why it matters: The people named in a proposal are often not the people who arrive. Asking for named CVs up front lets you hold the vendor to the team you evaluated. Strong answer: Named individuals with CVs, their current allocation, a start date for each, and a willingness to list them as key personnel in the contract. Weak answer: "Sample profiles", "a team like this", or "we will assign the right people once the contract is signed". ### 2. Who is the single accountable lead on your side, how much of their time is allocated to us, and what decisions can they make without escalating? Why it matters: A lead spread across many accounts cannot make timely decisions. Knowing their allocation and authority tells you how fast problems get resolved. Strong answer: A named lead with a stated percentage of time on your account, a list of decisions they own, and the escalation route for the rest. Weak answer: "Your account manager will coordinate everything" with no allocation or decision rights stated. ### 3. How will your testers work inside our team day to day: which of our meetings do they attend, whose backlog do they work from, and who on our side assigns their work? Why it matters: Testing works well when testers sit in the delivery team rather than receive builds over a wall. The answer shows whether the vendor expects to be embedded or separate. Strong answer: A concrete working model: which ceremonies they join, the shared tools and backlog they use, and a named buyer contact who prioritises their work. Weak answer: "We adapt to any process" with no description of how, or a model where testing happens only at the end of each release. --- ## Process and methodology _How the vendor decides what to test, documents it and fits into your delivery cycle._ ### 4. Please share an anonymised test strategy or test plan you have written for another client. What sections does it contain? Why it matters: A real document shows how the vendor plans testing, far better than a description of their method. ISO/IEC/IEEE 29119-3 sets out what test plans and strategies are expected to cover. Strong answer: A redacted document covering scope, risks, test levels and types, entry and exit criteria, environments, data, roles and schedule, broadly in line with ISO/IEC/IEEE 29119-3. Weak answer: A marketing slide describing "our proprietary methodology" or a template with no sign it has been used on a real project. ### 5. How do you fit testing into an agile, iterative delivery cycle? At what point in a sprint or release does your team start testing a change? Why it matters: Testing that starts only after development ends turns every sprint into a queue. Government digital guidance expects testing to run throughout delivery, not at the end. Strong answer: Testers involved in refinement, tests written alongside development, regression run on every build, and a description of what happens when a story fails testing late. Weak answer: "We have a dedicated testing phase after development" presented as the default for an iterative team. ### 6. What test environments do you need from us, and how do you make sure results in those environments reflect how the product behaves in production? Why it matters: Tests run in environments unlike production give false confidence. The vendor's environment needs also affect your cost and timeline. Strong answer: A list of environments, access and data required, how differences from production are identified and tracked, and who maintains each environment. Weak answer: "We will use whatever you have" with no view on how environment differences affect test results. --- ## Automation and code ownership _Who owns the test code, which framework it uses, and whether you can maintain it without the vendor._ ### 7. Where will the test automation code live? Will it be in a repository that we own and control from the first commit? Why it matters: If the test code sits in the vendor's repository, leaving them means losing it or paying to get it back. Owning the repository from day one removes that lever. Strong answer: Code committed to the buyer's repository under the buyer's organisation account from the start, with the vendor given access rather than the other way round. Weak answer: "We host it in our secure repository and share access" or "we hand over the code at the end of the contract". ### 8. Who owns the intellectual property in the test code, test cases, data and documentation you produce for us, and does any of it depend on your own libraries or accelerators? Why it matters: Vendor libraries embedded in your tests can leave you unable to run or change them without a licence. Ownership needs to be written into the contract. Strong answer: Clear transfer or assignment of all deliverables to the buyer, a list of any pre-existing vendor components, and a perpetual licence to use those if they remain. Weak answer: "Our proprietary framework speeds things up" with no statement of who owns it or what happens to it when the contract ends. ### 9. Which automation framework would you use for our product, and why that one? What would make you choose differently? Why it matters: The framework should fit your stack and your team's skills, not the vendor's habit. A reasoned choice also shows the vendor understands trade-offs. Strong answer: A choice tied to your languages, platforms and team, with trade-offs named, for example why Playwright, Cypress, Selenium or Appium fits or does not, and a preference for open standards and open-source tools you can maintain yourself. Weak answer: The same framework for every client regardless of stack, or a closed tool that only the vendor can run. ### 10. How do you define, detect and deal with flaky tests? What do you do with a test that passes and fails on the same code? Why it matters: Flaky tests teach a team to ignore red builds. How a vendor handles them shows whether their automation will stay trustworthy over time. Strong answer: A definition of flakiness, how it is measured, quarantine rules, an owner and deadline for each fix, and reporting of flaky rates to the buyer. Weak answer: "We just rerun failed tests" or "our tests are never flaky". ### 11. How will the automated tests run in our CI pipeline, and how long should a full run take before it slows our developers down? Why it matters: Automation that does not run on every change finds problems late. Run time matters because slow suites get skipped. Strong answer: Tests triggered on each change in the buyer's pipeline, a target run time, a plan for splitting fast and slow suites, and results visible to developers. Weak answer: Tests run manually by the vendor on their own machines, or nightly only, with results sent by email. ### 12. How do you keep test code maintainable? Do you have coding standards, code review and a way of structuring tests that our developers could follow? Why it matters: Test code is code. If it is hard to read or change, it becomes a cost you carry long after the vendor has gone. Strong answer: Written coding standards, peer review on every change, a described structure such as page objects or similar patterns, and linting in the pipeline. Weak answer: "Our engineers know what they are doing" or recorded scripts with no review process. ### 13. How do you decide which tests to automate and which to keep manual, and how will you report automation coverage to us? Why it matters: Automating everything is expensive and automating the wrong things adds maintenance without value. You need a rationale and a way to see progress. Strong answer: Criteria for automation such as repetition, risk and stability, a view on the test pyramid, and coverage reported against agreed risks or features rather than test counts alone. Weak answer: "We will automate every test" or coverage reported only as a growing number of test cases. ### 14. If we ended the contract tomorrow, could our own engineers run, read and extend the test suite without you? What documentation would they have? Why it matters: This tests whether you are buying a capability or renting a dependency. Public digital guidance warns against arrangements that lock the buyer in. Strong answer: A README and setup guide in the repository, documented architecture, no hidden dependencies on vendor systems, and an offer to demonstrate a clean setup by your team. Weak answer: "Our team would handle the transition" or documentation that exists only in the vendor's internal wiki. --- ## Tooling _How the vendor's work runs inside your tools and pipeline rather than theirs._ ### 15. How will your tools connect to our existing pipeline, issue tracker and chat channels? Is there anything we would need to install or change? Why it matters: Tools that do not fit your pipeline create manual handoffs and duplicated work. Knowing the integration work up front avoids surprise cost. Strong answer: A description of each integration, what the buyer needs to set up, and an example from another engagement of results flowing into the issue tracker automatically. Weak answer: "Our tools integrate with everything" with no specifics about your setup. ### 16. How is test progress shown to us during a cycle? Can we see a live dashboard in our own tools, and which measures does it show? Why it matters: Visibility during a cycle lets you act before release day. Measures tied to delivery outcomes are more useful than counts of tests run. Strong answer: A live view in buyer-accessible tools showing progress against plan, open defects by severity, pass rates on the main branch and pipeline run times. Weak answer: A weekly slide deck, or dashboards that report only the number of test cases executed. --- ## Security and data _Security posture, certifications with verifiable scope, and handling of test and personal data._ ### 17. If you process personal data for us, will you sign a data processing agreement that covers every point in GDPR Article 28(3), including acting only on our documented instructions, confidentiality, security, assistance, deletion or return, and audits? Why it matters: Article 28 sets the minimum content of a controller to processor contract. A vendor that resists these terms is telling you how it will behave when something goes wrong. Strong answer: Has a standard processing agreement mapped to each Article 28(3) point, accepts your template or reviews it quickly, and names who handles data protection requests on their side. Weak answer: Says a confidentiality clause in the main contract is enough, or refuses audit and assistance obligations. ### 18. How do you control access to our environments: named accounts, multi-factor authentication, least privilege, managed devices, and removal when someone leaves the team? How quickly is access removed? Why it matters: Testers usually hold wide access to pre-release systems. Offboarding gaps are a common source of exposure when teams rotate. Strong answer: Describes named accounts only, enforced multi-factor authentication, company-managed and encrypted devices, a stated time to revoke access after a role change, and periodic access reviews it shares with you. Weak answer: Uses shared test accounts, allows personal laptops, or relies on the buyer to notice when someone leaves. ### 19. How do you manage supply chain risk in your own tools and dependencies, such as test frameworks, plugins, browser extensions and cloud services used on our work? Why it matters: Test tooling runs inside your pipelines and environments. A compromised dependency on the vendor side can become your incident. Strong answer: Keeps an inventory of tools and dependencies used on client work, has an approval process for new ones, applies updates on a schedule, and can explain how this maps to a recognised supply chain practice. Weak answer: Lets each tester install whatever tools they like, or has never been asked about this. --- ## Reporting and communication _What you see each week, how problems surface, and how the team talks to yours._ ### 20. How exactly do you define and count your defect metrics, such as severity, priority, defect leakage, reopen rate and fix verification time? Where are those definitions written down? Why it matters: The same metric name can mean different things between vendors. Without written definitions you cannot compare vendors or track change over time. Strong answer: Provides written definitions with formulas, explains who sets severity and how disputes are resolved, and agrees to align the definitions with yours in the contract. Weak answer: Reports counts of test cases executed or bugs found with no definitions, or changes definitions from month to month. ### 21. How do you report flaky or unreliable automated tests, and how do you keep them from hiding real failures in our reports? Why it matters: Flaky tests make pass rates look better or worse than reality and train teams to ignore failures. Strong answer: Tracks flaky tests separately with a stated threshold, quarantines and fixes them with an owner and a date, and shows the flaky count in each report. Weak answer: Reruns failures until they pass and reports the final result as green. ### 22. How will you tell us about risks you see in our product or process, such as untestable requirements or unstable environments, and how quickly? Why it matters: A testing vendor sees problems early. The value is lost if those observations wait for a monthly report or are softened to keep the client happy. Strong answer: Has a risk log you can see, raises blocking risks the same day with a proposed action, and can give an example of a risk it raised with a client that changed a plan. Weak answer: Says it focuses on execution and leaves risk to the client, or reports risks only in retrospectives. --- ## Commercials _How you pay, what you get for it, and which remedies apply when service levels are missed._ ### 23. Is your price based on time and materials, fixed price or outcomes, and exactly which activities are inside the price and which are billed separately? Why it matters: The pricing model decides who carries the risk of the work taking longer. Hidden extras, such as environment setup or tool licences, change the real cost. Strong answer: States the model, lists what is included and excluded, and gives a priced example for a typical month or release using your scope. Weak answer: Quotes a day rate only, or says extras will be agreed as they come up. ### 24. How do you handle change requests: who can raise them, how are they estimated and approved, and what happens to work that starts before approval? Why it matters: Scope changes are normal in testing. A clear change process stops small additions from turning into unexpected invoices. Strong answer: Describes a written change process with an impact estimate, a named approver on your side, and a rule that no unapproved work is billed. Weak answer: Says changes are handled informally, or bills extra work after the fact. --- ## Exit and continuity _How you leave, what you take with you, and which legal entity you are actually contracting with._ ### 25. What exactly will you hand over when the contract ends: test cases, automation code, test data, environments configuration, defect history, documentation and credentials? Why it matters: If the handover list is not agreed at the start, you may find at exit that key assets live only in the vendor's tools or heads. Strong answer: Provides a written inventory of deliverables handed over on exit in usable formats, confirms you own them, and agrees to include the list in the contract. Weak answer: Says everything will be handed over without listing it, or treats frameworks and test assets as its own property. ### 26. When we take over or move the automation suite, can it run in our own pipeline without your licences, private libraries or internal tools? Why it matters: An automation suite that depends on the vendor's proprietary components is hard to move and becomes a reason you cannot leave. Strong answer: Confirms the suite runs on open or buyer-licensed components, keeps it in your repository from day one, and shows a build running in your pipeline. Weak answer: Relies on a proprietary framework or accelerator you can only use while the contract runs. ### 27. Who will own the intellectual property in test cases, scripts and documentation created for us, and will the contract assign it to us in writing? Why it matters: Work made by a contractor is not automatically owned by the client in every jurisdiction. Ownership needs clear written terms. Strong answer: Assigns ownership of all deliverables created for you in the contract, lists any pre-existing components with a perpetual licence for you to keep using them, and has employee and subcontractor agreements that support the assignment. Weak answer: Grants only a licence while the contract runs, or says ownership will be discussed at the end.