How to read a QA vendor's sample test summary report

Updated 2026-09-24

A test summary report, which ISO/IEC/IEEE 29119-3 calls a test completion report, should state what was tested against the plan, what was not tested and why, how the results compare with the exit criteria, the open defects by severity, the risks that remain, and what blocked progress. When a vendor sends a sample, read it for what it leaves out: a report with only pass counts and charts tells you little about the risk of releasing.

What to check

  1. Check that the sample states scope against a plan, including what was not tested and why.
  2. Check that results are compared with named exit criteria, not only listed.
  3. Check that open defects are listed by severity, with the severity definitions stated.
  4. Look for a residual risk section written in plain words that a product owner could act on.
  5. Check whether flaky or skipped automated tests are reported separately from passes and failures.
  6. Ask whether the sample is a real, anonymised report or one prepared for the sales process.

Ask the vendor

  • Can you show us a real, anonymised weekly status report from a current engagement, with nothing added for the sales process?
  • What does your test completion report at the end of a release or phase contain, and can you show one?
  • How exactly do you define and count severity, priority, reopen rate and fix verification in your reports?
  • How do you report flaky or unreliable automated tests, and how do you keep them from hiding real failures?
  • Will we have direct read access to the test management data and defect tracker behind these reports?

Why ask for a sample report

Reports are what you will actually see of a testing vendor most weeks. A sample from a real engagement shows how the vendor thinks about risk, how honest it is about gaps, and whether you will be able to make a release decision from what it sends you. It also shows whether the vendor's reporting rewards activity or information.

Ask for two samples through the questionnaire: a weekly status report and an end-of-release completion report. Ask that they be real and anonymised, not prepared for the sale.

What a test summary report should include

ISO/IEC/IEEE 29119-3 describes the test completion report as the document that summarises testing at the end of a project, level or cycle. The items below follow that structure, phrased as what you should look for.

Section What it tells you Question to ask yourself
Scope tested What was tested, against which plan and build Does it name the build and the plan?
Deviations What was planned but not done, and why Is anything missing without explanation?
Not tested Areas deliberately left out Would I have left those out?
Results against exit criteria Each criterion, met or not met Are the criteria measurable?
Defects Open and closed, by severity, with definitions Are the high-severity items described in words?
Residual risks What could still go wrong, in plain language Could a product owner act on this?
Blockers Environments, data, access or late builds that slowed testing Is my side named where it caused delays?
Measures Execution, pass, fail, blocked, skipped, flaky Are skipped and flaky separated from passed?
Assets What was produced or updated, and where it lives Is it in my repository and tools?
Lessons What to change next cycle Is it specific?

Read for scope honesty first

The most useful part of a report is often what was not tested. A report that shows full execution of every planned test on every cycle is either describing a very small scope or leaving things out. Look for:

  • a "not tested" or "out of scope" section, with reasons;
  • deviations from the plan, such as tests dropped because an environment was down;
  • the build or version tested, so you can tell whether results apply to what you are releasing.

If the sample has none of these, ask the vendor how it would tell you that an area was skipped. The answer tells you more than the sample.

Read exit criteria as tests of the vendor

Exit criteria should be written before testing starts and evaluated in the report. Compare these two versions:

Weak: "All planned test cases executed. Testing complete."

Stronger: "Exit criteria: no open severity 1 or 2 defects (met); regression suite for changed modules executed with no unexplained failures (not met, see risk R3); performance of checkout within the agreed threshold on staging (not tested, environment unavailable)."

The first can always be met. The second tells you what you are accepting if you release.

Read defects by severity, not by count

Check that the report defines severity and priority, and uses the definitions consistently. Then read the open high-severity items. Each one should say what the user would experience, where it was found, and its status. A report that shows a chart of defect totals by week, with no description of the serious ones, is built to show effort.

Be wary of reports where the main success measure is the number of defects or test cases logged. That pattern rewards volume rather than lower release risk, and the red flags checker flags it.

Read automation results carefully

Automated test results hide problems in two ways. Flaky tests, which pass and fail on the same code, can be rerun until green. Skipped tests can be counted as not failing. Google's testing team has written publicly about how flaky tests erode trust in results and need to be tracked and fixed rather than retried.

Check that the sample shows passed, failed, skipped and flaky tests as separate numbers, and that it says what is being done about the flaky ones.

What to do with what you find

Score reporting in the scorecard from what the samples show, not from what the proposal promises. Put the gaps into your contract: the report contents, the frequency and your read access to the underlying data. The reporting section should also match the vendor's test strategy, since the report is where you check the strategy was followed. For the measures that belong in service levels, see SLA metrics that matter.

Tools for this

Questions buyers ask

What is the difference between a status report and a summary report?
A status report shows progress during a cycle: what has been run, what is blocked, what has been found so far. A summary or completion report is written at the end of a cycle, release or phase and evaluates the result against the exit criteria. Ask for a sample of each.
Is a high defect count a good sign?
Not on its own. A report that leads with the number of defects or test cases rewards volume. What matters is whether serious risks were found early and whether the remaining risk is clear to you.
Should the vendor recommend whether to release?
The vendor can give an opinion, but the decision should sit with your release owner, based on evidence in the report. A report that says only 'ready' or 'not ready' without the evidence gives you nothing to challenge.

Sources

  1. ISO/IEC/IEEE 29119-3:2021 Software testing, Part 3: Test documentation, ISO/IEC/IEEE
  2. ISO/IEC/IEEE 29119 series overview, ISO/IEC JTC 1/SC 7
  3. Flaky Tests at Google and How We Mitigate Them, Google Testing Blog