Queensland Health payroll: what the inquiry found on testing

In March 2010 Queensland Health, the state's largest department, switched to a new payroll system. Many staff were not paid, or were paid the wrong amount. A Commission of Inquiry led by the Hon Richard Chesterman reported on 31 July 2013. Its report is unusually detailed about testing: it quotes the test plan's entry and exit criteria, records when they were changed, and explains why that mattered. It is one of the clearest public records of what happens when acceptance criteria give way to a deadline.

As elsewhere on this site, the organisation that built the system is referred to as the contractor, and the firm engaged to run user acceptance testing as the test provider. The buyer side was the State of Queensland, acting through a central IT agency, and Queensland Health.

A hard payroll, an urgent timetable

The Commission describes Queensland Health as the most difficult department for which to design a payroll system. About 80,000 staff were employed under two Acts of Parliament, 12 industrial awards and six industrial agreements, creating more than 200 separate allowances and as many as 24,000 different combinations of pay.

The contract was signed on 5 December 2007 and contemplated replacing the payroll by the end of July 2008. The old system was obsolescent and would lose vendor support, and the Commission says the resulting sense of urgency, which it calls unwarranted, shaped procurement, contract management and project management throughout.

Price chosen, capability not tested

The Commission found that price was the principal factor in selecting the contractor. The remaining budget was about $108 million. The selected bid was $98 million; another bidder priced the whole scope at about $176 million at the same stage (about $175 million at the earlier request-for-proposal stage). Whether the selected contractor could deliver at its price, and why the bids differed so much, were never thoroughly examined by those advising the selection.

How the test gates were lowered

The test plan set clear criteria for user acceptance testing (UAT). To enter UAT, all Severity 1 and Severity 2 defects from unit and system testing had to be resolved. To exit, all Severity 1 and Severity 2 defects from UAT and concurrent test phases had to be resolved.

The Commission records what happened to those criteria:

• When the system entered the third UAT phase in May 2009, it had numerous Severity 2 defects, so the entry criteria were not met.
• It entered the fourth UAT phase in September 2009 with 11 Severity 2 defects outstanding, with the project directorate's agreement.
• A number of Severity 2 defects were reclassified to a lower severity by agreement. The Commission found the record gave no justification beyond the agreement itself.
• On 12 May 2009 the project board minuted a decision to disregard all current language such as severities and defects, and to refer to issues as either a go-live problem or not.
• On 24 December 2009 a contract change replaced the exit criteria with no Severity 1 defects and a management plan for Severity 2, 3 and 4 defects, and deemed the system to have passed. The test provider's final UAT report noted some 26 known Severity 1 and 2 defects at that point.

An expert witness told the Commission that changing a defect's severity can be acceptable if it is well documented and considered by the project board, but that in the end there is still a defect which has to be fixed.

The Commission's conclusion is blunt. The risk of lowering the standards was made explicit by the test provider. If the parties had kept to the controls they set at the start, the system would never have reached the point of a go-live decision. It got there through conscious decisions at earlier stages to lower the bar and alter the criteria. Different forms of testing were also run at the same time, so the project was not required to pass check points before moving on.

The result

When it went live, the system was seriously deficient, and the problems were obvious well before go-live. The Commission found it cost more than four times the contract price in payments to the contractor alone and took three times longer to deliver than scheduled. In September 2010 the State settled, releasing the contractor from claims in return for fixing some 35 defects, after legal advice that damages might be worth as much as $88 million.

The Commission also found that governance was spread across many committees with no single point of accountability, and that these bodies tended to push on, compromising go or no-go gates. Its wider lesson was that the State could not outsource risk: buying from the private sector requires the buyer to manage projects and contractors actively and competently. In its August 2013 response, the Queensland Government accepted the Commission's first recommendation, that there be forward planning for all legacy systems, so that replacements are not decided in haste.

Lessons for a buyer of testing services

• Write entry and exit criteria into the contract, with the severity definitions they depend on.
• Make any change to criteria or severity a recorded decision by a named accountable owner, with the reason and the risk accepted.
• Keep the test provider's independent assessment on the record and in front of the go-live decision makers.
• Do not let test phases overlap to save time unless you decide what each phase still proves.
• Examine a price that is far below others. Ask the bidder to explain its assumptions before you rely on it.

What this means for your selection

Put acceptance gates and severity definitions into the contract and the statement of work from the RFP tool. When a bid is far below the rest, use the cost model and the red-flags checklist to test whether it is realistic, and score delivery risk alongside price in the scorecard. The guides on software testing statement of work, how to evaluate RFP responses and SLA metrics that matter go into the detail.