TSB 2018: what the regulators said about testing and suppliers

Over the weekend of 20 to 22 April 2018, TSB moved most of its customer services and data from the IT platform of its former parent group onto a newly built platform. Problems began soon after go-live and affected digital banking, telephone banking, branches and payments. In December 2022 the Financial Conduct Authority (FCA) and the Prudential Regulation Authority (PRA) fined TSB a combined £48.65 million: £29.75 million from the FCA and £18.9 million from the PRA. The regulators' announcement says all of TSB's branches and a significant proportion of its 5.2 million customers were affected, and that TSB did not return to business as usual until December 2018.

The FCA Final Notice runs to more than a hundred pages. Much of it is about testing evidence and about how a bank oversaw the company that built and ran its platform. Both are directly relevant to anyone buying testing services or relying on a supplier's test results. Following the regulators' approach, this post refers to that company as the supplier. It was part of the group that had acquired TSB.

What the regulators found

The regulators' joint announcement summarises the case in one sentence: TSB failed to organise and control the IT migration programme adequately, and it failed to manage the operational risks arising from its IT outsourcing arrangements with its critical third-party supplier. The FCA notice is careful to say that the direct technical causes were configuration, capacity and coding issues, but that there were failings in planning, testing, risk management and outsourcing along the way.

Testing ran out of time

The notice describes testing phases that did not keep up with the plan. The various types of testing were meant to run largely in sequence. As they fell behind, they ran in parallel or changed in scope to meet the timeline. Testing only concluded the day before the decision to migrate was taken.

For a buyer, the lesson is that a test plan is only a control if the order and exit points in it are kept. Once phases overlap to protect a date, the results of one phase no longer inform the start of the next.

A reduced performance test, decided as a technical matter

The most specific finding concerns non-functional testing. The new platform ran across two data centres in an "Active-Active" configuration. TSB planned to run end-to-end performance tests in both data centres in that configuration. Around the end of February 2018, after a testing forum discussion, TSB accepted a counter-proposal from the supplier to test in only one data centre, because testing across both would have interrupted services already live on the platform.

The FCA says this decision was taken outside the appropriate governance forum, was treated as a purely technical decision, and that the risks of not testing in Active-Active were not identified or reported, so no mitigations were considered. It also says that had this testing been conducted, it is likely that a configuration problem in the data centres, which made internet and mobile banking unavailable for periods after go-live, would have been found.

Readiness evidence that was not evidence

The notice describes several ways in which assurance looked stronger than it was:

• Responsibility for attesting that non-functional requirements had been met was split between business functions and IT in February 2018. The FCA says this change does not appear to have been drawn explicitly to the Board's attention.
• Instead of design documentation, TSB agreed to accept configuration documents, which could not be used to verify that the infrastructure had been built to the original design. In February 2018 TSB decided not to require the full set of those documents either. The FCA notes this placed extra importance on testing as a mitigant.
• The supplier provided a letter confirming non-functional readiness, referencing confirmations from its critical subcontractors. The FCA describes these as to some extent forward-looking statements of good intention rather than statements of fact, and all but one were caveated with outstanding tasks or tests.

Oversight of the supplier and its subcontractors

The FCA found that TSB did not, at the outset, run a formal comprehensive due diligence exercise on the supplier's capability to deliver and operate the platform. The supplier relied on 85 third parties, TSB's fourth parties, and by February 2018 TSB had not ensured that the supplier's own supplier management model was fully developed. TSB also did not reassess the supplier's capability after problems with services that had already gone live on the platform.

Lessons for a buyer of testing services

• Own the test scope. Any reduction in performance, resilience or failover testing is a risk decision. Route it through the same governance that approves go-live, with the risk written down.
• Ask what the test environment proves. A test in one data centre does not prove a two-data-centre configuration. Require the vendor to state which production conditions each test does and does not reproduce.
• Accept evidence, not assurances. A readiness letter is not a test report. Require results against stated exit criteria, with open items listed.
• Look through to subcontractors. Ask who else is in the delivery chain and how the vendor oversees them.
• Reassess when early signals appear. Problems in a pilot or early release are a reason to recheck capability, not a detail to manage later.

What this means for your selection

Write exit criteria and scope-change rules into the contract and the statement of work, and check the vendor's approach to non-functional testing with the questionnaire. Use due diligence to map subcontractors, and the red-flags checklist to spot readiness claims without evidence. The guides on performance testing vendor questions, test strategy document contents and how to read a vendor sample report cover what good evidence looks like.