HealthCare.gov: contract oversight lessons from GAO-14-694

HealthCare.gov, the federal health insurance marketplace, had a troubled launch in 2013. In July 2014 the US Government Accountability Office (GAO) published report GAO-14-694 on how the Centers for Medicare & Medicaid Services (CMS), the buyer, planned and oversaw the contracts behind it. The report is not about code. It is about how a buying organisation set up, directed and checked the work of its contractors, and it reads as a list of controls that any buyer of software delivery or testing services can apply.

Following the approach used across this site, the organisations that built the systems are referred to here as the contractors.

What GAO found

GAO's summary is direct: CMS undertook the development of HealthCare.gov and its related systems without effective planning or oversight practices, despite facing challenges that increased both the level of risk and the need for oversight. CMS officials described the work as a first-of-its-kind effort with compressed time frames.

Four findings matter most for a buyer.

Work started before requirements were known. To be expedient, CMS issued task orders to develop the federally facilitated marketplace (FFM) and the federal data services hub when key technical requirements were unknown, including the number and composition of states to be supported and the number of potential enrollees. The number of users is the core input to any performance test.

The contract type put cost risk on the buyer. CMS used cost-reimbursement contracts, which GAO notes created additional risk because CMS was required to pay the contractor's allowable costs regardless of whether the system was completed.

Oversight tools existed but were not used. CMS did not develop a required acquisition strategy to identify risks and mitigations, and did not use available information, such as quality assurance plans, to monitor performance and inform oversight.

Nobody was clearly in charge of approving extra work. Because of unclear guidance and inconsistent oversight, there was confusion about who had the authority to approve contractor requests to spend funds on additional work. New requirements and changing decisions led to delays and wasted contractor effort.

Costs moved with the requirements

GAO reports that from September 2011 to February 2014, obligations for the FFM rose from $56 million to more than $209 million, and data hub obligations rose from $30 million to nearly $85 million. GAO attributes this primarily to changing requirements, made worse by oversight gaps. After launch, CMS awarded a new contract to another firm for $91 million in January 2014 to continue FFM development; by June 2014 costs on that contract had risen to over $175 million, while key capabilities remained unavailable.

Launch without verification

CMS delayed key governance reviews. An assessment of FFM readiness moved from March to September 2013, weeks before launch, and did not receive the required approvals. GAO's conclusion is the sentence most relevant to testing: CMS launched HealthCare.gov without verification that it met performance requirements.

Concerns raised, consequences not applied

CMS did identify problems. In April and November 2013 it wrote to the FFM contractor about product quality and responsiveness to CMS direction. In September 2013, CMS program officials were concerned enough to move operations to the contractor's offices to give on-site direction. But CMS chose to forego actions such as withholding fee, in order to focus on the launch date. In the end CMS declined to pay about $267,000 in requested fee, about 2 percent of the $12.5 million in fees paid to that contractor.

GAO's recommendations

GAO made five recommendations to CMS. In summary, CMS should:

1. Assess the causes of continued cost growth and delayed functionality and develop a mitigation plan.
2. Ensure quality assurance surveillance plans and other oversight documents are collected and used to monitor contractor performance.
3. Formalise the roles and responsibilities of contracting officer representatives and other oversight staff, including the limits of their authority to direct contractors.
4. Ensure acquisition strategies are completed when required and address requirements, contract type and acquisition risks.
5. Ensure IT projects obtain governance board approvals before proceeding with development.

CMS concurred with four and partially concurred with one.

Controls a QA buyer can take from the report

• State the load before you buy the test. Put expected users, transactions and peak patterns in the RFP. If they are unknown, make establishing them the first deliverable.
• Write a surveillance plan and use it. Define how you will check the vendor's work: which reports, which metrics, which samples you will review, and who reviews them. Then review them on a schedule.
• Name who can direct the vendor. Record who may approve extra scope or spend, and in writing tell the vendor that nobody else can.
• Make readiness a gate with evidence. Require test results against performance requirements before a go-live decision, and do not move the gate without a recorded risk decision.
• Tie money to acceptance. Use payment and fee terms that you are prepared to apply when quality falls short.

What this means for your selection

Build the load assumptions, the surveillance plan and the acceptance gates into the RFP and the contract before you sign. Score each vendor's reporting and escalation approach in the scorecard, and check their performance testing answers with the questionnaire. The guides on KPIs for a QA vendor, managing an outsourced QA team and software testing statement of work turn these controls into documents.