Case Study 36.2 — Priya's Implementation Recovery: When the Go-Live Went Wrong

Background

Lakewood Securities, a mid-size US broker-dealer headquartered in Chicago with approximately 1,200 registered representatives and $28 billion in assets under management, decided in early 2023 that its legacy compliance reporting infrastructure could no longer support its regulatory obligations. The firm was subject to FINRA and SEC reporting requirements, including FOCUS reports, blue sheet submissions, CAT reporting, and Form PF filing for its affiliated investment adviser. Its existing infrastructure was a patchwork of three separate systems — a legacy order management system feeding a modified reporting module, a separate trade reconciliation database, and a spreadsheet-based aggregation layer maintained by two compliance analysts — that the firm's Chief Compliance Officer, David Marchetti, described to the board as "duct tape and prayer."

Marchetti ran a structured vendor selection process. Six vendors were evaluated. A formal RFP was issued in June 2023. A proof of concept was conducted in August and September 2023 with the two finalists. A vendor was selected in October 2023: AxisReport, a RegTech compliance reporting platform that had recently expanded into the US market from its UK base and had strong references from comparable broker-dealer clients in the UK and two US pilot clients.

The implementation was scheduled to go live on March 1, 2024 — a date chosen because it aligned with the end of the firm's fiscal year and a period of relatively lower trading volumes. The contract was signed. The implementation began.

By April 15, 2024 — six weeks after go-live — Lakewood Securities had filed 47 manual exception reports with FINRA, spent $180,000 in emergency consultant fees, and brought in Priya Nair from her Big 4 team to conduct a post-go-live recovery assessment. David Marchetti's "duct tape and prayer" infrastructure had been replaced by something that was, for the moment, worse.

What the Root Cause Analysis Found

Priya spent the first two weeks of her engagement doing what she always did in a recovery situation: reading backwards from the failures to their causes, without accepting the first causal explanation offered. The immediate explanation from the project team was "the data mapping was wrong." That was true but incomplete. The question was why the data mapping was wrong and why it was not caught before go-live.

She identified four root causes, each of which had contributed independently to the failure, and all of which had a common upstream cause: a compressed, under-resourced User Acceptance Testing phase.

Root Cause 1: Missing Data Mapping Documentation

The AxisReport platform required specific field mappings from Lakewood's order management system to produce FINRA-compliant FOCUS report output. Fifteen of those mappings were non-standard — they required transformation logic (aggregation, currency conversion, a specific treatment of inter-account transfers) rather than simple field-to-field mapping. During the implementation's Build phase, a contractor hired by the vendor had built the transformation logic. The contractor had documented the logic in a working notes document stored on the vendor's project share drive.

That document was never formally reviewed by Lakewood's compliance or technology team. It was never incorporated into the implementation's technical documentation package. When the contractor left the vendor's project team in January 2024 — six weeks before go-live — the documentation left with him in practice, because no one at the vendor or the client knew it existed in its specific location.

In UAT, the FOCUS report output had been validated at a high level — "the numbers look approximately right" — but had not been validated field by field against the FINRA format specification. When the first production FOCUS report ran on March 7, 2024, three of the fifteen non-standard transformations produced incorrect output. Two produced rounding errors that were individually small but aggregated to material discrepancies. One produced incorrect treatment of a specific transfer category that caused the FOCUS report to understate the firm's net capital position.

Root Cause 2: Abbreviated UAT

The implementation plan, baselined in November 2023, had allocated four weeks to UAT with a structured test plan covering 127 test cases. The test plan had been drafted by the vendor's implementation team and reviewed, but not significantly modified, by Lakewood's project manager.

In January 2024, the implementation fell behind schedule. The delay was traced to unexpected complexity in the CAT reporting integration, which required a custom API connector that had not been anticipated in the implementation plan. The vendor proposed — and the client accepted — compressing UAT from four weeks to ten days and reducing the test plan from 127 test cases to 68. The decision was documented in a project status update that was circulated to the steering committee but not formally approved by the CCO. David Marchetti, reviewing the status update two days before the abbreviated UAT began, had noted a question: "Is this enough time?" He did not receive a response before the UAT clock started.

Of the 68 UAT test cases executed, 61 passed. Seven were marked as "acceptable defects" — defects that the team assessed as unlikely to cause material failures in production. Three of the seven were related to the FOCUS report transformation logic.

Root Cause 3: No Rollback Plan

The implementation plan contained a go-live section that described the activation sequence — which systems would be turned off, in what order, and when the new platform would be activated as the system of record. It did not contain a rollback plan. There was no documented answer to the question: "If we activate the new system on March 1 and discover a critical failure on March 2, what do we do?"

The answer, as it turned out, was to attempt to reconstruct the legacy system — which had been decommissioned on February 29 — from backups that had not been tested for restoration completeness. The reconstruction took 11 days. During those 11 days, Lakewood's compliance team was running its reporting obligations from spreadsheets and memory, with consultant support. The 47 FINRA exception reports were filed during this period.

Root Cause 4: Training Was Transactional, Not Procedural

The platform training delivered to Lakewood's compliance team was a three-day classroom session conducted by the vendor's training team in February 2024, two weeks before go-live. The training covered how to use the platform — how to navigate the interface, how to run reports, how to configure report schedules. It did not cover the underlying data logic — why the platform produced the numbers it produced, what the transformation rules were, what the output should look like when it was correct. When the platform produced incorrect FOCUS report output, the compliance analysts had no frame of reference to identify the error. The numbers "looked approximately right" because the analysts did not know what exactly right looked like at the field level.

The 90-Day Stabilization Program

Priya's recovery plan was structured around three workstreams that ran in parallel for 90 days.

Workstream 1: Immediate Remediation

The first priority was to produce correct regulatory output while the underlying platform issues were resolved. Priya's team established a parallel validation process: every regulatory report produced by AxisReport was validated against a manual recalculation before submission. This process was expensive and unsustainable, but it was the only way to ensure that no further exceptions were filed while the root causes were addressed. The parallel validation ran for 45 days.

Simultaneously, Priya engaged AxisReport's technical team to reconstruct the missing transformation logic documentation. This required a week of forensic work by AxisReport's technical lead, who was able to recover the original contractor's working notes from the vendor's archive. The transformation logic was documented, reviewed by both Lakewood's compliance team and Priya's, corrected in three places where it was found to be incorrect, and formally signed off as the authoritative data mapping specification.

Workstream 2: System Remediation

The three transformation defects identified in the FOCUS report were fixed in a hotfix release deployed on March 22, 2024. The broader test coverage gap — the 59 UAT test cases that had been cut — was addressed through a post-go-live testing sprint that ran through April. All 127 original test cases were executed against the production system; four additional defects were found and remediated.

Workstream 3: Process and Governance

Priya worked with David Marchetti to establish a sustainable operational governance structure for the platform: a weekly platform review meeting covering data quality, exception rates, and open issues; a monthly report to the CCO on SLA performance and open defects; and an annual platform review process feeding into the vendor assessment framework.

She also drafted, for the first time, a platform operations manual: a document that described the data flows into the platform, the transformation logic, the expected output, and the validation steps that compliance analysts should perform before submitting any regulatory report. The operations manual became the training foundation for two compliance analysts who joined after the crisis.

Cost and Consequence Summary

Total remediation cost: approximately $400,000 — exceeding the annual AxisReport license fee.

Key Lessons

Priya's analysis, delivered to Lakewood's board in June 2024, identified the following structural lessons for future implementations:

UAT is a compliance activity, not a project milestone. Compressing UAT to meet an implementation deadline transfers risk from the project to the production environment. The CCO must have explicit go/no-go authority over UAT completion. No implementation deadline should be allowed to override a materially inadequate UAT.

A rollback plan is not optional. For any system that supports a regulatory reporting obligation, the implementation plan must include a tested rollback procedure. "We can restore from backup" is not a rollback plan. "We can restore from backup, we have tested the restoration, the restoration takes X hours, and here is the operational procedure for the period during which neither system is available" is a rollback plan.

Technical documentation is a contractual deliverable, not a project byproduct. Data mapping documentation, transformation logic, integration specifications — these documents must be formally delivered, reviewed, and approved as part of the implementation before go-live. They must be stored in a location controlled by the client, not the vendor.

Training must build understanding, not just familiarity. Compliance staff who operate a regulatory reporting platform must understand what correct output looks like, why it looks that way, and how to identify deviations from it. System familiarity training that does not address the underlying compliance logic is insufficient for a platform on which regulatory submissions depend.

This case study is based on a composite of multiple real-world implementation recovery situations. All names, institutions, and specific financial figures are fictional.