Case Study 15.1: Priya's Credit Model Audit — Discovering the Override Problem at Northgate Bank

Background

Northgate Bank plc was not the kind of institution that appeared on FSB systemic importance lists or generated banner headlines during stress events. It was, by most measures, a solid and unremarkable mid-sized UK retail and commercial bank: £14 billion in total assets, a network of 47 branches concentrated in the East Midlands and Yorkshire, a retail mortgage book of approximately £6 billion, and a commercial lending portfolio totalling £2.8 billion spread across owner-managed businesses and lower-market SMEs.

Northgate had been using its current commercial credit scoring model — built in 2017 by an internal analytics team working alongside a now-defunct consultancy — for seven years. The model was a logistic regression scorecard, well-documented by the standards of the time, validated at development, and signed off by the then-Chief Risk Officer. In the intervening years, annual validation had been described in board risk committee packs as "satisfactory" or "no material issues identified."

In the spring of 2024, Northgate's board risk committee received a letter from the PRA. The regulator's model risk supervisory team had noted, during a broader thematic review of credit model governance at mid-tier UK banks, that Northgate's model validation reports had been produced by the same internal analyst — the Head of Risk Analytics — who had co-developed the model. The PRA asked Northgate to commission an independent external validation within 90 days.

Northgate's Chief Risk Officer, Gregory Ashford, called the Big 4 firm where Priya Nair worked.

The Engagement Begins

Priya had been a RegTech consultant for four years by the time the Northgate engagement came in. She had worked on IFRS 9 implementation at two insurance groups, a model risk governance programme at a building society, and a credit bureau data quality review for a consumer lender. She was, in her own quiet estimation, good at finding things that people had not intended to be found.

The Northgate engagement was scoped as an independent model validation covering:

1. Conceptual soundness of the commercial credit scorecard
2. Methodology review (variable selection, binning, scorecard calibration)
3. Performance assessment against current commercial lending data (2022–2024)
4. Model use and governance review
5. Override analysis

Gregory Ashford had been straightforward in the initial call: "We think the model is fine. The validation is about satisfying the PRA, not because we have concerns. You'll have full access to everything you need."

Priya had heard variants of that sentence on approximately every engagement she had ever worked. She wrote in her engagement notebook: "Full access. They think it's fine. Start with the overrides."

Week One: The Data Room

Priya's team — herself, a senior associate named Kishore, and a data analyst named Bethany — set up in a windowless conference room on the third floor of Northgate's Leicester office on a Monday in April.

The model development documentation was, to Priya's eyes, genuinely reasonable for its vintage. The 2017 build had followed standard scorecard methodology: WoE transformation of nine input variables, logistic regression, scaling to a 300–900 range, calibration to a base odds of 40:1. The development Gini had been 0.48; the hold-out Gini had been 0.43. Acceptable for commercial lending.

Bethany pulled the current performance data. The model was scoring approximately 3,200 commercial lending decisions per year. On the current data (2022–2024, approximately 6,400 decisions with 12-month outcomes observable), the Gini had fallen to 0.36 — down from the development-time 0.43 hold-out, but still technically above the bank's stated minimum threshold of 0.30.

"Model's in acceptable territory," Kishore noted, scanning the metrics. "Gini's down but not failed."

Priya was staring at a different column in Bethany's spreadsheet.

"What's the override column?"

Bethany pulled the raw data extract. Each decision record had a field called model_recommendation (Approve / Decline / Refer) and a field called final_decision (Approve / Decline). Bethany had flagged cases where the two fields did not match.

The number was 1,092 out of 3,200 decisions in the most recent annual period.

34.1%.

The Override Analysis

"That can't be right," Kishore said. He pulled the SQL query and ran it again. The number was the same.

Over the following two days, Priya's team built an override analysis framework. They segmented the 6,400 historical decisions into four groups:

1. Model approved, approved in final: No override, model decision followed. (n = 2,861)
2. Model declined, declined in final: No override, model decision followed. (n = 1,347)
3. Model declined, approved in final: Favourable override — relationship manager approved against model recommendation. (n = 984)
4. Model approved, declined in final: Unfavourable override — relationship manager declined against model recommendation. (n = 108, mostly large exposure credit committee reviews)

Group 3 — the favourable overrides — was where the credit risk lived. Priya's team tracked actual defaults for all four groups over the following twelve months.

The 4.8% default rate for the favourably overridden population was 2.3 times the 2.1% default rate for the non-overridden approved population.

Priya wrote the finding in her notebook: The model is being applied correctly to approximately 66% of decisions. The 34% of decisions where it is being overridden show default rates more than twice as high as model-recommended approvals. Either the relationship managers have information the model does not — which would imply the model is missing material risk factors — or the overrides are generating credit losses that appropriate model adherence would have avoided.

The Override Log Problem

Priya requested the override justification records. Gregory Ashford, when asked, was confident: "Relationship managers are required to document override reasons in the lending management system. That's been policy since 2019."

Bethany spent a day and a half extracting the data. What she found was technically consistent with the policy — every override record had an entry in the override_reason free-text field — but the content of those entries was, charitably described, sparse.

A sample:

Of the 984 favourable overrides in the review period, 731 (74.3%) had override reasons consisting of ten words or fewer. 89 override records contained only a single word or phrase. None of the override records documented the specific credit risk factors that justified overriding the model's recommendation.

There was no override log in the sense that SR 11-7 would recognise — no systematic record of override frequency by relationship manager, by geography, by exposure size, or by outcome. The bank could not have told you which relationship managers had the highest override rates, or whether their overrides performed better or worse than average.

"The policy exists," Priya told Gregory Ashford in the midpoint briefing. "The records don't. You have a documentation policy without a documentation culture."

Understanding the Driver

Before recommending remediation, Priya wanted to understand why the override rate was so high. She arranged structured interviews with five relationship managers and two regional directors.

The answers were consistent.

"The model was built in 2017," said one regional director. "The East Midlands has changed since 2017. We've got a lot of manufacturers who pivoted to green tech. The model doesn't know about that."

"There are some businesses we know really well," said a relationship manager who had worked in the same market for eleven years. "The owners have been with us for fifteen, twenty years. Their financials are lumpy — seasonal businesses, construction cycles — but they've never missed a payment in twenty years. The model doesn't see that."

"Honestly? Sometimes we get pressure from business development," said a third. "Not formal pressure. But it's there."

This gave Priya three categories of override driver:

1. Legitimate information not in the model: Relationship-specific knowledge, local market expertise, sector dynamics not captured in the model's variables. This was genuinely information the model did not have.

2. Model staleness: The model was seven years old. The commercial landscape had changed — sectors, business structures, macroeconomic conditions. Variables that were predictive in 2017 might have drifted.

3. Relationship and commercial pressure: Informal incentive structures favouring loan approval, without adequate counterbalancing credit discipline.

Category 1 was a model limitation problem. Category 2 was a model governance problem. Category 3 was a culture and incentive problem.

All three had the same consequence: systematic erosion of model discipline, documented by the 2.3x higher default rate.

The Validation Findings

Priya's team presented six findings in their validation report:

Finding 1 (Critical): Model override rate of 34.1% materially exceeds industry norms and the bank's own implicit policy expectation. Overridden loans default at 2.3x the rate of non-overridden loans.

Finding 2 (Critical): Override justifications are inadequate — they do not document the specific credit factors that support the override decision. This constitutes a material deficiency in model use controls under SR 11-7.

Finding 3 (High): No override log exists at the level required to identify patterns by RM, geography, or sector. Management cannot identify whether some relationship managers are systematically generating higher-risk overrides.

Finding 4 (High): The model has not been re-validated against current economic conditions. Performance metrics have declined (Gini from 0.43 to 0.36). PSI analysis of key input variables suggests moderate population shift in three of nine model variables.

Finding 5 (Medium): Validation has not been independent — the same individual has produced validation reports as co-developed the model. All prior validations are considered compromised from a governance perspective.

Finding 6 (Medium): The model was developed without a data set that included the post-2019 period. Its calibration does not reflect pandemic-era credit behaviour or current interest rate environment.

The Override Governance Framework

Priya's recommendations centred on a structured Override Governance Framework, which she developed collaboratively with Gregory Ashford and Northgate's Head of Commercial Lending over the final two weeks of the engagement.

Component 1: Override Categorisation

Overrides were classified into three permitted types, each requiring a different level of documentation and approval:

Type A and B overrides were expected to constitute the majority. Type C overrides were capped at 5% of total decisions per quarter (above this, automatic escalation to the Risk Committee).

Component 2: Override Log Requirements

Every override record must contain: - Borrower ID and loan ID - Relationship manager (RM) ID - Model recommendation and score - Override type (A/B/C) - Specific rationale (minimum 50 words for Type A/B; full credit paper for Type C) - Approval hierarchy sign-off - Timestamp

The override log was to be stored as a structured data table (not a free-text field), enabling systematic analysis.

Component 3: Override Performance Monitoring

Quarterly override monitoring report to Credit Risk Committee covering: - Override rate by region, business line, RM, and exposure size - 12-month default rates of overridden vs non-overridden approvals (by override type) - RM-level override performance league table (shared with regional directors, not published to RMs) - Breach reporting: any RM with a override default rate exceeding 2x the portfolio average triggers a review

Component 4: Model Recalibration Trigger

If the aggregate override default rate exceeds 3.0% in any rolling 12-month period (vs the current 2.1% for non-overridden approvals), this triggers a mandatory model re-validation within 90 days.

Implementation: Six Months Later

Six months after delivering the framework, Priya received a call from Gregory Ashford. The override rate had fallen to 18.7% in the first quarter following implementation — not to the 10–15% typical of well-governed credit institutions, but a material improvement. More importantly, the quality of override documentation had transformed: the average override rationale was now 87 words, attached to specific supporting documents in 71% of cases.

The first quarterly override performance report had surfaced a finding that Northgate's credit committee had not anticipated: one relationship manager in the Sheffield office had an override rate of 61% and an override default rate of 8.9% — more than four times the portfolio average. An immediate review of that RM's book had been commissioned.

"We thought the problem was the model," Gregory told Priya. "It turned out the model was telling us something true, and we weren't listening."

Northgate's next PRA model risk supervisory engagement, twelve months after the initial letter, found that the override governance deficiencies had been remediated to the regulator's satisfaction.

Key Lessons

1. Override rates are a model risk indicator. An override rate above 20% in a well-designed scoring system is a signal that something is wrong: the model is not fit for the population, the override culture is dysfunctional, or both.

2. Override performance tracking is non-negotiable. Without data on how overridden loans perform relative to non-overridden approvals, management cannot tell whether overrides are adding value (RMs adding genuine information the model lacks) or destroying it (relationship pressure overriding sound credit discipline).

3. Override documentation quality is not a compliance exercise. It is risk data. Insufficient documentation is not a technicality — it means the bank cannot learn from its override decisions, cannot hold RMs accountable, and cannot identify concentration risks in override behaviour.

4. Model staleness and override rates interact. As models age, their predictive accuracy for a changing population tends to decline. Relationship managers, sensing that the model "doesn't get it," increase overrides. This becomes a self-reinforcing dynamic: the worse the model, the more overrides, the more the model becomes irrelevant. Regular re-validation is not bureaucratic overhead — it is how you maintain model credibility.

5. SR 11-7 applies to model use, not just model construction. A technically sound model that is systematically ignored through undocumented overrides is a model risk management failure, regardless of the model's Gini coefficient.

Discussion Questions

1. Priya identified three categories of override driver: legitimate information, model staleness, and commercial pressure. Which of these three is the most dangerous from a model risk management perspective, and why?

2. The override governance framework caps Type C overrides at 5% of decisions per quarter. What are the trade-offs of a hard cap versus a soft limit requiring escalating approval levels?

3. If you were Northgate Bank's PRA supervisor, what would you expect to see in the bank's Pillar 2 capital assessment (ICAAP) to reflect the model risk identified in this case?

4. The case identifies that an RM with a 61% override rate and an 8.9% default rate was identified through the new monitoring framework. What actions should the Credit Committee take, and what are the people-management sensitivities involved?

Chapter 15 of Regulatory Technology (RegTech): A Practitioner's Guide