Key Takeaways — Chapter 29: Algorithmic Fairness and Bias in Compliance Systems

1. Bias enters machine learning systems through five distinct pathways, each requiring a different diagnosis and remediation.

Historical bias, measurement bias, label bias, representation bias, and aggregation bias are not the same problem and do not yield to the same solution. Historical bias means the model reproduces discriminatory patterns from historical decisions. Measurement bias means differential monitoring intensity is mistaken for differential risk. Label bias means human prejudice in training labels is inherited by the model. Representation bias means the model performs worse on populations underrepresented in training data. Aggregation bias means strong aggregate performance conceals poor performance for demographic subgroups. Effective remediation depends on correctly identifying which pathway is responsible for the observed disparity.

2. There are four primary mathematical definitions of fairness, each capturing a different moral intuition.

Demographic parity requires equal positive prediction rates across groups. Equalized odds (Hardt et al. 2016) requires equal true positive rates and equal false positive rates across groups. Calibration requires that a model's predicted probabilities are equally accurate across groups. Counterfactual fairness asks whether an individual would have received a different decision if their protected characteristic had been different, all else equal. These definitions are not interchangeable, and a model that satisfies one definition may fail others significantly.

3. The impossibility theorem means fairness is a choice among competing values, not a single optimisation target.

Chouldechova (2017) and Kleinberg et al. (2016) independently proved that when base rates differ across demographic groups, demographic parity, equalized odds, and calibration cannot all be satisfied simultaneously. This is not a temporary limitation of current algorithms — it is a mathematical impossibility. Compliance professionals must therefore make explicit choices about which fairness criteria take priority, grounded in regulatory requirements, ethical considerations, and the specific harms the system can cause. The choice cannot be delegated to data scientists and should not be hidden in technical documentation.

4. The four-fifths rule provides a practical, regulatory-actionable threshold for demographic parity assessment.

Originating in US employment discrimination law, the four-fifths rule holds that if the approval rate for a minority group is less than 80% of the approval rate for the majority reference group, this constitutes potential evidence of disparate impact requiring investigation and justification. The rule is widely applied as a fairness benchmark in algorithmic systems operating in the UK, EU, and US, and provides compliance teams with a concrete, quantifiable trigger for escalation. A parity ratio below 0.80 does not automatically establish unlawful discrimination — but it creates an obligation to investigate.

5. The NIST Face Recognition Vendor Test (2019) demonstrated that commercial facial recognition algorithms exhibit large, structured demographic disparities.

NIST tested 189 algorithms from 99 developers against a dataset of 18.27 million images and found false non-match rates up to 100 times higher for African American and Asian faces than for Caucasian faces in some systems. These disparities reflect representation bias: training datasets that did not adequately represent the demographic diversity of the populations the systems would ultimately process. Any KYC system incorporating facial verification should be assumed to carry this risk unless the vendor can demonstrate demographic-disaggregated performance data showing otherwise.

6. The UK Equality Act 2010 applies to algorithmic systems producing disproportionate adverse impacts.

Indirect discrimination under the Equality Act — where a provision, criterion, or practice applies equally to everyone but has disproportionate adverse effects on people sharing a protected characteristic — does not require discriminatory intent. An automated KYC system that rejects customers with African-heritage names at 3.8 times the rate of Anglo-Saxon-name customers is producing an outcome that is prima facie indirect racial discrimination regardless of whether any individual acted with discriminatory intent. The nine protected characteristics include race, religion or belief, sex, age, and disability, all of which may be proxied by algorithmic features.

7. The FCA's Consumer Duty requires firms to demonstrate good outcomes for all customers, including from automated systems.

Consumer Duty Principle 12 (PS22/9) requires firms to deliver good outcomes for customers in the form of products and services that meet their needs, fair value, clear communications, and effective support. The Duty applies to automated as well as manual processes, and the firm — not the technology vendor — bears responsibility for the outcomes those systems produce. The Consumer Duty also requires firms to give particular attention to customers with characteristics of vulnerability, which may include customers from communities facing structural barriers to financial services access.

8. Remediation of detected disparate impact requires a structured programme addressing training data, features, thresholds, and ongoing monitoring.

When a fairness violation is identified, effective remediation involves auditing training data for composition gaps and label biases; examining model features for proxy encoding of protected characteristics; considering threshold adjustment by demographic group as an interim measure; pursuing data augmentation to address representation gaps; applying fairness-constrained training algorithms where appropriate; and implementing ongoing fairness monitoring using metrics such as parity ratios and equalized odds gaps. The remediation programme should be documented and proportionate to the severity of the disparity.

9. Financial firms are responsible for the fairness of vendor-supplied algorithmic systems operating on their customers.

Regulatory responsibility does not transfer to a vendor when a firm procures an algorithmic system. If a third-party KYC verification system produces discriminatory outcomes for a firm's customers, those are the firm's regulatory compliance failures. Vendor contracts for algorithmic systems in regulated financial services contexts should require: demographic-disaggregated performance reporting; regular fairness assessments; disclosure of training data composition; and remediation timelines for identified disparities. Fairness performance should be treated as a contract SLA metric alongside accuracy and availability.

10. Ongoing fairness monitoring must be integrated into model risk management as a first-class performance obligation.

A fairness assessment at model deployment is necessary but not sufficient. Model performance drifts over time as the deployment population changes, as the world changes, and as patterns in the data evolve. Fairness monitoring must be periodic, automated where possible, and structured to provide early warning before a regulatory violation is reached. Compliance teams should establish trigger thresholds more conservative than the regulatory minimum (for example, a parity ratio below 0.85 rather than 0.80) to allow investigation and remediation before a four-fifths rule violation occurs. Trend tracking over time is as important as point-in-time assessment.