Case Study 19-1: NYC Local Law 144 — The First US Mandatory AI Bias Audit Law

Overview

New York City Local Law 144 of 2021 (LL 144) represents a watershed moment in AI governance: it is the first law in the United States to require that AI systems used in consequential decisions be subject to independent bias auditing and public disclosure. The law's passage, implementation, and early experience have illuminated both the promise of mandatory AI auditing and the significant limitations that arise when audit requirements are designed without full technical depth or regulatory resources.

This case study examines the legislative history of LL 144, its specific requirements, the implementing rules issued by the NYC Department of Consumer and Worker Protection (DCWP), early compliance experience, the law's substantive limitations, and its influence on other jurisdictions considering similar requirements. It draws on the law's text, the DCWP's implementing rules, early enforcement actions, academic analysis, and the experiences of affected employers and vendors.

Legislative History

The Political Context

LL 144's passage occurred in a specific political context: a period of intense public and legislative attention to AI bias in hiring, following widespread media coverage of Amazon's hiring algorithm (2018), ProPublica's COMPAS investigation (2016), and growing advocacy by civil rights organizations about algorithmic discrimination in employment, credit, and housing.

New York City's Human Rights Law — which prohibits employment discrimination on the basis of race, gender, national origin, disability, and numerous other characteristics — applies to all employers in the city with four or more employees. As AI tools in hiring became more widespread, civil rights advocates argued that the existing legal framework was insufficient: employers could use discriminatory AI tools while individual job applicants, who were unaware they were being screened by an AI system and had no information about how the system worked, had no practical ability to bring discrimination claims.

The Algorithmic Accountability Act, proposed at the federal level by Sen. Ron Wyden and Rep. Yvette Clarke in 2019, proposed a national bias audit requirement but failed to advance through Congress. New York City moved independently.

The Legislative Drafting Process

LL 144 was introduced in the New York City Council in 2021 by Council Member Laurie Cumbo. The drafting process involved engagement with civil rights advocates, employment law attorneys, AI researchers, and technology industry representatives, including HR technology companies and their lobby organization, the HR Policy Association.

The key negotiating tension was between the civil rights advocates who wanted comprehensive bias auditing with strong disclosure requirements and the industry representatives who were concerned about compliance costs, trade secret exposure, and the technical feasibility of the audit requirements as initially drafted. The final law reflected significant compromises: it covered only automated employment decision tools (not other consequential AI domains), it allowed employers to define what counts as an AEDT with some latitude, and it did not specify detailed technical requirements for audits, leaving those for subsequent rulemaking.

LL 144 was enacted in November 2021 and signed by Mayor Bill de Blasio. An effective date was initially set for January 1, 2023, following a delay from an original effective date of January 1, 2022, to allow for rulemaking and compliance preparation.

Requirements Under LL 144 and DCWP Rules

Scope: What Is Covered?

LL 144 applies to employers in New York City who use an "automated employment decision tool" (AEDT) to "substantially assist or replace discretionary decision making" in employment decisions, including decisions about who to interview, who to hire, and who to promote.

The DCWP's 2023 implementing rules defined an AEDT as a machine-based tool that uses machine learning, statistical modeling, data analytics, or AI to generate a simplified output — including a score, a classification, or a recommendation — that employers use to make employment decisions.

This definition excludes several categories of software that employers initially feared would be covered: video interview platforms that merely record and transmit video (without AI analysis), applicant tracking systems that merely organize and route resumes, and scheduling tools that manage interview logistics. It covers tools that use AI to analyze resume content, assess candidate video interviews, analyze written responses, or otherwise generate evaluative outputs that employers use in selection decisions.

The DCWP rules clarified that the law applies when an employer uses an AEDT for a position for which the ultimate performance of work will be in New York City, or when the applicant has notified the employer that they are applying from a New York City location.

Bias Audit Requirements

The law requires employers to obtain a "bias audit" before using an AEDT and at least annually thereafter. The DCWP rules define a bias audit as an impartial evaluation by an independent auditor to assess the AEDT's performance, including its disparate impact, across sex, race/ethnicity categories as defined by the U.S. Equal Employment Opportunity Commission, and intersectional categories.

The audit must calculate:

  • Selection rates for each race/ethnicity category and each sex category, and for each intersectional category (combination of race/ethnicity and sex)
  • Impact ratios for each category, defined as the selection rate for the category divided by the selection rate for the highest-scoring category

An impact ratio below 0.8 (the "four-fifths rule" from EEOC's Uniform Guidelines on Employee Selection Procedures) is a standard indicator of adverse impact. The audit must also disclose the date of the audit, the historical data used, and the number of individuals assessed.

The "independent auditor" requirement excludes the AEDT vendor and the employer itself from conducting the audit. The DCWP rules did not establish credentialing requirements for auditors — anyone who is independent of the vendor and employer can conduct an audit — a significant gap that has allowed compliance-oriented audits of limited rigor.

Disclosure Requirements

Employers must publish the bias audit results on their publicly accessible website in a "clear and prominent" location. The disclosure must remain on the website for at least the period during which the employer uses the AEDT and for six months after they stop using it.

The disclosure must include: a summary of the results of the most recent bias audit; the date of the audit; the name of the tool; and a statement of whether the employer relied on historical data from the employer, historical data from the vendor, or test data.

Candidate Notification

Before using an AEDT to screen a candidate for employment, employers must notify candidates that an AEDT will be used and must describe the job qualifications and characteristics the AEDT will use to evaluate them. The notification must be provided at least 10 business days before the AEDT is used.

Candidates must also be informed of their right to request an alternative selection process. The law does not specify what alternative processes must be available — only that employers must accommodate such requests.

Early Implementation Experience

Compliance Challenges

Implementation of LL 144 has been uneven. A survey of large employers and HR technology vendors conducted by industry researchers in 2023 found widespread confusion about what constitutes an AEDT, which vendors' tools fall within the law's scope, and how to conduct compliant audits. Several large employers reported difficulty determining whether specific tools — particularly video interview analysis platforms that generate structured assessments — were covered.

The DCWP initially lacked the enforcement resources to systematically investigate non-compliance, and early enforcement actions were limited. Civil rights advocates expressed frustration that many employers were not complying, were posting disclosure notices that were not clearly accessible, or were conducting audits that did not meaningfully test for bias.

The Audit Market's Response

The bias audit requirement created a market for audit services almost overnight. Several firms — BABL AI, Parity AI, Holistic AI, and others — offered LL 144 compliance auditing services. Early audits varied significantly in quality, reflecting the absence of credentialing requirements and standardized audit methodology.

Some early audits were conducted with limited data: firms audited vendors' tools using proprietary historical data not disclosed publicly, and the resulting "audits" provided minimal information about how the tool performed for New York City-specific populations. Critics noted that the DCWP rules' requirement to disclose the date, the data, and the results — but not the methodology — created significant latitude for superficial compliance.

A survey of publicly posted bias audits conducted by academic researchers found that many audit disclosures were difficult to find on employer websites, used inconsistent methodologies, and provided results that were difficult for non-technical users to interpret. The proponents of the law acknowledged these limitations but argued that even imperfect disclosure was preferable to no disclosure.

What the Audits Found

Of the early bias audits conducted under LL 144, several found selection rate disparities for certain demographic groups. A small number of employers modified their use of tools or their tool configurations in response to audit findings. In most cases, employers published audit results without commenting on whether they had taken any action in response to identified disparities.

The absence of any requirement that employers remediate identified disparities — or even respond to them — is a significant limitation of the law. LL 144 requires auditing and disclosure, not correction. An employer can publish audit results showing adverse impact ratios well below 0.8 and continue using the tool without modification. The law's theory is that public disclosure will create market and reputational pressure for remediation; early experience suggests this mechanism operates slowly if at all.

Substantive Limitations

What LL 144 Doesn't Cover

Several substantive limitations of LL 144 have been identified by researchers and practitioners:

Outcome quality is not assessed. LL 144's bias audit focuses on selection rates: does the AEDT select candidates of different demographic groups at different rates? It does not require assessment of whether the AEDT's selections are actually predictive of job performance. A tool that selects men at higher rates than women for software engineering roles is flagged by the audit; a tool that selects all candidates at equal rates but selects poorly-performing candidates is not flagged. This limitation means that a tool can satisfy LL 144's requirements while providing little predictive value and potentially serving as a racially neutral veneer on discriminatory hiring.

Intersectional analysis is incomplete. While LL 144 requires calculation of selection rates for intersectional categories (race/gender combinations), the analysis is limited to the categories defined by the EEOC's EEO-1 reporting categories. Non-binary gender identities and many ethnic categories are not captured. Intersectional analysis that would reveal, for example, whether Black women fare specifically worse than either Black men or white women is not required in sufficient depth.

Only employment is covered. LL 144 covers only AI tools used in employment decisions. AI systems used in credit decisions, housing, healthcare, education, and public benefits — all of which have significant potential for bias-based harm — are not covered.

Only AI hiring is covered, not other employment decisions. LL 144 covers hiring and promotion decisions but not performance management, scheduling, or termination decisions, even though AI is increasingly used in all of these contexts.

Self-definition of what is covered. The AEDT definition has significant ambiguity, and some employers have defined their tools as outside the law's scope without regulatory review of that determination.

No validity requirement. EEOC's Uniform Guidelines require that selection procedures be valid predictors of job performance. LL 144 does not require validity evidence — only that selection rates not differ dramatically by demographic group. A tool that screens out all candidates at equal rates would satisfy LL 144 while satisfying no plausible employment objective.

The Gaming Problem

Any audit requirement that specifies what metrics will be measured creates the possibility of optimization to those metrics — gaming the audit. If LL 144 requires that selection rates not differ by more than the four-fifths rule, tools can be modified to produce technically compliant selection rates without addressing underlying bias in how candidates are evaluated. For example, a tool that evaluates resumes and produces a biased ranking could be calibrated to produce equal top-line selection rates across demographic groups by adjusting the threshold applied to each group — a form of demographic parity adjustment that may not reflect genuine removal of bias.

This gaming problem is not unique to LL 144; it is inherent in any audit requirement that specifies a limited set of metrics. More robust auditing would examine multiple metrics, assess validity, and evaluate outcomes — reducing the opportunity for gaming. This requires more sophisticated audit requirements than LL 144 currently provides.

Influence on Other Jurisdictions

Despite its limitations, LL 144 has had significant influence:

Illinois enacted the Artificial Intelligence Video Interview Act in 2020, requiring employers that use AI to analyze video interviews to notify candidates and to report demographic data on interviewees to the Department of Commerce. Several additional Illinois AI employment bills have been introduced.

Colorado enacted SB 205 in 2024, a comprehensive AI non-discrimination law covering AI systems used in employment, education, lending, housing, healthcare, and insurance. The law requires developers and deployers of AI systems to take reasonable care to avoid algorithmic discrimination and requires deployers to notify affected individuals when a high-risk AI system is used in a consequential decision.

California has considered multiple AI audit bills, including proposals to extend bias audit requirements to a broader range of consequential AI applications.

Federal proposals. The EEOC has signaled that it views existing employment discrimination law as requiring ongoing validity evidence for AI hiring tools, effectively mandating a form of audit without new legislation.

The EU AI Act, which designates AI systems used in employment decisions as "high-risk" requiring conformity assessment, represents a more comprehensive audit requirement than LL 144 and will apply to tools used by EU employers — including US-based companies with EU operations.

Analysis and Implications

LL 144 demonstrates both the promise and the difficulty of mandatory AI bias auditing. Its promise: it established the principle that AI systems in employment decisions are subject to independent review and public disclosure. Its difficulty: narrow scope, limited audit standards, no validity requirements, and inadequate enforcement resources have produced early compliance that is uneven and audit quality that is variable.

The law's experience suggests several design principles for more effective mandatory AI auditing:

  1. Specify audit methodology standards, not just required metrics, to reduce gaming and variation in audit quality.
  2. Require auditor credentialing or establish minimum auditor qualifications to ensure audit quality.
  3. Require validity evidence alongside disparate impact analysis, to ensure that tools are actually predictive of legitimate job qualifications.
  4. Include remediation requirements when significant adverse impact is found, rather than treating disclosure as sufficient.
  5. Provide adequate enforcement resources to investigate non-compliance, including proactive investigation rather than complaint-driven enforcement only.
  6. Extend coverage to the full employment decision lifecycle and to other consequential domains.

Discussion Questions

  1. LL 144 requires disclosure of bias audit results but does not require employers to remediate identified disparities. Is disclosure alone a sufficient accountability mechanism? What additional requirements would you add?

  2. The absence of auditor credentialing requirements under LL 144 has produced variable audit quality. What credentials, if any, should be required of AI bias auditors, and who should establish and enforce those credential requirements?

  3. The four-fifths rule (requiring selection rates to be at least 80% of the highest-scoring group's rate) was developed for traditional employment selection procedures in the 1970s. Is it an appropriate metric for AI-based selection tools? What modifications, if any, would make it more appropriate?

  4. Several employers have argued that certain AI tools they use do not qualify as AEDTs under LL 144's definition, effectively self-exempting from the law's requirements. How should the DCWP respond to employer self-definition? What enforcement mechanisms are available?

  5. Compare LL 144's requirements to the EU AI Act's requirements for AI systems used in employment decisions. What are the most important differences? Which approach is more likely to produce genuine improvements in AI fairness in employment?