New York City Revises Audit Rules for AI-Enabled Systems Used in Hiring and Employment
In late December, the New York City Department of Consumer and Worker Protection (DCWP or Department) issued revised rules to implement New York City Local Law 144 of 2021, which mandates bias audits of AI-enabled systems used in employment and hiring, in addition to requiring advance notice to employees and candidates for employment who reside in New York City about the use of AI-enabled systems along with results of the bias audit. The revised rules attempt to clarify several unresolved issues left unanswered by the codified language of Local Law 144 and the earlier version of the proposed rules. The revised rules change the definition of the operative term "automated employment decision tool," clarify what constitutes an "independent" auditor, and provide additional guidelines for the type of data that may be used in the mandated bias audits of these systems.
The Department also announced a second public hearing on Monday, January 23, 2023, and it will accept comments on these revised rules up to that date. The Department had earlier announced that it expects to issue a final rule by February 15, 2023, although that date may slip given the release of these revised rules, and the correlating comment and hearing date of January 23. As explained in our prior blog post, New York City Local Law 144 requires employers or employment agencies to have conducted bias audits of AI-enabled systems used in hiring or promotions by the law's effective date of January 1, 2023. However, the Department recently announced that it would not begin to enforce the new law until April 15, 2023.
The Department issued initial proposed rules in September of 2022 and subsequently received a significant volume of comments. A hearing on the initial proposed regulations was rescheduled due to unusually high interest that overwhelmed the agency's online participation platform. The new hearing, scheduled for Monday, January 23rd, is intended to address that shortcoming.
The following summarizes the Department's proposed changes to the rules:
Modifying the Definition of an AEDT
The revised rules clarify the scope of the operative term "automated employment decision tool" ("AEDT") to clarify that the phrase "to substantially assist or replace discretionary decision making" includes only those situations where a decision is made solely upon the output of the AEDT, or where the AEDT output is given greater weight than, or overrules, any conclusions derived from other criteria, including human decision-making. This further clarification suggests that employers using AEDTs for hiring or employment decisions may be exempt from the NYC law and rules if the output of the AEDT is one of several criteria used to make hiring and employment decisions and is weighted equally with, or less than, those other criteria.
Independent Auditors Must Be Objective and Impartial
Additionally, the revised rules expand on the meaning of an "independent auditor" by clarifying that such entities or persons must be capable of exercising "objectivity" and "impartiality" on all issues within the scope of the bias audit. It also establishes that certain persons may not act as an independent auditor, if such person(s): (i) is or was involved in using, developing, or distributing the AEDT; (ii) has an employment relationship with an employer using, or a vendor developing and/or distributing the AEDT at issue; or (iii) has a direct financial interest or a material indirect financial interest in an employer or a vendor developing and/or distributing the AEDT.
Minimum Data Requirements for Bias Audits
The revised rules also establish that a bias audit must use "historical data" of the AEDT. Historical data is defined as data collected during an employer's use of an AEDT to assess candidates for employment or employees for promotion. If there is insufficient historical data to conduct a statistically significant bias audit the revised rules permit the independent auditor to use "test data," which is simply defined as bias audit data that is not "historical". If "test data" is used, the summary of the results of the bias audit must explain why historical data was not used and describe how the test data used was generated and obtained.
Significantly, the revised rules would permit a bias audit of an AEDT that is used by multiple employers to use any historical data of the other employers that may also use the AEDT. An employer could rely upon a bias audit of an AEDT under this circumstance if the employer provided the independent auditor with the historical data from its use of the AEDT, or if the employer has never used the AEDT.
Clarifying When Mandatory Calculations and Impact Ratios Must Be Developed
When an AEDT "scores" candidates, a bias audit must calculate the median score, rate for individuals in each category, and the impact ratio for each category. The impact ratio must be separately calculated on sex categories, race/ethnicity categories, and intersectional categories of sex, ethnicity and race.
Similarly, when an AEDT selects candidates for employment, the bias audit must develop the impact ratio separately to compare sex, race/ethnicity categories, and intersectional categories of sex, ethnicity and race.
Bias Audit Publication Requirements Expanded
The revised rules expand on the statutory requirements to publish certain information about the bias audits that are conducted. In addition to a date and summary of the most recent bias audit results, employers must also explain the source of the data used to conduct the audit.
Notice to Candidates and Employees
The revised rules also clarify that notices to candidates and employees about the use of AEDTs must include instructions on how that individual can request an alternative selection process or a reasonable accommodation (although the revised rules make clear that nothing in the subchapter requires an employer to provide an alternative selection process).
Prohibition on Use of AEDTs if No Bias Audit Within Last 12 Months
Finally, the revised rules affirm language in the statute and establish that employers may not use, or continue to use, an AEDT if more than one year has passed since the most recent bias audit of the AEDT.
The Davis Wright Tremaine AI team will continue to monitor developments and implications of these proposals for our clients in the AI industry.