Overview
Readers of DCI blogs know all about New York City’s (NYC’s) Local Law 144 (LL-144). Briefly, this law applies when an Automated Employment Decision Tool (AEDT) is used to hire or promote individuals for a job located in NYC or (remotely) associated with an office in NYC. The core components are (1) an annual bias audit, including adverse impact ratios, conducted by an independent auditor with results published on an organization’s website, and (2) applicant notifications, including that the selection process uses an AEDT, the job qualifications and characteristics it evaluates, the type and source of data used, the data retention policy, and instructions for requesting an alternate process or accommodation.

Although there is still ambiguity and nuances around coverage, this law likely applies not only to selection procedures using artificial intelligence (AI) methods (e.g., machine learning, natural language processing) but also to non-AI selection procedures that involve complex, automated algorithms. As we approach the one-year anniversary of enforcement and “round two” of bias audits for some, our goal with this blog is to (1) describe some potential unintended consequences of bias audit analyses, and (2) consider what independent auditor expertise is most useful for bias audits.

NYC LL-144 Bias Audit Analyses and Unintended Consequences
Recent reports suggest that there has been little actual NYC LL-144 enforcement.¹ However, some bias audit results have been published as the law requires, and these analyses, if done incorrectly or without consideration of broader implications, may increase an organization’s risk. One example is when an auditor applies inappropriate aggregation methods, which may distort the results² and lead to inappropriate interpretations or flawed organizational responses. For example, results distorted by incorrect data structuring and analysis methods may mask meaningful disparities that exist, or may artificially produce disparities that do not exist (if conducted at appropriate units of analysis). As a result, organizations may make important talent decisions based on mis-specified analyses that do not map onto the operational reality of an AEDT’s use.

Further, the NYC LL-144 publication requirements for bias audit reports contain every piece of information necessary to compute other adverse impact metrics, including statistical significance tests. Since bias audit results are publicly available and are tracked, anyone can examine the adverse impact results and calculate statistical significance tests, which will routinely be statistically significant for large, aggregated samples.³,⁴ Finally, since numerous other laws and regulations involve adverse impact analyses, it may be difficult for employers, vendors, or their auditors to argue against this type of inappropriate, over-aggregated analyses when they themselves have published a report where this was done.

This is just a single example of where inexperience with adverse impact analyses can lead to inappropriate analyses and second-order risk consequences. Bias audits require careful planning, and the requisite knowledge and expertise, to conduct analyses that comply with NYC LL-144, map onto the actual use of an AEDT, and minimize any unintended consequences outside of this law.

What Expertise Is Important for Auditors to Possess?
Per NYC LL-144, independent auditors may not have: 1) been involved developing the AEDT, 2) been employed by the organization or vendor, or 3) a financial interest in the organization or vendor. However, the law is silent on auditor expertise requirements, such that if these exclusions do not apply, in theory anyone can assert themselves as an independent auditor.

Given this, one may wonder, “what makes someone qualified to do high-quality bias audits?” Bias audits are greatly enhanced if the auditor has knowledge and expertise in:

1.  The legal and professional frameworks that have been associated with employee selection for decades. This benefits not only how the audit is conducted, but also provides perspective on the ripple effects that may stem from the audit and how to approach compliance with the myriad of existing and emerging federal and state/local regulations. Federal agencies have repeatedly asserted that the long-standing legal frameworks for evaluating employee selection procedures for unlawful discrimination are applicable to AI-based tools in the same way as non-AI-based tools.⁵ State and local AI laws represent only a fraction of the relevant employment laws, and compliance with one law does not mean compliance with another. Engaging an independent auditor with this expertise provides a more comprehensive, forward-facing perspective that goes beyond a single local law.
2. The fundamentals and nuances of adverse impact analyses that allows for appropriate planning, data structuring, and analyses aligned with the specific use case, in a manner that is consistent with accepted practices in adverse impact measurement.⁶ On the surface, a bias audit may appear straightforward. However, there are intricate complexities and considerations involved in conducting high-stakes adverse impact analyses that are aligned with contemporary practices and go beyond the text of NYC LL-144 (e.g., evaluating multiple statistical and practical indicators of adverse impact, considering unit of analysis, sample sizes, data aggregation methods). This is particularly relevant in vendor-level bias audits⁷ which involve data from multiple employers and in different use contexts. Experience in conducting these analyses across a wide variety of employment outcomes and situations allows an auditor to properly structure and analyze data, explain the results and implications, and prioritize next steps. This allows an organization to not only comply with NYC LL-144, but also proactively mitigate potential second-order risk from other laws or regulations.

The Uniform Guidelines on Employee Selection Procedures (1978), which federal agencies use as the evaluative framework for all selection tools challenged under Title VII and Executive Order 11246, include a much wider range of topics for which auditors should be well-versed. For example, the Uniform Guidelines call for consideration of job-relatedness evidence, as well as consideration of fairness from multiple perspectives. While NYC LL-144 is silent on these and other issues, existing and emerging laws and regulations are not. It is beneficial if auditors possess expertise in selection procedure design, measurement principles, and validation strategies to evaluate key factors (e.g., what is being measured, what is being predicted, how it is operationally used, the jobs for which it is used, applicant sample characteristics). Consideration of this framework is useful both for legal defensibility of a selection procedure and for evaluating if it is adding the intended value to the process.

Auditors with deep expertise in these areas are capable of considering the broader landscape and going beyond simply checking the box for NYC LL-144.⁸ This is particularly useful given the patchwork of state and local laws on the horizon. These areas of expertise become even more critical when conducting vendor-level bias audits as the analyses required and factors that should be considered are significantly more complex and nuanced.

Given the required areas of expertise for bias audits, Industrial/Organizational (I/O) psychologists are particularly well suited to conduct this type of work. I/O psychology is a broad-spanning field that involves the application of scientific principles and procedures to understand and improve the workplace—within I/O psychology, some specialize in employee selection and have deep expertise in these areas critical for conducting high-quality bias audits.

Summary and Encouragement to … “Choose Wisely”
This is still early on in regulating the use of AI in employee selection. DCI’s State Legislation Tracker provides an overview of laws in this space that have been proposed and enacted, some of which may incur greater organizational burden. Many of the proposed laws include the need for algorithmic explainability and transparency, applicant notifications, and adverse impact analyses (e.g., annual “bias audits”). Some of these consider job-relatedness (validity), and the provision of an alternative, non-AEDT selection procedure for requesting applicants. Coupled with the significant increase of federal activity and guidance for use of AI in selection and their stated applicability of long-standing frameworks, this clearly underscores the value of partnering with seasoned experts to perform audits. It is more important than ever to safeguard against unintended risk when complying with different regulations and DCI I/O psychologists stand ready to help organizations navigate this critical audit work.

DCI will continue to monitor developments.

References

¹ https://citizensandtech.org/research/2024-algorithm-transparency-law/ and https://www.law360.com/employment-authority/articles/1808951.

² Morris, S. M., Dunleavy, E. M., & Lee, M. (2017). Many 2×2 tables: Understanding multiple events in adverse impact analyses. In S. B. Morris & E. M. Dunleavy (Eds.), Adverse impact analyses. Data statistics and risk (pp. 147-168). Routledge press.

³ Oswald, F. L., Dunleavy, E. M., & Shaw, A. (2017). Measuring practical significance in adverse impact analysis. In S. B. Morris & E. M. Dunleavy (Eds.), Adverse impact analyses. Data statistics and risk (pp. 92-112). Routledge press.

⁴ Jacobs, R., Murphy, K. & Silva, J. (2013). Unintended Consequences of EEO Enforcement Policies: Being Big is Worse than Being Bad. Journal of Business and Psychology (28).

⁵ 2024 OFCCP guidance re AI; 2023 EEOC guidance re AI; 2022 EEOC guidance re AI and ADA; 2022 DOJ guidance re AI and ADA; 2024 Joint statement re equal opportunity laws and AI; 2024 OMB Memo re AI; Executive Order 14110

⁶ Morris, S. B. & Dunleavy, E. M. (2017). Adverse impact analysis: Understanding data, statistics, and risk.

⁷ “Vendor-level bias audits” are cross-employer adverse impact analyses that may be used by vendor customers in some situations (e.g., prior to AEDT implementation since no employer historical data are available; employers who have their data included in the vendor audit analyses).

⁸ Note that this list of critical areas of expertise for conducting NYC bias audits does not include deep knowledge of artificial intelligence or machine learning methods. While on the surface this may seem relevant, in actuality this expertise has no meaningful relevance to NYC bias audits, and has limited relevance for the legal frameworks related to adverse impact analyses.