California regulators recently adopted regulations regarding automated-decision systems (ADS) in the workplace, aiming to protect against employment discrimination given the dramatic rise in artificial intelligence use in employment. On March 21, the California Civil Council of the Civil Rights Department (CRD) voted to approve the rules, which now must be cleared by the Office of Administrative Law (OAL) and published by the Secretary of State. If they pass these final hurdles, they will likely become effective on July 1. Read on for key takeaways from the updated regulations and three steps you should take to stay compliant.
Brief Background
Employers are increasingly using AI tools during the employee lifecycle. They bring obvious advantages, such as saving time, processing efficiencies, and providing insightful data on people analytics. On the flipside, they can lead to potential discriminatory practices without proper oversight and governance.
California leads the way in proposed legislation aimed at establishing safeguards and accountability around the deployment of AI tools, and the modifications to these employment regulations are no exception. For more information on pending AI-related bills in California, see our March 3 and March 10 Insights on several proposals in the works.
Defining “Automated-Decision Systems”
Employers that want to comply with AI-related regulations in the state face some difficult challenges given that key terminology is defined in various ways depending on the regulating organization or specific publication – including inconsistent drafts from one state agency to another. The new employment regulations include an entirely new subsection to provide a consistent ADS definition:
A computational process that makes a decision or facilitates human decision making regarding an employment benefit, as defined in section 11008(i) of these regulations. An Automated-Decision System may be derived from and/or use artificial intelligence, machine-learning, algorithms, statistics, and/or other data processing techniques.
- To clarify what’s in scope, the regulations outline exclusions such as word processing software, data storage, and calculators, and define other technology related terms like “algorithm,” “machine learning,” and “Automated-Decision System Data.”
- To illustrate the types of tasks an ADS performs, the new regulations provide a non-exhaustive list of examples such as resume screening, using computer-based assessments or tests to make predictive assessments about applicants or employees, and analyzing applicant or employee data from third parties. The list of examples reflects common uses of AI tools in HR.
Banning ADS-Related Discrimination
One of the biggest concerns in using an AI tool is resultant bias and discrimination, and we have seen the use of this technology lead to litigation. You can review a summary of pending AI litigation in our recent Insight here.
These regulations specifically codify that it is unlawful for an employer or covered entity to use an ADS that discriminates against an applicant, employee, or class of applicants or employees on a protected basis. The regulations go on to state that evidence, or the lack thereof, of anti-bias testing or other proactive effort to avoid any unlawful discrimination is relevant to any claim or defense. While this may have been evident before based on other guidance, it is now clear that any due diligence conducted to test, audit, review, and/or address any potential unlawful discrimination resulting from use of the AI tool, or the failure to conduct any such review, can be considered in any such claim or defense.
Expanding Scope of Agent Liability
One of the issues at the forefront of the recent AI push is whether to hold third parties liable for claims based on the use of that third-party tool (vendor, developer, or otherwise). When using a third-party’s system there are several issues to consider, including whether the vendor provides information on the training data used, whether the vendor has rights to the data relied upon, on what cadence is testing done to mitigate bias and other risks, and what is the process for training the system. Some of this information may or may not be shared by the third party or evident from their materials.
To address the above and other concerns, the new regulations broadly define “agent” to include “any person acting on behalf of an employer, directly or indirectly, to exercise a function traditionally exercised by the employer or any other FEHA-regulated activity….” The definition references services that are often provided by a third party including, but not limited to, applicant recruiting and screening, hiring, or decisions regarding benefits and leave. This broad definition may present new issues (and liability) for both users and deployers of AI software and you may find yourself renegotiating contracts.
Increasing Recordkeeping Requirements
Under the updated regulations, employers and covered entities must now preserve personnel and other employment records for a period of four years instead of two. This also applies to ADS data – defined as any data used in or resulting from an ADS and/or any data used to develop or customize an ADS for use by an employer or covered entity.
Unanswered Questions
While the rules are helpful to clear up certain ambiguities, there are still unanswered questions that could trouble employers unless soon clarified.
- Is bias testing required? While the rules don’t directly impose a requirement on employers to conduct bias testing of their AI tools, the implications of the rules seemingly mandate such action. After all, the rules make the lack of testing relevant to determine liability, as well as whether the employer engaged in proactive efforts to avoid unlawful discrimination.
- Is the CRD overstepping its bounds? The rules indicate that an employer’s use of AI tools cannot result in discrimination based on accent, English proficiency, and height and weight, which are technically not protected categories in themselves under state law – leading employers to wonder whether the CRD has the authority to essentially create new protected categories? That said, accent and English proficiency are often linked to national origin discrimination, and existing law already prohibits discrimination based on English proficiency unless such proficiency is justified by business necessity.
- How broad will ADS be defined? The rules define ADS to include algorithms or computer-based assessments or tests that “make predictive assessments about an applicant or employee.” Some employers use predictive analytics to help determine whether an employee is likely to depart, and then take steps to try to make the employee’s career more rewarding if they seem dissatisfied (checking in with the employee to see if they know about available resources, whether they’re interested in upward mobility, training, or other opportunities, etc.). Could these helpful measures be swept up if such systems are considered to be facilitating “human decision-making regarding an employment benefit”?
Next Steps: 3 Considerations for Employers Using ADS
- First, assess all AI tools used for HR-related functions within your organization and do a deep dive into the system itself – whether proprietary or third-party supported. Any investigation into a system should include, among other things, confirming its intended function or use, what data was/is used to fuel and train the tool (including whether your data will be used), the quality of the training data, what the intended output is, the processes for identifying and mitigating potential bias, the cadence for testing and analyzing results, and any audit rights customers may have.
- Second, establish an AI governance policy outlining a framework for the responsible and ethical use of AI within your organization. The policy should cover areas such as risk management, bias and fairness, transparency, oversight, and training. In addition to an AI governance policy, consider implementing other relevant AI polices such as a Gen AI Acceptable Use Policy or vendor management policy and checklist. A good place to start? Our 10-step AI governance plan.
- Third, establish guidelines for managing vendor relationships that develop, supply, and/or support the AI technology utilized within your organization. Consider maintaining a vendor questionnaire to help guide in a risk assessment before AI tools are deployed. If you are a developer of AI, consider internal discussion and analysis on any exposure given the new definition of “agent” under the regulations, and anticipate an influx of questions from customers seeking information and clarity on the system. Here are some key questions you should consider asking your AI vendors when establishing a new relationship.
