The Proposed Amendments to California’s Employment Legislation Regarding Automated-Decision Systems

In the wake of New York City’s bias audit mandate for automated employment decision tools (Local Law 144), states like California have introduced legislation to address the potential harms that can result from using artificial intelligence or automated systems in hiring. The legislation targets the use of automated systems or tools in the workplace and employment-related decisions, where bias or discrimination against protected classes is of particular concern. To address bias and discrimination, California has proposed modifications to its employment regulations to extend non-discrimination practices to automated-decision systems. The modifications were first proposed in March 2022  for public workshop and then updated in July 2022 and February 2023. In this blog post, we outline the key things to know about this legislation, focusing on the second and third versions of the text.

Who does California’s proposed legislation on automated-decision systems in hiring affect?

Employers with five or more employees are subject to this regulation, which includes employees outside of California, but they are not covered by the act protections if the prohibited activity did not occur in California. Vendors, or agents, acting on behalf of an employer are also considered an employer under the regulation.

What is the definition of automated-decision systems according to the proposed amendments to California’s employment legislation?

An automated-decision system (ADS) is a computational process, including one derived from machine learning, statistics, or other data processing or artificial intelligence techniques, that screens, evaluates, categorises, recommends, or makes or facilitates employment-related decisions. This includes systems used to direct job advert targeting, screening resumes, analysis of facial expressions, word choice, and voices in video interviews, computer-based tests and game-based assessments, and the measurement of constructs such as personality, aptitude, cognitive ability, or cultural fit through automated tests.

The most recent version of the text also clarifies that an ADS is not word processing software, spreadsheet software, and map navigation systems.

What is Automated-Decision System (ADS) data?

Referred to as machine-learning data in the first version of the text, automated-decision system (ADS) data is that which is used to develop or apply machine learning, algorithms, or artificial intelligence as part of an ADS. This includes training data, data provided by applicants or employees or information about applicants or employees that has been analysed by an ADS, and data produced by an ADS.

Under the proposed amendments to California’s employment legislation, it is prohibited to use automated-decision systems that limit, express a preference for, or screen out applicants based on protected characteristics or proxies of characteristics unless there is an affirmative defence for using this criterion.

Artificial intelligence and machine learning

Artificial intelligence is a machine learning system that can make predictions, recommendations, or decisions that influence real or virtual environments when given a set of human-defined objectives. Typically, the developer relies partly on the computer’s analysis of data to determine the criteria to use to make decisions.  

Machine learning is an application of artificial intelligence where a system can automatically learn and improve based on data or experience without the need for explicit programming, with systems learning from analysis and automatically applying it to future calculations or tasks.  

Using automated-decision systems to make decisions based on criminal history

An applicant must be notified if an employer plans to withdraw an employment offer based on the applicant’s criminal history, and the decision to withdraw the offer involves using an ADS.  The applicant must be provided with a copy or description of any report or information from the operation of the automated decision system, related data, and assessment criteria used as part of an automated -decision system resulting in the withdrawn employment offer.

Restrictions on conducting medical or psychological exams of an applicant

Before an offer is extended to an applicant, procedures to conduct a medical or psychological exam, including by using an ADS, are not permitted. This includes using tests of optimism, emotional stability, extraversion, intensity, and tests of mental ability to make a medical or psychological enquiry.

Which characteristics are protected under the proposed amendments to California’s employment legislation?

The legislation prohibits discrimination based on characteristics including race, national origin, gender, accent, English proficiency, immigration status, driver's license status, citizenship, height or weight, national origin, sex, pregnancy or perceived pregnancy, religion, and age unless they are shown to be job-related for the position in question and are consistent with business necessity.

The major contribution of the modifications, other than defining the terms associated with ADSs, is to explicate that these prohibitions also apply to the use of ADSs. However, the most recent version introduces some novel obligations, including providing evidence of bias testing and other similar efforts to avoid unlawful discrimination. This includes information on the quality, recent, and scope of these efforts, as well as the results of the testing and any responses.

How long does a company need to retain data for under the proposed amendments to California’s employment legislation on ADS?

Anyone involved in the advertisement, sale, provision, or use of a selection tool, including an ADS, must retain records of the assessment criteria used for each employer or entity that is provided with the tool for at least 4 years after the tool is last used.

Prior to the modifications, the retention period was only two years. The most recent version of the modifications, however, extends these requirements; records must be kept of the training set, modelling, assessment criteria, and outputs from the ADS for at least four years from the last date the ADS was used by the employer or other covered entity.

Are companies required to have an audit for the proposed amendments to California’s employment legislation on ADS?

While this legislation does not require that employers must conduct bias audits of their automated-decision systems like legislation in New York City does, many companies are opting to conduct an audit on the automated-decision systems to identify bias and assess and manage risks. Having an audit and taking steps to mitigate any bias revealed as part of the audit can help reduce harm and legal risks.

Schedule a demo with us and find out about how Holistic AI can help you manage risks of your automated-decision systems.

‍

Written by Ayesha Gulley, Public Policy Associate at Holistic AI & Airlie Hilliard, Senior Researcher at Holistic AI.