On January 13, 2025, California Attorney General Rob Bonta issued a legal advisory (the Advisory) providing guidance to healthcare providers, insurers, vendors, investors, and other healthcare entities that develop, sell, and use artificial intelligence (AI) about their obligations under California law, including under the state’s consumer protection, anti-discrimination, and patient privacy laws. The Advisory highlights how California laws:
- prohibit unlawful, unfair, and fraudulent business practices, including the marketing of AI that does not comply with state or federal law;
- prohibit AI from practicing licensed professions such as medicine;
- prohibit discrimination based on certain protected traits through the use of AI; and
- protect the use and disclosure of patient information, including information that is accessed or disclosed by AI.
While acknowledging the broad potential of AI to help improve patient and population health, reduce administrative burdens, and facilitate appropriate information sharing, the Advisory highlights several risks posed by the adoption of AI in healthcare.
Action Items
Healthcare-related entities that either develop and sell or use AI systems should carefully review the Advisory and related California laws to ensure that such systems are lawfully designed and implemented. At a minimum, we encourage healthcare-related entities to take the following steps:
- Implement Risk Identification and Mitigation Processes
- Development Information: Developers should assess how their AI systems were developed, including the data used for training.
- Diligence and Risk Assessment: All healthcare-related entities should conduct due diligence to evaluate AI systems for potential risks of noncompliance with California law.
- Mitigate Risk: If any compliance issues are identified, use of the noncompliant technology should be limited until such compliance issues can be addressed.
- Monitor and Validate
- Ensure that AI systems are regularly tested, validated, and audited to ensure that their use is safe, ethical, and lawful, and reduces human error and biases.
- Train Staff on Proper Usage
- Train staff on the proper use of clinical algorithms and other AI-powered tools to ensure they are applied appropriately. Training should also focus on identifying and addressing potentially adverse outcomes caused by these tools.
- Be Transparent
- Be transparent with patients about whether patient information is being used to train AI and how providers are using AI to make decisions affecting health and healthcare.
Examples of Potentially Unlawful Uses of AI in Healthcare
The Advisory suggests that the following uses of AI in healthcare may be unlawful in California:
- using generative AI to draft patient notes, communications, or medical orders that include erroneous or misleading information, including information based on stereotypes relating to race or other protected classifications;
- determining patient access to healthcare using AI that makes predictions based on patients’ past healthcare claims data, resulting in disadvantaged patients or groups that being denied services on that basis;
- double-booking a patient’s appointment, or creating other administrative barriers, because AI systems predict that the patient is the “type of person” more likely to miss an appointment; and
- conducting cost-benefit analyses of medical treatments for patients with disabilities using AI systems that are based on stereotypes that undervalue the lives of people with disabilities.
Further, on February 10, 2025, AB 489 was introduced as the latest California bill aimed at regulating AI, addressing growing concerns that AI-generated communications may mislead or confuse patients into thinking that they are interacting with a licensed healthcare professional.
California’s Consumer Protection Laws
Several California consumer protection laws apply to the use of AI in healthcare, including California’s Unfair Competition Law and professional licensing laws.
Unfair Competition Law
California’s Unfair Competition Law (UCL) prohibits unlawful, unfair, and fraudulent business practices, and includes protections against false advertising and anticompetitive practices.1 Practices that deceive or harm consumers are covered under the UCL, and include the creation, marketing, or dissemination of an AI system that does not comply with other state laws, including civil rights, privacy, false advertising, and competition laws.
As an example, if a company uses an AI tool to submit inaccurate or upcoded (i.e., fraudulent) claims for reimbursement to Medi-Cal, such use of AI would violate laws governing Medi-Cal, and thereby violate the UCL as well. Such applications are discussed in more detail in a related legal advisory issued by the California Attorney General.
Professional Licensing Laws
California’s professional licensing laws provide standards that licensed professionals must meet to obtain and maintain a professional license, many of which apply to the use of AI in healthcare. For example, only human medical professionals can practice medicine in California, and such professionals cannot delegate the practice of medicine to AI.2The Advisory discusses how the use of AI to make decisions about medical treatment, or to override licensed providers’ medical decisions, may violate such laws in addition to California’s Unfair Competition Law and prohibition on the corporate practice of medicine.
For instance, a lay telehealth company cannot use an AI agent to render patient diagnoses and issue treatment decisions. Instead, diagnosis and treatment should be provided by a human medical professional employed by a professional medical corporation through a corporate practice of medicine-compliant structure. To incorporate AI into such an arrangement, the professional medical corporation could use the lay corporation’s AI technology to help inform medical decisions, so long as a licensed medical professional retains final decision-making authority.
California’s Anti-Discrimination Laws
California law prohibits discriminatory practices by entities receiving state support (e.g., healthcare entities that receive reimbursement from Medi-Cal) on the basis of sex, race, color, religion, ancestry, national origin, ethnic group identification, age, mental disability, physical disability, medical condition, genetic information, marital status, or sexual orientation.3 The Advisory discusses how this prohibition extends to the use of AI systems, and provides the following example:
“An AI system that makes less accurate predictions about demographic groups of people who have historically faced barriers to healthcare (and those whose information may be underrepresented in large datasets), though facially neutral, may have a disproportionate negative impact on members of protected groups.”
The Advisory warns healthcare entities that such disparate impact discrimination is prohibited by California’s anti-discrimination mandates,4 and that, although a policy or tool may be facially neutral, healthcare entities may not ignore or avoid data regarding inequity relating to protected classifications. Instead, such entities may be required to take steps to overcome the effects of past discrimination and/or prevent new discrimination.5
For example, California’s anti-discrimination laws would likely prohibit the use of generative AI to draft patient notes, communications, or medical orders that include erroneous or misleading information, including information based on stereotypes relating to race or other protected classifications. Therefore, healthcare entities should be mindful of anti-discrimination laws when choosing to implement an AI system, and developers and vendors of such systems should be mindful that healthcare entity customers will likely choose to implement AI systems that can be shown to be nondiscriminatory and should design and test their AI systems accordingly.
The California Attorney General is actively investigating potential discrimination by commercial decision-making platforms used by California healthcare entities. Additionally, the U.S. Department of Health and Human Services Office for Civil Rights recently issued a “Dear Colleague” letter with guidance on how certain healthcare entities must ensure nondiscrimination of AI and other emerging technologies in healthcare, though the Trump administration is not likely to actively enforce such protections.
California’s Privacy Laws
The Advisory outlines how several privacy laws in California provide heightened protections for consumers, on top of federal health privacy laws. For example, California’s Confidentiality of Medical Information Act and Information Practices Act (CMIA) protects the use and disclosure of patient health and medical information, adding heightened protections for sensitive information such as behavioral health and reproductive healthcare information.6The CMIA requires healthcare providers to obtain patient consent before disclosing medical information, and the Advisory notes that dark patterns (user interfaces designed or manipulated to subvert or impair user autonomy, decision making, or choice), including those generated by AI, cannot be used to obtain such consent.7
Other patient privacy laws, including the Genetic Privacy Information Act (GPIA),8Patient Access to Health Records Act,9Insurance Information and Privacy Protection Act (IIPPA),10and California Consumer Privacy Act (CCPA),11may also be implicated by the use of AI in healthcare. For example, the GPIA prohibits disclosure of genetic test results without patient permission,12while the IIPPA grants rights to healthcare consumers for determining reasons for adverse insurance decisions.13The CCPA imposes numerous requirements on non-HIPAA covered entities, such as direct-to-consumer health apps. Taken together, developers, sellers, and users of AI systems that fail to take adequate steps to ensure patient privacy and autonomy rights for California residents may be found in violation of California’s existing privacy laws.
Federal/State Divide in AI Regulation and Oversight
While the Trump administration has indicated the federal government will be hands-off when it comes to AI regulation, oversight, and enforcement (see “Removing Barriers to American Leadership in Artificial Intelligence”), California is doubling-down in this area. California has recently enacted 17 bills covering the use and regulation of AI technology, including a law requiring covered providers to provide an AI detection tool (SB 942), a law requiring certain providers that use generative AI to provide certain disclosures to patients (AB 3030), and a law requiring developers of generative AI systems to make certain disclosures about training data (AB 2013). Additionally, the California Privacy Protection Agency initiated a formal automated decision-making technology rulemaking, with the comment period closing on February 19, 2025.
Other states are taking an active stance towards AI as well. The Texas AG has already brought one AI-related enforcement action against Pieces Technologies, alleging that the company deployed its AI healthcare technology products at several Texas hospitals after making a series of false and misleading statements about their accuracy and safety. Utah’s AI Policy Act, which went into effect in May 2024, requires disclosures when consumers are interacting with certain AI systems. Colorado’s AI Act, which goes into effect February 1, 2026, includes transparency, governance, and other requirements on high-risk AI systems, defined as those that make, or are a substantial factor in making consequential decisions. And Massachusetts has issued an advisory clarifying that state consumer protection laws apply to AI developers and users.
[1] Bus. & Prof. Code, § 17200 et seq.
[2] Bus. & Prof. Code, Division 2
[3] Cal. Gov. Code, §§ 11135, 12900; Cal. Code Regs., tit. 2, §§ 14000, 14020; Cal. Civ. Code, § 51(b); Cal. Ins. Code § 1861.03; Cal. Health & Saf. Code § 1317.3.
[4] Cal. Code Regs., tit. 2, § 14027(b)(3).
[5] See, e.g., Cal. Health & Saf. Code § 131019.5.
[6] Cal. Civ. Code, §§ 56.10, 56.26, 56.101, 1798.25.
[7] Cal. Civ. Code, § 56.18(b)(6).
[8] Cal. Civ. Code, §§ 56.17, 56.18.
[9] Cal. Health & Saf. Code § 123110.
[10] Cal. Ins. Code § 791.
[11] Cal. Civ. Code § 1798.100.
[12] Civ. Code, §§ 56.17, 56.18, et seq.
[13] Ins. Code, § 791.
