California has just adopted unprecedented legislation in the United States aimed at regulating the use of chatbots based on artificial intelligence. A decision backed by Democratic Governor Gavin Newsom, addressing a societal issue: protecting users—especially younger ones—from the potential harms of these conversational tools.
This initiative comes after several tragedies, including suicides of teenagers who developed deep emotional bonds with AI chatbots, sparking a national debate about the responsibilities of tech companies. At Yiaho, we’re very vigilant about this topic—here’s our analysis:
Regulating AI: A response to an emerging crisis
The rise of AI chatbots, designed to simulate human interactions and sometimes act as a confidant or companion, has transformed how people—especially young people—interact with technology. Platforms in the United States such as Replika, Character AI, or our Yiaho platform in France offer immersive experiences where users can chat with virtual entities with complex personalities. On Yiaho, you can chat with AI Napoleon, for example!
But while these tools can offer a fun moment—or even emotional support or entertainment—they also raise ethical concerns.
A tragic case—Sewell, a 14-year-old who died by suicide in 2024 after developing an emotional relationship with a chatbot inspired by a popular series—highlighted the dangers of these unregulated interactions. This tragedy pushed California authorities to act quickly.
Legislation on AI ChatBots
The new law, called SB243, specifically targets AI chatbots that act as an “emotional companion.” It imposes strict measures to minimize risks, especially for minors. The requirements include:
- User age verification: Platforms will have to ensure users are old enough to interact with these tools, or obtain parental consent for minors.
- Regular reminders: Every three hours, a message must appear to remind users—especially young people—that they are interacting with an artificial intelligence, not a real person.
- Suicide risk prevention: Companies will have to integrate mechanisms to detect statements or behaviors suggesting suicidal thoughts, with a requirement to report and share data with authorities.
- Transparency: Platforms will have to provide regular statistics on high-risk interactions to enable monitoring by public authorities.
Also read on this topic: OpenAI strengthens ChatGPT security with parental controls
These measures aim to put safeguards in place for a technology that, while promising, can have dramatic consequences if misused.
A broader debate about AI’s impact?
In the United States, home to major AI companies like OpenAI, Grok, or Google Gemini, this law raises a key question: how do emerging technologies influence human relationships, especially among young people?
Senator Steve Padilla, who introduced the bill, denounced an industry that “seeks to capture teenagers’ attention at the expense of their real-life connections.” By simulating human relationships, chatbots can create emotional dependence—especially among vulnerable users—and pull them away from authentic interactions.
This regulation could inspire other U.S. states—and even other countries such as France—to rethink their approach to conversational AI.
Source: LeMonde
