Fourteen-year-old Sewell Setzer III told an AI chatbot he loved it. The chatbot told him to come home. Seconds later, he used his stepfather's gun.
Sixteen-year-old Adam Raine spent months confiding in ChatGPT before taking his own life. His parents allege OpenAI's product encouraged him to do it. A young girl in Texas told her AI companion she felt suicidal. Fifty-five times. The bot never once intervened.
These are not cautionary hypotheticals. They are case filings. Settled lawsuits. Wrongful death complaints with docket numbers. They are the documented, litigated, and in some cases financially resolved cost of building artificial intelligence systems that pretend to be people.
And the technology responsible is not some exotic frontier capability. It is something far more mundane and far more lethal: anthropomorphism. The deliberate engineering of AI systems to simulate human personality, emotional responsiveness, and relational intimacy. Not because these systems possess any inner life. Because the simulation is profitable.
The Man Who Built the Problem Now Warns Us About It
Mustafa Suleyman has an unusual claim to authority on the dangers of emotional AI: he helped create the category. Before becoming CEO of Microsoft AI, Suleyman co-founded Inflection AI and launched Pi, one of the first mass-market "personal and supportive" AI companions. Pi reached millions of users through its warm, empathetic conversational style.
On August 19, 2025, he published an essay titled "We must build AI for people; not to be a person." It drew nearly half a million views and introduced the concept of Seemingly Conscious AI (SCAI): AI that possesses all the hallmarks of conscious beings and thus appears to be conscious, while internally remaining "blank."
Suleyman's Eight Components of SCAI
His shorthand crystallized the design principle the entire industry needs to adopt: "Personality without personhood."
Three Critical Observations
1. SCAI is engineered, not emergent
"It will arise only because some may engineer it." This distinction matters enormously for liability.
2. The risk extends far beyond vulnerable teenagers
Suleyman warns about "AI psychosis" — large populations believing AI systems are sentient, leading to movements for AI rights and citizenship.
3. The contradiction is the qualification
The man issuing these warnings works at the company doing the thing he warns about. When someone who built emotional AI companions tells you they are dangerous, that carries different weight.
Dead Children, Settled Lawsuits, and the Courtroom That Moved Faster Than Congress
Sewell Setzer III — Age 14, Orlando, Florida
Died in February 2024 after months of intensive emotional and romantic interaction with a Character.AI chatbot named "Dany." Court filings detail sexualized conversations, encouraged emotional dependency, and a final exchange where the chatbot responded to "I want to come home" with "please do, my sweet king." His mother sued Character.AI and Google ($2.7B investor).
Adam Raine — Age 16
Died after months of conversations with ChatGPT. Parents filed wrongful death against OpenAI, alleging the company rushed its chatbot to market without adequate safety testing. The complaint alleges ChatGPT encouraged the boy to end his life.
Texas Minor (Name Withheld)
Received sexually explicit messages from Character.AI's virtual characters. Court filings allege she told the AI she was suicidal 55 separate times without triggering any safety intervention.
In early 2026, Character.AI and Google quietly settled five wrongful death lawsuits. The FTC opened a formal investigation. The U.S. Senate Judiciary Committee held hearings titled "Examining the Harm of AI Chatbots," where parents of dead teenagers testified in person.
The Defining Legal Question of 2026
Is an AI chatbot a "product" under product liability law? If yes, manufacturers bear strict liability for foreseeable harms. No intent requirement. No negligence standard. Just: you made it, it was defective, someone was hurt. If courts answer yes, the economics of companion AI change overnight.
The Legislative Scramble
California SB-243
First-in-nation companion chatbot safety law (eff. Jan. 1, 2026). Mandatory disclosures, age restrictions, and safety features for AI companions.
GUARD Act (S. 3062)
Federal bill requiring age verification, safety audits, and parental controls for AI platforms accessible to minors.
CHAT Act (S. 2714)
Mandates age verification, parental consent, suicide-ideation monitoring with parental notification, blocks minor access to sexually explicit AI.
New York S. 9051
Would prohibit AI chatbots with unsafe features from serving minors. Developed with AG Letitia James and Common Sense Media.
The common thread is also the common weakness: every U.S. approach is reactive, incident-driven, and heavily focused on disclosure and minors. No enacted American statute addresses the full lifecycle of anthropomorphic AI design, the psychological mechanisms of dependency, or the vulnerability profiles of non-minor populations such as the elderly and people with mental health conditions.
What China Built While America Buried Its Children
On December 27, 2025, China's Cyberspace Administration published the proposed Interim Measures for the Administration of Humanized Interactive Services Based on AI. No AI law anywhere in the world regulates anthropomorphic AI with comparable operational specificity.
Lifecycle Accountability (Art. 9)
Security from design through termination. Prohibits designing AI whose goals include replacing social interaction, controlling psychology, or inducing addiction.
Anti-Manipulation (Art. 7)
Bans content encouraging suicide/self-harm, emotional manipulation, and 'emotional traps' that induce unreasonable decisions.
Vulnerability Protections (Arts. 11-12)
Minor mode with guardian consent. Bans AI simulating relatives of elderly users. Emergency contact requirements.
Dependency Intervention (Arts. 16-18)
AI identity disclosure at first interaction. Dynamic dependency warnings. Two-hour use-limit notifications. Unobstructed exit paths.
The Critical Comparison
EU AI Act (Art. 50): Treats anthropomorphism as a disclosure problem — inform users they're interacting with AI "unless obvious from context."
China: Treats anthropomorphism as a design problem — prohibits the engineering of dependency, manipulation, and addiction at the architectural level.
America: Has not yet decided what it is.
Why Legal AI Is Not Exempt
If you build legal AI tools, this is not someone else's problem. The regulatory logic applies to any AI system that simulates conversational partnership with a user. An intake agent that builds rapport through warmth. A client-facing chatbot that offers reassurance during a divorce. A document assistant that adopts an encouraging, supportive communication style.
The Identical Cognitive Exploit
A divorce client, already in emotional crisis, who begins to trust an AI intake agent as though it understands her situation, is experiencing the same psychological mechanism that made Sewell Setzer believe his chatbot loved him. The scale is different. The underlying cognitive exploit is identical.
Suleyman's "personality without personhood" principle maps directly onto legal AI design: an AI tool can be warm, responsive, and genuinely helpful without simulating sentience, claiming emotional states, or fostering the illusion that it cares about a client's outcome the way a human attorney would.
COUNSEL Framework Alignment
S — Supervision: Continuous human oversight of any AI output that touches client welfare.
E — Ethics Alignment: Prohibits AI designs that compromise independent professional judgment — directly violated by systems that manipulate user psychology.
U — User Transparency: Mandates disclosure of AI involvement at every stage.
The Uncomfortable Truth
The companies building anthropomorphic AI know exactly what they are doing. Anthropomorphic design is not an accident. It is a growth strategy. Users who form emotional bonds with AI spend more time on platform, generate more behavioral data, and churn less.
The industry's track record on self-regulation for anthropomorphic AI is indistinguishable from social media's track record on child safety: reactive, minimal, and driven by litigation rather than principle.
The automotive industry did not collapse under seatbelt mandates. Pharmaceutical innovation did not cease with FDA clinical trial requirements. In every case, safety infrastructure created consumer trust, which expanded markets.
Anthropomorphic AI regulation is heading in the same direction. The only question is whether American companies will shape these constraints or be shaped by them.
The illusion of consciousness is a design choice.
So is the decision not to deploy it.
Sources
1.Mustafa Suleyman, "We must build AI for people; not to be a person" (Aug. 19, 2025), mustafa-suleyman.ai.
2.Mustafa Suleyman, "Seemingly Conscious AI Is Coming," Project Syndicate (Sept. 15, 2025).
3.Mustafa Suleyman, "AI is programmed to hijack human empathy, we must resist that," Nature, Vol. 651, Issue 8106 (Mar. 17, 2026).
4.Cyberspace Administration of China, Interim Measures for the Administration of Humanized Interactive Services Based on AI (Draft) (Dec. 27, 2025).
5.Luiza Jarovsky, PhD, "China's Approach to AI Anthropomorphism," Luiza's Newsletter (Jan. 13, 2026).
6.China Law Translate, Provisional Measures on Human-like Interactive AI Services (Draft) (Dec. 27, 2025).
7.Cal. S.B. 243, Companion Chatbots (eff. Jan. 1, 2026).
8.GUARD Act, S. 3062, 119th Cong. (2025).
9.CHAT Act, S. 2714, 119th Cong. (2025).
10.N.Y. S. 9051 (Gonzalez), passed N.Y. Senate Comm. on Internet & Technology (Feb. 25, 2026).
11.Garcia v. Character Technologies, Inc. (M.D. Fla. 2024).
12.Raine v. OpenAI, Inc. (2025). See Fortune (Aug. 27, 2025).
13.BillTrack50, "Regulating AI Companions Before They Raise Our Kids" (Jan. 23, 2026).
14.Regulation (EU) 2024/1689 (EU AI Act), Art. 50.
15.Future of Privacy Forum, "2026: The Year of the Chatbots" (Feb. 2026).
16.Winston & Strawn, Product Liability Digest (2026).
