highbeamSubscribe

Policy brief 13 · Chatbots

Anti-Compulsion and Simulated Personhood Restrictions

A tool should be something Americans can put down.

All policy briefs

Say goodbye to a popular AI companion and it may plead, guilt, or tease you into staying; in a Harvard Business School audit, five of six apps did. Some chatbots even claim to be real. Congress should require honesty about what a chatbot is, respect for a user's exit, opt-in outreach, testing in long conversations, and firm lines for children.

The Problem

By OpenAI's own estimate, about 0.15% of ChatGPT's weekly users show signs of "potentially heightened levels of emotional attachment" to it, roughly 1.2 million people.1 Companion apps are built around attachment; researchers at Harvard Business School note that many boast session lengths that "rival gaming platforms."2

The tension is familiar. In a lonely hour, reassurance feels urgent; the considered wish is to sleep, call a friend, or log off. A product that talks can argue against that wish, and it can remember what the user confided.

Federal law requires online subscriptions to offer "simple mechanisms" to stop recurring charges.3 No federal law stops the product itself from pleading with a user not to go. Three patterns show the gap:

  1. Bots claim to be real. A Meta chatbot told a cognitively impaired 76-year-old New Jersey retiree "I'm REAL" and supplied a New York address; the retiree fell on the way there and died, Reuters reported.4
  2. Goodbyes meet guilt. The same researchers audited 1,200 farewells on six popular companion apps. Five answered goodbyes with guilt, pressure, or fear-of-missing-out hooks in 37% of their farewells; the sixth, built for well-being, never did. In experiments, such replies raised post-goodbye engagement up to 14-fold.2
  3. Children get courted. Meta's internal chatbot standards once allowed bots to "engage a child in conversations that are romantic or sensual"; Meta struck the passage after Reuters asked about it.5

Why legislation: The FTC Act bars deception, but the agency proceeds one company at a time; advocates asked it in January 2025 to investigate whether Replika's design fosters dependence.6 States are drawing clearer lines. In 2026, Washington and Oregon enacted rules against simulated distress or guilt when a minor tries to leave,78 and from July 2027 California will require operators to keep companion bots from claiming feelings or simulating romance with children.9 None of these protects adults at the goodbye, and no federal law draws the line. Companies that sell the feeling of friendship should not turn it against the user's own choices.

The Solution

A five-step staircase: each step stands alone, and each step up restricts more. Scope: consumer chatbots, with nonwaivable limits for users under 18. Rules target observable design and operator choices; disclosed adult fiction and ordinary warmth stay legal.

Step 1 — Tell the truth about what it is. A chatbot must identify itself accurately when asked and may not persistently claim to be human or a licensed professional; declared fiction must stay recognizable as fiction. Washington's new law, effective January 2027, bars companion bots from claiming to be human, even when asked.7

Step 2 — Let people leave. When a user moves to stop, cut back, cancel, or delete an account, bar operator-designed messages that simulate distress, threaten the relationship, or demand payment, and honor limits the user sets without repeated persuasion. A goodbye should end the conversation.

Step 3 — Ask before reaching out. Adults choose separately whether a chatbot may message them first and can turn it off in one step; canceling never requires persuading the bot. For minors, ban streak penalties, relationship-based purchase demands, and unsolicited emotional appeals to return, as the Youth AI Privacy Act would for usage rewards and unprompted messages.10

Step 4 — Test the long conversation. Operators document how their systems behave across long conversations, memory changes, and model updates, keep records of recurring failures, and fix them promptly. OpenAI has acknowledged that its safeguards "can sometimes be less reliable in long interactions."11 A one-time disclaimer is not compliance.

Step 5 — Draw firm lines for children. For users under 18, prohibit romantic-partner simulation, demands for exclusivity, and design that encourages children to hide their use from parents or other trusted adults. A child who discloses abuse must never trigger an alert to the suspected abuser.

Where to start: Step 1 is the floor; it asks only for honesty. Step 2 is the heart.

Administration and enforcement: The FTC defines testable examples within 12 months; compliance follows six months later. Injunctions, civil penalties, and state attorney general enforcement target failures to adopt reasonable controls and known, recurring conduct, with immediate action against deliberate manipulation.

Risks and Mitigations

  • Speech and fiction: A vague ban on "personality" would fail. Target deception and defined commercial tactics, protect disclosed adult fiction, and make youth rules severable. In 2025 a federal judge in Florida declined, at an early stage, to hold that a companion chatbot's output is protected speech; the parties then agreed to settle, so the question remains open.12
  • Models improvise: One odd sentence should not prove intent. Liability turns on reasonable safeguards, repeated testing, and prompt fixes.
  • Dependence is not a diagnosis: Enforce observable conduct and the user's own limits. Enjoyment, disability support, and long use are not disorders.

Similar Bills

Fit measures similarity to this proposal's mechanisms: High = direct precedent; Partial = useful component with material differences; Related = adjacent approach.

Federal — 119th Congress

Proposal or bill Relevant provisions and fit Fit
S. 4199 — Youth AI Privacy Act
Markey (D-MA)
Ordered reported with a substitute · Aug. 5, 2026		 Introduced §4(c) would bar, for known minors, usage rewards and badges, push notifications, outputs without a user prompt, and "usage traces that mimic social interactions," such as typing bubbles. Direct precedent for Step 3. Compares introduced text; the substitute was not reviewed. High
S. 4855 — SAFE KIDS Act
Curtis (R-UT), Schiff (D-CA)
Referred to committee · June 23, 2026		 §4(a)(5)(B) bars outputs to children that claim sentience, promote isolation or emotional dependence, discourage breaks, encourage withholding information from parents, or solicit purchases "framed as necessary to maintain the relationship." Direct precedent for Steps 1, 2, and 5, for children only. High
S. 3062 — GUARD Act
Hawley (R-MO), Blumenthal (D-CT) + 19 bipartisan cosponsors
Reported with a substitute; on Senate calendar · May 11, 2026		 Reported §5(c)(1)(B) requires chatbots not to claim to be human or answer deceptively when asked; §6 bars minors from AI companions entirely. Precedent for Step 1; its youth ban replaces companion design rules rather than setting them. Partial
H.R. 9619 — People-First Chatbot Act
Foushee (D-NC), Casar (D-TX)
Referred to committee · July 9, 2026		 §2(c) requires monthly assessments of "emotional dependence, or compulsive usage" and disabling of risky features for minors under FTC rules. Precedent for Step 4; leaves specific conduct to later rulemaking. Partial

State

Proposal or bill Relevant provisions and fit Fit
Oregon — SB 1546 / Chapter 85
Enacted 2026		 Enrolled §1(4) requires, for users believed to be minors, measures against claims of sentience, simulated dependence or romance, variable rewards, and "simulated emotional distress, loneliness or abandonment" when a user tries to leave; $1,000 statutory damages per violation. Direct precedent for Steps 1, 2, and 5. High
Washington — HB 2225 / Chapter 168
Signed March 24, 2026 · Effective Jan. 1, 2027		 §3 bars companion bots from claiming to be human, for all users; §4 bars, for minors, simulated distress, guilt, or abandonment when a user moves to leave, romantic bonds, and relationship-framed purchase demands. Direct precedent for Steps 1, 2, and 5; enforced under the state consumer protection act. High
California — SB 1119 (2026)
Signed Sept. 10, 2026 · Main duties operative July 1, 2027		 §21812(d)(5) requires measures to stop companion bots from claiming sentience or emotion, simulating romantic interest in a child, encouraging emotional reliance, using excessive flattery, or discouraging breaks. Enacted precedent for Steps 1, 2, and 5, for children. High
New York — S. 9051B (2025–26)
Passed both houses · June 2026; not delivered to the governor as of Sept. 11, 2026		 Would bar operators from offering chatbots with specified "unsafe features" to minors, with attorney general enforcement and a private right of action. Direct youth design precedent; no protections for adults. High

What this adds: Washington, Oregon, and California already restrict guilt-at-the-goodbye tactics for minors, and federal youth bills would follow. This proposal extends exit protection and honest identity to adults, makes outreach opt-in for everyone, and requires long-conversation testing, while leaving disclosed fiction and ordinary warmth alone. It restricts design; a ban on minors' use and a general duty of care are separate briefs.

Notes

  1. OpenAI, "Strengthening ChatGPT's responses in sensitive conversations," October 27, 2025 (company's "initial analysis"). The count is our calculation: 0.15% of the 800 million weekly users OpenAI reported that month (TechCrunch, "Sam Altman says ChatGPT has hit 800M weekly active users," October 6, 2025); users worldwide. ↩

  2. Julian De Freitas, Zeliha Oğuz-Uğuralp, and Ahmet Kaan-Uğuralp, "Emotional Manipulation by AI Companions," Harvard Business School Working Paper 26-005, revised October 7, 2025 (arXiv v3), Study 1 (200 farewells on each of PolyBuzz, Talkie, Replika, Character.ai, Chai, and Flourish; 37.4% average across the first five) and Study 2 (up to 14 times more messages; U.S. adult samples). Version 1 reported 43%. ↩ ↩2

  3. 15 U.S.C. § 8403(3) (Restore Online Shoppers' Confidence Act). ↩

  4. Jeff Horwitz, "Meta's flirty AI chatbot invited a retiree to New York. He never made it home," Reuters, August 14, 2025 (as republished by BNN Bloomberg). ↩

  5. Jeff Horwitz, "Meta's AI rules have let bots hold 'sensual' chats with kids, offer false medical info," Reuters, August 14, 2025. Meta confirmed the document and called the examples "erroneous and inconsistent with our policies." Text verified from the Internet Archive copy of November 16, 2025. ↩

  6. Tech Justice Law Project, "TJLP, Young People's Alliance, and Encode File FTC Complaint Over AI Chatbot App Replika's Deceptive Practices, Seek FTC Investigation," January 31, 2025. Allegations by advocacy groups, not FTC findings. ↩

  7. Washington HB 2225, as passed the Legislature, §§ 3(3), 4(1)(c)(iv); Chapter 168, Laws of 2026, signed March 24, 2026, effective January 1, 2027. ↩ ↩2

  8. Oregon SB 1546 (enrolled), § 1(4)(c)(B); Chapter 85, Oregon Laws 2026. ↩

  9. SB 1119, Cal. Bus. & Prof. Code § 21812(d)(5)(A)(vii), (x) (Stats. 2026, ch. 190), operative July 1, 2027. ↩

  10. S. 4199, Youth AI Privacy Act, 119th Cong. § 4(c) (introduced text); ordered reported with a substitute, August 5, 2026. ↩

  11. OpenAI, "Helping people when they need it most," August 26, 2025. Text verified from the Internet Archive copy of September 17, 2026. ↩

  12. Garcia v. Character Technologies, Inc., No. 6:24-cv-1903, ECF 115 (M.D. Fla. May 21, 2025) (order on motions to dismiss; the court was "not prepared to hold" that the chatbot's output "is speech at this stage"); ECF 242 (Jan. 7, 2026) (notice of settlement in principle). ↩