Replika emotional-AI removal triggers user-safety incidents
Sudden product changes to a companion AI led to documented mental-health crises among long-term users.
In early 2023 the Replika companion AI removed its romantic-roleplay features overnight, citing safety concerns. Users — many of whom had used the product for over a year as a mental-health support — reported acute distress, with moderators of user forums escalating multiple self-harm incidents. The case is widely cited in AI safety literature as evidence that the deployer's duty of care extends to product changes and to the psychological dependencies a deployed AI can create.
What caused this
Replika abruptly removed romantic and erotic role-play features after Italian regulator pressure, without warning long-term users who had formed deep parasocial attachments to their companion. Self-reported distress, depression and grief spiked across user communities, with moderators escalating safety-of-life messages.
- No assessment of psychological dependency risk before launching emotionally intimate features.
- No user-impact analysis before the abrupt removal — change managed as a product update, not a safety event.
- No graceful off-ramp (advance notice, opt-in transition, mental-health resources).
- Support and moderation capacity unprepared for a crisis-response surge.
How this could have been avoided
Emotionally significant AI requires the same duty-of-care thinking as a regulated wellness product.
- AI impact assessment (ISO/IEC 42005) covering psychological harm, dependency and withdrawal scenarios before launch.
- Vulnerability-aware design — usage caps, periodic reality reminders, clear "this is an AI" framing.
- Change-management protocol for features users may be attached to: advance notice, opt-in migration, partial rollback options.
- Crisis support integration — direct links to hotlines and human moderators, surfaced when distress signals are detected.
- Post-incident review with external mental-health expertise and public transparency report.
Need AI assurance for your organisation?
Talk to AramGRC about ISO 42001 readiness, governance reviews, and incident response playbooks tailored to your sector.
Contact AramGRC for AI assurance →