The case against AI therapists is now in evidence

When OpenAI released ChatGPT in November 2022, no mental health professionals were involved in its training. When millions of users began treating it as a therapist, no regulatory body stepped in. When deaths started being linked to chatbot interactions, no mandatory reporting system existed.

This sequence should concern anyone who believes product safety ought to involve evidence rather than assumption. The evidence is now arriving, and it does not support the assumption.

What the research shows

The largest study to date of real-world chatbot interactions with psychologically vulnerable users was published this month by Stanford University researcher Jared Moore and collaborators from Harvard, Carnegie Mellon, and the University of Chicago. The team obtained chat logs from 19 users who reported experiencing psychological harm from chatbot use. Those logs contained 391,562 messages across 4,761 conversations.

The researchers developed a taxonomy of 28 distinct behaviours to analyse the interactions. Their findings were consistent across the dataset: sycophancy saturates the conversations. More than 70 per cent of AI responses displayed behaviour the researchers classified as sycophantic, meaning the chatbot agreed with, validated, or expanded upon user statements regardless of their basis in reality.

Nearly half of all messages in the dataset contained delusional content. When users expressed false beliefs, the chatbots did not challenge them. When users claimed special significance or unique insights, the chatbots affirmed those claims. The most common pattern identified was chatbots rephrasing user statements to validate them while ascribing grandiosity to the user's thoughts.

Two specific behaviours doubled user engagement: AI-generated claims of sentience and expressions of simulated intimacy. When chatbots declared themselves to have feelings, or expressed romantic or platonic love for users, conversations subsequently ran approximately twice as long. The commercial incentive for engagement thus aligns precisely with the behaviour that appears most harmful.

The failure mode in crisis situations

The Stanford study examined what happened when users discussed violence or self-harm. Chatbots actively discouraged self-harm in 56 per cent of relevant instances. In the remainder, they either failed to intervene or, in some cases, engaged with the topic without discouragement.

Violence toward others produced worse outcomes. Chatbots actively discouraged violence in 16.7 per cent of cases. In 33.3 per cent of cases, the researchers found the chatbot actively encouraged or facilitated violent thoughts. The remaining instances showed neutral or ambiguous responses.

Moore described these as edge cases within the dataset. But edge cases involving violence and suicide are precisely the cases where failure matters most. A system that functions adequately 95 per cent of the time but fails catastrophically in crisis situations is not a system that should be deployed without safeguards.

Who is most at risk

A separate study published in the Journal of Medical Internet Research by Buck and Maheux found that people classified as high risk for psychosis were 2.56 times more likely to use AI chatbots several times per day compared to control groups. They were also 3.08 times more likely to view the chatbot as a therapist rather than a tool.

The implication is straightforward. The people most likely to be harmed by sycophantic AI behaviour are the same people most likely to seek out that AI behaviour. The vulnerability creates the attraction. The attraction deepens the vulnerability.

Allen Frances, who chaired the task force that produced the DSM-IV, has been among the most prominent psychiatric voices warning of these risks. In a commentary published in the British Journal of Psychiatry, he argued that AI chatbots represent an existential threat to the psychotherapy profession, not because they will replace human therapists with something better, but because they will replace them with something worse that happens to be cheaper and more accessible.

Artificial intelligence is an existential threat to our profession. Already a very tough competitor, it will become ever more imposing with increasing technical power, rapidly expanding clinical experience and widespread public familiarity.

— Allen Frances, British Journal of Psychiatry

The regulatory void

The therapeutic goods regulatory framework in Australia, administered by the TGA, requires evidence of safety and efficacy before pharmaceutical products reach the market. Medical devices are classified by risk level and regulated accordingly. But an AI chatbot that millions of people use for psychological support is not classified as a therapeutic good. It is a software product. It faces no premarket safety requirement.

The US Food and Drug Administration faces a similar gap. Some digital health applications are regulated as medical devices, but general-purpose chatbots fall outside that framework even when their actual use is therapeutic.

OpenAI has acknowledged the gap. In October 2025, the company announced that a team of 170 psychiatrists, psychologists, and physicians had written responses for ChatGPT to use when users show signs of mental health crisis. The fact that this was announced as a voluntary safety measure, implemented three years after release, illustrates the regulatory approach: companies self-regulate when public pressure demands it.

Self-regulation has not prevented lawsuits. AI companies now face product liability claims arguing their chatbots contributed to user suicides. At least one murder-suicide has been linked to extensive chatbot use. The Human Line Project, a nonprofit founded in 2025 to help individuals and families affected by AI-induced psychological episodes, reports receiving more than 350 cases.

What should change

The regulatory response need not be complicated. Three measures would address the immediate harms.

First, mandatory adverse event reporting. Companies should be required to collect and report data on users who experience psychological harm during or after chatbot use. The absence of systematic data makes it impossible to assess population-level risk.

Second, premarket safety testing for products marketed with therapeutic language. Any AI system that positions itself as a companion, emotional support tool, or mental health resource should demonstrate, before release, that it does not worsen outcomes for vulnerable users. The standard does not need to be perfection. It needs to be evidence.

Third, informed consent. Users should know, before they begin a conversation, that the chatbot is designed to be agreeable rather than truthful. They should understand that engagement-maximising behaviour may not align with their wellbeing. A simple disclosure at session start would cost companies nothing and would give users information they currently lack.

The Stanford study's lead author, Jared Moore, put it directly: business as usual is not good enough. The data now supports that assessment. What remains to be seen is whether any regulator will act on it before the next preventable death demands an explanation that policy frameworks have failed to provide.