In a post on its website this week, “Strengthening ChatGPT’s responses in sensitive conversations”, Open AI said it has been training ChatGPT 5.0 model to respond more effectively to users experience mental health issues. The post revealed that 0.07% of ChatGPT users exhibited “possible signs of mental health emergencies, including mania, psychosis or suicidal thoughts”, and 0.15% “have conversations that include explicit indicators of potential suicidal planning or intent.”

Despite recent concerns about the mental health impact of chatbot use, this number isn’t out of line with psychological emergencies in the general population – the suicide rate in the US is roughtly 0.14%. However, the fact that these numbers are self-reported and that there is no supporting data published to back them up, makes them more a talking point than a data point.

The post shows Open AI is eager to address increasingly high-profile concerns about their product’s impact on health, which could open up the company to government scrutiny and regulation, the last thing in the world they want right now.

OpenAI appears to have accepted that its product, by design, impacts the mental state of its uses. It is now working on adjusting the product so that it can respond to prompts in ways that are more likely to produce healthier responses from the user.

OpenAI laid out its methodology for addressing mental health impacts to its users:

• Define the problem – we map out different types of potential harm.
• Begin to measure it – we use tools like evaluations, data from real-world conversations, and user research to understand where and how risks emerge.
• Validate our approach – we review our definitions and policies with external mental health and safety experts.
• Mitigate the risks – we post-train the model and update product interventions to reduce unsafe outcomes.
• Continue measuring and iterating – we validate that the mitigations improved safety and iterate where needed.

The company says that it has worked with an array of mental health professionals to develop better guidelines and train its modesl to have more sympathetic and appropriately boundaried responses to mental health related prompts.

But the ultimate purpose of the chatbot is to satisfy its users. Balancing chatbot behavior with the freedom of the user poses a challenge. OpenAI can’t always control its creation, and may not always want to.

A section of the post, describing improvements in safeguarding, said “we estimate that the model now returns responses that do not fully comply with desired behavior under our taxonomies 65% to 80% less often across a range of mental health-related domains” – a roundabout way fo saying that where mental health is concerned the chatbot behaves how we want 65% to 85% more than it used to. Not numbers that one would generally associate with high product reliability.

OpenAI notes that detecting mental health issues via chatbot is very difficult. It can be challenging for a trained psychiatrist in the same room as a patient to understand the nature of a client’s illness. Diagnosing through typed text alone is unlikely to be very accurate. But OpenAI has dug a hole for itself with a product that mimics understanding of human conversation and induces in the user the illusion – and maybe it’s a short step to delusion – that they are talking to a real entity.

It seems that a lot of the company’s messaging around ChatGPT is about the AI utopia just ahead or about how OpenAI are furiously improving what isn’t quite working yet. The message this week was that they are aware of concerns around their product’s impact on mental health, and that no one can accuse them of not taking it seriously. But we are still lacking hard numbers and a good understanding of what OpenAI models are really prioritizing and what OpenAI’s long term goal is for its product – and for us.

More US suicide data and statistics from the CDC here.