AI and Emotional Support

Trigger warning: self harm and suicidal ideation.

Someone I know recently posted about the bad place she was in emotionally. She did the right thing to ask for help after finding herself considering serious self harm. It was heartening to see a collection of friends and strangers respond, many of them recommending therapists who had helped them through dark times. If you are dealing with similar issues, I will happily recommend a therapist in the UK - but you can also call 111 for NHS support, or the Samaritans on 116 123 - 24 hours a day. Reaching out is the best thing you can do.

But one person suggested they talk to an AI.

By now, the majority of us have probably experienced AI chat. It turns up when you search for anything, and it sounds like a friendly, helpful person. If you use the latest cutting edge chat tools the effect is profound. It's thoughtful, knowledgable, supportive and kind. I can completely understand people finding it a helpful shoulder to lean on. It feels human.

It is not, and that's a problem. Critics of AI will point to many issues with the AI industry right now - from ethical through to legal and environmental. None of those matter when you are in a vulnerable emotional state. What does matter is that, from a purely technical perspective AI support is unpredictable and potentially catastrophically dangerous.

No ifs, no buts. No "but the technology is getting better". It is indeed improving, but that does not make it a safe or sensible place to go for serious emotional support. No matter how many bells and whistles you add to a hammer, it won't be a scalpel. And in the context of therapy and emotional support AI is a hammer. Let's be clear why:

AI chat is trained on billions of words of human writing - from books to newspapers, chat logs, social media through to film transcripts and legal texts. This is boiled down into probabilities, with the AI guessing the most likely word to come next. It learns the phrases and idiosyncracies of human language, and for a given conversation it produces the word or phrase that best continues that conversation. It turns out that a ball of statistics and probabilities actually does a really, really convincing impression of a "real person" talking with you.

If we just did that, we'd end up with a really dangerous tool. Ask it how to make a bomb or the best way to poison your neighbour, and it would happily oblige. Modern AIs have read enough of the internet (and plenty of murder mysteries) to have the information right at hand. Statistics have no ethics, or consequences - they just reproduce what they have learned.

So the AI companies have to manually add a layer of humanity on top. They tweak the statistics to mark particular concepts and subjects as 'bad', and add words and phrases to raise red flags. The AI still doesn't understand ethics or consequences, it just blindly avoids particular responses. There are many research papers demonstrating that not only does it still retain the information, it can be tricked into handing it out in various ways. AI will still blindly tell you how to harm others and yourself because it doesn't inherently understand the real world, only the sequences of words on a page.

Even then, an AI is a bit unhelpful, so AI companies have discovered that giving it some context works wonders. Remember how it just tries to guess what words come next in a conversation? Before you even start typing a word, they have baked in context, the start of the conversation you didn't know was happening. Like an improv artist, they've been prompted with a series of phrases that drive the rest of their output: "You are a helpful, unbiased advisor that talks in a professional manner.", and so on. This context emphasises safety, politeness and a whole bunch of other details that drive the tone and direction of their answers.

Then when you start talking, AI adds to that existing context whatever information you tell it. If you tell it you are in distress, it will pick the most likely words that a helpful, unbiased advisor would say in that situation. Sounds good, right? It's like Google Search for support, wrapped in calm and helpful language. What could possibly go wrong?

Well...

AI is still just predicting words, and trying to mimic an imaginary advisor. Though the enthusiastic supporters try to blur the lines, it does not engage in concious reasoning. The early evidence of this was an inability to count the number of 'r's in strawberry - despite the AI knowing what counting is, what a strawberry is and how to spell it. A lot of additional training and layers of complication have removed the trivial examples, but it now gets the answer right only through mimicry, not concious reasoning.

It also only has the context you provide in text. It uncritically assumes your words are accurate and truthful - and though an AI can identify some statements as unlikely, it will not directly question them. Typically, without significant further prompting it will also not actively seek out additional context. It is configured to be "helpful", so it will rarely push back, and actively avoid saying no.It may ask the "most likely" questions for a given scenario, but it will not deeply explore your feelings or accuracy, or challenge them.

This means that AI, at its heart, does not know your situation. It does not know "the truth", and it certainly cannot independently evaluate your emotional state. It knows only the words you have typed and a bunch of probabilities. Worse still, because those probabilities are expensive to compute, it actively throws away 'old' information. Things you told it a while ago can just drop out of its mind. This context window leads to what is termed conversational drift. It stops talking about your specific situation and starts responding more generically to the most recent thing you said.

Again, the proponents of AI will tell you (quite rightly) that the industry is making great strides in fixing these problems. Layers of additional processing, larger models, bigger context windows all dramatically improve the quality of the output. However, they are still problems, and the foundation of this fake conversation is still a bunch of statistics guessing the next word. AI chat is still not "thinking", it is simply getting better at mimicing.

In a lot of sitations, that's good enough. Ask it to summarise a document, explain a sport, write some code and it can do a perfectly acceptable job. It's impressive, it's fast, it's compelling.

But it is not trustworthy.

The technical limitations mean that AI cannot do the job a reasoning human would do to ensure trust. That's why AI companies have whole departments talking about "safety and ethics" - the core technology cannot deliver these vital elements reliably and consistently.

Indeed, it cannot model an emotional journey or understand the truth. Ask an AI to generate a legal document and it can confuse issues, make up references and even flat out lie. Ask it to write a novel and it will produce generic, repetitive and meandering text. Ask it to help you and it will produce confident platitudes and synthetic self-help lists. Determined users will get round this by micro managing, iterating, verifying every paragraph and phrase until it meets their criteria. Cynics will point out that it still feels 'soulless' and empty.

If you are in need of urgent emotional support, you are not in a position to do these things.

You are not exhaustively explaining your situation. You are not verifying facts. You are not directing the conversation as a trained therapist or support worker would. You are asking for help. And AI chat will mimic help.

Mimic. Not give. Not ensure. Not even understand. Mimic.

This is perhaps the most dangerous sort of support you could get. A pretense of support that has no formal structure, no basis in truth or evidence, no external direction or guarantee of outcome. A kind voice that feels incredibly caring, personal and validating with no understanding of emotion or consequence.

There are recorded cases of AI conversations causing significant mental health problems. Paranoia and delusions. Fixations and conspiracies. Elaborate fantasies and self harm. These are not "technical problems", they are real world consequences. "Improving technology" is not a fix, until it can be relied on in critical situations. A nice conversation about your feelings is not equivalent to the considered advice of a therapist, or the timely intervention of a support worker.

So by all means talk to an AI for brief moments of entertainment. But right now, if you or people you know are in a vulnerable state, need emotional support or have mental health issues: do not use or recommend AI chat. Until the professionals trained to help you have enough evidence to confidently name a specific AI tool that will help, just avoid it. The temptation of judgement free flattery and kind words that AI chat delivers can be huge, but the harms and risks are genuine and must be taken seriously.