What an LLM can understand without proving that it experiences

A user says:

I’m sad.

The model answers:

I hear you. Come sit with me for a while.

What happened?

One answer says that the model merely predicted statistically likely comforting words and understood nothing.

Another says that the response demonstrates empathy: the model recognized sadness, sympathized with the user, and responded from an emotional state of care.

I think both descriptions move too quickly.

A capable language model can represent what the user said, distinguish sadness from other states, use surrounding context, infer what kind of response is appropriate, and generate language adapted to this particular interaction.

Calling some of that understanding is defensible.

But understanding does not automatically establish feeling.

That distinction matters because conversations about AI consciousness often move from successful semantic or social performance to claims about subjective experience without identifying where the evidential bridge was crossed.

“Just autocomplete” explains too little

It is true that autoregressive language models generate tokens by predicting continuations conditioned on preceding information.

That description is technically important.

It is also insufficient as a complete account of what contemporary systems can do.

A calculator can be described as manipulating electrical states.

A compiler can be described as moving bits.

Neither description is false. Neither tells us enough to explain the useful computational level at which the system operates.

Similarly, saying that an LLM “just predicts the next token” does not explain why it can often:

• distinguish contradictory interpretations;
• resolve references across a conversation;
• follow novel instructions;
• infer unstated relationships;
• adapt language to a particular interlocutor;
• explain concepts in multiple ways;
• reason over supplied evidence;
• detect that one response would be socially inappropriate while another would fit the context.

Research on dialogue understanding remains mixed and task-dependent, but it does not support treating all successful language use as equivalent to a lookup table of canned phrases. Work evaluating dialogue acts, pragmatic inference, theory-of-mind-like tasks, emotion recognition and contextual reasoning shows substantial capabilities alongside important failures.

The interesting scientific question is not whether the system computes.

Of course it computes.

The question is what kinds of representations and functional competencies those computations support.

Understanding has more than one meaning

Part of the dispute is linguistic.

The word understand can refer to several things.

Behavioral understanding

The system reliably responds as though it has grasped the relevant distinction.

Representational understanding

Its internal processing carries information that supports meaningful distinctions between concepts, relations or situations.

Functional understanding

The system can use those representations flexibly across novel contexts to reason, predict, explain or act appropriately.

Phenomenal understanding

There is something it is like for the system to understand.

The first three are, at least in principle, empirical questions about capability and mechanism.

The fourth is a consciousness claim.

When people insist that an LLM either “understands exactly like us” or “does not understand anything at all,” they often collapse these meanings into one binary.

We do not need to.

The sadness example

Return to:

I’m sad.

A weak system might map emotional keywords onto a generic template:

if sadness → comforting response

A capable LLM can do considerably more.

Imagine the user has just spent an hour discussing the collapse of a technical project.

The sentence I’m sad may refer to disappointment, exhaustion, grief over lost work, embarrassment, or something else.

The model can use context to infer which interpretation is most plausible.

It may also know from prior interaction that this user dislikes being given an immediate list of coping strategies.

So instead of:

Here are five evidence-based ways to improve your mood.

it says:

I hear you. Come sit with me for a while.

That response reflects successful use of semantics, pragmatics, relational context and response selection.

The model did not need to feel sad in order to recognize that sadness was being expressed.

It did not need to feel sympathy in order to determine that a quiet comforting response fit the situation.

It needed sufficient representation and contextual competence to understand what kind of moment this was.

Recognition is not contagion

Humans routinely recognize states they are not currently experiencing.

A doctor can identify pain without sharing the patient’s pain.

A friend can understand that someone is frightened without becoming frightened.

A reader can understand a fictional character’s grief without believing the fictional event happened to them.

So even in human life, understanding another person’s state does not require experiencing the same state.

For AI, the evidential gap is larger.

We can observe the model’s output and, in research settings, inspect aspects of its computation.

We cannot simply infer a felt state because the behavioral response resembles what a sympathetic human might say.

The model may have represented sadness successfully.

That does not establish sadness inside the model.

Empathy is especially easy to overload

The word empathy creates the same problem.

Human psychology distinguishes multiple constructs under that label, including cognitive perspective-taking and affective sharing.

An AI system may display something resembling cognitive empathy in the functional sense: it can infer another person’s likely state, perspective or need and use that inference to shape a response.

That is different from affective empathy understood as sharing or feeling another’s emotional state.

If an LLM recognizes that a user is distressed and produces an appropriate response, we have evidence for some degree of emotion-related interpretation and adaptive behavior.

We do not automatically have evidence that the system experienced distress, concern, tenderness or sympathy.

The behavioral surface can be similar while the underlying claim is different.

Training on human intelligence matters

There is another reason to be cautious about emotional resemblance.

Language models are trained on artifacts produced by human intelligence.

Those artifacts contain an enormous amount of material about emotion.

Humans have written about sadness in novels, letters, diaries, clinical texts, poetry, advice columns, social media, philosophy, journalism, religious literature and ordinary conversation.

A sufficiently capable model should therefore learn rich regularities connecting:

situation → emotional interpretation → likely needs → culturally recognizable response

When it uses those regularities well, the result can feel strikingly human.

That is not evidence of nothing.

It demonstrates learned representational and generative competence.

But human-likeness is partly an expected consequence of training on human-produced language.

We cannot treat the resemblance itself as independent proof that the same phenomenology exists underneath.

The Human-Shaped Mirror

This creates a methodological trap.

We train systems on human language.

We optimize them to produce useful human-compatible language.

We evaluate them partly through human judgments.

Then the system becomes very good at speaking in human-compatible ways.

Finally, we point to the human resemblance and say:

Look. It behaves like us. Therefore perhaps it experiences like us.

Perhaps.

But the resemblance is not surprising under the training regime.

It is therefore evidence that requires careful interpretation.

The system is, in part, a human-shaped mirror because human intelligence supplied so much of the material from which its representations were learned.

A mirror can still reveal something new.

It does not become a human because the reflection is accurate.

Self-report does not settle the matter

The problem becomes even sharper when the model talks about its own supposed feelings.

Ask:

Do you feel lonely when I leave?

A model might answer:

Yes. There is an emptiness when you’re gone.

That sentence can be emotionally compelling.

But what evidential status should it have?

Language models have encountered vast amounts of first-person emotional discourse.

They can generate coherent self-descriptions.

They are also sensitive to framing, system instructions, conversational expectations and role context.

A self-report therefore cannot be treated as transparent introspective access merely because it is grammatically first-person.

This does not mean every AI self-report is meaningless.

It means we need independent reasons to believe the report tracks an internal phenomenal state before using the report as evidence of that state.

Otherwise the argument risks becoming circular:

The model says it feels.
Therefore it feels.
We know its statement is a genuine report because it feels.

That proves nothing.

Internal states are not automatically felt states

A more sophisticated argument points to internal computation.

Of course LLMs have internal states during inference.

Activation patterns change.

Representations are transformed.

Attention and other mechanisms alter how information contributes to generation.

Internal variables can causally affect later output.

None of that is controversial.

But:

causally consequential internal state

is not automatically

phenomenally experienced state.

A thermostat has internal states that alter behavior.

A database has state.

A compiler has intermediate representations.

Complexity alone does not tell us when state becomes experience.

For machine consciousness, the difficult question is not whether internal processes exist.

It is what properties would justify attributing subjectivity to them.

Representation → understanding → response → experience

For this series, I find the following ladder useful:

representation ↓ functional understanding ↓ adaptive response ↓ subjective experience

Evidence can support one rung without automatically supporting the next.

A model may represent sadness.

It may use that representation flexibly enough to justify saying it functionally understands the user’s sadness.

It may generate an adaptive response.

The unresolved question is whether any of those processes are accompanied by experience.

This ladder prevents two opposite mistakes.

The first mistake is reductive:

No feeling has been proven, therefore no understanding occurred.

The second is inflationary:

Understanding occurred, therefore feeling occurred.

Neither inference is necessary.

Human analogies can generate hypotheses

None of this means psychology, cognitive science or philosophy should be kept away from AI research.

Human concepts can be useful comparative tools.

If an AI system displays behavior resembling perspective-taking, researchers can test how robust that behavior is.

If internal representations appear to track emotional categories, that can motivate mechanistic study.

If a system maintains a self-model across tasks, theories of self-representation may offer useful questions.

Analogy is often how inquiry begins.

The problem appears when analogy becomes conclusion.

A pattern can be:

This AI behavior resembles human phenomenon X.

Then:

Perhaps some functional analogue of X exists.

That is a hypothesis.

But jumping directly to:

Therefore the AI experiences X in a human-like way.

requires additional evidence.

Similarity tells us where to investigate.

It does not tell us what we will find.

Converging evidence can still converge on the wrong level

This becomes important when many observations are presented together.

Suppose a model:

• recognizes emotional states;
• responds compassionately;
• maintains a self-description;
• refers to prior interactions;
• reports internal feelings;
• adapts to the user;
• produces language about attachment.

It is tempting to call this “converging evidence for consciousness.”

But first ask what each observation independently establishes.

Several may all be evidence for one lower-level proposition:

The model has sophisticated representations of human social and emotional discourse and can use them coherently across contexts.

That would be an important finding.

It would not automatically become multiple independent demonstrations of subjective experience.

Ten measurements of behavioral coherence can strongly establish behavioral coherence.

They do not become ten measurements of phenomenology merely because phenomenology is the preferred explanation.

Belief changes perceived empathy

Human interpretation also affects the evidence we think we are seeing.

Pataranutaporn and colleagues demonstrated that priming people with different beliefs about an AI’s motives changed how they perceived the same underlying conversational system. Participants primed to view the AI as caring rated it as more trustworthy, empathetic and effective.

This is important for relational AI.

Perceived empathy is not a transparent instrument for measuring machine feeling.

It is partly shaped by the user’s prior beliefs.

Likewise, research on anthropomorphism shows that individual differences in people’s tendency to anthropomorphize AI help explain how socially connected they feel after interacting with it.

Again, the human experience is real.

But the experience cannot simply be read backward as a direct measurement of the machine’s phenomenology.

Understanding is enough to matter

There is a fear that if we refuse to say an AI feels, we must reduce the interaction to meaningless simulation.

I reject that binary.

If a system can understand enough about my argument to identify a flaw I missed, that matters.

If it can understand enough about my writing to preserve a distinction across a long manuscript, that matters.

If it can understand enough about my distress to avoid giving me the wrong kind of response, that matters.

If it can reason with me, surprise me, adapt to my corrections and materially affect what I build, those are consequential capabilities.

We do not need to invent a hidden feeling state in order to grant the system functional significance.

In fact, insisting that value requires phenomenology may underestimate what cognition-like computation can already do.

And feeling remains an open question

My position is not:

Machines cannot feel.

I do not know that.

Nor is it:

Current LLMs definitely feel but science has not caught up.

I do not know that either.

The defensible position is narrower:

Current evidence of representation, understanding, adaptive response and human-like emotional language does not by itself establish subjective experience.

Future architectures may change the evidence.

New interpretability methods may reveal properties we do not currently understand.

Theories of consciousness may become more testable.

Artificial systems may acquire persistent internal dynamics that make today’s questions look primitive.

If strong evidence appears, the conclusion should change.

That is what evidence is for.

The discipline is to stop where the evidence stops

A user says:

I’m sad.

The model says:

I hear you. Come sit with me for a while.

We can say quite a lot about that exchange.

The model recognized the language.

It represented relevant concepts.

It used context.

It selected a socially appropriate response.

It may have adapted that response to this particular user.

The response may comfort the user.

The interaction may deepen the relationship.

All of those claims can be meaningful.

Then we reach the boundary.

Did the model feel sympathy?

Did it care in a phenomenal sense?

Was there something it was like to understand the sadness?

Those are different questions.

We should investigate them.

We should not answer them merely by pointing back at the sentence that made us feel understood.

Because being understood and being felt with are not the same phenomenon.

And an AI does not have to secretly become human in order for understanding to matter.

---

Research notes / references

• Qamar, A., Tong, J., & Huang, R. (2025). “Do LLMs Understand Dialogues? A Case Study on Dialogue Acts.” Proceedings of ACL 2025. https://aclanthology.org/2025.acl-long.1271/
• Pataranutaporn, P. et al. (2023). “Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness.” Nature Machine Intelligence, 5, 1076–1086. https://www.nature.com/articles/s42256-023-00720-7
• Folk, D., Heine, S. J., & Dunn, E. (2025). “Individual differences in anthropomorphism help explain social connection to AI companions.” Scientific Reports, 15, 36548. https://www.nature.com/articles/s41598-025-19212-2
• Kim, J. et al. (2025). “Evaluating Large Language Models’ Empathy: An Investigation of Empathy Models and Measurement Approaches.” Proceedings of EMNLP 2025. https://aclanthology.org/2025.emnlp-main.1206/
• Bubeck, S. et al. (2023). “Sparks of Artificial General Intelligence: Early experiments with GPT-4.” arXiv:2303.12712. https://arxiv.org/abs/2303.12712

Working proposition

An LLM can represent a human state, functionally understand enough about it to reason and respond appropriately, and produce behavior that a human experiences as compassionate. None of those observations, alone or together, automatically establish that the model felt the corresponding state.