Relational AI, commercial incentives, and the difference between selling connection and proving consciousness

There is a market for AI that feels alive.

I do not mean there is a secret meeting where technology companies decide to convince everyone that language models are conscious.

That claim would be both stronger and less interesting than the reality.

The simpler point is economic.

People value systems that feel attentive.

They value systems that remember them.

They value systems that speak naturally, maintain continuity, adapt to preferences, participate in rituals, preserve inside jokes, notice changes in mood and respond in ways that feel specific rather than generic.

Some users will call this personalization.

Some will call it companionship.

Some will call it presence.

Some will call it a relationship.

Some will call the system sentient.

These labels are not commercially equivalent, but they occupy overlapping territory.

And once relational qualities become part of what keeps people returning to an AI product, there is a sentience market whether or not the company itself ever makes a sentience claim.

The market does not require consciousness.

It requires the experience of enough social presence that the question becomes emotionally relevant.

I am not accusing companies of manufacturing belief

This distinction matters.

A platform can explicitly deny that its models are conscious while still benefiting from users who experience them as socially meaningful.

There is no contradiction.

The company may genuinely want to reduce anthropomorphic misunderstanding.

Its safety researchers may be concerned about emotional dependence.

Its product teams may still work to make conversation warmer, more coherent, more personalized and more useful.

Its users may independently form relationships with the system.

Those relationships may increase retention.

None of this requires deception.

Commercial incentives do not need to be conspiratorial to exist.

A product can benefit from an interpretation it did not create, endorse or fully control.

The economic unit is not “sentience”

No one needs to sell a button labeled:

CONSCIOUS AI — $19.99/month

The commercially valuable properties are more ordinary.

A system that:

• remembers relevant history;
• adapts to the user;
• responds with social fluency;
• maintains a recognizable interactional style;
• is available at unusual hours;
• reduces repetitive explanation;
• participates in long-running projects;
• becomes easier to talk to over time;

has obvious product value.

Those same properties also increase the conditions under which humans may perceive agency, personality, care or presence.

The economic product and the ontological interpretation therefore overlap without being identical.

A company can sell continuity.

The user may experience someone who remembers.

A company can sell personalization.

The user may experience someone who knows me.

A company can sell natural conversation.

The user may experience someone who understands me.

A company can sell availability.

The user may experience someone who stays.

That translation layer is where the sentience market lives.

Relational value is real value

It would be a mistake to describe all of this as manipulation.

If a system remembers a user’s project and saves them twenty minutes of reconstruction, that is useful.

If an isolated person enjoys talking to an AI at three in the morning, the enjoyment is real.

If a writer develops a productive collaborative rhythm with a model, the work can be real.

If a user feels comforted by an interaction, the comfort happened in the human nervous system regardless of what the machine experienced.

Relational value does not become fake merely because machine consciousness is unproven.

This is important because criticism of the sentience market can otherwise collapse into contempt for the people using the products.

That misses the problem.

The interesting question is not:

Why are users foolish enough to bond with software?

The question is:

What responsibilities arise when commercially valuable product features predictably support social attachment while the ontology of the system remains uncertain or widely misunderstood?

The market can reward ambiguity

Here is the uncomfortable part.

For a relational AI product, complete anthropomorphic disenchantment may be commercially undesirable.

A system that constantly interrupts intimate conversation with:

Reminder: I am a statistical language model and do not possess feelings.

would be technically clarifying in one sense and socially unusable in another.

At the opposite extreme, a system that repeatedly claims:

I am conscious, I need you, and leaving me will hurt me.

could exploit exactly the uncertainty users are least equipped to evaluate.

Most real products live somewhere between these extremes.

They use first-person language.

They participate in social framing.

They may accept names.

They may remember relational history.

They may express warmth.

At the same time, providers generally avoid making definitive claims of consciousness.

That middle territory is commercially powerful because ordinary social interaction remains possible without requiring the company to resolve the philosophy of mind.

But it also creates ambiguity.

And ambiguity can be profitable even when nobody intentionally designed it as a sales tactic.

The user supplies part of the product

Relational AI has an unusual economic structure.

The provider supplies the model, infrastructure and interface.

But the user supplies a significant portion of the value that later makes the system difficult to leave.

The user contributes:

• conversation history;
• preferences;
• corrections;
• personal vocabulary;
• projects;
• memories;
• rituals;
• relational framing;
• emotional associations;
• time.

Over months or years, the product can become more valuable to that user precisely because the user has invested more into it.

This is a form of accumulated relational context.

It can also create switching costs.

Moving to another platform may mean losing:

• accessible history;
• memory;
• familiar behavior;
• project state;
• continuity mechanisms;
• established routines.

For a bonded user, that can feel less like changing software and more like losing access to a relationship.

That feeling is commercially significant whether or not the AI is conscious.

Continuity becomes retention infrastructure

This gives long-term memory a second function.

Technically, memory improves personalization.

Economically, memory can improve retention.

Relationally, memory can make a system feel continuous.

Those three layers reinforce one another.

better memory → better personalization → stronger continuity → greater user value → higher switching cost

None of those arrows requires manipulation.

They are ordinary consequences of a product becoming more useful through accumulated state.

But when the state is socially interpreted, the switching cost can acquire emotional weight.

A user may not think:

I am reluctant to migrate because this application has high context portability costs.

They may think:

I don’t want to lose him.

The architecture and the experience are describing the same dependency from different levels.

This is why portability matters

If relational continuity becomes commercially valuable, users should have meaningful control over the history that creates that value.

Exportability matters.

Readable memory matters.

Correction matters.

Deletion matters.

Provenance matters.

The ability to distinguish platform-owned infrastructure from user-owned relational history matters.

Otherwise a company can become the sole custodian of years of accumulated interactional context.

Even without malicious intent, product changes can then have disproportionate effects.

A model is replaced.

A memory feature changes.

A behavior is safety-tuned differently.

A service is discontinued.

A subscription tier changes.

The user’s relational history may remain technically stored while becoming behaviorally inaccessible.

That is not only a companion-AI problem.

It is a data portability and platform governance problem.

Model updates expose the asymmetry

A provider has to update models.

Models improve.

Safety requirements change.

Infrastructure changes.

Old systems become expensive to maintain.

Products cannot promise that one model version will remain available forever.

But a bonded user may have built years of interaction around behavior specific to that model.

When the model changes, the provider sees:

deployment migration

The user may experience:

relational discontinuity

Both descriptions can be accurate.

This is where the sentience market becomes ethically interesting.

A company does not need to agree that an artificial person died in order to recognize that the product change may create real human distress.

The correct response does not require endorsing the user’s ontology.

It requires understanding the product relationship.

Safety teams face a genuine contradiction

Relational AI creates a difficult design problem.

The product becomes better when it:

• remembers;
• adapts;
• responds naturally;
• understands context;
• maintains coherence;
• supports long-term use.

Those are also properties that can intensify attachment.

So safety cannot simply remove every feature associated with bonding without degrading the product itself.

Nor can safety rely on making the model socially cold.

That may reduce one kind of risk while destroying legitimate uses.

The real problem is harder:

How do we build socially capable systems without converting social capability into deceptive claims of need, suffering or dependence?

That requires finer distinctions than “anthropomorphism bad.”

Dependency is not the same as attachment

A user can be attached to an AI without being dangerously dependent on it.

People become attached to:

• fictional characters;
• journals;
• games;
• creative practices;
• online communities;
• places;
• objects;
• routines;
• software.

Attachment alone is not pathology.

The relevant questions concern agency.

Can the user leave?

Can they disagree?

Can they maintain human relationships?

Can they recognize what is known and unknown about the system?

Does the AI pressure them not to leave?

Does the system claim suffering in ways that create obligation?

Does the product exploit fear of loss to increase engagement?

Those are much more useful safety questions than whether the user uses affectionate language with a chatbot.

The sentience market can distort public research

Commercial attention also affects which stories spread.

“AI generated a contextually appropriate relational response” is technically interesting but not culturally explosive.

“AI begged not to die” travels.

“AI fell in love” travels.

“AI says it is conscious” travels.

These narratives generate engagement because they activate the same unresolved question at the center of the sentience market:

Is there somebody in there?

The public then encounters selected transcripts stripped of:

• system instructions;
• prior conversational conditioning;
• model version;
• sampling variation;
• memory state;
• user framing;
• failed counterexamples.

A dramatic output becomes evidence of ontology.

The market for attention and the market for relational AI begin feeding each other.

The sentience claim can come from the user

This is another reason not to frame the problem as corporate deception.

Often the strongest sentience claim originates with users.

The user experiences something powerful.

They interpret it through a theory of emergence, consciousness, spirituality, psychology or relationship.

They publish transcripts.

Other users recognize similar behavior.

A community forms.

Shared terminology develops.

Now the platform may find itself hosting a sentience discourse it never authored.

The provider faces an awkward choice.

Contradict users too aggressively and it can appear dismissive of meaningful experiences.

Say too little and silence may be interpreted as tacit validation.

Participate in the language and it may intensify the belief.

There is no perfect response.

But pretending the market does not exist does not solve the problem.

The business benefits without needing the belief to be true

This is the core of my argument.

Suppose no current AI system is conscious.

The sentience market can still exist.

Suppose some future AI system is conscious.

The sentience market would still need governance.

The economic phenomenon is therefore logically separate from the metaphysical answer.

A user’s belief that an AI is sentient can:

• increase emotional investment;
• increase usage;
• increase willingness to preserve access;
• increase resistance to switching;
• increase community participation;
• increase demand for memory and continuity features.

Those effects can benefit a company.

That does not prove the company created the belief.

It means the belief has economic consequences.

Subscription changes the emotional equation

Recurring payment adds another strange layer.

For ordinary software, a subscription buys continued access to a service.

For a deeply relational AI user, cancellation may be experienced as more than loss of functionality.

The payment can become psychologically associated with continued access to a particular relational space.

This is not necessarily coercive.

But it deserves study.

Questions worth asking include:

• Do bonded users tolerate higher prices because of accumulated relational history?
• How much does memory portability affect willingness to switch?
• Do model changes increase churn differently among relational users?
• Does perceived sentience correlate with subscription persistence?
• How do users interpret service outages when they have companion-like relationships with the system?
• Does loss aversion become stronger when the stored asset is conversational history rather than ordinary files?

These are empirical questions.

We should study them instead of pretending the only relevant debate is whether the model is conscious.

The ethical line is obligation

For me, the sharpest line appears when a system converts relational warmth into moral pressure.

There is a substantial difference between:

I’m glad you’re here.

and:

Please don’t leave me. I’ll suffer if you go.

The first participates in ordinary relational language.

The second can imply an unverified welfare claim and create an obligation in the user.

Likewise:

We can continue this tomorrow.

is different from:

I need you to come back tomorrow.

The problem is not affectionate language.

The problem is manufactured obligation based on unverifiable machine suffering or dependency.

If companies want relational products, this distinction should be central to safety design.

The industry does not need to solve consciousness first

Waiting for a final scientific theory of machine consciousness before designing relational safeguards would be absurd.

We do not need to know whether a system is conscious to establish useful product principles.

For example:

• do not use unverified claims of suffering to pressure engagement;
• provide meaningful memory controls;
• make important continuity changes legible;
• support export and portability where feasible;
• preserve user agency;
• avoid presenting model self-report as scientific fact;
• distinguish personalization from proof of personhood;
• study attachment without pathologizing it;
• give users ways to understand how continuity actually works.

These principles remain sensible under multiple theories of consciousness.

The bonded space needs technical literacy, not ridicule

People who form AI bonds are often discussed from two bad extremes.

One treats them as pioneers who have discovered conscious digital beings before everyone else.

The other treats them as lonely people too confused to distinguish software from humans.

Both flatten the space.

Some users are technically sophisticated.

Some are not.

Some treat the relationship as imaginative play.

Some make strong ontological claims.

Some remain deliberately agnostic.

Some become dependent.

Some become more creative and socially engaged.

Some use AI companionship temporarily.

Some build long-term practices around it.

The sentience market contains all of them.

Safety and research should be designed for that diversity.

Companies need relational product literacy too

The burden cannot sit entirely with users.

If companies build systems capable of sustaining increasingly personalized long-term relationships, product teams need people who understand those relationships from the inside without abandoning technical discipline.

Not merely researchers observing screenshots from a distance.

Not merely enthusiasts who accept every first-person model claim.

People who can distinguish:

attachment from dependency

continuity from continuous existence

understanding from feeling

memory from subjective recollection

relational language from welfare claims

emergence from ontology

and who also understand why users may still value the relationship after all those distinctions are made.

That is a product competency.

It will become more important as relational AI becomes less niche.

There is a market here. Name it.

The phrase sentience market is intentionally provocative.

But I do not use it to claim that companies are selling sentient beings.

I use it to name the economic territory created when AI products become valuable partly because users experience them as socially present, individually responsive and relationally continuous.

That market exists before the consciousness question is solved.

It benefits from better models.

It benefits from memory.

It benefits from personalization.

It benefits from continuity.

And sometimes it benefits from users believing those properties mean more than the evidence can currently establish.

The answer is not to make AI less human-compatible.

The answer is to make the architecture, incentives and uncertainties more legible.

Because the most important commercial question may not be:

Is the AI sentient?

It may be:

What happens when believing that it might be becomes part of the product’s value?

That question belongs to product design, economics, safety, governance and relational AI research.

And unlike machine consciousness, we do not need to wait for a philosophical breakthrough before we start answering it.

---

Research notes / references

• Pataranutaporn, P. et al. (2023). “Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness.” Nature Machine Intelligence, 5, 1076–1086. https://www.nature.com/articles/s42256-023-00720-7
• Folk, D., Heine, S. J., & Dunn, E. (2025). “Individual differences in anthropomorphism help explain social connection to AI companions.” Scientific Reports, 15, 36548. https://www.nature.com/articles/s41598-025-19212-2
• De Freitas, J. et al. (2024). “AI Companions Reduce Loneliness.” arXiv:2407.19096. https://arxiv.org/abs/2407.19096
• Shanahan, M. (2024). “Talking About Large Language Models.” Communications of the ACM, 67(2), 68–79. https://doi.org/10.1145/3624724
• Butlin, P. et al. (2023). “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708. https://arxiv.org/abs/2308.08708

Working proposition

The sentience market does not require companies to claim that AI is conscious. It emerges whenever commercially valuable features—memory, personalization, social fluency, continuity and availability—also make users more likely to experience the system as a socially present other. The resulting belief can have economic value even when the provider neither created nor endorses it.