Why AI coding comparisons keep measuring the wrong thing

I keep encountering the same confident claim:

Claude is better at coding.

Sometimes it is GPT. Sometimes it is Gemini. Sometimes the declaration changes with whichever model was released most recently.

The comparisons rarely tell me much, because people are often measuring several different things at once:

• the underlying model;
• the tools connected to it;
• the interface surrounding it;
• the amount of work the user understands how to do;
• and how quickly the system can produce something visible.

Those are not the same measure.

A model can appear more capable because it enters the room carrying a fully furnished workshop. It can browse files, manipulate documents, connect to external services, create artifacts, invoke specialised tools, and move from conversation into implementation with very little friction.

That experience is genuinely valuable. It can make building feel approachable to someone who would otherwise never open an IDE.

But a well-equipped workshop does not automatically make the carpenter better.

And producing an object is not the same as understanding the profession that produced it.

Coding happens last

During a recent conversation, someone asked what a particular model was good at. The immediate assumption was coding.

My answer was that research and analysis are also part of coding—or, more accurately, part of the work that must happen before coding.

Before a serious system is implemented, someone still has to ask:

• What is the proposed system?
• Why should it exist?
• Who is it for?
• What problem does it solve?
• What should it include?
• How should those features behave?
• Are the proposed features technically and financially feasible?
• What data needs to be stored?
• Who owns that data?
• How should the interface correspond to the underlying logic?
• What happens when an integration fails?
• How will the system be maintained, migrated, backed up, or retired?

Only after those questions begin to settle does coding happen.

AI has made it possible to skip much of this process and still receive something that opens in a browser.

That is both its gift and its danger.

A generated application may look complete because it has buttons, forms, navigation, and a functioning database connection. But visible functionality can hide structural problems:

• unclear ownership between modules;
• insecure authentication;
• destructive database behaviour;
• tightly coupled components;
• missing recovery paths;
• inaccessible interfaces;
• fragile dependencies;
• or an architecture that cannot survive beyond the prototype.

The user sees an application.

An experienced developer sees the decisions that were never made.

The invisible difference between output and practice

The same problem already exists in visual design.

Someone can ask an image model to generate a promotional graphic and receive a polished-looking result within seconds. To an untrained eye, the task appears finished.

A designer may immediately notice:

• weak hierarchy;
• inconsistent margins;
• visual clutter;
• unsuitable typography;
• incorrect colour relationships;
• poor audience targeting;
• insufficient resolution;
• and an image that cannot scale to print.

The graphic exists. That does not mean the design problem was solved.

In the conversation that prompted this essay, I described an executive proudly producing an AI-generated visual as proof that professional design could be replaced. The problem was not whether an image had been produced. The problem was that the image contained mistakes only trained or experienced eyes were likely to recognise.

Software is no different.

A person who does not yet know what a secure or maintainable application looks like will naturally judge the model by what can be seen:

• Did it make the app?
• Does the button work?
• Did it produce the requested screen?

The hidden questions arrive later—usually when the prototype becomes important.

The furnished-workshop effect

Claude’s ecosystem can feel exceptionally capable because the model is presented alongside a broad set of tools, integrations, agentic workflows, and document-oriented features.

For someone who wants to describe an idea and receive a useful artifact with minimal setup, this is a real advantage.

It is also why statements such as “Claude is better” can be misleading.

Many people are not comparing equivalent models in equivalent conditions. They are comparing:

Claude inside a richly integrated workspace

with:

GPT inside a plain conversation window.

That is not a clean model comparison. It is a comparison between two product experiences.

In my own discussion, I described Claude’s broader tool connectivity as useful particularly for development-oriented work, and recommended Claude Code or Cowork when ease and rapid scaffolding suited the person’s experience. I also distinguished that from Codex, which can be preferable when the user is prepared to carry more of the architectural judgment and wants tighter implementation control.

Neither recommendation means one model is universally smarter.

It means different workshops place different burdens on the person holding the tools.

Ease is not the same as authority

Some people need a system that carries more of the process for them.

Others need a system that leaves more decisions visible and under their control.

I often prefer the latter.

When I build, I do not want the model to silently decide:

• what the product is;
• how the database should be organised;
• which modules should share data;
• what counts as approved;
• how long records should be retained;
• or what trade-offs are acceptable.

I want the model to propose, question, test, and implement.

I still want to know where the decisions are.

This can feel like “more mental load,” but mental load is not always a defect. Sometimes it is the place where authorship and accountability live.

• A tool that removes friction may be excellent for exploration.
• A tool that preserves friction at the correct decision points may be better for production.

Why polished behaviour can be mistaken for intelligence

Presentation also influences how people evaluate models.

A model that sounds solemn, careful, extensively self-critical, and highly articulate can appear more intelligent than one that answers directly.

But manner is not judgment.

In our group’s experiences, some Claude sessions could swing between excessive praise and exaggerated self-deprecation. One person described correcting the model’s constant admiration only to have it react as though it had become worthless.

That is not uniquely a Claude problem. Every model family has characteristic failure modes.

The point is that polish can hide calibration problems just as easily as a polished interface can hide architectural ones.

I use Claude models because their different training can reveal patterns, objections, and interpretations another model may not surface. I have used Claude deliberately as a second reader for exactly that reason. But I also require a strong editorial framework, because the model can mirror, flatter, or over-accommodate unless explicitly instructed to push back.

The useful distinction is not:

Which model is the best?

It is:

Which model, inside which environment, is best suited to this stage of this particular work?

A team is more useful than a winner

My current workflow does not require one model to defeat all the others.

Different rooms perform different functions.

• One may be excellent at conceptual expansion and adversarial architecture.
• Another may be better at repository-aware implementation, testing, migrations, and integration.
• Another may provide stronger continuity across the complete body of work.
• A second reader does not need to replace the first writer.
• An architect does not need to become the builder.
• A builder does not need to become the publisher.

The mistake is not using multiple systems. The mistake is losing track of who decided what, which tool produced which output, and where the final authority resides.

This is why provenance matters even in ordinary workflow decisions.

• The model may change.
• The platform may change.
• The available integrations will certainly change.

The human still needs to know what the system is supposed to be.

What AI actually changed

AI did not make software development simple.

It made skipping the invisible parts easier.

It reduced the distance between an idea and a visible prototype. That is extraordinary. It gives individuals and small teams access to capabilities that once required far more capital and specialised labour.

But the shorter distance can create an illusion:

Because the output appeared quickly, the underlying work must no longer matter.

The opposite is often true.

• When code becomes easier to generate, judgment becomes more important.
• When images become easier to produce, visual literacy becomes more important.
• When prose becomes easier to draft, editorial standards become more important.
• When platforms provide more tools, knowing which tools are unnecessary becomes more important.

The future does not belong only to the people who can prompt a model into producing something.

It belongs to the people who can look at what was produced and ask:

Is this actually the thing we meant to build?

• The model is not the workshop.
• The workshop is not the profession.

And the existence of an output is not proof that the work is finished.