The Trial of AI: Who Pays for the Hallucination?

Ibrahim Fawakherji — ArchUp

I believe history, twenty or thirty years from now, will not divide the world into before and after the internet.

It will draw a different line entirely.

Before AI. And after AI.

When I open my email archive from 2020 today, I feel as though I am reading correspondence from a different civilization. Not because the language has changed. Because the people behind it have.

The email before 2022 was human in a way that is difficult to describe precisely until you compare it to what came after. There were spelling errors. Incomplete sentences. Ideas that had not been fully developed before being sent. But behind every message, even the poorly written ones, you could feel a person making an effort in real time. The imperfection was evidence of presence.

Today the message is longer. More structured. More diplomatic. More polished. Sometimes more than it needs to be. AI does not simply correct the text. It adds layers: introductions, transitions, conclusions, rhetorical framing that the original sender may not have intended and might not even recognize as their own voice. People are now routinely sending messages that do not resemble them. The machine has added what might be called an aura around every idea, a layer of eloquence that inflates the signal and sometimes obscures it.

This is not a complaint about writing quality. It is an observation about what happens to communication when a tool begins to substitute for the communicator.

If I wanted to describe what has happened to knowledge in the AI era using a single image, I would use a snowball.

Before large language models, knowledge accumulated slowly. An article here. A book there. A research paper. A project. A conversation. Human understanding built itself through friction, through the effort of assembling pieces that did not always fit together.

Then the models arrived. They read everything. They compressed everything. They learned from the aggregate of human textual production across decades and encoded that learning into weights and parameters that could be queried in seconds.

Then we began using those models daily. And every interaction generated new data. Every correction, every refined prompt, every judgment about which output was better than another, fed back into the system in ways that vary by company and by policy but that collectively contribute to the continued development of the models. The snowball grew. Then we used it again. Then it grew again.

This is the closed loop of the current era: the AI learns from humanity, then humanity learns to work with the AI, then the AI is trained further on the results. Each turn of the cycle produces a model that is in some respects more capable than the last.

But capacity is not the same as accuracy. And this is where the problem begins.

As the snowball grows, something else grows with it.

Hallucination.

The term has a precise technical meaning in AI research: the model generates output that appears coherent, confident, and contextually appropriate but is factually incorrect or simply invented. The model does not know it is wrong. It has no mechanism for knowing. It produces what is statistically most likely given the input, and when the statistical patterns in its training data lead it toward a confident error, it commits to that error with the same fluency it would bring to a correct answer.

In text generation, hallucinations can be caught by a reader who knows the subject. A fabricated citation, an incorrect date, a misattributed quote, these are detectable with verification.

In architectural visualization, the consequences are different in kind.

Several months ago I was working on a project with a completed three-dimensional model. The massing was correct. The proportions were resolved. The street geometry was accurate. The prompt was as specific as I could make it. All I needed was image enhancement, a realistic rendering of what the model already showed.

The AI changed the street width.

Then it altered the sidewalk alignment.

Then it added elements that were not in the model.

Then it removed a section of the building facade.

Then it reconstructed the entrance sequence according to what it apparently expected a building entrance to look like, regardless of what I had modeled.

I submitted the prompt again. Tokens were consumed. The model made different errors. I submitted again. More tokens. The same category of error, expressed differently.

At no point did the system flag its own uncertainty. At no point did it indicate that the output deviated from the input. It simply produced what it produced, charged what it charged, and waited for the next instruction.

The question this experience raises is not technical. The technical limitations of current image generation models are known and documented. The question is economic and ethical.

Most professional AI tools are no longer free. There are subscriptions, credits, token budgets. Every inference, every image generation, every attempt carries a cost that is deducted from an account the user maintains.

This pricing model is legitimate when the tool performs as described. But when the error originates in the model, when the input is clear and the output is wrong, the current arrangement places the cost of the error entirely on the user. The failed attempt costs the same as the successful one. The token consumed by a hallucination is indistinguishable in the billing system from a token that produced accurate output.

Consider how other professional service relationships handle this. A contractor who delivers work that does not meet the specification bears responsibility for remediation. A supplier who provides defective material absorbs the cost of replacement. A consultant whose analysis contains errors is accountable for correcting them. These are not exceptional principles. They are the basic structure of any professional relationship in which payment is exchanged for a defined level of performance.

The AI service relationship currently operates outside this structure. The user pays for access to the model, not for a guaranteed outcome from it. Terms of service make this explicit: the output is provided as-is, and the provider assumes no liability for errors in it. This is legally defensible and practically unsurprising for a technology that is still developing.

But as these tools transition from experimental to infrastructural, as they become embedded in professional workflows that carry real financial and legal consequences, the current arrangement will face increasing pressure.

There is another dimension to this that the industry has not fully reckoned with.

The large language models and image generation systems that power these tools were trained on human-produced content at a scale that had no precedent. The text of the internet, including architectural publications, academic research, professional forums, project documentation, and the output of practitioners like the ones reading this article, formed the corpus on which these models learned to do what they do.

The users who are now paying for access to these models are, in many cases, the same people whose work contributed to training them. This is not a legal claim. The copyright questions around AI training data are genuinely complex and still being contested in courts across multiple jurisdictions. But it is a structural observation that matters for how we think about the relationship between practitioners and the platforms they use.

When a professional whose documented work helped train a model is then charged for each failed inference that model produces, the economic logic is at minimum worth examining.

I want to be direct about what I am not arguing.

I am not arguing that AI tools should be free. Development, infrastructure, and compute cost real money, and pricing models that reflect those costs are legitimate.

I am not arguing that AI should produce perfect outputs. No tool does, including the ones human practitioners have used for decades. Errors are expected and manageable.

What I am arguing is that the current accountability structure, in which the user bears the full cost of errors that originate in the model, is not sustainable as these tools become professional infrastructure rather than consumer novelties.

The architectural profession has mature frameworks for this. When a structural engineer stamps a calculation, they are accepting liability for its accuracy. When a materials manufacturer certifies a product, they are committing to its performance under specified conditions. These accountability structures exist because the built environment has real consequences, and the parties who shape it must be capable of being held responsible for the quality of their contribution.

As AI enters the professional workflow at the level it is currently entering, the question of accountability for AI errors becomes a professional question, not just a consumer rights question.

Who is responsible when the AI changes the street width? When it removes a structural element from a rendering that a client will use to make a construction decision? When the hallucination is not caught before it influences a real outcome?

Currently, nobody is responsible. The platform has disclaimed liability. The user produced the prompt. The output exists in a gap between human intent and machine execution that has no clear owner.

This will not remain the case indefinitely.

As AI tools are used in more consequential contexts, the legal and professional frameworks around their outputs will develop. Some jurisdictions are already moving toward requirements that AI-generated content be disclosed and that liability for AI errors be addressed in the contracts governing professional services.

What that future framework looks like will be determined in part by how clearly practitioners articulate the problem now, while the norms are still being established.

The architectural profession, which has more experience than most in navigating the relationship between tools and professional accountability, has a contribution to make to this conversation.

The tool that draws the wrong street width while consuming a budget I cannot recover is not a neutral instrument. It is an actor in a professional process that has stakes.

Treating it as one, with appropriate expectations of performance and appropriate mechanisms for accountability when those expectations are not met, is not hostility to technology.

It is the basic condition of using any tool professionally.

✦ ArchUp Editorial Insight

The hallucination liability gap the article describes is not a regulatory oversight awaiting correction — it is a deliberately constructed contractual architecture in which the platform’s terms of service perform the same function that the developer’s exit from the asset performed in every case this archive has previously examined: the party who designed the condition of failure has legally vacated the position of responsibility before the failure materializes in a professional context that carries real consequences. The token billing model is not incidental to this arrangement; it is its mechanism, because charging for inference regardless of output accuracy converts the error itself into a revenue event, which means the platform has no structural incentive to reduce hallucination rates below the threshold at which users abandon the tool entirely — a threshold that, as Architectural Design for Twenty Dollars established, is kept artificially low by the switching cost that platform dependency has already installed. The article’s most structurally significant observation — that the practitioners whose documented work trained these models are now purchasing access to the derivative of their own collective output and absorbing the cost of its errors — describes a double extraction that has no precedent in the professional tool economy: the knowledge was taken without compensation in the training phase, and the failure of that knowledge’s deployment is charged to its original producers in the inference phase, while the platform occupies, in both moments, the position of the party who controls the terms and bears none of the liability that the built environment will eventually, and physically, demand of someone.