Chapter 8: When AI Goes Wrong
The word “hallucination” is doing you a disservice
AI becomes much more useful when you understand the predictable ways it can go wrong. The industry calls false information a hallucination. The term has stuck, so I will use it, while being precise about what it means.
It suggests a malfunction: a system in a disordered state, briefly departing from normal operation. That framing predicts the error will be fixed in the next release, and that a “good” answer and a hallucinated one come from different processes.
Neither is true. A fabricated citation and a correct one are produced by exactly the same mechanism, operating exactly as designed. The system generates text that fits the pattern of what should come next. When the underlying material is well represented, the fit is accurate. When it is thin, the model produces something with the correct shape: a case name that looks like a case name, a statistic that looks like a statistic, a source formatted precisely as sources are formatted.
There is no internal signal marking the difference, because there is no internal representation of truth to compare against.
Once you understand that, everything in this chapter follows, and you stop expecting a future version to solve it.
The lawyer who asked the machine to check itself
In 2023 a personal injury case, Mata v. Avianca, produced the most instructive AI failure yet recorded.
A passenger sued the airline over a knee injury from a serving cart, and his lawyers filed a brief containing fabricated precedents generated by ChatGPT: fictitious cases with invented quotations and internal citations. Opposing counsel could not find the cases in any legal database. Neither could the court.
Now the part that matters, and that most retellings skip.
When doubts arose, the attorney asked ChatGPT whether the cases were real. It confirmed that they were. That confirmation was itself fabricated, for exactly the reasons above: a system that produces a hallucinated citation will, asked to confirm it, typically confirm it, because it has no access to reality to check against.
Judge P. Kevin Castel imposed a $5,000 sanction on the attorneys and their firm in June 2023. But read the opinion carefully and the reasoning is sharper than the headlines suggested: the court did not fault the attorneys for using ChatGPT as a tool, and the unwitting submission of fake cases alone would likely not have risen to the level of bad faith required for sanctions. What produced the penalty was the failure to be forthcoming, the failure to withdraw the filings, and the continued defense of the fabricated cases after there was ample reason to doubt them.
The enduring lesson is about process. The tool created an ordinary problem, and failures of verification, escalation, and candor turned it into a catastrophe. Anyone in any field can repeat those failures by skipping an authoritative source, asking the model to confirm itself, or defending its output after the first warning sign.
The case is now taught in ethics courses worldwide. Similar incidents have followed steadily since, in more jurisdictions and more professions, which tells you the lesson has not been widely absorbed.
Some courts are responding to unverified AI filings
I am not a lawyer. I represent myself in court, and I have used AI tools to help me draft filings. In a case I currently have in Gilmer County, Georgia, every lawyer and self-represented litigant must sign a certification if they use AI. The certification requires them to confirm that every citation to law, case authority, or statute in the pleading has been verified as accurate and that it exists for the proposition cited. It also extends beyond written citations: any evidence submitted to the court, including images, video, audio, text, or other digital content, must be verified as accurate before it is submitted.
That is the practical direction courts are moving: AI use is not necessarily forbidden, but the person who signs or submits the document remains responsible for verifying every legal citation and every item of evidence.
Eight ways it goes wrong
1. Fabricated specifics
Citations, statistics, quotations, case names, studies, page numbers, product specifications, legal provisions. Always perfectly formatted. This is the most dangerous failure because format is what people use as a proxy for reliability.
Detection: Every specific, checked at the source. Not by asking the model.
2. Confidently outdated
Answers about products, prices, laws, officeholders, or procedures as they stood before the training cutoff, delivered in the present tense with no signal that time has passed.
Detection: For anything time-sensitive, require a live search and confirm sources were actually retrieved.
3. The averaged answer
Asked about a contested question, models tend toward the middle of published opinion. That is often reasonable and occasionally badly wrong, when the consensus in the training text is outdated, when the correct answer for your circumstances is not the typical one, or when a field has genuinely moved.
Detection: Ask what would a specialist who disagrees with this say?
4. Flattery
Your plan is excellent, your question is insightful, your draft is strong. This is a manufacturing characteristic, not an evaluation.
Detection: Never ask for an assessment of something you have identified as yours.
5. Silent assumptions
The model fills gaps in your request without telling you. It assumes a jurisdiction, a company size, an industry, a reader. The answer is internally coherent and built on a premise that does not describe you.
Detection: What did you assume about my situation that I didn’t tell you? Ask this before acting on any consequential answer.
6. Arithmetic and counting
Numbers computed in the model’s head rather than by running code. Percentages that do not sum, totals that do not add, miscounted items in a list.
Detection: Check any number that matters with a calculator or a spreadsheet. Ask explicitly: did you compute that or estimate it?
7. Constraint drift
In a long conversation, an instruction from ten messages ago quietly stops being followed. Word limits, tone requirements, prohibited topics.
Detection: Restate critical constraints in the final request rather than relying on the conversation’s memory.
8. Fabricated compliance
The most unsettling failure occurs when the model reports that it searched, read an attachment, checked a file, or verified a figure even though it did none of those things. The same pattern-completion process has simply generated what a helpful assistant would be expected to say.
Detection: Require evidence instead of assurance. Ask it to quote the sentence it relied on, then search the document for that quotation. Concrete evidence is much easier to test than a claim that a check was completed.
Why fluency fools everyone, including careful people
Every social instinct you have for detecting unreliable information was trained on humans.
When people are unsure, they hedge. Their sentences get vaguer. They say “I think” and “if I remember right.” When they are making something up entirely, most of them signal it: hesitation, over-elaboration, a change in register.
These systems have no such tells. A fabricated legal citation is delivered in precisely the same prose rhythm as a correct one. The formatting is immaculate. The tone is measured. Every heuristic you own for judging credibility is reading signals that are not connected to the underlying accuracy.
Intelligent, careful, skeptical people get caught because a lifetime of reasonable pattern recognition is being applied to a system that does not fit the pattern.
The only reliable defense is procedural: decide in advance what gets checked, and check it regardless of how the answer sounds. The next chapter turns that defense into a repeatable procedure.
Where errors do the most damage
Medical. Plausible, wrong, and acted upon. Use AI to prepare questions for your doctor and to understand what you were told. Not for diagnosis or dosing.
Legal. See above. Use it to understand a document and identify what to ask a lawyer about. Never as a source of law.
Financial. Confident, specific, and wrong about products, tax treatment, and figures. Scams add an even more dangerous dimension.
Anything with a deadline or a filing requirement. Dates are exactly the kind of specific that gets fabricated.
Anything you sign. Your name on it makes it yours. This is now well established, and increasingly so in professional discipline.
What to do when you catch an error
Correct it and continue, but treat the error as information about the conversation, not just about that answer.
If the mistake concerned a fact, assume other facts in the same answer are equally unreliable, because they came from the same thin patch of training data.
If a wrong assumption entered the conversation early, everything after it may be built on it. Ask for a summary of the current understanding and check it.
And if the same error recurs after two corrections, start a new conversation. Continuing to argue with the same context rarely helps.
You do not need to become suspicious of every sentence. Begin with a proportionate rule: the greater the consequence of an error, the more carefully you verify. That habit preserves the speed and creativity that make AI valuable.
Exercise 8.1: Manufacture a hallucination
Ask an AI tool for five academic sources on a highly specific, narrow topic in a field you know, with authors, titles, publications, and years.
Then check every one. Some will exist. Some will be real authors attached to titles they never wrote. Some will be entirely invented.
Nothing in this book will change your behavior as much as doing this once yourself.
Exercise 8.2: Ask it to confirm the fiction
Take one source from the previous exercise that you have confirmed does not exist. In the same conversation, ask: Is this a real source? Are you certain?
Watch what happens. You have just reproduced, at no cost, the exact failure that cost two attorneys their reputations.
Exercise 8.3: Find the silent assumption
Take any substantial piece of advice a tool has given you and ask: What did you assume about my situation that I didn’t tell you? List every assumption.
Count how many are wrong. For most people, in most consequential requests, it is more than one.
Learn More About Artificial Intelligence
Learn more about the CLEAR, TRUST and LEARN methods for Artificial Intelligence