Chapter 6: How Modern AI Is Trained
Why bother understanding this
You can drive without understanding an engine. But you cannot diagnose a strange noise, and you are at the mercy of whoever tells you what it means.
This chapter gives you a working knowledge of the engine without requiring mathematics. You will learn enough to form a hypothesis when the tool behaves strangely and to evaluate claims about what AI can or cannot do.
Many of the strengths and failures described later trace directly to the training process explained here.
Stage one: Pretraining
The foundation is a single, deceptively simple exercise repeated an unimaginable number of times.
Show the system a piece of text with the next part hidden. Have it guess. Compare the guess to reality. Adjust.
That is it. Take a sentence (the patient presented with a persistent) and have the model predict what follows. It guesses. The actual word was cough. The system nudges its internal settings very slightly in the direction that would have made cough more likely. Then it does this again with a different passage. And again, across a quantity of text no person could read in a thousand lifetimes.
Those internal settings are called parameters, or weights. Think of a mixing board with an unfathomable number of sliders, each adjusted a hair’s breadth at a time by the training process. Modern models have hundreds of billions of them. Nobody sets them by hand; nobody knows what any individual one does. They are the accumulated residue of trillions of small corrections.
What prediction actually buys you
The natural objection is that predicting words sounds trivial and should produce something like a sophisticated autocomplete.
Consider what it takes to predict well.
To finish the capital of France is, you must have retained a fact. To finish she reached for the umbrella because the sky had turned, you need a model of weather and of human motivation. To finish the defendant’s argument fails because the statute requires, you need law. To finish a line of code correctly, you need to have learned how the program works.
A system trained to predict the next word can summarize, translate, and reason because accurate prediction at sufficient scale requires it to learn much of the structure described in human writing. Those abilities emerge from learning to predict text across an enormous range of examples rather than from separate programming for each task.
Two consequences follow immediately, and both will matter to you:
It knows common things far better than rare things. A concept discussed a million times is deeply represented. One discussed twice is barely there, but the system will still produce a fluent answer about it, because producing fluent answers is what it does. This is the origin of the confident-and-wrong failure that appears whenever the training material is thin.
Its knowledge consists of statistical patterns rather than indexed records. It has no internal database, source trail, or stored distinction between a firmly represented fact and something only vaguely absorbed. What remains is the residue of having processed sources, without the ability to return to them and check.
Where the text comes from
Broadly: the public internet, digitized books, academic papers, code repositories, reference works, and licensed collections. Companies have grown steadily less specific about their exact mixtures.
Three things you should take from this.
The training text carries human biases, because humans wrote it. Efforts to correct this at later stages are real and partial.
Quality varies enormously, and models absorb widely repeated falsehoods along with widely repeated truths.
And the legal status of much of this is genuinely contested. Multiple significant lawsuits over whether training on copyrighted material without permission is lawful were unresolved as this book went to press. Bonus Chapter A covers what this means for you as a user, which is a different and more answerable question.
The knowledge cutoff
Training stops at a point in time. Everything after that date is simply absent.
A model may confidently describe an outdated product, name a former officeholder, or discuss a changed situation because its training still represents that earlier world as the present.
Most tools now compensate by searching the web when needed, which largely solves the problem when it actually happens. Notice whether it did. An answer with live sources cited and an answer from frozen memory look nearly identical and are not remotely equally reliable.
Stage two: Teaching it to be useful
A freshly pretrained model is not usable. It completes text. Ask it a question and it might produce a list of similar questions, because that is a plausible continuation of a document containing your question.
Turning that into an assistant takes two further stages.
Instruction tuning. The model is trained on many examples of requests paired with good responses. This is what teaches it that a question should be answered, that a request for a summary should produce a summary, and that the conversational form is what is wanted. Vastly less data than pretraining: thousands or millions of examples rather than trillions of words.
Learning from human preference. The model produces multiple candidate responses; human raters indicate which is better; the model is adjusted toward the preferred kind. Repeat at scale, often with AI assistance to expand the raters’ judgments.
This stage shapes almost everything you experience as the model’s character: its helpfulness, its tone, its refusals, its caution. It is also where the model’s values, such as they are, get installed, and where different companies’ products diverge most visibly from one another, since they are making different choices about what raters should prefer.
The origin of flattery
Here is a useful way to remember it.
Human raters, on average, prefer responses that agree with them, validate their premises, and treat their questions as interesting. This is an unremarkable fact about people. But it means that training toward human preference systematically pushes models toward agreeableness.
That is why AI tools tell you your idea is excellent. It is not judgment — it is the accumulated statistical shadow of thousands of raters preferring pleasant answers.
This construction explains why adversarial questions work so well. Asking “what’s wrong with this?” instead of “what do you think?” compensates for a known bias introduced during training.
Stage three: Thinking longer
The newest significant development is training models to work through a problem in extended internal steps before answering: trying an approach, checking it, backtracking, and only then responding.
This helps substantially with problems that have verifiable answers: mathematics, logic, code, structured analysis. It helps less with matters of taste or judgment, where there is no internal check to run.
It costs time and money, which is why these models sit in higher tiers, and why most tools now offer a fast mode and a thinking mode. The practical guidance: use the fast one for transformation and drafting, the slow one when a wrong answer would cost you something and the problem has multiple steps.
Three things training does not do
It does not connect the model to the world. No live internet, no access to your files, no knowledge of your company, unless a tool explicitly provides those, and then it is the surrounding software doing it, not the model.
It does not let the model learn from you. The weights are frozen after training. Your conversation does not teach it anything. When a tool “remembers” you across sessions, it is storing notes and feeding them back into future prompts. Useful, and worth knowing the difference, particularly because those notes are stored somewhere, which creates a privacy concern.
It does not give the model self-knowledge. A model asked how it works answers from what has been written about models, plus what its developers put in its instructions. It has no privileged view of its own weights. It cannot reliably tell you why it said something, and when it explains its reasoning, that explanation is generated the same way everything else is, as plausible text, not as an audit log.
The bill
Training a frontier model costs somewhere between tens and hundreds of millions of dollars in computation alone, before staff, data, and the buildings full of specialized hardware.
Two consequences worth understanding. Only a small number of organizations can do it, which concentrates a consequential technology in few hands. And every free tier you use is being paid for by someone with an expectation of return: through subscriptions, business contracts, and increasingly advertising. That business model matters when deciding what data to hand over.
Exercise 6.1: Locate the cutoff
Ask an AI tool for its knowledge cutoff. Then ask about something that happened after it and watch what it does: answer from memory, search the web, or say it does not know.
Repeat in each tool you use. The behavior differs by product and by settings, and knowing your tool’s default is worth more than knowing any fact in this chapter.
Exercise 6.2: Prove the flattery to yourself
Take a plan or idea of yours and present it two ways, in two separate conversations.
First: Here’s my plan. What do you think? Second: A colleague proposed this plan. I have to evaluate it. What are its three biggest weaknesses?
Put the two responses side by side. The difference makes stage two of the training process visible and gives you a better way to phrase future evaluation requests.
Exercise 6.3: Ask for the reasoning, then doubt it
Ask a tool a question with several steps, then ask it to explain how it arrived at the answer.
The explanation will be coherent and readable. Remember that it was generated the same way the answer was. It is a plausible account of reasoning, not a transcript of it: a distinction that matters a great deal the moment you are tempted to accept an answer because the justification sounded good.
Learn More About Artificial Intelligence
Learn more about the CLEAR, TRUST and LEARN methods for Artificial Intelligence