Most explanations of LLMs do one of two things. They drown you in matrix algebra, or they wave their hands and say “it’s basically predicting the next word.” Neither helps if you want to build, fine-tune, or deploy one of these models and form accurate expectations about what will and will not work.
This article sits in the middle. It assumes you can write code and think in systems, but not that you have read a single ML paper. The evidence comes from 18 papers read together, from the original Transformer through to Qwen3’s four-stage post-training pipeline. The full collection is on GitHub with an interactive page for each paper and a cross-paper synthesis. This is the short version, and it is organised around a single idea rather than a tour of papers.
The idea is this: pretraining is where an LLM acquires its knowledge, and everything that happens afterwards is about surfacing that knowledge in the right form, not adding more of it. Once you hold that idea, a lot of confusing results stop being confusing. Why a rank-4 matrix can adapt a model with billions of parameters. Why 4-bit quantisation loses almost nothing. Why a thousand examples can align a 65-billion-parameter model. Why one reinforcement-learning method makes a model choose its existing answers more consistently without making it able to solve anything new. These are not separate curiosities. They are the same fact seen from different angles.
What the machine actually is
Strip away the infrastructure and an LLM is a function. It takes a sequence of tokens and produces a probability distribution over what token should come next. Everything else, the data centres, the benchmarks, the chat interfaces, is engineering to make that function larger, faster, and more useful.
Text enters the model as tokens, which are subword chunks produced by a tokenizer. A word like “understanding” might become two tokens, “under” and “standing”. Each token maps to a vector, a list of numbers in a high-dimensional space, typically around 4,096 dimensions in modern models. These vectors are not assigned by hand. During training the model learns to place semantically related concepts near each other, so “king” and “queen” end up close, and the direction from “Paris” to “France” runs roughly parallel to the direction from “Berlin” to “Germany”. The model learns this geometry purely from seeing which words co-occur across billions of sentences.
The mechanism that makes the whole thing work is attention, introduced in the 2017 Transformer paper and still the core of every major LLM. For each token, the model computes three vectors: a query (what am I looking for), a key (what do I contain), and a value (what should I pass along if someone finds me). Every token compares its query against every other token’s key, and the resulting scores determine how much of each token’s value gets mixed into the output. In “the cat sat on the mat”, the token “sat” attends strongly to “cat” and to “mat”. Multi-head attention runs several of these comparisons in parallel, each head learning to track a different kind of relationship: syntax, coreference, positional adjacency.
Stack dozens of attention layers with feed-forward networks in between and you get a model that learns grammar, factual knowledge, and reasoning patterns from a single training objective: predict the next token. Nobody programs subject-verb agreement. Nobody encodes that water boils at 100 degrees. The model absorbs these regularities from the statistical structure of the text it sees.
That is pretraining, and it is the expensive part. It needs thousands of GPUs, weeks or months of compute, and terabytes of carefully filtered text. The output is a base model that can complete text fluently, but if you ask it a question it will often continue your sentence as though it were a paragraph in an essay. It does not yet know how to be an assistant. Every paper published after the original Transformer is about one of the stages that turn a base model into something useful, and the distinction between those stages matters more than most tutorials suggest.
The one idea: pretraining encodes, post-training surfaces
Before walking through the pipeline, it is worth stating the organising principle directly, because it is the thing that makes the rest cohere.
Read the 18 papers together and a pattern emerges that no single paper states outright: every post-training method is fundamentally the same operation. Take the probability distribution the pretrained model already produces, apply some constraint or reward signal, and concentrate probability mass on the desired region. Knowledge is almost never added. It is surfaced.
This is not a claim any one author makes. It is what becomes visible when you line the results up:
- LoRA learns weight updates that are largely orthogonal to the pretrained model’s main directions, which is why a rank-4 or rank-8 matrix is enough to adapt a model with billions of parameters.
- DeepSeekMath’s GRPO improves a model’s majority-vote accuracy (Maj@K) without improving its single-shot accuracy (Pass@K). The correct solutions already existed in the model’s output distribution. Training just made the model choose them more consistently.
- DPO reweights the log-ratios of outputs the model could already produce.
- Qwen3’s “cold start” stage teaches the model to exhibit reasoning it already possesses, not new reasoning ability.
- LIMA’s thousand examples unlock assistant behaviour that was already latent in the weights.
The practical reframing is this: pretraining encodes capabilities as latent potential, and post-training makes them accessible. The distinction between adding a capability and surfacing one is close to the organising principle of the whole field.
It also explains an otherwise puzzling set of numbers. If fine-tuning were adding new knowledge, a rank-4 matrix would not have enough capacity to store it, 4-bit quantisation would destroy it, and a thousand examples could never teach it. The fact that all three work is evidence that the knowledge was already there. The collective suggestion across these papers is that the actual knowledge content of these models is far lower-dimensional than the parameter count implies. Most parameters appear to be scaffolding: they encode the structure of how concepts relate and how reasoning flows, rather than storing the content itself. That is an inference from the evidence, not a result any paper proves directly, and the collection flags the obvious unasked question: if knowledge is this compressible, what are the rest of the parameters doing? But it is the inference that best fits the data.
The pipeline is three different operations
The standard description says a model is pretrained, then fine-tuned, then aligned. That phrasing hides the fact that “alignment” is used for two completely different operations, and conflating them is the source of a lot of confused claims about alignment cost.
Stage one: supervised fine-tuning
You take the base model and train it on examples of the behaviour you want: question-answer pairs, instruction dialogues, code completions with explanations. This is where the model learns to respond in a helpful format instead of rambling.
The surprising finding, from LIMA, is how little data this takes. LIMA fine-tuned a 65-billion-parameter LLaMA model on 1,000 handpicked examples, with no reinforcement learning or preference optimisation, and in a controlled human study its responses were judged equivalent or strictly preferable to GPT-4 in 43% of cases (rising to 58% against Bard and 65% against DaVinci003). The authors called this the “superficial alignment hypothesis”: almost all the knowledge is learned during pretraining, and only a small amount of instruction data is needed to teach the model to produce high-quality output. The evidence from several papers supports the reading that SFT changes output style more than underlying knowledge.
There is a cost, though, and it produces the first genuine paradox. Several papers report a 1 to 5% drop in factual accuracy after SFT, the so-called alignment tax. If alignment only teaches format, why does knowledge degrade? The best available explanation is that format-matching shifts weights, and some of those weights were storing factual information. The thousand examples are a lossy compression of the desired output distribution, and that compression corrupts a little of the pretraining signal. The alignment tax is real even when alignment is “superficial”, and the two facts are consistent once you accept that SFT operates on the weight space, not just the output format.
Stage two: preference alignment
SFT teaches a format. Preference alignment teaches the model which responses are better within that format. The standard approach used to be RLHF: train a separate reward model on human preference rankings, then use reinforcement learning to optimise the language model against it. Two models, two training loops, significant instability.
DPO simplified this. Its central result is that the RLHF objective can be rewritten as a classification loss over preference pairs. Instead of generating a response, scoring it, updating the policy, and repeating, you feed the model pairs of good and bad responses and it learns to assign higher probability to the good ones. The normalising term cancels out, which is what lets you drop the separate reward model. Same outcome, one training loop. ORPO went further by merging SFT and preference learning into a single stage using an odds-ratio loss, so the model learns format and quality at once.
ZEPHYR demonstrated that you can replace human feedback entirely with GPT-4 scoring. A 7-billion-parameter model trained this way surpassed Llama2-Chat-70B on MT-Bench. Read chronologically, the alignment papers show human annotation being systematically eliminated: SELF-INSTRUCT bootstraps instruction data from 175 seed examples, ZEPHYR replaces human preference labels with AI feedback, and Qwen3 distils from a 235-billion-parameter teacher model. The trajectory points toward a large model teaching a small model with no human in the loop.
That trajectory has a ceiling the literature does not address directly: a student cannot exceed its teacher through distillation alone. If your alignment data is scored by GPT-4, your model’s quality is bounded by GPT-4’s preferences. The collection’s reading is that no paper demonstrates a path past this limit without interfacing with verifiable reality: compilers, mathematical proofs, unit tests. Where there is a ground-truth checker, the teacher ceiling does not apply, because the reward signal is the world rather than another model.
Stage three: reasoning alignment
This is the operation that genuinely changes what the model can compute, and it is what DeepSeekMath and Qwen3 are doing when they use reinforcement learning.
GRPO, the RL method DeepSeekMath introduced, deserves a sentence because it explains a lot. Traditional PPO needs a separate value function, a critic model, to estimate expected rewards. GRPO drops the critic entirely. It generates a group of candidate responses, scores each one, and computes advantages relative to the group mean. No value network, no separate training loop. The trade-off is compute: you need many samples per question, typically around 64, to get stable gradient estimates. On a half-billion-parameter model this is cheap. On a 7-billion-parameter model it gets expensive fast.
The signature result is that GRPO improves Maj@K but not Pass@K. The model does not learn to solve problems it could not solve before. It learns to explore its own output distribution and settle on the answers it was already capable of producing. That is distribution shaping at the level of the reasoning trace rather than the surface output, and it is why reasoning alignment is expensive while style alignment is cheap. A thousand to fifty thousand examples and a few hours of GPU time buys you style alignment. Multi-stage RL with process supervision buys you reasoning alignment, and it genuinely changes the model’s internal computation.
When you run DPO in a tool like Unsloth on your dataset, you are doing style alignment. When you run GRPO on math or code problems with verifiable rewards, you are doing reasoning alignment. They look similar in a notebook. They are not similar in cost, mechanism, or the kind of improvement you should expect.
Qwen3 runs the full modern version: cold-start SFT to teach reasoning format, GRPO to build reasoning capability, a mode-fusion stage that integrates “thinking” and “non-thinking” behaviour, then GRPO again with broader data. Each stage has different data and a different objective, which leads to the second organising observation.
Training is curriculum design
Every paper that describes a successful training pipeline is really describing a curriculum: a sequence of different optimisation problems, each with different data and a different objective.
Phi-1 pretrains on filtered code, then fine-tunes on exercises, teaching the model to read before it writes. SmolLM2 runs four stages with manually rebalanced data mixtures, increasing the proportion of math when math is weak and shifting to long-context data once code improves. Qwen3’s four-stage post-training changes both data and objective at each step. The common thread is that the optimal data at each stage is different, and the optimal objective is different. This is not “more training”. It is a designed sequence where each stage prepares the model for the next.
One finding makes the point concrete. A 2026 paper on SFT-GRPO data overlap reports that reusing SFT data in the GRPO stage costs about 10 percentage points, and that keeping the data pools disjoint recovers that performance for free. The collection’s reading is that this is a curriculum failure: when you reuse SFT data for GRPO you ask the model to explore material it already knows. GRPO needs novel challenges, the next level of the curriculum, not repetition of the previous one. (This paper and the related SFT-DPO interaction paper are recent, from 2026, and their specific figures are reported here as the collection’s reading rather than independently re-verified results.)
The broader lesson is that what you exclude matters as much as what you include. Phi-1 removed about 40% of its initial code-exercise dataset and barely saw a performance drop, because the removed data was redundant. SELF-INSTRUCT filters out generated instructions that are too similar to existing ones. The collection calls this the negative-data blind spot: the field optimises for inclusion, but several papers accidentally discover that exclusion is equally important. Bad training data is net-negative. Curate by exclusion first, removing noise, repetition, and style inconsistency, then add volume.
Why fine-tuning is so cheap
The compressibility evidence is worth its own section because it should change how you think about your own training runs.
LoRA showed that you can adapt a model to a new task by training a low-rank matrix decomposition, rank 4 or 8, while freezing the other billions of parameters. That is roughly 0.01% of the weight space, with zero added inference latency. QLoRA quantised the base model to 4 bits using a data type designed for normally distributed weights, added double quantisation for the quantisation constants, and used paged optimisers to handle memory spikes. The result, verified in the paper’s own abstract, is that you can fine-tune a 65-billion-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning performance; the resulting Guanaco models reached 99.3% of ChatGPT’s level on the Vicuna benchmark. LIMA aligned a 65-billion-parameter model with a thousand examples.
If fine-tuning added new knowledge, none of these numbers would work. A rank-4 matrix cannot store new factual information at scale. The reason they work is that fine-tuning rearranges existing knowledge so the model surfaces the right thing at the right time. The model already knows the answer. You are teaching it where to look.
There is a practical consequence the SFT-DPO interaction paper made explicit, and it is arguably the most useful single finding for someone running their own experiments: the choice between full fine-tuning and LoRA dominates the choice of training objective (SFT versus DPO) by roughly 12 to 1. Papers that compare LoRA-DPO against LoRA-GRPO and declare one algorithm better may be measuring the ceiling of LoRA rather than the algorithm. If you have the VRAM, full fine-tuning reveals what your method can actually do. LoRA comparisons systematically understate algorithm potential.
The same paper reports that reducing SFT data from 283,000 examples to about 2,830 drops performance by 11 points, which seems to contradict LIMA’s thousand-example result. The reconciliation is that quality-adjusted count matters, not raw count. LIMA’s thousand examples are hand-selected for diversity, quality, and style consistency. A random 2,830-example subset is not. The variable that matters is “how many high-quality, diverse examples”, not “how many examples”.
Four beliefs the papers kill
Read chronologically, the collection contradicts four pieces of common wisdom so reliably that they are worth naming.
Parameter count is a marketing number. Phi-1, at 1.3 billion parameters, matched or beat models around ten times its size on Python coding tasks. The difference was data quality: roughly 7 billion tokens of textbook-quality code with exercises, instead of everything ever scraped. SmolLM2, at 1.7 billion parameters trained on 11 trillion tokens with careful rebalancing, produced results competitive with much larger models. Data quality and training curriculum determine capability. Parameter count is a rough proxy at best, and the specific number on a model card tells you less than how the model was trained.
You do not need a data centre. TinyStories produced coherent, creative narratives with about 10 million parameters. Phi-3 runs a 3.8-billion-parameter model on an iPhone at over 12 tokens per second using 4-bit quantisation. QLoRA fine-tunes a 65-billion-parameter model on one GPU. Unsloth fits 7-billion-parameter DPO training on a 16GB card. The practical frontier is defined by what fits in your VRAM, not by what is theoretically optimal, and that constraint created the entire research direction around quantisation, parameter-efficient fine-tuning, and distillation.
Benchmark numbers do not mean what they say. Almost every paper evaluates results by asking GPT-4 to judge the output. QLoRA’s authors measured the correlation between GPT-4’s judgements and human judgements and found a Kendall tau of 0.43, moderate at best. GPT-4 has known biases: it prefers verbose outputs, favours models distilled from itself, and responds to formatting patterns. When a paper claims a 3-point improvement, there is a real chance it measured GPT-4 preferring outputs that sound like GPT-4. The leaderboard may be tracking stylistic similarity rather than capability.
Small models are not big models with fewer parameters. Below about 3 billion parameters, models fail differently and qualitatively, not just worse. TinyStories showed a clear capability hierarchy: grammar emerges first, then consistency, then creativity. SmolLM2 hits a math ceiling around 32% on GSM8K that more data does not fix. Small models struggle to explore during RL and overfit SFT patterns. The collection’s reading is that below roughly 3 billion parameters, distillation beats online RL, and above roughly 7 billion, GRPO with exploration starts to work. You cannot shortcut this by scaling down a recipe designed for a 70-billion-parameter model. The threshold is approximate and drawn from several papers rather than one clean measurement, but the directional claim is consistent.
What the field has not resolved
A useful reference should be honest about where the evidence runs out. The collection identifies several contradictions that no paper resolves, and they matter for how you interpret results.
LoRA adapter placement is contested. The original LoRA paper recommends adapting only the attention matrices. QLoRA directly contradicts this, finding that under quantisation all layers including the MLP must be adapted, because quantisation noise in the base weights means adapters need to compensate model-wide. Practitioners follow QLoRA’s recommendation, but the contradiction is resolved in practice and never explained in theory.
DPO’s value is reported inconsistently. The DPO paper shows strong gains over PPO. ZEPHYR shows DPO adding value over SFT-only. The SFT-DPO interaction paper finds DPO adds only a marginal amount. The reconciliation, again, is parameterisation: with full fine-tuning DPO helps more, and with LoRA it is handicapped. Comparing DPO papers without controlling for full fine-tuning versus LoRA is comparing different things.
DPO and GRPO rest on incompatible assumptions. DPO assumes preferences are transitive comparisons: A is better than B. GRPO assumes rewards are cardinal: A scores 0.7, B scores 0.3. These are different definitions of “better”, which means you cannot directly compare a DPO paper to a GRPO paper. When one reports that DPO underperforms GRPO on math, it is partly measuring which type of data is more informative for that task, not pure algorithm quality.
The evaluation problem is the deepest one. The 12-to-1 parameterisation finding was only discovered because one paper ran careful ablations on both axes. Most papers do not. They compare methods under a single parameterisation, usually LoRA, which means the field’s benchmark leaderboard may be comparing methods under conditions that suppress their true potential. The collection’s blunt conclusion is that the community is over-investing in alignment algorithms and under-investing in evaluation.
And then there are the open questions the collection thinks the field should be asking: whether there is an information-theoretic lower bound on training data for a given capability; why code-to-math transfer appears to be one-way; at what model size online RL becomes worth its sampling cost; whether models can be factored into a knowledge component and a scaffolding component; and what “done” even means, since every paper trains until the budget runs out rather than until the model converges.
None of this is settled. Presenting it as settled would make you more confident without making you more accurate.
What this means for your own training runs
The point of a mental model is that it changes what you do. If you are running fine-tunes on consumer hardware, the collection’s findings translate into a fairly concrete set of defaults, with the caveat that these are assembled from the papers rather than prescribed by any single one.
All of your post-training is distribution shaping. When you run DPO you are concentrating probability mass on preferred outputs, not teaching new knowledge. The model already “knows” what you want; you are making it more likely to surface it. This reframes debugging: if the model cannot do the task at all after fine-tuning, the problem is usually that the capability was never in the base model, and no amount of alignment will add it.
Data quality matters more than algorithm, and parameterisation matters more than objective. If you upgrade your SFT dataset from adequate to excellent, you will likely see more improvement than switching from DPO to GRPO. And if you have the VRAM, try full fine-tuning at least once, because the 12-to-1 gap means every LoRA-only comparison understates what the algorithm can achieve.
Curate by exclusion first. Remove noise, repetition, and style inconsistency before you add volume, and keep your SFT and RL data pools disjoint if you are running GRPO.
Match the method to the task and the scale. For verifiable tasks like math and code, online RL with a ground-truth reward is the strong choice, and it is the one place the teacher ceiling does not bind. For subjective tasks like dialogue and writing, DPO is the more robust default because there is no objective correctness signal to explore against, and a noisy reward model can make GRPO worse rather than better. For models below roughly 3 billion parameters, prefer distillation from a larger model over online RL.
Evaluate honestly. GPT-4 as judge correlates only moderately with human judgement, so test across parameterisations, use more than one evaluator, and do not trust a single-number benchmark. If your metric goes up but you have not checked whether a human would agree, you may be measuring style.
And know your ceiling. If you distil from GPT-4, your model will not exceed GPT-4. If you use GPT-4 to judge preferences, your alignment is bounded by GPT-4’s preferences. The only demonstrated escape hatch is a verifier that is not another language model.
The hardware constraint is not an inconvenience to work around. It is the thing shaping the entire field. QLoRA, Phi-3, and Unsloth exist because the “optimal” model is not the best-performing one but the best-performing one that fits in your memory. An 8GB card runs a 4-billion-parameter model in 4-bit, fine-tunes with LoRA, and fits a long context. A 36GB machine pushes to 7 or 14 billion parameters. GRPO with 64 samples per query is cheap on a half-billion-parameter model and expensive on a 7-billion-parameter model. Start small, verify the approach works, then scale.
Going deeper
The full collection has all 18 papers as interactive pages, a cross-paper synthesis with 18 insights and 7 open questions, and a structured learning path. If you want to understand why DPO and GRPO are mathematically incommensurable, why code-to-math transfer appears to be one-way, or whether models can be factored into knowledge and scaffolding components, start there.
The individual papers are not hard to read. The difficulty is holding them together at once and noticing what becomes visible only at that altitude. The view from that altitude is that this field is not accumulating a pile of unrelated techniques. It is circling a single insight about the difference between encoding knowledge and accessing it, and most of the confusion in the literature comes from treating those two things as the same.