Training vs Inference

One Factory, Two Businesses

Every AI chip in the world is doing one of exactly two jobs. Training is how a model is made: show it trillions of examples and adjust its internal numbers (weights) until it predicts well. Inference is how a model is used: freeze those weights and answer queries. The industry's favorite analogy is a refinery versus gas stations. Training a frontier model is a one-off capital project — months of time on tens of thousands of chips, a cost paid once and amortized forever after. Inference is retail: billions of small transactions whose profitability lives or dies on unit cost. The unit is the token — roughly three-quarters of an English word — and the two businesses even quote prices in different currencies: training capacity trades in dollars per GPU-hour, inference in dollars per million tokens ($/Mtok). The reason this module exists: the two workloads stress opposite parts of a chip. Training wants raw arithmetic (FLOPS) and a network that can keep tens of thousands of chips in lockstep. Inference — specifically its decode phase — wants memory bandwidth and cheap capacity. A chip, rack, or datacenter optimized for one can be mediocre at the other, and as we will see, the industry's center of gravity has swung decisively toward inference.

Pretraining: the Largest Synchronized Machine Ever Built

Pretraining is the famous part: next-token prediction. Feed the model a slice of the internet, ask it to guess the next token, measure the error, and nudge every weight to reduce it — repeated quadrillions of times. Computationally, each step is dominated by enormous matrix multiplications, which perform a huge amount of arithmetic on every byte fetched from memory. That makes pretraining compute-bound: the limiting resource is floating-point operations, which is why training chips race on FLOPS and why lower-precision number formats (FP8, FP4) that multiply arithmetic throughput matter so much.

The catch is the word 'every.' Modern runs split identical copies of the model across tens of thousands of GPUs, each chewing a different slice of data. After every step — every few seconds — all copies must average their proposed weight updates in a collective operation called gradient synchronization (all-reduce). The entire cluster marches in lockstep: one slow GPU, one flaky optical link, one crashed node stalls the whole machine. This is why training clusters obsess over networking and reliability, and why 'goodput' (useful work actually completed) diverges so far from paper specs.

The honest efficiency yardstick is MFU — Model FLOPS Utilization, the share of a chip's theoretical arithmetic actually spent on the model's math. Even elite labs live between 30% and 55%. And it moves: SemiAnalysis's benchmarks on a 128-H100 cluster show BF16 MFU climbing from 34% to 54% in twelve months purely from software — Nvidia's kernel engineers, no new silicon — a 57% free throughput gain that holds nearly flat when scaled to 2,000+ GPUs. In cost terms, training GPT-3 175B fell from 72¢ to 54.2¢ per million tokens over 2024, or $216k to ~$163k for the full original 300B-token run.

Post-Training and RL: the Wall Comes Down

Around 2024 pretraining hit the data wall — the internet had been read. Capability gains shifted to post-training: supervised fine-tuning on curated examples, RLHF, and above all reinforcement learning on verifiable rewards. The mechanism is simple to state: give the model a task where success is machine-checkable — does the code pass the tests? is the math answer right? — let it generate many attempts ('rollouts'), score each one, and shift the weights toward what worked.

Notice what those rollouts are: inference. RL training spends most of its compute generating hundreds of candidate answers per question, which means production-grade inference now runs inside the training loop. The clean wall between the two workloads has fallen — and hardware demand followed, tilting toward memory-heavy, inference-friendly chips even within 'training' budgets.

The proof this vector works: OpenAI post-trained the same GPT-4o-class base into both o1 and o3 — nearly all the headline capability gains of the reasoning era came from post-training, not a bigger pretrain. And the vector is young: Qwen, China's leading open-weights family, still spends only ~5% of its pretraining compute on post-training. As that ratio rises industry-wide — SemiAnalysis expects it will — the compute mix of making models will look ever more like the compute mix of serving them.

Inside One Request: Prefill, Decode, and the KV Cache

Now the workload that pays for everything. When you send a prompt to a model, inference happens in two radically different phases.

Prefill: the model reads your entire prompt at once. All input tokens flow through the network in parallel, in big dense matrix multiplications — computationally this looks like training, i.e., compute-bound. Prefill's output is not text; it is the KV cache: the attention mechanism's stored 'reading notes' (keys and values) for every token seen so far. The cache exists so the model doesn't reread the whole conversation from scratch for each new token; the price is memory — it grows with context length, and a long agentic session can pin gigabytes of precious HBM per user.

Decode: generation, one token at a time. Each new token requires streaming essentially all of the model's active weights plus the entire KV cache out of memory, to do a comparatively tiny amount of arithmetic. Bytes moved dwarf math done: decode is memory-bandwidth-bound, and the chip's expensive FLOPS sit idle waiting on HBM.

Operators fight this with batching: serve dozens of users in one pass so the weight-streaming cost is shared. Bigger batch = cheaper tokens — but each user waits their turn, so interactivity (tokens/sec/user) drops. Think of a kitchen cooking in ever-bigger batches: cost per plate falls, diners wait longer. Every serving stack picks a point on this cost-vs-speed curve, which is why the same model can cost several times more when served 'fast.'

Because prefill and decode want opposite chips, frontier labs (OpenAI, Anthropic, xAI, DeepSeek) run disaggregated serving in production — separate GPU pools per phase — and silicon has followed: Nvidia's Rubin CPX is a prefill-only part with 128GB of cheap GDDR7 instead of HBM (>50% lower memory cost per GB), attacking the memory-cost wall directly. On the model side, DeepSeek's Multi-Head Latent Attention (MLA) compresses the KV cache by ~93.3% — the single biggest reason its tokens are so cheap.

$/Mtok: the Number That Now Designs Chips

Add it up and inference has one economic scoreboard: dollars per million tokens, at a given speed. Every optimization in this module — MFU, batching, KV-cache compression, disaggregation, FP4 — cashes out on this curve, and SemiAnalysis now benchmarks entire racks on it: a GB300 NVL72 rack serves DeepSeek V4 at $0.156 per million output tokens at 50 tokens/sec/user, partly because its 72-GPU NVLink domain keeps mixture-of-experts traffic on one fast fabric where 8-GPU servers must spill onto slower networks.

Two twists complete the economics. First, sticker prices lie: agentic workloads read far more than they write (Claude Code runs ~300 input tokens per output token) and constantly re-read the same context (90%+ cache-hit rates), so Anthropic's Opus lists at $5/$25 per Mtok but realizes ~$0.99 blended. Caching is not plumbing; it is pricing. Second, the curve falls relentlessly: tokens of GPT-3 quality are ~1200x cheaper than at GPT-3's launch, and the savings are reinvested in bigger models and longer agent sessions — Jevons paradox as a business plan.

This is why inference economics now design chips. HBM capacity and bandwidth became the largest item in the GB300 package bill-of-materials; 'world size' (how many chips share one fast domain) became the headline spec; and entire product lines — prefill-only GDDR7 parts, SRAM-based speed demons like Groq's LPU, which Nvidia effectively absorbed via a $20B IP-license-and-acquihire deal — exist only because what buyers actually purchase is not peak FLOPS but $/Mtok at a target interactivity.

Own illustration · Yicheng Yang

Nvidia's GB200 NVL72 rack at COMPUTEX 2024 — 72 GPUs cabled into a single NVLink domain. 'World size' (how many chips share one fast fabric) became a headline spec because rack-scale interconnect slashes $/Mtok on mixture-of-experts inference. — Source: Wikimedia Commons — 极客湾 Geekerwan (CC BY 3.0)

A Google TPU v3 board. Google's systolic-array TPU line — scaled today to 9,216-chip Ironwood pods — trained Gemini 3 end-to-end: proof that frontier training no longer runs on GPUs alone. — Source: Wikimedia Commons (CC BY-SA 4.0)

Key Numbers

Sources

• SemiAnalysis — H100 vs GB200 NVL72 Training Benchmarks - Power, TCO, and Reliability Analysis
• SemiAnalysis — DeepSeek Debates: Chinese Leadership On Cost, True Training Cost, Closed Model Margin Impact
• SemiAnalysis — AI Value Capture - The Shift To Model Labs
• SemiAnalysis — DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200
• SemiAnalysis — Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack
• Wikimedia Commons — 极客湾 Geekerwan (CC BY 3.0)
• Wikimedia Commons (CC BY-SA 4.0)
• SemiAnalysis — RL Environments and RL for Science
• SemiAnalysis — InferenceX V2: Nvidia Blackwell vs AMD MI355X
• SemiAnalysis — Nvidia: The Inference Kingdom Expands
• SemiAnalysis — Groq Inference Tokenomics: Speed, But At What Cost?
• SemiAnalysis — Scaling Laws: O1 Pro Architecture, Reasoning Training Infrastructure
• Nvidia — GB300 NVL72 product page (spec table: '72 NVIDIA Blackwell Ultra GPUs, 36 NVIDIA Grace CPUs')
• Anthropic — Claude Docs Glossary (Tokens)

Related sector deep-dives

Optional — jump ahead if this module is all you needed; otherwise continue the ladder above.