The Two Ends of the LLM Scaling Curve (And Why the Middle Is Mostly Dead)

There is a question that surfaces repeatedly in any serious discussion about foundation model strategy: at what parameter count does a model stop being worth building? The answer has changed twice in five years, and both times the correction pointed in the same direction — away from the middle.

The current landscape is bifurcating. On one end, compact 3.8B dense models overtrained on trillions of curated tokens are running on consumer hardware with benchmark scores that would have required 30B+ parameter networks two years ago. On the other end, Moonshot AI's Kimi K3 has pushed open-weight frontier intelligence to 2.8 trillion total parameters — matching proprietary systems like Anthropic's Claude Fable 5 on complex agentic tasks. The 70B-to-200B range that once defined "large" has been squeezed from both sides. Understanding why requires tracing the math from first principles.

The Two Ends of the LLM Scaling Curve

From Kaplan to Chinchilla: What the Compute Equation Actually Says

For standard autoregressive Transformers, the cost of a complete forward and backward training pass follows a well-established approximation:

$C\approx 6ND$

where $N$ is parameter count and $D$ is training tokens, with $C$ measured in FLOPs. This single formula governs every budget decision in pre-training.

In 2020, Kaplan et al. found that scaling $N$ returned faster loss improvement than scaling $D$. The industry responded rationally: build larger models on modest data. GPT-3 (175B parameters, ~300B tokens) is the canonical example of this era.

Hoffmann et al. closed that chapter in 2022. Running over 400 controlled experiments across the 70M–16B parameter range, they modeled pre-training loss as a parametric power law:

$L(N,D)=E+\frac{A}{N^{\alpha }}+\frac{B}{D^{\beta }}$

Here $E$ is the irreducible entropy floor of the training distribution — the minimum loss achievable regardless of scale — and $A$, $B$, $\alpha$, $\beta$ are empirically fitted constants. Minimizing $L(N,D)$ subject to a fixed compute budget reveals that loss-optimal training requires scaling $N$ and $D$ in equal linear proportion. The practical result: roughly 20 training tokens per parameter, $D\approx 20N$.

Chinchilla (70B parameters, 1.4T tokens) validated this. It matched or surpassed Gopher (280B parameters, dramatically fewer tokens) on every evaluated benchmark while consuming a fraction of the compute. The correction seemed decisive.

It wasn't the last one.

The Cost Function Chinchilla Forgot

Chinchilla minimizes pre-training loss per FLOP. That's the right objective for a lab paper. A deployed product has a different cost function:

$C_{\text{total}}=6N{D}_{\text{train}}+2N{K}_{\text{inference}}$

where $K_{\text{inference}}$ is the total token volume generated across the model's operational lifetime. Inference cost scales linearly with $N$ — every parameter in the model is a recurring tax on every query it serves, forever.

As $K_{\text{inference}}$ grows, the math tips decisively. Spending more on pre-training to produce a smaller, better-trained model amortizes across billions of cheaper inference passes. This is the overtraining insight: cross-entropy loss continues decaying logarithmically past 1,000 tokens per parameter, past 10,000 tokens per parameter, with no firm saturation boundary observed empirically.

The token-to-parameter ratios tell the story:

LLaMA 3 8B's 1,875:1 ratio sits nearly two orders of magnitude past Chinchilla optimal. Phi-3-mini at 868:1 achieves performance previously requiring 30B+ parameter architectures — not because it breaks any scaling law, but because it trains on a carefully filtered corpus that maximizes information density per token. Frontier LLM classifiers filter the web data; synthetically generated reasoning and coding examples fill the remainder. The result: a 3.8B model that fits on a phone and reasons at a level that would have required a server cluster in 2022.

The Memory Wall at Scale

The overtraining strategy breaks cleanly for models up to roughly 70B parameters. Beyond that, a different physics problem takes over: memory bandwidth.

In a dense Transformer, every token activates 100% of the network's parameters on every forward pass. Inference VRAM scales linearly with $N$. At 70B parameters in BF16, that's ~140 GB of static weight memory — already beyond single-GPU capacity. At 671B it's ~1.3 TB. A 5T dense model in BF16 would require 10 terabytes of weight storage before a single token is processed.

Sparse Mixture-of-Experts (MoE) architectures resolve this by separating two quantities that dense models keep locked together: total parametric capacity $N_{\text{total}}$ and compute cost per token $N_{\text{active}}$. A learned router assigns each token to a fixed subset of expert feed-forward networks; the rest sit in memory but never fire.

The gap between the two columns is not a deficiency on either side — it's a deliberate fork. The 3.8B model is designed for the device in your pocket. The 2.8T model is designed for cluster-scale agentic workflows and frontier reasoning tasks.

Inside Kimi K3: What Scaling to 2.8 Trillion Actually Requires

Kimi K3 is the first open-weight model in the 3-trillion-parameter class. Getting to that scale without making inference commercially impractical required solving several problems simultaneously.

Extreme routing sparsity. K3 uses 896 expert subnetworks per MoE layer, activating only 16 per token — a 3.7% active parameter ratio. Inference FLOPs scale to $2\times 104\text{B}$ per token rather than $2\times 2.8\text{T}$. The remaining 96.3% of parameters are hosted in memory but idle, functioning as a massive specialized knowledge store distributed across cluster nodes.

MXFP4 Quantization-Aware Training. A 2.8T model in BF16 would require approximately 5.6 terabytes of static weight VRAM — physically unmanageable on any current cluster. K3 applies Microscaling FP4 (MXFP4) weights with MXFP8 activations natively during pre-training, not as a post-training approximation. This compresses static weight storage to approximately 1.4 TB while preserving training stability that post-training quantization cannot deliver at this scale.

Kimi Delta Attention and the 1M context window. Standard Multi-Head Attention scales quadratically with sequence length: $O(L^2)$ in both compute and memory. At a 1-million-token context window, the KV cache alone would exceed available cluster VRAM under standard attention. K3 addresses this with Kimi Delta Attention (KDA), a hybrid linear attention mechanism that replaces standard attention in designated layers while maintaining full expressive capacity in the remaining ones. This brings KV cache memory scaling down from quadratic to approximately linear in those layers.

Quantile Balancing for expert load. At 896 experts, conventional auxiliary load-balancing losses — which penalize routing imbalance by adding a secondary loss term — degrade model convergence at scale. K3 uses Quantile Balancing instead: tokens are routed based on score quantiles across the expert population, keeping expert utilization even without modifying the primary training loss. The result is stable routing without the quality penalty that load-balancing losses impose.

Attention Residuals (AttnRes). Rather than standard sequential residual accumulation through each layer, AttnRes allows layers to selectively retrieve intermediate representations from earlier layers in the stack. This smooths gradient flow across deep MoE networks — a practical necessity when pipeline depth runs into the hundreds of layers.

What Happens at 5 Trillion Parameters

Whether a 5T parameter model would outperform current frontier systems depends almost entirely on whether it's dense or sparse.

Dense 5T: Weight storage in BF16 is 10 terabytes before training begins. AdamW optimizer states add another ~60 TB of FP32 memory per parallel instance. Under Chinchilla-optimal pretraining ($D=20N$), that's 100 trillion training tokens:

$C=6\times (5\times 10^{12})\times (10^{14})=3\times 10^{27}\text{ FLOPs}$

A cluster of 50,000 enterprise GPUs at 50% MFU would require months of continuous execution drawing hundreds of megawatts. Every additional layer of pipeline parallelism grows the pipeline bubble fraction — the proportion of GPU time spent idle waiting for gradient synchronization rather than executing FLOPs. And on a 50,000-node cluster, Mean Time Between Failures drops to hours, forcing either expensive synchronous checkpointing (which consumes training time) or asynchronous sparse snapshotting protocols like MoEtion. Even if successfully trained, serving a dense 5T model costs $2\times 5\text{T}$ FLOPs per generated token — commercially impractical for any interactive application.

Sparse MoE 5T — say, 5T total parameters activating 80B to 120B per token across 512 to 1024 routed experts — is a viable engineering target. Inference FLOPs are comparable to a mid-sized dense model. Building on K3's MXFP4 QAT and hybrid linear attention techniques, static weight storage falls to roughly 2.5 TB under FP4. The main challenge isn't hardware; it's data.

The Data Problem Is Harder Than the Hardware Problem

Reaching 100 trillion training tokens under Chinchilla-optimal rules hits a physical ceiling. Reliable estimates put the total pool of high-quality, unduplicated human-written text on the public internet between 10 and 30 trillion tokens. The remaining 70 to 90 trillion tokens in a hypothetical 5T model's training corpus must come from synthetic generation.

Recursive pretraining on uncurated synthetic text causes model collapse: tail distributions erode, systematic errors compound through generations, output diversity homogenizes toward the center of the generating model's distribution. The Phi-3 lineage demonstrates the mitigation path — strict automated filtering, formal verification for reasoning traces, unit-test validation for code — but scaling that pipeline to 70+ trillion synthetic tokens while maintaining training stability is an unsolved operational problem.

The performance ceiling of the next generation of frontier models will be set not by parameter count, not by cluster size, but by who builds the best synthetic data pipeline. A 5T sparse MoE trained on 100 trillion tokens of rigorous, verified synthetic data would represent a genuine step beyond current frontier systems. A 5T model trained on uncurated synthetic fill to reach token count targets probably wouldn't.

---

The bifurcation of the scaling curve is already visible in the models shipping today. Compact overtrained dense models are becoming standard inference targets for on-device applications. Multi-trillion parameter sparse MoEs are becoming the reference architecture for frontier intelligence. What neither camp has fully solved is the data problem — and the organizations that crack high-volume synthetic corpus generation with reliable quality guarantees will write the next chapter of this curve, regardless of parameter count.

References

1. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
2. Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). arXiv:2203.15556.
3. Sardana, N. et al. (2024). Accounting for Inference in Language Model Scaling Laws. arXiv:2401.00448.
4. Abdin, M. et al. (2024). Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219.
5. Meta AI. (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
6. DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.
7. Moonshot AI. (2025). Kimi K3 Tech Blog: Open Frontier Intelligence. kimi.com.
8. Rajbhandari, S. et al. (2024). MoEtion: Efficient and Reliable Sparse Checkpointing for Mixture-of-Experts Models. arXiv:2412.15411.
9. Qwen Team. (2025). Qwen2.5 Technical Report. arXiv:2412.15115.