The Bitter Lesson: Computation Over Cleverness
Again and again, the elaborate hand-built method loses to the simple method that scales. Human insight, painstakingly encoded, is overtaken...
Year: 2026
Scaling: When Quantity Becomes Capability
Make a system larger, feed it more, and something changes in kind, not just degree. Beyond mere improvement, new abilities...
Interpretability and the Wish to See Inside the Black Box
A system can be accurate and still be opaque. It gives answers without reasons, competence without account. The desire to...
Specification Gaming: Optimizing the Letter, Not the Spirit
Ask precisely for the wrong thing and you will receive it. Optimizers are literal genies, satisfying the objective as written...
The Alignment Problem: Goals We Cannot Fully Specify
We build systems to pursue objectives, but our true wishes exceed what we can write down. A goal stated precisely...
Exploration Versus Exploitation: The Explorer’s Dilemma
Act on what you know, or seek what you don't? Every agent that learns from its choices faces this fork....
Transfer: Knowledge That Moves Between Tasks
What is learned in one place can pay off in another. Skills acquired on one task often lighten the learning...
Distillation: Compressing One Mind Into a Smaller One
A large model's competence need not stay large. Its behavior can be poured into a smaller vessel that mimics it...
The Lottery Ticket Idea: Sparse Networks Hidden in Dense Ones
Inside a large trained network may hide a tiny one that could have learned the task alone. The full apparatus...
Double Descent: When Bigger Models Learn Better Again
Conventional wisdom says a model too large will overfit. Yet past a certain size the rule inverts, and enormous models...
