Kolmogorov Complexity: The Length of the Shortest Description
The complexity of an object is the length of the shortest program that produces it. A string of a million...
Year: 2026
Minimum Description Length
The best theory is the one that, together with the data it explains, can be stated most briefly. This turns...
Occam’s Razor: The Case for Simplicity
Faced with rival explanations that fit equally well, prefer the simpler. This ancient counsel is not mere taste; simpler theories...
Mutual Information as a Measure of Relationship
How much does knowing one thing tell you about another? The question has a precise answer, a single number that...
The Information Bottleneck
Learning may be a matter of forgetting the right things. A good representation keeps what is relevant to the task...
Generative Versus Discriminative: Modeling the World or the Boundary
To tell cats from dogs, one can learn what separates them or learn what each is. The first draws a...
Robustness Versus Accuracy
A model tuned to be right on average may be fragile to the unusual. Push it slightly off the familiar...
Ensembles: The Wisdom of Many Models
One model has one set of blind spots; many, averaged, cover for each other. Combine diverse imperfect predictors and the...
Overconfidence in Machines and Minds
Certainty is seductive and often unearned. Both minds and models tend to believe more firmly than the evidence allows, and...
Calibration: Knowing How Much to Trust a Guess
A confident wrong answer is worse than an uncertain one. A system should not only guess but know how much...
