Shannon and the Birth of Information

In 1948, a Bell Labs engineer named Claude Shannon published a paper that quietly reshaped the century. He showed that information, like energy, could be measured, encoded, and transmitted with provable limits.

Shannon defined the bit as the fundamental unit of information. He introduced entropy as a measure of uncertainty, and showed how any message could be compressed, corrected, and communicated across a noisy channel.

Shannon's core insights

1. 1Information is the reduction of uncertainty, measurable in bits
2. 2Every channel has a capacity; beyond it, errors are unavoidable
3. 3Redundancy in a message allows error correction
4. 4Compression is possible up to the entropy of the source, and no further

Shannon also built playful machines. He made a mechanical mouse named Theseus that could solve a maze using relays, widely considered one of the first learning machines. He juggled, rode a unicycle, and proved that his intellectual range matched his rigor.

“Information is the resolution of uncertainty.”

The big idea: AI is applied information theory. Every loss curve, every tokenizer, every compression trick traces back to Shannon's 1948 paper.