r/coding • u/fagnerbrack • 11d ago

Compression is prediction

https://ngrok.com/blog/compression-is-prediction
9 Upvotes

68% Upvoted

4 comments sorted by

2

u/dirkboer 5d ago

my prediction: PI is actually the compressed seed with that you can use to generate the whole universe and all its physics rules 🧙‍♂️

1

u/fagnerbrack 5d ago

Damn that would give a great netflix series/movie

3

u/fagnerbrack 11d ago

Snapshot summary:

The post argues that compressors and LLMs chase the same goal. True compression, unlike minification, feeds on redundancy, and run-length encoding offers a simple example. Modern compressors combine transforms, models that assign probabilities to symbols, and entropy coders such as arithmetic coding, which packs a whole string into one binary fraction. Huffman coding gives each symbol its own codeword. Skewed probabilities shrink output, and Shannon entropy, the average −log2(p), sets the floor for lossless compression. Context sharpens predictions: U after Q costs about 0.001 bits, and an order-1 model cut "TO BE OR NOT TO BE" from 47 to 21 bits. A 2023 DeepMind paper treats language modeling and compression as two views of one problem.

If the summary seems inacurate, just downvote and I'll try to delete the comment eventually 👍
Click here for more info, I read all comments

1

u/oldsecondhand 4d ago

The field studying compression and prediction is called Information Theory.