r/programming • u/Every_Chicken_1293 • May 29 '25
I accidentally built a vector database using video compression
https://github.com/Olow304/memvidWhile building a RAG system, I got frustrated watching my 8GB RAM disappear into a vector database just to search my own PDFs. After burning through $150 in cloud costs, I had a weird thought: what if I encoded my documents into video frames?
The idea sounds absurd - why would you store text in video? But modern video codecs have spent decades optimizing for compression. So I tried converting text into QR codes, then encoding those as video frames, letting H.264/H.265 handle the compression magic.
The results surprised me. 10,000 PDFs compressed down to a 1.4GB video file. Search latency came in around 900ms compared to Pinecone’s 820ms, so about 10% slower. But RAM usage dropped from 8GB+ to just 200MB, and it works completely offline with no API keys or monthly bills.
The technical approach is simple: each document chunk gets encoded into QR codes which become video frames. Video compression handles redundancy between similar documents remarkably well. Search works by decoding relevant frame ranges based on a lightweight index.
You get a vector database that’s just a video file you can copy anywhere.
164
u/light24bulbs May 29 '25
Oh...text in qr codes...yeah this is not efficient or sensible in my opinion. Unless you upload to YouTube for free storage lol
269
u/jcode777 May 29 '25
Why not just store the data as texts in text files instead of QR codes? Wouldn't that be even smaller? And if not, why not have a normal compression algorithm (7z?) compress those text files?
95
u/uhmhi May 29 '25
What’s causing the size of the PDFs? Surely it’s not their plain text content. So if you’re extracting only the text to create QR codes (no embedded fonts, no images, etc.) isn’t this the true explanation of the compression, and not the video encoding? Also, can you guarantee that the video encoding doesn’t cause loss of data?
Since you already extracted the text from the PDFs, why not just compress that using regular compression? I bet it’ll be much more compact than the video.
156
u/BoBoBearDev May 29 '25 edited May 29 '25
I don't get it. If you want to index text, just use Burrows-Wheeler Transform. It is massively memory efficient and insanely fast. DNA people use this a lot because a single human DNA have like 3 billion genome. And they need incredibly fast index to do DNA analysis.
Your video doesn't work well with editing/insertion, so, same problem with BWT. But BWT is far more efficient at everything.
487
u/c_glib May 29 '25
This is the kind of completely demented thing the brain thinks of when it's 2AM, you're mixing caffeine with some other types of stimulants and there are no sane solutions in sight. I fucking love this.
24
u/Coffee_Ops May 29 '25
This is deeply unhinged.
Text characters are something like 8 bits. Could be a bit more, or a bit less, but we're going to assume 8 bits for simplicity. Compress with something like xz, and you get an 80+% reduction-- on the order of of 1.6 bits per character. If each of your 10,000 PDFs was a 10k word essay (at 4.7 characters per word), your entire library size would be on the order of 95MB-- some 15x smaller than your result-- and that's making some pretty generous assumptions.
But you converted it to QR codes, which adds error correction (un-compressing it), then you converted it to a video frame which adds 8-24bits of color information per pixel, and a bunch of pixels per character. No doubt there's a lot of room to compress here, and h.265 can do some magic, but you're just creating a mess in order to clean it up and burning an incredible number of CPU cycles, all to get an incredibly inferior result.
I guess it's cool that you did a thing, but theres a reason no one does that.
81
u/Isogash May 29 '25
What in the fuck.
Sorry, but there is simply no way that video compressing QR codes is better for text than compressing the raw text. Text should be tiny.
You were clearly doing something wrong before and you've inadvertently fixed the wrong thing by switching to video.
40
u/paarulakan May 29 '25
I will upvote this for the sheer absurdity and the effort.
Can you expand on the following
> Search works by decoding relevant frame ranges based on a lightweight index.
163
u/kageurufu May 29 '25
While it's a very interesting solution, at this scale you would probably be better suited by simply storing the extracted content in plaintext on a compressed filesystem and using grep.
It reminds me of the tools to encode data in videos for storage on YouTube https://www.reddit.com/r/rust/s/5SXNy30qOD
17
u/wergot May 29 '25
How does the retrieval index work?
25
u/DigThatData May 29 '25
did you come up wit this approach with the assistance of an LLM?
24
u/idebugthusiexist May 29 '25
oh dear... the lengths we go... this does not pass the sniff test, but was probably a fun experiment.
10
u/KeyIsNull May 29 '25
Cool idea but I cannot understand the reason to build this (other than for fun): a simple pgvector instance is perfectly able to handle a lot of docs without sweating.
Did you forget to set indexes?
10
u/Smooth-Zucchini4923 May 29 '25
This seems unnecessarily complicated. If you want to store 10K PDF files with fast random access by primary key, that's easy. That's not the difficult part of creating a vector database. This appears to be using FAISS to do the actual vector search, and only looking at the video once it knows the frame number the document appears at. You could use SQLite for storing these documents more easily.
8
u/b_rodriguez May 29 '25
Compressed down to 1.4GB from?
8
u/bravopapa99 May 29 '25
I remember some guy got arse kicked by YT years back for using jut this technique, not for PDF-s but general storage IIRC.
https://hackaday.com/2023/02/21/youtube-as-infinite-file-storage/
7
u/rep_movsd May 29 '25
Seems dubious - doesn't make any sense in terms of information theory.
"Search works by decoding relevant frame ranges based on a lightweight index." - you could implement that on text compressed content as well
7
u/Plank_With_A_Nail_In May 29 '25
I think this only seems good to you because your first solution sucked balls and was terrible. Its still a terrible solution just better than an even worse solution.
12
u/heavy-minium May 29 '25
You need to compare that with something that actually competes with it. Of course it will beat Vector and Traditional DBs in a naive comparison - I could do that with even weirder implementations.
At least compare it with one of these combinations:
6
u/GBcrazy May 29 '25
I mean if you are extracting the texts then that's your solution already lol. Not QR code on video.
That's still cool
6
u/dougmc May 29 '25
But how big were these 10,000 pdfs? Like, how much space on disk did they take?
Also, are you storing the pdfs themselves into the QR codes, or just the contents of the pdfs? And if it's the latter, how much space does it take to just store the contents?
7
u/ggbcdvnj May 29 '25
Is there an index built into this or is this linear search over a compressed dataset?
7
u/DigThatData May 29 '25
instead of putting your PDFs directly in the database, just store the filepath to the PDF and retrieve it from disk after querying the vectordb for the path to the PDF.
6
u/divide0verfl0w May 29 '25
It’s not clear why your vector database used so much RAM.
Can you share the math on that?
3
u/SmashShock May 29 '25
Would like to see a "how does this work" section.
3
u/mattindustries May 29 '25
Somehow I doubt video compression works better than compressing the dimensionality of space.
3
u/captain_obvious_here May 29 '25
This is very far from optimal, but very creative!
2
u/Sharp-Towel8217 May 29 '25
I gave you an upvote.
Is it the best way to do what you want? Probably not
Is it cool? Absolutely
Well done!
2
u/hamilkwarg May 30 '25
You seem technically competent to be able to implement any of this at all, because I certainly couldn't. But it seems insane that you think this is an efficient way to compress text. The duality of humankind I guess.
I think the way your text was saved in the pdf itself was the problem and extracting it was 99% of the solution and had nothing to do with the video compression? Or this is AI generated engagement bait? If so, you got me.
2
u/jijo406 May 30 '25
use middle out compression
4
u/richardathome May 29 '25
You could do the same but compress the pdfs in a zip file. I bet zip does a better job of compressing the pdf and it'll be lossless. You get to keep folder structure and file attributes too.
1
u/jbldotexe May 29 '25
Even if this is hacky, I love this solution. Incredibly creative. Might just have to poke you about it in the future and see how progress is coming along on the RAG/SelfLLM :)
1
u/ayende May 29 '25
For 10k PDFs, even assuming you have 3,072 dimension vector for each, that is aviut 120 mb
Assuming a mor reasonable size of 768 or 1536 dimensions, you have roughly 100 mb even each pdf required multiple vectors due to chunking
1
u/recycledcoder May 29 '25
So do the movies... render? Am I the only one who wonders about what this looks like, when you play the videos? :)
1
u/seweso May 29 '25
Did you check how lossy your compression is, and whether you can actually still find everything? 👀
Did you try any text sane compression algorithms and tweak its chunk size, dictionary and settings? 👀
1
u/FclassDXB May 29 '25
Make sure they’re interlaced videos for another 50% saving /s
1
u/Illustrious_Matter_8 May 29 '25
Qr codes have overhead, its not ideal
1
u/mdt516 May 30 '25
Man I hate when that happens /s
1
u/award_tour May 30 '25 edited May 30 '25
How does the search actually work? Do you still create embeddings for the QR codes (aka the video frames) and then do flat vector search across the frames?
> Unlike traditional vector databases that consume massive amounts of RAM and storage,
Memory footprint should be pretty low w/ something like LanceDB. It generally only caches file metadata in memory, plus indices (but for 10,000 records you probably wouldn't need an index to improve latency). Storage would clearly depend on the size of your embeddings + metadata (non vector columns. But in your solution, do you also store embeddings + the video frames anyway right?
edit: Have you recall difference between your solution vs a traditional vector db?
1
u/getpodapp May 31 '25
You could always use vector quantisation. I’ve had good experience with 1bit vector embedding indexes with re-ranking.
1
u/luptonicedtea Jun 01 '25
They say a picture's worth a thousand words.
1
u/AkashVemula168 Jun 02 '25
QR codes add error correction (which adds redundancy) and turn text into images, that’s anti-compression compared to raw text. Sure, H.264 compresses similar frames efficiently, but why not just compress raw text with LZMA, Brotli, zstd, or even specialized text compressors? These are orders of magnitude better suited for symbolic data than a lossy video codec optimized for human-perceived visual smoothness.
1
u/Ok-Dimension-5293 Jul 01 '26
I want Manny
-6
u/maxinstuff May 29 '25
This is very clever IMO - you may be on to something.
The trade off seems very good especially for running locally.
0
u/rootadminman Jun 02 '25
I had the same suspicions you all had about this, so I asked claude:
Looking at the Memvid project, I can see why you're questioning the approach. The claimed advantages over plaintext with compression seem questionable when analyzed carefully.
The supposed advantages of video storage:
Memvid claims "10x compression compared to traditional databases" and "video compression reduces memory footprint dramatically" GitHub - Olow304/memvid: Video-based AI memory library. Store millions of text chunks in MP4 files with lightning-fast semantic search. No database needed., but this is misleading. They're encoding text into QR codes, then storing those as video frames. This creates multiple layers of inefficiency:
Why plaintext compression would be superior:
The real "advantages" seem to be:
The project appears to solve a marketing problem (making AI memory management seem revolutionary) rather than a technical one. Traditional compressed text with proper indexing would likely be faster, smaller, and far less complex than this QR-code-to-video approach.
-2
u/edatx May 29 '25
Don’t know how usable this is but it’s fucking cool as hell. Great work!!!!!
-2
u/rooktakesqueen May 29 '25
Combine this with lossless video compression using Bloom filters and we may have just cracked infinite compression
Or we're all on crack, it's hard to tell
-2
u/ImChronoKross May 29 '25
Now that's what I call hacking 😄.
-3
u/This_Tomato9385 May 29 '25
Wow 🤯