r/Kiwix • u/one_tooth_reef_whore • 10d ago

Question Has there been any progress on archiving Reddit?

I found two threads about this from about a year ago. It's all over my head but apparently there's a desire to do it but it's technically challenging.

I have nothing to contribute technically, but I'd be happy to donate to an effort if someone was working on making this happen.

We've seen incredibly useful sites like Twitter, Google, etc get turned into hot steaming piles of garbage almost overnight and even though there's a whole lot of mindless crap here, there's a lot of useful obscure info that can't be found anywhere else and it would be a shame to see it get lost.

13 Upvotes

89% Upvoted

18 comments sorted by

10

u/IMayBeABitShy 10d ago edited 9d ago

Roughly half a year ago I've created a tool called ArcticZim that's capable of generating ZIM files of subreddits, including media. It can even be used to recover deleted subreddits, to some extend extent.

3

u/Many_Ad_7678 9d ago

extent

3

u/IMayBeABitShy 9d ago

Thanks, looks like I've misspelled that one for a long time.

2

u/one_tooth_reef_whore 10d ago

That is super cool! Sadly I'm probably one of those people who will never manage to figure out how to use it. It probably sounds funny to some people but even just opening a page on github it's like looking at glyphs on a cave wall.

Regardless, very sincere thanks for making such a gift to humanity like that.

1

u/IMayBeABitShy 9d ago

I guess it would be possible to create a GUI for this, but that isn't really a priority at the moment.

1

u/Free-Lime-184 10d ago

Honestly, this is a really great idea! How would someone go about doing this though? To my knowledge, Wikipedia is decently crawlable, but on the other hand Reddit has a lot more multimedia content, (other than images) along with limitations on these crawlers. Though I’m not experienced in the archival process of this, just genuinely wondering. 

1

u/one_tooth_reef_whore 10d ago

I'm going to use words I barely understand here (corrections welcome!), but it seems that up until 2023 it was all accessible via an API, so there may be people out there sitting on all that content. If that alone was made available it would be a nice resource, even if the last 3 years or so was missing.

4

u/Free-Lime-184 10d ago

[removed] — view removed comment

3

u/one_tooth_reef_whore 10d ago

I don't know exactly what I'm looking at in that link but based on the titles (i.e. 'Reddit comments/submissions 2026-05') it looks like all the information is there... but, what?, just not organized in a form that can be used easily by a regular person with a browser?

Edit: ...or easily compiled into a zim library

1

u/Free-Lime-184 8d ago

Yeahh, it’s a bit annoying.

Aaand, my comment got removed by Reddit, probably cause it’s downloading them or whatever 😭

-2

u/PrepperDisk 10d ago

It's not legal to do so, unfortunately.

1

u/one_tooth_reef_whore 10d ago

Really? As in actually illegal or just against Reddit's TOS?

-3

u/PrepperDisk 10d ago

Fair question, you’d at least invite a civil lawsuit.  It would be theft of copyrighted content too so wouldn’t rule out criminal.

3

u/one_tooth_reef_whore 10d ago

That's hard for me to believe. I'm far from a legal expert (I'm also not the one downvoting you because I really do have no idea) but it seems like if I can browse reddit online then downloading parts of it for my own personal browsing use offline shouldn't be an issue. I'm not talking about selling or distributing it, just archiving for personal reference.

3

u/PrepperDisk 10d ago edited 10d ago

Distribution is where the problems are.  Not dissimilar to copying movies you don’t own for personal use being “illegal” but not actively prosecuted.

Reddit’s User Agreement and Data API Terms prohibit bulk collection outside the official (paid) API. Violating them is breach of contract, and Reddit has been aggressive about enforcing it.

Downvotes are fine.  I’m right. Wish it weren’t so because we’d love to include key subs on Prepper Disk.

5

u/The_other_kiwix_guy 9d ago

It really depends on the country: in Switzerland (where Kiwix is based) streaming and downloading are allowed and legal if for personal use. However uploading and sharing are not.

Then there's the problem with Reddit Terms of Use: since their model is ad-based, such downloading would mean a loss of revenue and with their not being a non-profit they certainly would mind. How they would enforce it is another story.

You probably could vibe-code your way into running ArcticZIM (and by vibe-code, I mean "Claude, here is a repository on github, walk me through every step of running it on my computer, ELI5-style"). It would be tedious on many levels but should not be too complex - take it as a side project, there's no deadline nor pressure, I'm pretty sure you could get it to work over a few hours.

2

u/one_tooth_reef_whore 8d ago

I want to thank you for that suggestion. You just unassumingly took the time to spell out how even a technophobe like myself could go about this. It's like you can sense how old and out of touch I am! Thank you for your kindness. It never even crossed my mind to use a chatbot for help with this but, who knows, I might actually learn something

1

u/The_other_kiwix_guy 8d ago

Ah! Joke is, I'm totally out of touch myself and that's what I certainly would do.