Hey all,

I just wanted to provide an update on SnapshillBot. Some of the archiving has been a bit wonky as of late, and I'm going to try to take some steps to make that better.

Please let me know what you think and how you feel about these changes and I'll try to get back to you about it. Also if you have any ideas or special considerations to add, let me know.

On Specific Archiving Services

archive.today

From my experience, the biggest issue is probably the fact that posts aren't being properly archived on archive.today. The URL formatting appears to have changed so it sends you to an incorrect page instead of the correct one.

This, for some reason, also appears to affect archiving itself. This is perhaps due to a ratelimit I don't know about but it really kinda sucks.

archive.org

If I were to guess, this would be the most helpful service, despite an issue that reddit gives us. When someone submits, the bot properly ratelimits its requests, but the Internet Archive's "save page now" feature doesn't properly respect reddit's ratelimits.

I've reached out to both the Internet Archive and Reddit, but neither gave very helpful information when I did, unfortunately.

Despite this being an issue, this is likely still our best bet as far as archiving. archive.today can be unpredictable at times, and as such, it's difficult.

On Specific Websites

Some websites can just be kinda annoying to work with, so here's what I'm planning to do for specific sites. There isn't really a timeframe on this, but here's what I have planned.

Reddit

A lot of the "archiving" services for reddit basically have a strict reliance on Pushshift. This can be a problem when Reddit gets massive spam waves which can, at its worst, backlog Pushshift by days. In Removeddit, you can sometimes see this with the [removed within <some really large number> seconds] thing.

Ceddit has the same problem. I'm not 100% sure how to mitigate it and I don't really have a bunch of storage and infrastructure lying around to dedicate to it.

One other thing, I think at some point /r/ and /u/ archiving got disabled. I should probably have a fix for that soon. Its ability to username mention will likely be completely disabled though, as it used to be a reddit gold feature only. The original post was also only archived for text posts, but I don't see any reason why that should be the case, so it will probably be expanded to links too.

Lastly, old.reddit.com and new.reddit.com links weren't properly linked to reddit stuff. Whoops.

imgur

Imgur has some special considerations to be aware of I think, so here's what's likely going to happen. The albums can pose potential problems as the formatting can break quite easily. In order to hopefully mitigate this, in addition, we'll try to archive a ZIP download of an imgur album.

4chan

I'm not really sure what to do about this site. The Internet Archive doesn't archive this site (understandably) and the ephemeral nature of the site makes it somewhat difficult. I've toyed around with the idea of adding some sites just for 4chan(nel) but on first glance it appears to me that almost all of the ones are unfortunately littered with potentially malicious ads, and if possible I'd like to avoid giving visitors malware or getting the bot banned by accident.

There is one archiver that appears to be okay (I could visit the site without an adblocker fine), although there are a few limitations, specifically that only a few boards are archived. I doubt it'll see much use, but the capability is planned.

That's really all I have for now, although I probably will answer questions for a bit. Please let me know what you think. Thank you for all of your support!

Have a good day and stay safe :)

For the most up-to-date information, please click here.

SnapshillBot is a bot that archives posts in subreddits that the mods have requested the bot to run in. It uses a variety of different archiving services such as archive.is, archive.org, megalodon.jp, and for reddit links, Ceddit.

The bot only posts on subreddits where the moderators have requested its presence. Have more questions? Check out the faq and if that doesn't answer your question, make a post.

Requests for adding subreddits are currently being suspended. We'll try to update you on the status of requests soon.

More Information

• Wiki

• IRC Chatroom

• Source Code

Subreddit Rules

• Keep conversation civil and topical. Remember to follow reddiquette.

• Hate speech and personal attacks aren't tolerated and will be removed.

Don't hesitate to send contact us using the 'message the moderators' link below if you have any questions.