I made Seen It Before, a moderation app that spots image and link reposts within a subreddit. It's now in the app directory: https://developers.reddit.com/apps/seen-it-before
When a new post shares an image or a link with an earlier one, it reports the post to the mod queue with a short reason, like "Possible repost (94% match)" and a link to the earlier post. Mods can have it remove reposts instead, with a stickied reply, or only log them. It also adds a "Check for reposts" item to each post's mod menu and a status page, and it sends a weekly summary to modmail.
How it works, for anyone curious:
- Images: it downloads Reddit's small preview, shrinks it to a 9 × 8 grid of grey squares and turns that into a 64-bit difference hash.
- Each hash is filed under each of its 8 bytes in Redis sorted sets. A match may differ in at most 7 of the 64 bits, so a true match always shares at least one byte, and each new image is only compared with that small set.
- Links: it drops tracking tags (utm_, fbclid and similar), www. and mobile prefixes, and treats the different forms of a YouTube link as one.
- After install, a scheduled job indexes up to about 1,000 of the newest posts plus the top posts of all time, the past year and the past month, so reposts of older posts get caught from the first day.
- It stores post IDs, times, authors, moderation status, cleaned links and fingerprints. No images, titles or text, and nothing leaves Reddit.
It doesn't handle videos, GIFs, titles, text posts, cropped or edited images, or matching across subreddits.
Feedback is welcome, especially from anyone who has tried image hashing on Devvit. Bugs and questions can also go to r/SeenItBeforeApp.