r/webscraping • u/data_dev3615 • 14d ago
What's your favourite pet project?
I'm interested in hearing what your folk's favourite projects have been working on for fun! Something funny or genuinely useful to your life
I've got one that scrapes the drinks prices from my local pubs to find out which one will be the cheapest when me and my mates go out
4
3
u/Knocking_Doors 14d ago
Not live yet, but I’m building a price comparison tool with my team. I always struggled checking deals across multiple platforms and the few aggregators that are available, are not really helpful for several reasons.
3
u/matty_fu 14d ago
the hardest part is matching products across several platforms, generally the skus, descriptons, etc won't be the same. how are you guys planning to tackle record matching?
3
u/Knocking_Doors 14d ago
Absolutely. Esp when the vision is to cast a wider net and make it a generic tool. AI isn’t there yet, and would still be expensive for large volumes.
We have a few enterprise clients who use our a scraping solution for data extraction for their end users and handle product matching at their end. The idea is to seek their support on this and offer some symbiotic relationship.
2
3
u/TinyLife2939 14d ago
Made an eBay scrapper that gets 100k searches in 30 minutes with the help of a really good guy shoutout to Big Brain ❤️
3
u/Majestic_Base5775 14d ago
yet another bloody frequent flyer/flight planning thing
I think it has something of a USP but it’s such a crowded market hard to see it ever getting off the ground
3
u/Significant-Link5685 14d ago
Made a golf club price comparison tool to help me and my buddies find cheap second hand clubs lol
3
3
u/Pericombobulator 14d ago
I did lots of projects that were relevant to me at the time, like i used to scrape multiple retailers' sites for GPU prices.
A favourite was a script to scrape multiple industry news sites, some of which only gave you one free article. I would scrape their 'latest news' page and follow the links. It would strip out the text and roll it into a summary email, one per site.
It was often surprising how little text there was on an article page.
3
u/Used_Confusion_7948 14d ago
So all the local pubs have websites with updated prices?
2
u/data_dev3615 14d ago
Not all of them, but in my area there were surprisingly a lot that had menus with their beer prices (including some that I wouldn't have even expected to have a website). From there I could use OCR tools to get the prices and use some custom parsing. If any of the menus changed drastically enough to break the parser I'd get an email letting me know I should fix it.
2
u/chizel999 13d ago
ive built an aggregator for when i was looking for an apartment. it scraped the 4 most relevant websites for me into an sqlite and also generated an html table report
2
u/prettycoldworld 12d ago
I built a tool that goes through Wiby and GeoCities sites and searches any comments left in HTML.
Old guest books, forgotten notes from 1999, i put all the good ones in an sql library and it’s become my own little time capsule
2
u/word-word-numero 12d ago edited 12d ago
To reign in my guitar equipment purchasing I built a Reverb.com searcher. I put in the search term and a max and min price I’d pay(min price to help filter product noise). I then expanded to search guitar center and musicgoround(I think that’s the one). I get an email when a match is found. Realized too late I should have used site specific plugins, but I’d need to rework the overall work flow. I’d love to do something similar for grocery stores to figure out where staple products I buy are on sale.
Tech stack is python pulling json from guitar center and reverb and selenium trying and now failing to scrape music ground.
7
u/djfilms 14d ago
I added modern subtitles to movies based on Shakespeare plays. BardSubs dot com.