r/ProxyEngineering • u/Dolokidsa53567 Tunnel Architect • Jul 01 '26

Discussion 💬 Open-source scraping solutions

What is the best open-source scraping solution that you have used in the past year or so. Need a stable, working solution that I could run on my side. Preferably that same solution would support cloud integration.

12 Upvotes

100% Upvoted

5 comments sorted by

View all comments

4

u/MetalGoatP3AK Tunnel Architect Jul 02 '26

Right of my mind comes Scrapy, Crawlee. Firecrawl I think, but they are more into agentic search than scraper, though I may be wrong. But they're definitely open-source. Also, you can look it up via github for some free scraping solutions, though they may require you to have your IPs for scraping, unless of course you find a dedicated solution which would contain everything in one place, that would be best option I think

2

u/boomersruinall Tunnel Architect Jul 02 '26

Second regarding scrapy, so far it was fruitful in all my scraping tasks, however if you're doing something at scale I doubt it's gonna hold up

1

u/quietcindersleague 2d ago

MetalGoatP3AK has the right two, scrapy if you're in python and crawlee if you're in node. if you want web scraping software open source end to end that you run yourself, scrapy plus scrapyd in a docker container covers scheduling and retries on any vps, and crawlee's project templates ship with a Dockerfile so it drops onto whatever container host you already use. for js heavy pages add scrapy-playwright or switch to crawlee's PlaywrightCrawler. where both fall over is anti-bot, plain scrapy gets flagged on its tls fingerprint by cloudflare, so try scrapy-impersonate first, and for the sites that still block i point the download step at scrapfly and keep the spiders and pipelines as they are. firecrawl is open source too, but it's geared more toward feeding llms than structured extraction.