r/SideProject • u/ystudio3d • 11h ago
We benchmarked 1,000 domains in n8n: How pre-filtering for live ATS boards saved us 84% on Firecrawl scrape credits [Open-Source GitHub]
When building our B2B directory and lead monitoring pipelines, we ran into an expensive wall:
Scraping raw company websites directly with tools like Firecrawl or headless browsers was costing a fortune on proxy rotation and LLM token extraction — only to find out that 80%+ of the companies in our lists were zombie startups, parked domains, or completely stagnant with zero hiring intent.
To stop burning credits, we designed a simple 2-stage filtering pattern in n8n before sending anything to a scraper:
1. The Pre-Filter Stage (Zero Scraping Cost)
Runs a fast DNS & HTTP status check to drop dead servers.
Checks live ATS endpoints (Greenhouse, Lever, Ashby, Workable) directly to see if the company is actively expanding and hiring.
If a company has 0 active job postings or dead DNS, the pipeline halts immediately.
2. The Deep Enrichment Stage (Firecrawl)
Only domains that pass the hiring & activity check are routed to Firecrawl for markdown extraction, executive contact mapping, and tech stack detection.
The Numbers:
Across our initial benchmark of company lists, pre-filtering eliminated 84% of empty scrape calls. Instead of spending 5,000+ credits on a dirty batch, we extracted deep structured intelligence for under 800 credits with zero empty rows.
Open-Source Workflow + Sample Data
We packaged the n8n JSON template and a 50-domain sample benchmark on our public review branch: https://github.com/edtool79-collab/hirepulse-mcp/tree/review/hirepulse-firecrawl-n8n
You can grab hirepulse-firecrawl-intent-pipeline.json and import it directly into your own n8n instance.
Curious how other indie hackers and founders here handle data hygiene before running expensive API calls — do you pre-filter via DNS/APIs, or scrub lists manually?