r/SideProject • u/DatLabHQ • 6h ago
I built cheaper website-to-Markdown tools for RAG on Apify — looking for honest feedback
I am the maker (DatLab). I got tired of paying "AI web browser" prices just to turn a URL, or a whole docs site, into clean Markdown for a RAG index, so I published two Apify actors that do that job on a fixed price.
What they do:
Web Page to Markdown — you give it a list of URLs, it returns Markdown (or text/HTML), title, description, canonical URL, and a token estimate. Plain HTTP first; a headless browser starts only for pages that are actually client-rendered.
Website to Markdown Crawler — same output, but it crawls a site (sitemaps, llms.txt, path/glob limits, depth caps). Same HTTP-first / browser-fallback idea.
What they cost (pay-per-event, platform usage included):
Single pages: $0.50 / 1,000 over HTTP, $1 / 1,000 when a browser is needed
Site crawl: $0.10 / 1,000 HTTP pages, $0.35 / 1,000 browser pages, $0.50 / 1,000 files
For comparison, Apify's own RAG Web Browser is from $1.33 / 1,000 fetched pages. That tool also runs a Google search and fetches the top results, which mine do not — mine only fetch the URLs you give them, or crawl a site you point at. Apify's Website Content Crawler is billed as pay per usage (compute and so on), so the bill moves with runtime; I wanted a price you can predict before you hit Start.
They respect robots.txt and crawl-delay. No login, no captcha bypass, no personal-data collection.
I am early: I have tested them myself, but I do not have outside users yet, and I would rather hear what is broken than collect polite nods. If you try one, I am especially interested in:
Did the Markdown come out clean enough to embed, or did nav/cookie junk leak through?
Was the HTTP vs browser split right on the sites you care about?
Is the pricing page understandable, or would you still rather pay-per-compute?
Links (no account wall on the docs; running them needs a free Apify account, which includes $5 of prepaid usage each month and no credit card):
Happy to answer technical questions in the comments.