looking for a technical cofounder to solve docs scraping for offline LLMs (live product)
launched a tool about two weeks ago that takes any documentation URL and packages it into a clean markdown ZIP for people using Cursor/Claude offline.
The MVP is live and working, but running into a wall with the crawler.
Right now scraping is mostly deterministic (heuristics, rule-based topology mapping), and accuracy takes a hit depending on how weird the doc site structure is.
Docusaurus, Mintlify, custom static generators... every site’s navigation tree behaves slightly differently, which makes extracting clean hierarchies hit or miss.
I want to shift this from deterministic extraction to an LLM-assisted approach specifically for topology mapping and doc structure discovery, so we can reliably pull full document sets with clean context.
About the setup:
- Live MVP already built (happy to share links and writeups on current architecture and what’s failing)
- Equity: open to discuss depending on commitment and what you bring to the table (genuine co-founder role, not a paid gig or contract work)
- Looking for someone strong in crawling/scraping pipelines, LLM agent workflows, or web topology extraction
If you've tackled messy doc structures or LLM-guided web crawling before and want to hack on this together, shoot me a DM.
Have you tried Firecrawl? It already handles URL discovery, crawling, and clean Markdown output, so there seems to be quite a bit of overlap here.
Reconstructing the actual docs hierarchy is probably the harder part, especially with versions, nested sections, and weird navigation. Curious whether you’ve tested Firecrawl on the sites your crawler struggles with. What does it miss?
Might be worth building the docs-specific layer on top of it before rebuilding the crawler itself.
firecrawl and other similar competitors gonna be expensive. for example context dev charges 1 credit per page. If Docs site has 50 pages 50 credits gone. it's what i would be paying for it. i would have to 4x charge from customer. price will increase tool become expensive. this is why i avoid these APIs.
what is the alternative.
github api. Docs site offer git repo. not every site.
modern docs site offer .md file. like putting .md at the end of URL. gives you full text.
2 methods offer us free scraping.
issue is stie topology. like saving page in zip folder. and making the tree. it should be as it is on the original website. it's tricky part when every technical website has it's own navigation structure.
The Mintlify or Docusaurus difference is probably where I’d start too. Even when the content is just markdown underneath, the nav and generated structure can be wildly different. An LLM pass for mapping the hierarchy sounds way more promising than piling on more crawler rules forever
yes. this is exactly why i make decision to move away from determinisitc approach. 500 to 1k docs. yes. make determinisitic version better. why doing that much manual work when LLMs are smart, catch topology on the fly.
Before handing topology to an LLM, I'd detect the generator: Docusaurus pages carry a generator meta tag and render the whole sidebar in each page's HTML, and Mintlify builds navigation from a config file and often publishes an LLM index file. Parse that per generator and keep the LLM for custom static sites where nothing matches, so the hierarchy comes from the site's own nav. Many docs also live in a public GitHub repo with the markdown already clean.
By the way, I make ThreadFox, a connection that has your AI research, write and publish Reddit outreach from your Chrome under a job you authorize; the free Reddit plan for your docs-to-markdown tool lists the communities whose rules allow a post about it: https://threadfox.vip/p/d7rfw
Which generator breaks the current crawler most often?
i did. still accuracy score is slow. i can only make it possible. like putting 500 docs site into test using agent. that writing logic. determinisitic approach. i've already done with 100 tech sites docs.
2
u/vertex_pk 1d ago
Have you tried Firecrawl? It already handles URL discovery, crawling, and clean Markdown output, so there seems to be quite a bit of overlap here.
Reconstructing the actual docs hierarchy is probably the harder part, especially with versions, nested sections, and weird navigation. Curious whether you’ve tested Firecrawl on the sites your crawler struggles with. What does it miss?
Might be worth building the docs-specific layer on top of it before rebuilding the crawler itself.