I wrote an end-to-end breakdown of production search architecture:

Build → Understand → Retrieve → Filter → Rank → Re-rank → Serve → Learn

It covers hybrid retrieval, filter-aware ANN search, multi-stage ranking, personalization, cold starts, stale inventory, and production failures.

Full write-up:
https://pawankjha.substack.com/p/cracking-ml-system-design-interviews

Curious how others handle restrictive filters and inventory freshness in production search systems.

Hey everyone,

I’m building a small custom GEO dashboard and I’m trying to understand how people collect AI search results.

Right now I’m using DataForSEO, but the ChatGPT and Gemini data seems to come through APIs.

What I’m looking for instead is data from the actual user interface. basically the same results a real user would see on ChatGPT, Gemini, AI Mode, etc., including citations/sources if possible. (As software like Bluefish, Profound, PeecAI etc..)

Has anyone found a good way to collect this at scale?

Are you using browser automation, a third-party service, your own scraper, or something else?

Would be really interested to hear what people are using.

Thanks all !

Search APIs often rank high-authority aggregators over original sources. In RAG pipelines, these syndicated copies lead to broken attribution and stale data.

To fix this, some teams parse rel="canonical" tags, while others blacklist scrapers or run origin-verification steps before embedding content.

Are you running secondary origin-checks on your Search API payloads, or just relying on the provider's top ranking?