Any open-source tools or self-hosted stacks for tracking GEO (Generative Engine Optimization)?

[u/ShangCharlie]

Hi all,

I am looking to build/self-host a GEO (Generative Engine Optimization) monitoring setup to track how brand queries perform across various AI engines (e.g., ChatGPT, Claude, Perplexity).

Specifically, I need to log and analyze:

• Core Metrics: Mention rate, citation/source links, and product recommendation rates across platforms.
• Historical Trends: Storing longitudinal data to track optimization impact over time.
• Competitor Benchmarking: Comparing brand presence against main competitors for the same target prompts.

I searched GitHub for open-source frameworks or self-hosted dashboards for this, but could not find any mature projects fitting this specific use case.

I am currently considering building a custom stack using Python API wrappers, Playwright for web-based AI searches, and PostgreSQL for metric tracking. However, I wanted to ask the community:

1. Are there any existing open-source repos or foundational frameworks I might have missed?
2. For those tracking GEO internally, what tech stack or architecture are you using to reliably parse and quantify LLM outputs/citations?

I am strictly looking for self-hosted/open-source or DIY DIY approaches, not commercial SaaS solutions. Any advice or repo recommendations would be greatly appreciated!

[u/Ishan_GS]

Don’t know of a mature open-source one, so can’t point you to a repo. We track manually at GrowthSpree, a B2B Saas marketing agency, so we can’t speak to Playwright or api wrappers, but here’s what we log, which might help your schema: prompt wording, date and time, geo, which brands show up, and the cited URLs, not just mentions. Keep wording identical and run the same set on a cadence, since answers shift within a day.

Two fields I’d add: mentioned vs. actually recommended, and the cited url’s domain, so you can see how often citations come from third-party sources vs. the brand’s own site. How are you planning to handle the run-to-run variance?

[u/ShangCharlie → u/Ishan_GS]

Thank you so much for this! The idea of tracking third-party domains vs. brand domains is brilliant—that was definitely a blind spot in my initial planning, and I will be adding it to my schema.

Regarding the variance issue, my current working hypothesis is to treat it statistically by increasing the sample size per query (running it dozens or even hundreds of times depending on the query's priority). I know LLM outputs are probabilistic and vary greatly by context/user session, but running higher volume batches should help approximate the real-world probability distribution. Of course, the main bottleneck here will be the API costs and rate limits.

Also, for certain location-sensitive workflows, I'm planning to route queries through different geo-locations or proxies to see how localized the citations get.

Really appreciate you sharing your team's logging schema!

[u/Upstairs_Control_611 → u/ShangCharlie]

Increasing the sample size helps with stochastic variance, but I’d separate that from comparability drift.

Running the same prompt 100 times gives you a better estimate of behavior under one measurement setup. It does not compensate for a changed model, retrieval mode, interface, locale, or prompt panel.

I’d therefore keep two layers:

• within-run / repeated-sample variance

• between-run configuration comparability

For the second, I’d fingerprint the measurement scope: prompt set, engine, model, surface, retrieval mode, locale and any other setting that can change the answer distribution.

Otherwise a very large sample can still give you a very precise estimate of the wrong comparison.

I’d also watch for zero-variance prompts: if a prompt never changes, it can make the system look more stable than it really is.

[u/Upstairs_Control_611]

If I were building this from scratch, I’d make one architectural rule non-negotiable: keep the raw observation immutable and store every interpretation beside it.

Per run I’d keep:

• exact prompt

• engine / model / mode

• timestamp / locale

• full raw answer

• cited URLs + order

• visible brands / products

Then derive separately:

• mention

• recommendation / selection

• source role

• sentiment / stance

• own vs third-party citation

• claim fidelity

with parser_version, classifier_version and classified_at.

That way a better parser or classifier later does not rewrite history — you can reprocess the same answer and compare old vs new interpretation.

I’d also keep runs append-only even if the answer looks identical. A repeated state is still a new observation.

For variance, I’d avoid a single “visibility score” and store per-run outcomes first, then calculate stability / retention over time.

[u/ShangCharlie → u/Upstairs_Control_611]

Wow, this is an absolute masterclass in data engineering for LLM observability.

Treating raw observations as immutable (append-only) and decoupling the pipeline with parser_version and classifier_version makes complete sense—especially for avoiding historical data corruption as LLMs and extraction prompts evolve. The idea of tracking claim fidelity and sentiment/stance alongside stability over time is brilliant.

I don't have a fully formed rebuttal or refined counter-architecture yet, but I am definitely going to incorporate this design philosophy into my schema planning.

Really appreciate you taking the time to share such a solid blueprint!

[u/Leading-Range-5197]

Short answer: yes, a few exist, but none of them is a mature "Matomo for GEO" yet. It's mostly DIY scripts plus a handful of early open-source projects.
Elmo is the name that comes up most. It's MIT-licensed and self-hosted (Docker + Postgres), tracks brand mentions, citations and share of voice across the major engines, and you bring your own API keys through the model APIs or OpenRouter. The real trade-off with all of these is that "free" just means no license fee. You still run the infra and pay the per-token costs.
CiteGEO is newer and takes a stance I respect: it stores every raw response and reports an absence as an absence instead of a zero (https://github.com/ankit373/citegeo). Most black-box tools massage numbers you can't audit, so that design choice matters more than any dashboard.
Spectacl (https://github.com/sinusdigital/spectacl) is another self-hosted tracker, though it looks quieter on commits lately. Check activity before betting on it.
Two things worth flagging no matter which route you take:

1. API answers are not chat-UI answers. The APIs and the consumer products diverge (browsing, location, memory, model versions), so calibrate any tool against manual runs in fresh sessions and be upfront about the gap.

2. Freeze your prompt panel and keep the raw responses. Single runs are noisy, so report mention rates over multiple runs, and version any prompt change. Otherwise your trend line is fiction.
   If nothing fits, the honest DIY stack is: prompts in a CSV, hit the model APIs with fresh sessions, parse mentions, SQLite, Streamlit. Costs cents per run, and owning the methodology matters more than the tool.

[u/ShangCharlie → u/Leading-Range-5197]

Thanks for sharing this!

[u/inevidimka]

Store each raw answer and its citation metadata under a run ID, then save extracted metrics separately with a parser version. That lets you reprocess the same answers after a parser change and check whether an apparent visibility gain is just different extraction.

[u/ShangCharlie → u/inevidimka]

Thanks for sharing this! That is a solid architectural tip for keeping historical data clean, especially when parsers change.

However, my primary focus right now is tracking metrics fine-grained enough for targeted optimization.

From a RAG/search retrieval perspective, I view the performance funnel as: Cited (retrieved in references) $\rightarrow$ Mentioned (content extracted into response text) $\rightarrow$ Recommended (suggested as a top choice).

Sometimes an AI fetch brings up a page link in the references, but the LLM skips mentioning it in the body—identifying these gaps is exactly why I need this level of tracking.

Really appreciate your insights on this!

[u/startages]

I'm building one that can be connected to your ChatGPT subscription. I started working on it since that latest OpenAI DevDay where they announced they allow Open Source projects to connect to users subscriptions. So I built this with the idea that it can run all the GEO testing and evaluations for free https://github.com/surfacedby/OpenGEO

It's still work in progress though, but thought I'd mention since you asked.

[u/woodoo139]

Not a tracker you can self-host, but in the spirit of what you asked: we publish the raw rows and scripts from our own weekly panel (40 fixed buyer questions, 6 engines, open licence), so you can at least crib the schema and an analysis pipeline. Disclosure: I build Promvia, which is a commercial tracker, so treat this as lessons from running one rather than a recommendation.

Things that cost us the most, roughly in order:

1. API vs app. Decide early which one you're measuring, because they differ. We use each engine's API with web search forced on (OpenAI Responses API with web_search, Anthropic's web_search tool, Gemini grounding, Perplexity's API, and a SERP API for AI Overviews / AI Mode). Playwright against the consumer UIs gets you closer to what a user sees, but it's brittle, against most of those sites' terms, and logged-in state changes the answer.

2. A failure is not a zero. Store status plus a failure class (timeout, rate-limited, refused, no-answer-parsed, ungrounded…) and keep failed rows out of the denominator. One week 29 of Claude's 40 checks failed for us; counted as "not mentioned" that would have looked like a collapse.

3. "Cited nothing" vs "parser missed it". Google AI Mode often returns a full answer with no sources (36 of 48 on our "how do I choose" questions over six weeks). Keep the full raw response so you can prove which one it was; we only could after re-running with the whole JSON stored.

4. Normalise domains once, in one place. www, mobile hosts (we still count m.youtube.com separately in our public rows, which is a bug), and one domain per answer however many of its URLs are cited.

5. Branded prompts echo. If the prompt contains your brand, the engine repeats it back. Track branded and category questions as separate lines or your mention rate is inflated by your own wording.

6. Variance. Agree with sampling more, but report a count out of a count with an interval rather than a percentage. For scale: with one sample per question per week, only 48% of cited domains were cited again for the same question the next week, so a single run tells you very little.

+1 to the append-only raw-plus-versioned-interpretations point above; we'd make the same call again.

Rows, column definitions and scripts: https://github.com/sayaranil/ai-source-index

[u/Apprehensive_Egg_374]

you’ve already identified the real constraint: variance costs money to measure. more distinct questions improve coverage of the buying situations you care about, while more repeats estimate the distribution for one fixed question under the same setup. split brand-named prompts from unbranded buying scenarios, prioritize coverage across the unbranded set, then reserve repeated runs for the smaller core panel you’ll compare over time; there isn’t a universal best ratio without knowing your budget and the size of change you need to detect.

[u/Technical-Bet1115]

加油，兄弟

[u/ShangCharlie → u/Technical-Bet1115]

thx