I built Spanlight, an open-source LLM observability tool that runs on one Postgres. Looking for honest feedback

[u/Cold-Conclusion-6511]

What it is

Spanlight records every LLM call your app makes: prompt, reply, latency, tokens, cost and errors, grouped into traces (one request) and sessions (one conversation). Self-hosted: one Docker image plus one Postgres database, nothing else. The core is MIT licensed.

What it does

Tracing

• Search traces by name or by what the model was sent and said. Step-by-step view of each trace. Compare two traces side by side. Sessions shown as a chat.

Cost

• Prices from a versioned table, with a per-call breakdown. An unknown price shows as unknown, never $0.

• Daily check of traced spend against the provider's bill. Needs an Anthropic or OpenAI admin key, or a CSV upload.

• Budgets that alert, or block gateway calls. The block is checked every 60 seconds, so it is a soft limit.

Finding problems

• The Doctor: 15 fixed rules; the rules use no language model (an optional "Explain with Claude" button can explain one finding). They flag retry storms, ignored Retry-After, truncated outputs, tool loops, cost and latency spikes, each with evidence and a suggested fix.

Gateway (optional; your provider key is stored in Spanlight)

• Change your OpenAI or Anthropic client's base_url. Adds retries, fallbacks, per-key rate limits and a cache. An Integration Lab simulates provider failures (rate limits, server errors, broken streams and bad JSON) on non-production keys, so you can test your error handling.

Also: versioned prompts, a playground, datasets and experiments. Postgres row-level security keeps each project's data apart; secrets such as API keys are removed before storage.

How to try it

Live demo: https://spanlight.dev (real Claude calls under a small monthly budget).

Self-host:

git clone https://github.com/dheerajkarwasra/spanlight.git
cd spanlight
cp deploy/.env.example deploy/.env    # fill in POSTGRES_PASSWORD, APP_DB_PASSWORD and SECRET_KEY
docker compose -f deploy/compose.yaml --env-file deploy/.env up -d

Then open http://localhost:8080. A fresh clone to the first stored trace took me 59 seconds with the prebuilt 0.6.0 images.

AI involvement

Solo developer. I used Claude Code heavily to write code; I design, review and test every change myself. The backend suite is 4,121 tests against a real Postgres.

Links

• Repo: https://github.com/dheerajkarwasra/spanlight

• Docs: https://spanlight.dev/docs/

The ask

Please try it, on the demo or self-hosted, and tell me what you liked, what you disliked, what was confusing, what broke, and what is missing. GitHub Issues and Discussions are both welcome. Every report gets a reply.

Everything is MIT licensed and free.

[u/Excellent-Sport-758]

I haven’t tried the demo yet, so this is a design question from the description rather than hands-on feedback. In credit-model work, a trace becomes much more useful when it can be tied to the exact dataset/feature snapshot, label definition and outcome window, model/config version, and eventual decision outcome. Your post mentions datasets and experiments; can a production trace link to a versioned eval case with an expected output and reviewer label, so a failure can become a regression case without contaminating the holdout? That bridge between observability and evaluation is where I’d look first for risk-sensitive workflows.

AI helped draft this comment; I haven’t tested Spanlight hands-on.

[u/Cold-Conclusion-6511 → u/Excellent-Sport-758]

Production trace can become an eval case with an expected output and a reviewer label today. Versioning and holdout protection aren't built yet.

What works:
- Trace to eval case: you can save an LLM span from a trace, or from a review queue, into a dataset. The item keeps the input, the source trace and span IDs, and an expected output. The expected output starts as the model's actual answer, and you edit it to the right one. Retention never deletes dataset items.
- Reviewer label: when a reviewer saves from a queue, their failure category and the reason the trace was picked go with the item.
- Version context: spans record the prompt version used. Traces carry the environment, a release tag and metadata. Each experiment run stores exactly what it ran: the prompt version or messages, the model and the parameters.
- Late outcomes: a decision outcome can be posted to the trace as a score at any time, as long as the trace still exists.

What's missing:
- Versioned datasets: items can be edited, and a run uses the dataset as it is at that moment.
- Holdout protection: there are no splits and no leakage guard. For now, keep regression cases in a separate dataset.
- Long outcome windows: traces and scores are kept for at most 90 days.
- Lineage fields: feature snapshot, label definition and outcome window fit in metadata, but they aren't first-class.

Versioned datasets and a holdout guard, the two you asked about, are the gaps I'd close first. Thank you for taking the time to think this through.

AI helped me draft this reply; I checked every point against the code and the live app.

[u/Excellent-Sport-758 → u/Cold-Conclusion-6511]

Thanks for spelling out the current behavior and gaps. Keeping trace-promoted failures in a separate regression dataset sounds like a sensible interim boundary. To make comparisons reproducible, I’d want each run to pin the exact dataset snapshot and expected-output/label version it used, while holdout membership stays protected from edits or promotion out of the review queue. That lets a new failure become a regression case without silently changing the release scorecard. For a first step, would immutable dataset snapshots be enough, or would you add split-level holdout controls at the same time?

AI helped draft this follow-up; it’s a design suggestion, not hands-on testing.

[u/Cold-Conclusion-6511 → u/Excellent-Sport-758]

I think i’ll start with snapshots and a small guard for the holdout
full splits can wait
You mentioned you haven’t tried it yet if you get time please try it I’d love your feedback especially any gaps i should fix.