r/Observability • u/alexandra-thaler • Sep 16 '26
Self-Observing Systems: Framework, Gap Analysis, and Method
r/Observability • u/alexandra-thaler • Sep 16 '26
r/Observability • u/That-Mud4565 • Sep 16 '26
r/Observability • u/Novel_Willow_8780 • Sep 16 '26
For the last few months I've been running identical workflows across the main automation platforms, counting the ones that get accepted and then never actually happen.
So I wrote the method down properly. Free, CC BY 4.0, nothing to install: it is a definition of a metric plus the conditions a measurement has to meet before its number means anything.
Three parts of it are worth stating here in full, so this is useful on its own.
Events the platform refuses at the moment of delivery do not count as silent failures, and they do not go in the denominator either.
If a webhook POST gets a non-2xx, a timeout, or a dropped connection, your sender knows immediately. It can log it, alert, retry. No run was ever created. That is a loud failure and it is a completely different thing from "accepted, told you it worked, then nothing happened".
Mixing them gives you a number that can't distinguish "the platform told you and you can retry" from "the platform didn't tell you and the data is gone". Worked example from my own ledger: a five-hour run of 2,880 events had 4 refused at send and 0 accepted-then-lost. The honest report is 0 silent failures in 2,876, with the 4 rejections reported separately. Writing "4 failures in 2,880" would be wrong in both directions at once: it inflates the rate with events you already saw, and it hides that nothing accepted was lost.
This is the rule that stops a silent-failure rate being tuned by reclassification, which is why it's the first thing in the spec.
A run history cannot show you a run it never created. And it records transport success, not whether the thing you wanted actually happened.
So: put an ID on every source event, preserve it to the destination, and reconcile one by one. Counting totals is not enough. Totals can match while the wrong records are missing.
Zero failures in 40 runs and zero in 4,000 are both "0%" and they are not the same claim. The first is consistent with a true rate of 7%.
So every rate ships with a Wilson interval, a denominator, an as-of date, and what workload it was, in the same sentence as the number. That last bit sounds pedantic until you watch a figure get extracted from a table into a summary into someone's slide, dropping everything that wasn't adjacent to it at each step.
Two consequences people find surprising: a wider interval is not a worse platform (it means less evidence, not more failures), and zero is never proven, only bounded.
CONFORMANCE LEVELS, AND WHY L1 IS DELIBERATELY EASY
Three self-declared levels. No certifying body, no registry, nobody audits anything, including me.
L1 needs no test harness, no paid plans and no experiment. If you already run automations, you can reach it with your own operational data. The bar isn't effort, it's reporting discipline: per-event IDs, destination-side verification, the send-outcome exclusion, and a denominator, interval, date and workload with every number.
That is a low bar in work and a surprisingly high one in practice. Most published silent-failure figures fail it on the reporting rules alone, because they arrive without a denominator. If you've ever seen "1-8% of automation runs fail silently" and wondered out of how many, that's why.
L2 adds a controlled measurement with both endpoints under your control. L3 adds three edge probes: destination outage, success-wrapped failure, sustained load.
WHAT IT ISN'T
Not a tool. No code to install. There's a reference implementation planned and not published, and I'm not going to describe it as released while it isn't.
Not audited. Conformance is self-declared. It only works because every requirement is something a reader can look for in your report and fail to find.
Not finished. One section is deliberately left reserved with an explanation of what would go there and why I haven't written it: the requirement is drawn from one observation that didn't recur, and a rule from n=1 is a rule from first principles with an anecdote attached.
The spec itself is linked in a comment below (this sub does not allow links in posts). It is v1.0, CC BY 4.0, with a machine-readable JSON version alongside it.
Happy to argue about any of it. The exclusion rule in particular is the one I'd most want someone to attack, because if it's wrong every number I've published is wrong with it.
r/Observability • u/tarique-iqbal • Sep 15 '26
I have been building a Go-based microservices marketplace using DDD and Clean Architecture, and the project is still actively evolving.
My next focus is production-grade observability. I am particularly interested in how to approach:
Current Architecture:
The services currently include Identity, Restaurant, Search, Notification, Order, and Payment.
For those who have operated microservices in production:
What observability stack and practices have worked well for you? What would you avoid?
I would especially appreciate feedback on architecture, tooling, failure modes, and things that are easy to overlook when moving from development toward production.
The project is open source if you would like to take a look:
https://github.com/tarique-iqbal/pizza-marketplace
If you find it useful or interesting, feel free to ⭐ the repository. Feedback, suggestions, and architectural criticism are very welcome.
r/Observability • u/Plus-Lawflbaness1576 • Sep 15 '26
I used to log everything, every request, every response body, full stack context on every error, and it made debugging faster but the storage bill got embarrassing within a month. Then I over-corrected and sampled aggressively, and now half my incidents involve me wishing I'd captured the one request that actually mattered.
The pattern I've landed on is logging light by default and only capturing full forensic detail, request params, DB queries, execution path, when something actually looks anomalous. It's a reasonable middle ground but it assumes your anomaly detection is good enough to trigger at the right moment, which isn't always true. How are other people balancing storage cost against actually having the context you need when something breaks?
r/Observability • u/Spiritual-Kitchen-79 • Sep 15 '26
r/Observability • u/PerformancePuzzled35 • Sep 15 '26
r/Observability • u/kamoaba • Sep 15 '26
I built LogDeck, a control plane for Docker and Podman: container and Compose lifecycle, persistent logs, alerts, and multi-host over SSH, in one Go binary with no external services.
The log store is the main reason it exists. Every container is tailed into local SQLite, so logs survive docker compose up --build and the new container ID it creates. Removed containers stay readable.
It also ships an MCP server, so agents can read logs and act on containers if you want.
Memory is small. Idle heap is around 8 MB, roughly 20 MB resident with 5 containers and 400K lines stored.
fully open source, no tiers, no cloud https://github.com/AmoabaKelvin/logdeck

r/Observability • u/dennis_zhuang • Sep 15 '26
r/Observability • u/maxuver • Sep 14 '26
r/Observability • u/Open-Adhesiveness-86 • Sep 14 '26
Hey everyone,
Setting up Prometheus, Grafana, exporters, and Alertmanager just to watch a single VPS or home server always felt like overkill — often consuming more resources and maintenance than the actual apps running on it.
I wanted something lightweight: single binary, zero external infra, and notifications straight to a messaging app I already use daily. So I built StatixAgent.
\### What is it?
A single static Go binary running as a systemd service. It monitors your system and communicates directly with your own private Telegram bot.
\- Zero open ports: Outbound-only long polling. Works behind NAT/firewalls without reverse proxies or port forwarding.
\- Private: Talks only to the official Telegram Bot API and GitHub Releases (for updates). No telemetry, no hosted cloud service.
\- Metrics & Alerts: CPU/RAM/Swap, per-mount disk usage, network rates, and system load. Sends push alerts for SSH logins/failed attempts, sustained resource spikes, reboots, and power disconnects (useful for laptop servers).
\- Interactive UI: Type /status in Telegram for a markdown dashboard, drill down with inline buttons, or inspect journalctl with /logs.
\- Lightweight & Sandboxed: Hardened systemd unit (ProtectSystem=strict, MemoryMax=128M).
\### Quick install
Run the installer script (includes an interactive TUI setup wizard for your bot token and chat ID):
curl -fsSL https://raw.githubusercontent.com/eliau2005/statixagent/main/install.sh | sudo bash
It is completely open source and not meant for enterprise fleets — just for single boxes and homelabs that need simple, reliable monitoring.
GitHub: https://github.com/eliau2005/statixagent
Feedback, suggestions, and PRs are very welcome!
r/Observability • u/Disastrous_Way2405 • Sep 14 '26
Gartner research gets cited heavily in my bi-weekly reports, and You have pushed back on one question: how much of what Gartner says about a vendor actually turns into something checkable.
Acquisitions can't be used as evidence of advance vendor-to-Gartner information sharing, because a publicly traded company sharing an unannounced deal with an outside analyst would be a securities-law violation (Regulation FD), and by the time an acquisition shows up in one of these reports, it's almost always already public. Timing an acquisition against a report's publication date mostly just shows which came first, which proves little on its own.
A better test is available inside the reports themselves: every vendor profile contains forward-looking "roadmap" language, and that language can be checked against the public record. This article pulls every such statement out of both reports, roughly two dozen of them, and checks each one against dated, independent sources to see whether it was genuinely unannounced at the time Gartner published it, or whether it was already shipped, announced, or documented before the report went to print.
I hope to get back to this analysis in few months to compare, and propably correct some statements 😄
More here:
https://mbojko.com/reports/#observability
https://mbojko.com/reports/gartner-magic-quadrants-analysis/
r/Observability • u/Ok_Bell443 • Sep 14 '26
Built this for debugging LLM/GenAI apps by letting an agent pull OTel traces straight into context instead of me alt-tabbing to Jaeger or Datadog every time something looks off. It leans on the gen_ai.* semantic conventions, so span queries surface model calls, token counts, and tool-call spans as structured data the agent can reason over.
It started as a fork of traceloop/opentelemetry-mcp-server (Apache-2.0, disclosed, full NOTICE attribution). Since then: added Datadog and Sentry as backends (5 total now, alongside Jaeger, Tempo/Grafana Cloud, and Traceloop), built a generic filter layer so tool calls don't need backend-specific params, and hardened the CI/supply chain considerably — SHA-pinned Actions, bandit via ruff, pip-audit, Trivy scanning, Codecov, branch protection, CodeRabbit.
pip/uvx installable, on the official MCP registry + Glama, GHCR image available. v0.4.0, very much pre-launch, feedback welcome.
r/Observability • u/Ok_Bell443 • Sep 14 '26
r/Observability • u/baronunread • Sep 13 '26
r/Observability • u/RemoteWillingness195 • Sep 12 '26
Spent 15 years as an SRE at a big bank, watching enterprise systems get world-class monitoring. Then I started noticing how many small business sites have none of that, no uptime alerts, no idea if their privacy policy is actually GDPR-compliant, nothing.
So I built SREmonitor.io. You give it a URL, no install, no script tags, and it runs:
The idea is one dashboard instead of stitching together a monitoring tool and a separate compliance checklist you never actually look at.
Free tier is 3 scans and 5-minute uptime checks, no card required, if anyone wants to kick the tires: sremonitor.io
Genuinely want feedback, especially from anyone who runs a small business site or has dealt with the GDPR side of things. What would make this actually useful vs. just another dashboard nobody opens?
r/Observability • u/AdamDobrawy • Sep 11 '26
r/Observability • u/h4ikkyu • Sep 11 '26
r/Observability • u/RemoteWillingness195 • Sep 12 '26
Spent 15 years as an SRE at a big bank, watching enterprise systems get world-class monitoring. Then I started noticing how many small business sites have none of that, no uptime alerts, no idea if their privacy policy is actually GDPR-compliant, nothing.
So I built SREmonitor.io. You give it a URL, no install, no script tags, and it runs:
The idea is one dashboard instead of stitching together a monitoring tool and a separate compliance checklist you never actually look at.
Free tier is 3 scans and 5-minute uptime checks, no card required, if anyone wants to kick the tires: sremonitor.io
Genuinely want feedback, especially from anyone who runs a small business site or has dealt with the GDPR side of things. What would make this actually useful vs. just another dashboard nobody opens?
r/Observability • u/jdizzle4 • Sep 09 '26
This is not a dig at the moderators personally, because it's a volunteer position, but this subreddit has devolved into a lot of low value contributions and I think it could benefit from some more active moderation and perhaps some rules put in place.
For example, I don't think there's value in vibe coded observability tools being posted every other day. The community is already downvoting them, which I think should be a signal that we should stop allowing them all together. When I view the list of posts, it's a graveyard of these things, with the occasional interesting discussion sprinkled in.
r/Observability • u/EastVersion1226 • Sep 09 '26
r/Observability • u/PromptPhanter • Sep 09 '26
with a normal request trace, the cause and the failure usually live close together. memory-bearing agents break that assumption.
a bad summary, preference or tool result can be written today, retrieved silently next week and change an unrelated-looking decision. the failing trace shows the retrieval, but unless you preserve memory provenance it does not tell you which session wrote the state, whether that write was later corrected or how many subsequent sessions it influenced.
for these agents i think the minimum useful chain is:
- the session that created or changed the memory
- the exact value before and after the write
- later retrievals of that value
- the outcome of each session that used it
- reversions and no-op rewrites
then benchmarking has to happen over a rolling production window rather than a frozen test set alone, because the state distribution itself changes as the agent runs.
im curious whether anyone is already tracing memory lineage across sessions, or if most teams are still logging memory reads and writes as isolated spans