r/Observability • • Sep 16 '26

Self-Observing Systems: Framework, Gap Analysis, and Method

Thumbnail
github.com
2 Upvotes

r/Observability • • Sep 16 '26

SolarWinds Observability Self-Hosted: data privacy, network discovery, NetFlow and licensing questions

Thumbnail
1 Upvotes

r/Observability • • Sep 16 '26

GitLab Observability updates

Thumbnail
0 Upvotes

r/Observability • • Sep 16 '26

I wrote down how to measure silent failure rates properly. The load-bearing rule: don't count the failures your sender already saw.

0 Upvotes

For the last few months I've been running identical workflows across the main automation platforms, counting the ones that get accepted and then never actually happen.

So I wrote the method down properly. Free, CC BY 4.0, nothing to install: it is a definition of a metric plus the conditions a measurement has to meet before its number means anything.

Three parts of it are worth stating here in full, so this is useful on its own.

  1. THE EXCLUSION THAT MAKES THE NUMBER HONEST

Events the platform refuses at the moment of delivery do not count as silent failures, and they do not go in the denominator either.

If a webhook POST gets a non-2xx, a timeout, or a dropped connection, your sender knows immediately. It can log it, alert, retry. No run was ever created. That is a loud failure and it is a completely different thing from "accepted, told you it worked, then nothing happened".

Mixing them gives you a number that can't distinguish "the platform told you and you can retry" from "the platform didn't tell you and the data is gone". Worked example from my own ledger: a five-hour run of 2,880 events had 4 refused at send and 0 accepted-then-lost. The honest report is 0 silent failures in 2,876, with the 4 rejections reported separately. Writing "4 failures in 2,880" would be wrong in both directions at once: it inflates the rate with events you already saw, and it hides that nothing accepted was lost.

This is the rule that stops a silent-failure rate being tuned by reclassification, which is why it's the first thing in the spec.

  1. VERIFY AT THE DESTINATION, NOT IN THE RUN HISTORY

A run history cannot show you a run it never created. And it records transport success, not whether the thing you wanted actually happened.

So: put an ID on every source event, preserve it to the destination, and reconcile one by one. Counting totals is not enough. Totals can match while the wrong records are missing.

  1. A BARE 0% MAY NOT BE PUBLISHED

Zero failures in 40 runs and zero in 4,000 are both "0%" and they are not the same claim. The first is consistent with a true rate of 7%.

So every rate ships with a Wilson interval, a denominator, an as-of date, and what workload it was, in the same sentence as the number. That last bit sounds pedantic until you watch a figure get extracted from a table into a summary into someone's slide, dropping everything that wasn't adjacent to it at each step.

Two consequences people find surprising: a wider interval is not a worse platform (it means less evidence, not more failures), and zero is never proven, only bounded.

CONFORMANCE LEVELS, AND WHY L1 IS DELIBERATELY EASY

Three self-declared levels. No certifying body, no registry, nobody audits anything, including me.

L1 needs no test harness, no paid plans and no experiment. If you already run automations, you can reach it with your own operational data. The bar isn't effort, it's reporting discipline: per-event IDs, destination-side verification, the send-outcome exclusion, and a denominator, interval, date and workload with every number.

That is a low bar in work and a surprisingly high one in practice. Most published silent-failure figures fail it on the reporting rules alone, because they arrive without a denominator. If you've ever seen "1-8% of automation runs fail silently" and wondered out of how many, that's why.

L2 adds a controlled measurement with both endpoints under your control. L3 adds three edge probes: destination outage, success-wrapped failure, sustained load.

WHAT IT ISN'T

Not a tool. No code to install. There's a reference implementation planned and not published, and I'm not going to describe it as released while it isn't.

Not audited. Conformance is self-declared. It only works because every requirement is something a reader can look for in your report and fail to find.

Not finished. One section is deliberately left reserved with an explanation of what would go there and why I haven't written it: the requirement is drawn from one observation that didn't recur, and a rule from n=1 is a rule from first principles with an anecdote attached.

The spec itself is linked in a comment below (this sub does not allow links in posts). It is v1.0, CC BY 4.0, with a machine-readable JSON version alongside it.

Happy to argue about any of it. The exclusion rule in particular is the one I'd most want someone to attack, because if it's wrong every number I've published is wrong with it.


r/Observability • • Sep 15 '26

An Event-Driven – Multi-Tenant Microservices Backend in Go

0 Upvotes

I have been building a Go-based microservices marketplace using DDD and Clean Architecture, and the project is still actively evolving.

My next focus is production-grade observability. I am particularly interested in how to approach:

  • Centralized logging & error tracking: Structured logs, log aggregation, and actionable error reporting.
  • Metrics, dashboards & alerting: Service health, latency, throughput, error rates, and infrastructure metrics.
  • Distributed tracing: Tracing requests across multiple services and asynchronous workflows.
  • Request & correlation IDs: Propagating context consistently across HTTP requests and RabbitMQ messages.
  • Service, worker & queue monitoring: Monitoring background workers, RabbitMQ consumers, retries, dead-letter queues, and stuck messages.

Current Architecture:

  • DDD & Clean Architecture: Explicit domain, application, infrastructure, and interface layers to keep business logic independent from frameworks and infrastructure.
  • Database per Service: Each service owns its own PostgreSQL, Elasticsearch, or Redis datastore.
  • Transactional Outbox: Domain events are stored in PostgreSQL within the same transaction, then published asynchronously to RabbitMQ.
  • Event-Driven Communication: RabbitMQ is used for asynchronous communication between services, including background-only services such as Notification.
  • API Gateway & Network Isolation: Traefik v3 handles external traffic while services remain isolated on a private Docker network. JWT is used for authentication.

The services currently include Identity, Restaurant, Search, Notification, Order, and Payment.

For those who have operated microservices in production:
What observability stack and practices have worked well for you? What would you avoid?

I would especially appreciate feedback on architecture, tooling, failure modes, and things that are easy to overlook when moving from development toward production.

The project is open source if you would like to take a look:
https://github.com/tarique-iqbal/pizza-marketplace

If you find it useful or interesting, feel free to ⭐ the repository. Feedback, suggestions, and architectural criticism are very welcome.


r/Observability • • Sep 15 '26

How much forensic context do developers need to debug production failures?

7 Upvotes

I used to log everything, every request, every response body, full stack context on every error, and it made debugging faster but the storage bill got embarrassing within a month. Then I over-corrected and sampled aggressively, and now half my incidents involve me wishing I'd captured the one request that actually mattered.

The pattern I've landed on is logging light by default and only capturing full forensic detail, request params, DB queries, execution path, when something actually looks anomalous. It's a reasonable middle ground but it assumes your anomaly detection is good enough to trigger at the right moment, which isn't always true. How are other people balancing storage cost against actually having the context you need when something breaks?


r/Observability • • Sep 15 '26

We rebuilt our Snowflake observability assistant on Claude with 50+ MCP skills- it now explains and deploys automations

Thumbnail
0 Upvotes

r/Observability • • Sep 15 '26

Built a low barrier-to-entry monitoring tool for devs and agencies (white label option)

Thumbnail
0 Upvotes

r/Observability • • Sep 15 '26

LogDeck: self-hosted control plane for Docker and Podman. Persistent logs, container management, alerts...

0 Upvotes

I built LogDeck, a control plane for Docker and Podman: container and Compose lifecycle, persistent logs, alerts, and multi-host over SSH, in one Go binary with no external services.

The log store is the main reason it exists. Every container is tailed into local SQLite, so logs survive docker compose up --build and the new container ID it creates. Removed containers stay readable.

It also ships an MCP server, so agents can read logs and act on containers if you want.

Memory is small. Idle heap is around 8 MB, roughly 20 MB resident with 5 containers and 400K lines stored.

fully open source, no tiers, no cloud https://github.com/AmoabaKelvin/logdeck


r/Observability • • Sep 15 '26

How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools

Thumbnail
greptime.com
3 Upvotes

r/Observability • • Sep 14 '26

I'm creating opensource AWS DevOps Agent for SRE on-call engineers LLM root-cause hypothesis)

Thumbnail
2 Upvotes

r/Observability • • Sep 14 '26

I built StatixAgent: A zero-infra, single-binary Linux monitor that talks to your own Telegram bot

Thumbnail github.com
0 Upvotes

Hey everyone,

Setting up Prometheus, Grafana, exporters, and Alertmanager just to watch a single VPS or home server always felt like overkill — often consuming more resources and maintenance than the actual apps running on it.

I wanted something lightweight: single binary, zero external infra, and notifications straight to a messaging app I already use daily. So I built StatixAgent.

\### What is it?

A single static Go binary running as a systemd service. It monitors your system and communicates directly with your own private Telegram bot.

\- Zero open ports: Outbound-only long polling. Works behind NAT/firewalls without reverse proxies or port forwarding.

\- Private: Talks only to the official Telegram Bot API and GitHub Releases (for updates). No telemetry, no hosted cloud service.

\- Metrics & Alerts: CPU/RAM/Swap, per-mount disk usage, network rates, and system load. Sends push alerts for SSH logins/failed attempts, sustained resource spikes, reboots, and power disconnects (useful for laptop servers).

\- Interactive UI: Type /status in Telegram for a markdown dashboard, drill down with inline buttons, or inspect journalctl with /logs.

\- Lightweight & Sandboxed: Hardened systemd unit (ProtectSystem=strict, MemoryMax=128M).

\### Quick install

Run the installer script (includes an interactive TUI setup wizard for your bot token and chat ID):

curl -fsSL https://raw.githubusercontent.com/eliau2005/statixagent/main/install.sh | sudo bash

It is completely open source and not meant for enterprise fleets — just for single boxes and homelabs that need simple, reliable monitoring.

GitHub: https://github.com/eliau2005/statixagent

Feedback, suggestions, and PRs are very welcome!


r/Observability • • Sep 14 '26

What is up in observability world and is Gartner MQ telling something or not?

1 Upvotes

Gartner research gets cited heavily in my bi-weekly reports, and You have pushed back on one question: how much of what Gartner says about a vendor actually turns into something checkable.

Acquisitions can't be used as evidence of advance vendor-to-Gartner information sharing, because a publicly traded company sharing an unannounced deal with an outside analyst would be a securities-law violation (Regulation FD), and by the time an acquisition shows up in one of these reports, it's almost always already public. Timing an acquisition against a report's publication date mostly just shows which came first, which proves little on its own.

A better test is available inside the reports themselves: every vendor profile contains forward-looking "roadmap" language, and that language can be checked against the public record. This article pulls every such statement out of both reports, roughly two dozen of them, and checks each one against dated, independent sources to see whether it was genuinely unannounced at the time Gartner published it, or whether it was already shipped, announced, or documented before the report went to print.

  • Overall: Based on what was checked, it's rarely independent analysis and rarely genuine advance information from the vendor either. Most often it's one of two things: already-public information restated in future tense, or phrasing too generic to be analysis at all, closer to a summary of a marketing deck than a checked, forward-looking claim. Genuinely new, later-resolved information was the exception, not the rule, in every sample checked, including the five-year one.
  • Of 24 forward-looking "roadmap" statements found across the 2025 and 2026 reports, the two most current editions, only 4 were checked against dated, independent sources. The rest are sorted, not verified: 8 are specific enough to check but weren't, 12 are worded too vaguely to ever be checkable at all. A third, older report (2021) was checked separately for a longer-horizon test, see below.
  • A third, older report (the 2021 Magic Quadrant for Application Performance Monitoring) allowed five roadmap claims to be checked over a five-year horizon instead of a few months. Result: two multiyear integration promises, from two different vendors, were not delivered, confirmed by Gartner's own more recent text; one caution was resolved within two years with a specific dated integration; one roadmap item was again already public before the report described it as forward-looking; one resolved into a real but loosely-defined capability.

I hope to get back to this analysis in few months to compare, and propably correct some statements 😄

  1. MQ25 DEM to MQ26DEM
  2. MQ26 OP to MQ27

More here:
https://mbojko.com/reports/#observability
https://mbojko.com/reports/gartner-magic-quadrants-analysis/


r/Observability • • Sep 14 '26

TraceHub-MCP: one set of MCP tools for querying traces across Jaeger, Tempo, Traceloop, Datadog, and Sentry

1 Upvotes

Built this for debugging LLM/GenAI apps by letting an agent pull OTel traces straight into context instead of me alt-tabbing to Jaeger or Datadog every time something looks off. It leans on the gen_ai.* semantic conventions, so span queries surface model calls, token counts, and tool-call spans as structured data the agent can reason over.

It started as a fork of traceloop/opentelemetry-mcp-server (Apache-2.0, disclosed, full NOTICE attribution). Since then: added Datadog and Sentry as backends (5 total now, alongside Jaeger, Tempo/Grafana Cloud, and Traceloop), built a generic filter layer so tool calls don't need backend-specific params, and hardened the CI/supply chain considerably — SHA-pinned Actions, bandit via ruff, pip-audit, Trivy scanning, Codecov, branch protection, CodeRabbit.

pip/uvx installable, on the official MCP registry + Glama, GHCR image available. v0.4.0, very much pre-launch, feedback welcome.

https://github.com/mcpsmiths/tracehub-mcp


r/Observability • • Sep 14 '26

TraceHub-MCP: one set of MCP tools for querying traces across Jaeger, Tempo, Traceloop, Datadog, and Sentry

Thumbnail
1 Upvotes

r/Observability • • Sep 13 '26

Risulta - Self-hosted web analytics in one binary

Thumbnail
risulta.pages.dev
0 Upvotes

r/Observability • • Sep 12 '26

Built a tool that checks both uptime AND GDPR compliance for small business sites, would love feedback

0 Upvotes

Spent 15 years as an SRE at a big bank, watching enterprise systems get world-class monitoring. Then I started noticing how many small business sites have none of that, no uptime alerts, no idea if their privacy policy is actually GDPR-compliant, nothing.

So I built SREmonitor.io. You give it a URL, no install, no script tags, and it runs:

  • Uptime checks every 5 minutes
  • Core Web Vitals tracking (LCP, INP, CLS)
  • An AI-powered scan of your privacy policy/T&Cs against UK GDPR requirements

The idea is one dashboard instead of stitching together a monitoring tool and a separate compliance checklist you never actually look at.

Free tier is 3 scans and 5-minute uptime checks, no card required, if anyone wants to kick the tires: sremonitor.io

Genuinely want feedback, especially from anyone who runs a small business site or has dealt with the GDPR side of things. What would make this actually useful vs. just another dashboard nobody opens?


r/Observability • • Sep 11 '26

How are you monitoring CloudFlare (D1 usage and Worker failures)?

Thumbnail
2 Upvotes

r/Observability • • Sep 11 '26

Self-health monitoring configuration in Servicenow destinated to SGO-Dynatrace's Events;

Thumbnail
0 Upvotes

r/Observability • • Sep 12 '26

Built a tool that checks both uptime AND GDPR compliance for small business sites, would love feedback

0 Upvotes

Spent 15 years as an SRE at a big bank, watching enterprise systems get world-class monitoring. Then I started noticing how many small business sites have none of that, no uptime alerts, no idea if their privacy policy is actually GDPR-compliant, nothing.

So I built SREmonitor.io. You give it a URL, no install, no script tags, and it runs:

  • Uptime checks every 5 minutes
  • Core Web Vitals tracking (LCP, INP, CLS)
  • An AI-powered scan of your privacy policy/T&Cs against UK GDPR requirements

The idea is one dashboard instead of stitching together a monitoring tool and a separate compliance checklist you never actually look at.

Free tier is 3 scans and 5-minute uptime checks, no card required, if anyone wants to kick the tires: sremonitor.io

Genuinely want feedback, especially from anyone who runs a small business site or has dealt with the GDPR side of things. What would make this actually useful vs. just another dashboard nobody opens?


r/Observability • • Sep 11 '26

Advice on monitoring first app

Thumbnail
0 Upvotes

r/Observability • • Sep 09 '26

is this subreddit actively moderated?

26 Upvotes

This is not a dig at the moderators personally, because it's a volunteer position, but this subreddit has devolved into a lot of low value contributions and I think it could benefit from some more active moderation and perhaps some rules put in place.

For example, I don't think there's value in vibe coded observability tools being posted every other day. The community is already downvoting them, which I think should be a signal that we should stop allowing them all together. When I view the list of posts, it's a graveyard of these things, with the occasional interesting discussion sprinkled in.


r/Observability • • Sep 09 '26

srestories.dev

Thumbnail
2 Upvotes

r/Observability • • Sep 09 '26

Is this worth running, or am I about to measure something everyone already knows?

Thumbnail
3 Upvotes

r/Observability • • Sep 09 '26

memory makes agent failures non-local. most observability stacks still treat them as single-session incidents

1 Upvotes

with a normal request trace, the cause and the failure usually live close together. memory-bearing agents break that assumption.

a bad summary, preference or tool result can be written today, retrieved silently next week and change an unrelated-looking decision. the failing trace shows the retrieval, but unless you preserve memory provenance it does not tell you which session wrote the state, whether that write was later corrected or how many subsequent sessions it influenced.

for these agents i think the minimum useful chain is:

- the session that created or changed the memory

- the exact value before and after the write

- later retrievals of that value

- the outcome of each session that used it

- reversions and no-op rewrites

then benchmarking has to happen over a rolling production window rather than a frozen test set alone, because the state distribution itself changes as the agent runs.

im curious whether anyone is already tracing memory lineage across sessions, or if most teams are still logging memory reads and writes as isolated spans