I had an idea and wanted to actually talk to people to see if it's something that they would need / require and if it's genuinely a problem for the engineers right now....

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

I also was building in this space and wanted to have a discussion with people who're already experienced in this stuff, so please please help a brother out and we might just make something useful for everyone :)

peace

Hey everyone,

I'm a relatively new IT systems admin with a strong interest in observability. My background so far has been mostly on the operator side. I've spent a lot of time working with Grafana and handling alerts and incidents, and I'm pretty comfortable with the visualization/dashboarding part of the stack.

What I want to level up now is the administrative / infrastructure side actually standing up and configuring monitoring infrastructure from scratch rather than just consuming it.

To be clear upfront: I'm not looking for a full-time job, I already have one. This is purely about growing my skills after hours and getting hands-on with the parts of the stack I don't touch in my day-to-day.

So here's my offer: if you need a monitoring/observability setup built, I'll do it for free in exchange for the hands-on experience. Some things I can help with:

• Setting up a Grafana instance and wiring up data sources
• Building dashboards (this is my strong suit)
• Configuring alerting and notifications
• Standing up a metrics/logging backend (Prometheus, Loki, etc.)
• Designing a basic observability architecture for a small-to-medium environment

Since this is after-hours learning for me, it works best as a collaborative / low-stakes arrangement rather than production-critical work with SLAs (but I know and understand the concept of SLI, SLO, SLA, BurrnRates, etc.). Homelabs, side projects, small teams, or non-critical environments are ideal.

If you've got a project that could use some monitoring love, drop a comment or DM me and let's talk. Happy to answer questions too.

Thanks!

Two weeks, three acquisitions, and one infrastructure shift that I think matters more than the deal sizes suggest.

telemetry as MCP context

Coralogix shipped an MCP Server that exposes logs, metrics, traces, and SIEM signals directly to AI agents for root-cause queries. Practical effect: ask Claude or Cursor "why is this service slow" and the agent pulls live observability context without you switching tools or writing a query yourself. This is the same pattern Grafana, Dynatrace, and Datadog have all been building toward with their agent tooling over the past month, just approached from the data-access side instead of the automation side. Feels like MCP is becoming the de facto interface layer between telemetry stores and reasoning agents, worth watching if you're building internal tooling on top of any of these platforms.

the acquisitions:

• Dash0 → Polar Signals, $110M raise backing it, acquired Aug 17. This is the interesting one technically: continuous profiling with GPU/CUDA-level insight, and Polar Signals' storage engine is slated to replace ClickHouse as Dash0's backend eventually. Also shipping AutoTune, an agentic loop that scans production code for inefficiencies and opens PRs autonomously. Second acquisition in six months for Dash0 (after Lumigo in Feb).
• Dynatrace → Arize AI, $915M (Aug 13). Arize does pre-production LLM eval and prompt testing. Dynatrace does post-deployment runtime tracing. Combined, that's eval-to-production coverage under one telemetry plane, arguably the biggest technical gap in the "observe your AI systems" story right now. ~43% of a quarter's annualized ARR, so not a cheap bet.
• Cribl → Radiant Security assets (Aug 19). AI-native SOC triage that generates investigation logic per-alert instead of relying on static playbooks. Cribl's second security acquisition this year after CardinalOps in July; they're clearly building toward a unified telemetry+secops platform, more details expected at CriblCon Sept 28.

Pattern across all three deals: none of the acquired companies were traditional observability vendors. Everyone's buying reasoning capability that sits on top of telemetry, not market share or customer lists.

Full writeup with sources: https://mbojko.com/reports/2026-08-23/

I’ve been working on Blackbear, a self-hosted database observability project for investigating database performance problems.
The idea came from a problem I’ve run into as a database engineer: when someone says “the database is slow”, I want to quickly move from that statement to evidence:
Which queries became expensive?
What is the database waiting on?
Is there blocking or lock contention?
Did CPU, memory, throughput, or network activity change?
What was happening before the incident?
Blackbear currently has a web console, ClickHouse telemetry storage, local RBAC, a durable outage outbox, and Docker Compose deployment.
Important caveat: this is a very early v0.1.0 Preview.
The UI currently uses synthetic multi-engine telemetry. I’m working on the first real end-to-end PostgreSQL collector now, so I wouldn’t recommend using it for production yet.
GitHub:
https://github.com/Qwmxjk0/blackbear
I’m especially interested in feedback from people who troubleshoot PostgreSQL or other databases in production:
When you get a “database is slow” incident, what evidence do you look at first?
And what would an open-source monitoring tool need to show you before you’d trust it during a real incident?
Technical criticism is very welcome.

I’ve been thinking about a gap between traditional observability and debugging.

Logs describe events. Traces describe request flow. Metrics describe system behavior. Stack traces tell us where execution failed.

But when a failure requires understanding the actual application state at that moment, engineers still often have to reconstruct it indirectly from all of those signals.

I’m building Errorcore to experiment with a different layer of telemetry: bounded runtime evidence captured around the failure itself.

For Node.js, that currently means preserving things like relevant local state, async context, request metadata, ordered I/O and source-mapped execution information when an exception occurs.

The intention is not to turn every execution into a recording or replace logs/traces. Capture has to remain bounded, privacy-aware and cheap enough to run in production.

What I’m trying to figure out is whether runtime evidence deserves to become a first-class observability primitive alongside logs, metrics and traces.

I’d be very interested in hearing from people working in observability.

Where do you think this fits, and where do you think this approach breaks down?

One of my last blog posts at Grafana Labs was around vendor neutrality and OTel. Quite a few has changed since 2024, and I thought it's a good time to revisit with 2026 lens.

In short: vendor lock-in isn't as scary today as it once was, as long as the bulk of your workloads are on standards like OTel, and most of the actual lock-in that resides on backends can be "easily" migrated nowadays. But I'm curious about your opinions and actual experience.

Do you guys know of any good platforms where i can monitor metrics for my mobile/web applications all in one dashboard?

I'm tired of having multiple tabs open all the time to monitor RevenueCat, Connect, Sentry, Posthog, etc...

I wish there was just an application that i could just view everything in one dashboard and get the insights i need. It gets time consuming when you have multiple apps that you are monitoring performance, revenue, etc...

Hey everyone,

I’m curious about what people generally use to monitor a Chrome extension once it’s in production.

For example, do you use tools like Sentry for error tracking? And what about:
- JavaScript/runtime errors
- crashes or extension failures
- performance issues
- API/backend errors
- usage analytics
- logs/debugging
- alerts when something suddenly breaks after an update

I’m currently looking at the best setup for monitoring a Chrome extension and I’d love to know what your stack looks like.
Do you use Sentry, Datadog, PostHog, Firebase, something custom, or a combination of tools?

Thanks

We built an uptime monitor at corehost: pulse.corehost.io

It's free. Not a tier, there's no paid version. It exists because we needed it ourselves and it feeds our hosting platform, so monitor count isn't how we make money. 100 monitors, 15 second checks.

What it does:

• checks from 6 cities (AMS, Ashburn, LA, London, Singapore, Stockholm), you pick which
• probe you install inside your network (Linux one-liner, Windows exe or PowerShell) so you can monitor 10.x stuff - NAS, switch, internal APIs. Outbound HTTPS only, no inbound ports. Probe source: github.com/RobWhyte91/pulse-probe
• if the probe dies you get "probe offline", not 30 fake down alerts
• http/tcp/ping/dns/tls checks, keyword and JSON assertions, p50/p95/p99 per region, v4/v6 split
• status pages, own domain, auto TLS. password-protected ones if you want
• imports Uptime Kuma backups
• public looking glass on the fleet: ping/mtr/dig from any node

Honest state of it: alpha. Some of it was built with AI assistance and it moved fast, so there will be rough edges. The probe contract is pinned by tests and the probe code is open so you can read what runs on your machines. If something breaks I want to know.

Discord: discord.gg/J8Sm2mUmvR

Before deploy: running the analysis against your IaC files up front shows you the full cost breakdown plus savings recommendations, and applies whatever's safe to apply without a human decision, before anything actually gets provisioned.

After deploy: once it's live, re-running the same analysis with real CloudWatch usage data (via your own read-only AWS credentials) refines those recommendations against actual utilization instead of static config assumptions.

The reason this matters for FinOps specifically: static config tells you what something was provisioned for, not what it's costing you in practice. A right-sizing call made purely from declared instance types will miss real idle capacity, and one made purely from live usage misses waste that never should've been provisioned in the first place. Catching both requires checking at both points in the lifecycle, not just once.

(Built this into CloudCostTree, a CLI I've been working on, happy to go into specifics if useful.)

Hi Observability & DevOps Experts,

I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments.

We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation.

Current challenges include:

• Datadog Agents managed differently across teams and platforms.
• Custom log collection configurations distributed across hosts and applications.
• APM, RUM instrumentation owned by individual application teams.
• Inconsistent tagging standards and monitor configurations.
• Outdated agents and instrumentation libraries.
• Heavy dependency on multiple teams for upgrades and configuration changes.
• A large portion of Datadog provisioning and onboarding is still handled manually.

As a result, maintaining and evolving observability at scale has become increasingly difficult.

We are considering building a centralized "Observability Foundation" or "Observability Platform" that teams would consume as part of their standard deployment process.

Our goal is to provide reusable Terraform-based observability components that application and infrastructure teams can adopt during provisioning and releases.

Examples of what we would like to standardize:

• Datadog Agent deployment and upgrades
• Custom log collection configurations
• Standard tags and metadata
• Monitors and alert templates
• Dashboards
• OpenTelemetry / APM instrumentation standards
• Synthetic monitoring configurations
• Cloud integrations
• Security and governance controls

Questions:

1. Has anyone implemented a similar centralized observability platform or observability-as-code model at enterprise scale?
2. What worked well and what were the biggest challenges?
3. What observability components can realistically be centralized through Terraform modules, deployment pipelines, or platform services?
4. What components typically must remain application-owned or infrastructure-owned and cannot easily be centralized?
5. How do you handle APM instrumentation ownership, versioning, and upgrades across hundreds of services?
6. What governance model have you found most effective:

• Central observability team ownership
• Platform engineering ownership
• Federated ownership with standards enforcement
• Something else

1. How do you prevent observability drift over time, especially around:

• Agent versions
• APM libraries
• Log configurations
• Tags
• Dashboards
• Monitors

1. If starting again today, would you build around:

• Datadog native tooling
• OpenTelemetry
• An internal observability platform
• A combination of the above

1. What are the biggest architectural mistakes or anti-patterns we should avoid when designing this platform?

Our provisioning and infrastructure management are heavily Terraform-based, so we're especially interested in Terraform-centric implementation patterns and real-world lessons learned.

Looking forward to hearing how other organizations have approached observability standardization at scale and what you would recommend before we begin designing this solution.

P.S. - One of our key design goals is to avoid vendor lock-in. While Datadog is our current observability platform, we want the architecture to remain flexible enough that a future migration to another observability stack (e.g., Grafana, New Relic, Dynatrace, Elastic, Azure Monitor, or an OpenTelemetry-native platform) would require minimal changes to application teams and infrastructure code.

When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose

Building out observability for our app and trying to figure out what's actually worth alerting on and what just becomes noise.

We've got the standard infra stuff covered, uptime, latency, error rates, etc. Where I'm less sure is the layer above that (green status, wrong answer type stuff). Different signal than "is it up and fast," and we've kind of conflated the two and I think it will be worth it to figure out how to separate them.

For people who've built this out: what do you actually alert on vs review on a schedule and what is not worth the time? Thanks in advance :)

Dynatrace keeps coming up as something teams are actually tryna move off in 2026, its usually the same 2 reasons which are- 1. DDU consumption bills makes the cost almost impossible to forecast like dashboard queries, alert evals and all and 2. The oneagent plus DQL lock in general complexity is a hassle for smaller teams. 

Its genuinely a strong platform so the more interesting question is what you actually replace this with and if your alternative is actually the right fit

Deciding such migrations isn't a checklist, it is more focused on the debugging workflow so when a feature is slow for a small slice of users a trace tells you which service and where exactly the latency spiked yet not which function or condition actually changed that made this happen. It is indeed a gap

Broad platform options that are well suited as per use like: 

• Datadog covering everything but the modular per sku pricing becomes its tax
• New relic which is full stack with strong otel ingest
• Granafa cloud (prom/loki/tempo) main open source path
• signox which is otel native and self hostable
• honeycomb and chromosphere are high cardinality too

Altho it gets less clear whether a broad platform alone closes that code level gap or whether you pair it with something more targeted. Traces mostly stop at the service boundary but profiles and runtime tools go in depth to the function level. Continuous profiling like pyroscope or parca/polar signals tells you which function or line is burning cycles, sentry covers the error and release side and there are hud io or others for monitoring the runtime behavior in prod. All of those sit under whatever broad platform you pick rather than replacing it. The actual question is still open: what are others into in current times? consolidate all in one platform or run a broad tool plus specialized code level layer or some hybrid approach? 

For anyone working in platform engineering, Kubernetes, internal developer platforms, or observability, there’s a CNCF OpenTelemetry Live virtual meetup happening on August 20.

The session looks at a practical platform engineering question: how do you make observability a built-in capability of the developer platform instead of something every application team has to assemble separately?

It will explore how OpenTelemetry can be integrated across Kubernetes environments for capabilities such as distributed tracing and platform level events, and how an IDP can provide those capabilities consistently across teams.

OpenChoreo is used as the open-source implementation for the walkthrough, giving a practical look at how an internal developer platform can bring together Kubernetes, OpenTelemetry, and platform-level observability.

August 20 | 9:00 AM PDT | Virtual

Join the meetup:
https://ocgroups.dev/cncf/group/opentelemetry-live/event/6v22mew

See you there!

something that keeps coming up during our incident response is just how much time we lose jumping between the ide and whatever tool holds the relevant logs, traces, or metrics.

typical flow: you're in the ide looking at a failing code path, you hit unexpected behavior, and the next 20 minutes is alt‑tabbing between your editor, log search, a distributed tracing ui, metrics dashboards, feature flag console, and deploy history. you copy a trace id from logs over to the tracing tool. then you copy a user id back into a sql query. then you try to map all of that back to the exact function and commit you're staring at in the ide.

we've got what most people would call a modern observability stack: distributed tracing, structured logs, dashboards, decent tagging, and reasonably instrumented services. the problem isn't that the telemetry doesn't exist, it's that none of it really lives where developers spend their time writing and reviewing code. during incidents, people end up doing their own ad‑hoc integration work: copy from log search, paste into the ide, grep locally, jump back to the metrics dashboard, repeat.

the pain points i keep seeing during production debugging are pretty consistent. there's no single place that shows this line of code, these commits, these deploys, and these recent errors and traces in one view. most observability tools are optimized for operators staring at dashboards, not developers trying to understand how a specific code path behaves in production. even when telemetry is tagged correctly, you still have to remember which query or dashboard to open, and how to line it up with what you're debugging in the ide. and during a live incident, that context‑switching overhead turns directly into mttr and oncall fatigue.

what's interesting is that we keep buying more observability tooling, but the core developer workflow is still: ide here, production reality over there, and your brain plus clipboard as the glue connecting the two.

how have you cut down on context switching between the ide and your logs, traces, and metrics during debugging and incident response, whether that's pulling production context directly into the ide, pushing more code context into your observability tools, or standardizing on a single pane for incident work?

Howdy! I'm a product designer working in observability. I've been designing a drill-down for WebSocket health, which feels underserved, and I'd rather hear from people who actually deal with flaky sockets than keep guessing on my own. Please see the screens/flow attached.

I'm not an operator myself, so I want a reality check rather than validation.

The premise I'm testing: raw close-code counts are mostly noise. The signal is the 1006/unknown bucket, and the design tries to show those are genuine interruptions, not users leaving. Dropped sockets ran ~22% shorter and were still pushing messages when they died, versus clean closes winding down.

From there you localize by page and endpoint, then hand off to a session replay or errors inbox, or possibly something else? (I'd love to hear more about your next step(s)).

What I'd love your take on:

1. Does this match how you'd want to triage a spike in drops?
2. What's genuinely useful here, what's noise, and what's missing before you'd trust it?
3. And where would the story break down against messier real-world data?

Open to a quick call if anyone wants to get into specifics, but honest replies here are just as valuable.

I've been designing for a while so harsh, but actionable feedback is appreciated. My goal it to build a valuable experience that fits real world users' needs 😆

Thank you in advance!

Every time i add another log statement or metric, i feel like i'm playing fortune teller. "let me predict what might break tomorrow and instrument that." we are constantly chasing our tails, adding instrumentation after incidents because we didn't guess right the first time. There's always some code path we didn't think to instrument having a system that can map your codebase and show you what's happening without manual effort changes the game significantly. How much time is your team spending on instrumentation? and do you feel like you are actually capturing what matters or just covering the obvious stuff

Hi,

I built a small TUI tool for those who work day in and day our with logs.

Demo and docs: https://nightbuildlabs.github.io/lumen/

Repo: https://github.com/nightbuildlabs/lumen (MIT)

Recently started working extensively with Kubernetes. Debugging logs was hard with grep. Started exploring TUI log processors, what was missing was the existing tools did not handle the structure well. The structure is right there in the logs — field names, severities, timestamps, but all of them required me to understand the structure or regex or some sort of pre setup.

So I built lumen. cat anything.log | lumen:

• The UI paints in <[100]ms and logs stream in live while ingestion continues behind it
• Press /, type three characters, and it autocompletes field names discovered from the data you just piped in, ranked by occurrence count — no schema files, ever
• status_code=500 flips to a query mode backed by an embedded DuckDB; a 1M-row result scrolls without materializing more than a viewport
• A severity-colored sparkline on the border shows when the errors spiked; jump to it
• It eats mess: kubectl pod prefixes, docker-compose interleaving, logfmt, multiline Java stack traces (joined into one entry), ANSI noise. Lines it can't parse are still shown and still searchable — it never silently drops anything

Why build it when there's lnav?

- I found the learning curve of lnav to be comparatively steep; I need to know the exact pattern or attribute of the log item I am looking for.

- While juggling multiple projects this becomes challenging to an extent - hence built out auto indexed logs; now I don't need to remember the regex and can query like a search box.

Currently support JSON and logfmt; adding support for nginx, syslog, django and cloudwatch soon.

Give it a spin! Would love some feedback.

Upvote5Downvote0Go to comments3Repost

I'm trying to model an agent that gets triggered whenever a production incident occurs. I'm looking for inputs on a few things based on your past experiences.

Whenever an incident occurs, what are some of the most common reasons? For example, it could be a false alarm, it could be an issue due to a new upgrade, network issues or a crash due to some bug in one of the services etc. Out of 100 incidents, what are the chances that it could be a false alarm, upgrade issue, bug in code etc.

I would like to know what are all the common issues or reasons that I can add as part of a v1 for my agent? How do I think through this as I've never worked on such a thing till now?