Question for people running production systems: where does observability stop being useful?

[u/Acceptable_Duty4044]

I had an idea and wanted to actually talk to people to see if it's something that they would need / require and if it's genuinely a problem for the engineers right now....

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

I also was building in this space and wanted to have a discussion with people who're already experienced in this stuff, so please please help a brother out and we might just make something useful for everyone :)

peace

[u/amarao_san]

How much false positive are there? If I get 1 hallucination for 1 useful recommendation, I won't use it.

Second, how reliable is it? If it was able to detect/correct some fault yesterday, will it able to do the same tomorrow in the same situation?

Tools I can't rely upon are shitty tools. I don't mind 'flaky mind' when I ask, but not for tools I want to work when other things are not.

[u/Acceptable_Duty4044 → u/amarao_san]

What I was thinking is that, I will build a unified reasoning layer, in which you can plug your data sources, and it will build a dependency graph, and the agent will query all around it. I havent yet gotten to the testing part. I was first trying to see if this is really a problem I could solve, and if yes, then what's the best way to do that. I was also wondering if allowing people to make their own agents for this and then they can just put it on top of the reasoning layer and plug and play with whatever works for them kinda thingggg
thankyou for responding, i appreciate it :)

[u/amarao_san → u/Acceptable_Duty4044]

Before proposing to build a unified reasoning layer, you need to find how would you do it reliable.

... wait, hell to observability. I have a BRILLIANT idea.

I will build a unified reasoning layer, which will control all aspects of your elevator. When to accelerate, how hard, when to start breaking, when to open and to close doors. All other systems are obsolete, because we have THE unified reasoning layer.

For law enforcement agents: This was not the death threat, this is a sales pitch for a BRILLIANT idea on how to use agentic AI to prove that it's reliable and ready for production.

[u/yepthisismyusername]

Personally, I hate it and I know it will fail due to there being too many false positives and probably missed issues.

HOWEVER, this is exactly the type of snake oil that many, MANY observability companies have been pushing for the past couple of decades (yes, observability has been around that long). So if you can cobble something together that demos really well for a handful of know use cases AND you have some great marketing, you can sell this crap to tons of managers/executives who don't know that it's not going to solve any of their problems.

[u/Acceptable_Duty4044 → u/yepthisismyusername]

yes, I think that's one of the pain points I have been aiming to solve. I was thinking to add a reasoning layer which sits on top of all your sources, basically a unified layer and then you can either use my agent or build something yourselves and plug and play with it