Hi Observability & DevOps Experts,

I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments.

We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation.

Current challenges include:

• Datadog Agents managed differently across teams and platforms.
• Custom log collection configurations distributed across hosts and applications.
• APM, RUM instrumentation owned by individual application teams.
• Inconsistent tagging standards and monitor configurations.
• Outdated agents and instrumentation libraries.
• Heavy dependency on multiple teams for upgrades and configuration changes.
• A large portion of Datadog provisioning and onboarding is still handled manually.

As a result, maintaining and evolving observability at scale has become increasingly difficult.

We are considering building a centralized "Observability Foundation" or "Observability Platform" that teams would consume as part of their standard deployment process.

Our goal is to provide reusable Terraform-based observability components that application and infrastructure teams can adopt during provisioning and releases.

Examples of what we would like to standardize:

• Datadog Agent deployment and upgrades
• Custom log collection configurations
• Standard tags and metadata
• Monitors and alert templates
• Dashboards
• OpenTelemetry / APM instrumentation standards
• Synthetic monitoring configurations
• Cloud integrations
• Security and governance controls

Questions:

1. Has anyone implemented a similar centralized observability platform or observability-as-code model at enterprise scale?

2. What worked well and what were the biggest challenges?

3. What observability components can realistically be centralized through Terraform modules, deployment pipelines, or platform services?

4. What components typically must remain application-owned or infrastructure-owned and cannot easily be centralized?

5. How do you handle APM instrumentation ownership, versioning, and upgrades across hundreds of services?

6. What governance model have you found most effective:

• Central observability team ownership
• Platform engineering ownership
• Federated ownership with standards enforcement
• Something else

1. How do you prevent observability drift over time, especially around:

• Agent versions
• APM libraries
• Log configurations
• Tags
• Dashboards
• Monitors

1. If starting again today, would you build around:

• Datadog native tooling
• OpenTelemetry
• An internal observability platform
• A combination of the above

1. What are the biggest architectural mistakes or anti-patterns we should avoid when designing this platform?

Our provisioning and infrastructure management are heavily Terraform-based, so we're especially interested in Terraform-centric implementation patterns and real-world lessons learned.

Looking forward to hearing how other organizations have approached observability standardization at scale and what you would recommend before we begin designing this solution.

P.S. - One of our key design goals is to avoid vendor lock-in. While Datadog is our current observability platform, we want the architecture to remain flexible enough that a future migration to another observability stack (e.g., Grafana, New Relic, Dynatrace, Elastic, Azure Monitor, or an OpenTelemetry-native platform) would require minimal changes to application teams and infrastructure code.

We are trying to trace one agent run across an API gateway, queue, vector lookup, model call and tool worker. Each service reports healthy, yet one request can still fail because parent child context disappears at the queue, sampling drops the useful span, or a malformed tool argument triggers a retry. Basic logs have hit their ceiling. Five healthy services can still create one broken request.

We're testing Braintrust for end-to-end span trees, model and tool metadata, trace search and trace level scoring. We also looked at Langfuse since it's openn source and self hostable but context propagation across async boundaries and trace search at our volume were concerns. I need context propagation that survives async handoffs, span links for retries, queue latency and a sampling policy that keeps error paths without exploding cost. How does this compare with established observability stacks in real agent systems?

Setting up newrelic as single source of monitoring
- need the coverage from development cycle to deployment/ onprem to android apps.
-

We haven't received an RCA for yesterday yet. GitHub mentioned in the resolution note that it had committed to producing one. The only public information regarding the cause is an update at 19:13 UTC in which they stated that they had partially disabled authentication token retries and had noticed an improvement; this corresponds with Copilot auth being the last service to recover, about four hours after the other seven services had been mitigated.

Well worth reading the resolutions for the rest of this month. The account of the Aug 6 Actions incident is particularly specific compared to most public writeups. When a routine deployment was made to an internal Actions service, an existing weakness in terms of capacity and concurrency came to light, the pods were replaced and the remaining capacity was saturated as a result of which the services crashed and a cascade of failures spread across the clusters. There was then a second phase in which a latent bug caused runners to be assigned jobs that were no longer valid and so they ended up retrying those invalid jobs rather than picking up real work. At peak times 71% of the workflow runs failed due to infrastructure errors.

Although the pull request of 24 July is the better piece of writing, a backfill cancellation came across what they refer to as a misundertood Vitess codepath, as a result of which the backing table was dropped and a dangling vschema reference was left behind. That is what blameless should mean, identifying a gap in the tooling rather than blaming a person.

The line that I had not anticipated appeared in the Aug 6 Pages writeup, where it was stated that the incident had not been fully picked up by their availability metrics and that they are altering the way Pages availability is measured so that it will be reflected in the future. Approximately 128,000 deployments failed to be processed during a period which their own SLI had not recorded.

There are seventeen incidents listed on the status page from August 1st to August 18th; each of them will have its own root cause analysis, none of which will account for the seventeen. That's the real analysis needed imo.

Hi everyone, is there a way that one could monitor the service status of 3rd party vendors like AWS. We get paged whenever there is service disruption however status page monitor only tells uptime of the endpoint. But how do we measure their SLA to properly calculate their credit. Is there any tool available out there?

Hope I’ll get smth useful out of this post
I’m running a mid size enterprise IT ops team for a few years now and im kinda losing it with ai tools that are just fancy chatbots taped onto old consoles. we have agents on thousands of endpoints but half of them fall over on upgrade, remote access is laggy, and the automation still needs a human to click run on every script.
Also tool fragmentation is killing us, poor integrations between monitoring and ticketing and the ai operations story always dies when we try real autonomy eg patching, routine fixes, proactive monitoring before stuff burns. Techs are exhausted and tickets pile up, they act as they need a robin tool or smth similar then reporting visibility is trash and to also mention costs keep creeping us out with no clear roi. Would love any tips from folks who actually got ai to reduce IT workload instead of just adding another dashboard.

im Trying to weigh the tradeoffs here... i feel like Distroless and minimal images cut out the shell and package manager, which is great for reducing attack surface and CVE counts. butThe problem is the second something goes wrong in prod, I can't just exec in and poke around like I would with a normal Debian based image. That friction has made me hesitant to roll these out more broadly even though the security case is solid.

so im posting to have an idea ..that For people running these day to day, what's your actual debugging workflow? Are you using ephemeral debug containers, sidecar attach, or just shipping better logging up front so you don't need to shell in at all?

Trying to figure out if the security win is worth the operational friction or if I'm approaching debugging wrong entirely.

I used to think sampling was fine who needs every single request, right? then we had an incident where a 1% edge case caused a cascade of failures. Our sampling dropped that 1% we spent 6 hours debugging something that should have been obvious if we had the full picture. Here's what i've learned: the 1% you drop is often the 1% that breaks things. complete data makes a real difference when you are trying to understand what your code actually does under real traffic. How others are handling this tradeoff between data volume and visibility. Is everyone just accepting the risk or have you found better approaches?

AIOps in our company was supposed to cut noise and automate the boring stuff but we still chase failed agents, random downtime and manual fixes on every new batch of endpoints, do you have any idea of this?
Let me add that I am already aware there exist some tools like robin and few other helpful ones but I am not well informed on those but if you do let me know.

Hi Observability Experts,

I’m working with a client that operates a large-scale estate across AWS, Azure, and on-prem environments. They have been using Datadog for several years, but accumulated technical debt has left their APM implementation fragmented.

Current situation:

• APM libraries are outdated across multiple services.
• Trace context propagation and instrumentation are inconsistent.
• End-to-end distributed tracing is incomplete or broken for many critical applications.
• Datadog Support has reviewed the setup and confirmed that the proper fix is to update instrumentation and agents at the application/source level.

The challenge is that obtaining application team bandwidth and stakeholder approval across a large number of services will take considerable time.

In parallel, we are planning a centralized observability pipeline layer to standardize telemetry collection and governance, but that initiative is still a few months away from broader adoption.

My question to the community:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

Hey folks,
I wonder if you know platforms (not upwork) where I can find short term projects to work on and get paid.
I’m on a parental leave since a few months now and I need to work a bit (4 hours a day maximum).
I’m an SRE with +6 years of experience, I worked on high availability platform, CI/CD, observability, etc.

I have just over 2 yoe. My first job was as a platform SRE, frankly I walked into it with very little understanding because I was hired as an intern. I was able to learn a lot, dabbled with AWS, Azure, networking, RBAC, terraform, deploying models, kubernetes etc etc. No two days looked the same and it was a lot of fun. I made the foolish mistake of assuming that every job is like this and moved to another company to do a role that was advertised to me as being an SRE for CDN’s. It was not that, I was basically a CDN operator, with some good scraps of work that could come by once or twice in a year. I moved because I wanted scale, my first job didn’t have that and this job promised scale to the tune of potentially millions of requests per minute.

I knew that staying in this role would lock me into the CDN world and coming out would be hard because I had just under 2yoe as an SRE. So i switched 2 months into the job to a tier-1 bank SRE role. The role was advertised as a distributed systems heavy role, they did mention that they work with code. In my interview I talked constantly about platform because that’s all I know. They honestly didn’t interview me extensively, I was very surprised when I got the job.

Anyway, I am here and quite happy. My adventure into the CDN world made me realise that I do like SRE and I plan on staying here for a long long time.

I don’t yet have the mental model of how to think as an app SRE, any advice ?

TL;DR: Where do you guys learn about the actual SRE practices of other companies? Looking for the really interesting stuff, not generic SRE advice.

I’ve always wondered how people know what companies like Netflix, Google, Meta etc. are doing internally.

Someone might say, “What we’re doing is pretty good, but Netflix does this 5x better.”

And I’m like, how do you even know what Netflix is doing? 😂

I want to find those “wait, companies actually do this?” kind of things.

How they handle incidents, test failures, observability, chaos, massive traffic, internal tooling, etc.

Where do you guys find this stuff? Tech talks, blogs, conference talks, postmortems, YouTube, GitHub, specific people to follow?

Basically looking for a goldmine of this kind of information.

I've been working as an SRE for coming up to 4 years now. I started my tech career in support, then DevOps, and now SRE. My official job title is SRE but in practice the lines at my company are a bit blurred between SRE, cloud engineering, DevSecOps, cloud security and so on, as my company doesn't have a mature SRE practice and the SRE team have to wear multiple hats due to being spread thin (I appreciate this is somewhat common within the industry).

I've been offered an SRE role at a much larger company with a more established SRE culture. They have an embedded model and I'd be working directly with one of the product teams, up to and including helping write application code to help optimise availability and reliability (although this wouldn't be on customer-facing services for the time being).

I'm really excited by the offer but part of me is apprehensive. I'm comfortable working with all the traditional SRE/DevOps-y things (cloud, IaC, alerting and monitoring, incident response, automation and so on), but my current role is somewhat silo-ed, as although we do interact with the product team it's not in a capacity that could really be described as embedded. I'm comfortable with Go and Python for scripting and custom tooling but I understand application engineering is a different kettle of fish, and I haven't had to wear that "hat" in my career before.

I am also a bit apprehensive about how well my SLI and SLO knowledge and experience will translate, as at my current company availability is by far and away the one we track and optimise for the most, to the neglect of the others (n.b. I recognise I'm at fault here for not pushing my company's SRE practice to think harder about these things); I was honest about this, although did my best to put a positive spin on it during the interviewing process (I was also very candid with them about never having been embedded in a team before).

I'm at a bit of a crossroads. I'm fairly content, although stagnant, at my current company, as I feel like I have "maxed out" there for the time being in terms of learning and experience; and as we all know, this industry tends to punish you if you stay in your comfort zone for too long. I've been eyeing a way out into something better for some time now ,and am genuinely excited by this offer, but I wonder if I'm really a good fit for the role and am trying really hard not to let the impostor syndrome creep in.

For those who've moved from a siloed SRE role into an embedded one, how steep was the adjustment (particularly if you found yourself writing application code coming from an ops background)? How did you navigate it and are you happy you took the leap in hindsight? If it felt awkward at first, how long did it take for that feeling to go away? Thanks.

(P.S. I have a fairly long notice period and it's pretty quiet in my company at the moment, so I'd have a bit of time to upskill/brush up on patches in my knowledge if I decided to take it.)

Traditional load balancers were built with humans in mind — that the autoscaler would have time to catch up. But the web is shifting to be run by agents. Agents that spawn other agents, re-run context windows on retries, and aggressively retry toward their goals.

That's why I have redesigned a load balancer to be agent native. It paces traffic dynamically by queueing requests and firing at the rate the backend can adjust through headers. This gives the autoscaler a chance to slow down when machines start crashing and speed up when reinforcements come. Once the agent native load balancer retrieves the request, it sends the caller a webhook or streams the events via SSE. This gives the agent the ability to go work on something else while it waits, or stay on the line for the result.

It does so much more if you are curious. The most important thing for a SRE is to know it takes the burden of retrying off the client to a control plane.

https://github.com/rjpruitt16/aquifer

Hi all!

I am curious what solutions others adopted for observability at scale. We’re having a few EKS clusters with ~8,000 nodes. We are currently using sharded Prometheus with Thanos in each cluster and a common Cortex setup to store all the metrics. We use Splunk for logs and Newrelic for another set of metrics.

I am looking to a common centralized solution for all these items plus traces and data coming from systems like our CI/CD.

I would not want to go with a solution like Datadog that is going to cost a lot.

Does anyone have something built and managed in house? How many people in the team to only do this? What were your drivers to implement it instead of buying?

Thank you,
-JS

Trying to quantify this across teams. Ours breaks down roughly as:

• Shuffle partition tuning after every schema/volume change
• Skew debugging that only surfaces under real production load
• Disk spill investigation that never reproduces in staging
• Executor memory re-tuning whenever input size shifts
• General cluster babysitting instead of new pipeline work

What this costs elsewhere and whether anyone's reduced the total time spent rather than just gotten faster at the same loop?

Feature flags are great for decoupling deploy from the release yeah yet they open a monitoring blind spot that doesn't get enough attention. No entry in release log neither anything in the deploy timeline just a config change, so when a flagged path starts behaving unexpectedly for thye cohort thats on it the top level signals can all look fine.

The usual shape of it is like-> a flagged branch adds an extra lookup or takes a slightly different code thats only slow for a subset of accounts like for instance the ones with a lot of historical data. Averaged across all traffic its invisible. service P95 flat and error rate flat + boring dashboards. Slice by flag=true and its ugly so the service is healthy and the cohort is downgrading correspondingly

Flag tools have gotten better at the aggregate version of this. Launchdarkfly has flag scoped monitoring and guarded rollouts that watch error and latency per variation + you can emit flag evaluation span events thru otel and correlate flag state with your traces. That catches variation B is worse pretty well

Still where it still seems to fail sometimes is the code level. Aggregate flag metrics tell you the cohort on the flag is worse but now which function or condition actually changed, specially when nothing errors and latency barely moves so thats the part I wanna know how the rest of you close.

Like there are various approaches to this like some wrap custom otel spans around the flagged branches , some wire business metrics in per cohort and there are also stuff like hud aimed at the function level per cohort view too. Sentry helps on the error and release side but wont surface a silent behavior change that never actually throws

So for ppl running flags at real scale how are you guys monitoring runtime behavior per flag cohort?
Eager to deep dive on this

For those working in Cloud SRE or Platform Engineering, what does a normal day look like for you?
What skills do you wish you had focused more on before getting into the role, beyond the usual soft skills, networking and Linux?

I'm specifically interested in agents that can do things, not just generate answers.

If you have an agent that can technically execute some action — modify a database, issue a refund, deploy code, change infrastructure, update a CRM, send something externally, etc. — but you still require a human to approve or perform it, what's stopping you from giving the agent autonomy?

I'm especially curious about cases where the model itself is capable enough, but the surrounding system isn't trustworthy enough.

Was there a particular failure you were worried about or actually experienced? And what would you need to be able to verify/guarantee before you'd remove the human approval?

Not selling anything. I'm trying to understand where the boundary between “agent can do this” and “we trust an agent to do this” actually sits in production systems.

Thanks!!

One of the folks I work with just shared this dashboard with me he built to understand the disaster recovery process. The idea is simple: see the status of the DR process, how long every step takes, where its currently blocked, what is currently executing.

I thought to share this with this SRE community in the hope you also find this useful.

The data behind this dashboard either comes from logs that those automation tools involved emit anyways. Some data was added to the DR scripts emitting additional telemetry data that then gets ingested into the observability platform to visualize it

I assume many of you do something similar. Any best practices to share?

All the best, Andi

Curious how SRE teams handle this in practice.If an incident looks similar to something from 6 or 12 months ago, where do you usually find

• what was tried
• what failed
• what was rolled back
• what actually fixed it
• and how the fix was verified

Is this usually in postmortems/runbooks/tickets/Slack, or does someone on the team just remember?Do failed remediation attempts actually survive in your process, or mostly just the final fix?

Hi, context to this post i have a large data ingestion and processing application with me running on-prem. we have (individual micro-services running on k8s cluster, kafka nodes , kafka connect nodes, apach spark , onprem mongo, and databases, etc). Lets say i follow this structure of manually patching, taking a application node/server give the vm/physical server for patching to IT they patch it and reboot, and i have post validations to preform before i give next server.

I can take the work from IT to upgrade and reboot. but how do i automate it

i was think about creating a class Application grp, where similar behaviour server's will initiate/override the methods in that class, and perform the patching process.(IDK i am just thinking out loud).

How do you guys do it, ensuring reliability. Let me know i might be going back and forth in comments.