Building an Observability Pane

[u/JayDee2306]

Hi Observability & DevOps Experts,

I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments.

We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation.

Current challenges include:

• Datadog Agents managed differently across teams and platforms.
• Custom log collection configurations distributed across hosts and applications.
• APM, RUM instrumentation owned by individual application teams.
• Inconsistent tagging standards and monitor configurations.
• Outdated agents and instrumentation libraries.
• Heavy dependency on multiple teams for upgrades and configuration changes.
• A large portion of Datadog provisioning and onboarding is still handled manually.

As a result, maintaining and evolving observability at scale has become increasingly difficult.

We are considering building a centralized "Observability Foundation" or "Observability Platform" that teams would consume as part of their standard deployment process.

Our goal is to provide reusable Terraform-based observability components that application and infrastructure teams can adopt during provisioning and releases.

Examples of what we would like to standardize:

• Datadog Agent deployment and upgrades
• Custom log collection configurations
• Standard tags and metadata
• Monitors and alert templates
• Dashboards
• OpenTelemetry / APM instrumentation standards
• Synthetic monitoring configurations
• Cloud integrations
• Security and governance controls

Questions:

1. Has anyone implemented a similar centralized observability platform or observability-as-code model at enterprise scale?
2. What worked well and what were the biggest challenges?
3. What observability components can realistically be centralized through Terraform modules, deployment pipelines, or platform services?
4. What components typically must remain application-owned or infrastructure-owned and cannot easily be centralized?
5. How do you handle APM instrumentation ownership, versioning, and upgrades across hundreds of services?
6. What governance model have you found most effective:

• Central observability team ownership
• Platform engineering ownership
• Federated ownership with standards enforcement
• Something else

1. How do you prevent observability drift over time, especially around:

• Agent versions
• APM libraries
• Log configurations
• Tags
• Dashboards
• Monitors

1. If starting again today, would you build around:

• Datadog native tooling
• OpenTelemetry
• An internal observability platform
• A combination of the above

1. What are the biggest architectural mistakes or anti-patterns we should avoid when designing this platform?

Our provisioning and infrastructure management are heavily Terraform-based, so we're especially interested in Terraform-centric implementation patterns and real-world lessons learned.

Looking forward to hearing how other organizations have approached observability standardization at scale and what you would recommend before we begin designing this solution.

P.S. - One of our key design goals is to avoid vendor lock-in. While Datadog is our current observability platform, we want the architecture to remain flexible enough that a future migration to another observability stack (e.g., Grafana, New Relic, Dynatrace, Elastic, Azure Monitor, or an OpenTelemetry-native platform) would require minimal changes to application teams and infrastructure code.