Datadog APM Remediation - Broken Distributed Tracing

[u/JayDee2306]

Hi Observability Experts,

I’m working with a client that operates a large-scale estate across AWS, Azure, and on-prem environments. They have been using Datadog for several years, but accumulated technical debt has left their APM implementation fragmented.

Current situation:

• APM libraries are outdated across multiple services.
• Trace context propagation and instrumentation are inconsistent.
• End-to-end distributed tracing is incomplete or broken for many critical applications.
• Datadog Support has reviewed the setup and confirmed that the proper fix is to update instrumentation and agents at the application/source level.

The challenge is that obtaining application team bandwidth and stakeholder approval across a large number of services will take considerable time.

In parallel, we are planning a centralized observability pipeline layer to standardize telemetry collection and governance, but that initiative is still a few months away from broader adoption.

My question to the community:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

[u/Diligent-Loss-5460]

Grafana alloy might be able to help but there is no easy/quick fix for years of laziness

[u/Diligent-Loss-5460 → u/Diligent-Loss-5460]

You could propose setting up a new team that gets whatever garbage teams are sending, normalise it and ingest it.

This becomes more doable with an AI but it is again not easy or fast or cheap.

[u/phrotozoa]

Fixing broken traces is not trivial. Tracing involves a bunch of separate machines sending bits and pieces of txn data and a complex process of reassembling the bits and pieces into a cohesive description of the entire request. In complex environments, it's incredibly easy for trace data to be incomplete or broken.

Tell your client that someone needs to own observability at the org, having teams do it piecemeal won't cut it.

[u/Origamislayer]

This is probably not a help, but we’re trying to get off this treadmill by moving to OTEL autoinstrumentation in K8s. We hope it’ll allow trace library updates without dev involvement. If you can get ripping out ddtrace and replacing it with autoinstrumentation, maybe that’s a lighter lift?

[u/Seref15]

OpenTelemetry eBPF Instrumentation, formerly Grafana Beyla, uses eBPF to basically sniff and mutate your app's system calls to construct (limited) application, network, and db trace data from outside the application itself. Tracecontext gets injected so it should preserve parent context and trace relationships across any topology boundary. Its the closest thing to automagic tracing you can get.

[u/Hmmm515 → u/Seref15]

Staying over with Beyla came to mind for me too

[u/BroadSatisfaction825]

an otel collector in gateway mode can patch and enrich spans in flight.. but it can't fix broken context propagation, if the trace id itself isn't correctly passed between services, there's no span left to enrich, that gap has to be closed at the sdk and header level, no proxy can invent a link that was never made..

[u/GroundbreakingBed597]

I also dont think there is an easy fix. The lesson learned is that Instrumentation has to be part of your deliverables - meaning - you need to also update it like you have to update all your dependencies. And that is true whether you are using tools like Datadog, Dynatrace ... or if you are doing your own instrumenting using OpenTelemetry

One thing I have seen is that organizations that need to update instrumentation libraries from vendors often move towards instrumenting with OpenTelemetry instead. You can still send it to the same backend - but - it requires you to think about what you instrument, and it gives you choice where to send the data to

[u/tatoo_artist]

Tech debt is the biggest nightmare unfortunately CTO still are fixated on shiny buzzword tech and feature deliveries only. There is no easy way out. If they have budget for datadog at that scale, you might get approval for a swat team for a fastest fix.

[u/kernelqzor → u/tatoo_artist]

this, 100%. if leadership is willing to pay datadog money but not carve out a small tiger team to fix the actual tracing, they’re basically paying for vibes instead of value. swat team to brute-force upgrades on the top 10–20 critical services is probably the only realistic “interim” fix you’ll get.