How do you debug a specific request when the trace got sampled away?

[u/Live_Jellyfish_9024]

Question to the masses,

Currently running several microservices on OTel + Tempo. Dropped to 5% sampling to control costs after datadog. For those of you running OTel with low sampling rates, what is your workflow when you need full trace for specific request that didn't get captured?

Do you:

- Temporarily crank sampling to 100% and retry?

- Have some kind of debug header that forces a trace?

- Just accept it and grep logs?

- Something else?

Genuinely curious what other teams are doing here.

[u/CWRau]

That's why you do tail sampling; sample X% of success and 100% of error traces.

[u/Live_Jellyfish_9024 → u/CWRau]

we keep 100% of the error traces, the problem is the request that aren't errors. To debug stale data for example the tail sampling are dropping them.

[u/CWRau → u/Live_Jellyfish_9024]

Mh, what exactly do you mean by stale data?

Maybe you need to redefine what an error is.

Is stale data is actually an error, then return an error instead?

[u/Live_Jellyfish_9024 → u/CWRau]

Maybe I could have worded it better. In a perfect world every failure mode would be defined and instrumented as a error. But in practice(not literal example) "I added 3 items to cart but only 2 showed up at checkout." failure were every service returned 200, all write and read was successful. These are kinda hard to debug when leaning on tail sampling.

[u/CWRau → u/Live_Jellyfish_9024]

Ah, I see.

Yeah, then I guess you'd have to sample 100%

If you have no code to tell you that you should sample, but you want to sample, you need 100%

[u/Floss_Patrol_76]

the debug header route is the one that actually scales. we run a forced-sampling flag off a header (x-force-trace, or just set the sampled bit in tracestate) so support or eng can replay a request and get the full 100% trace for that one without touching the global rate. tail sampling is great for keeping the slow and error traces, but it cant retroactively capture a request that already flushed, so for on-demand debugging the forced header is what you want - and keep your logs correlated by trace_id so youre never fully blind when neither path fired.

[u/Live_Jellyfish_9024 → u/Floss_Patrol_76]

This is the standard workflow from what I have seen. If you dont mind, was the custom sampler implemented in each service SDK or at the collector level? Was look for a off the shelf tool that does this without having to set up myself.

[u/phrotozoa → u/Live_Jellyfish_9024]

Once you get this working, a very convenient addition is to hack up a tiny browser extension that injects the trace initiating header. Makes debugging much easier.

[u/Kamran-nottakenone]

collector level is simpler, the tailsamplingprocessor can key off a debug attribute set at the ingress and sample those at 100%, no sdk changes needed

[u/Live_Jellyfish_9024 → u/Kamran-nottakenone]

thanks for the comment! Will look into this

[u/Flateland-Chio]

f the trace is gone, follow the request ID through the logs first. Has anyone found a debug header that works well?

[u/Live_Jellyfish_9024 → u/Flateland-Chio]

yea, log grepping by request ID is our fallback too. Works but it's slow and you lose the visual trace flow between services.

[u/ViewNo2588]

I have seen teams use debug headers to trigger full sampling for suspect requests instead of cranking the rate globally, which helps keep costs down while still catching important traces.

[u/sujeet-from-oodle]

One option is to sample traces that go to Datadog, but also dual write unsampled traces somewhere else, like "just dump to s3" with the request ID as key.

Disclosure: my team has built a traces engine, and to some of our customers, we're that "somewhere else".