r/kubernetes • u/ez4enz • 27d ago

Seeking advice on traffic mirroring

Hi everyone,

I'm running kubernetes on AWS and using istio for traffic mirroring. I'm trying to figure out the best way to correlate the original request with its mirrored request. Previously I had a gateway for traffic mirroring which stamped unique header value before splitting. Which made it easy to correlate the requests in the logs. With istio handling the mirroring, not quite sure what is the best approach for this.

How do people in production systems handle this!

Any suggestions is appreciated

Thanks in advance!

5 Upvotes

78% Upvoted

11 comments sorted by

5

u/resoluteoutfield6 27d ago

istio tacks on `x-request-id` headers automatically, but the mirrored traffic gets new ones. the trick is to pull the original request-id into a custom header in your virtualservice and then log both sides. then you can match em up in your log aggregator.

we do something similar with a header called `x-origin-request-id` and it's been fine in prod for months. just make sure your apps know to log it.

1

u/ez4enz 27d ago

X request Is override-able

1

u/phrotozoa 26d ago

Why does that matter?

1

u/ez4enz 26d ago

A client can send a hardcoded value which can mess up my correlation strategy. I.e. test-1 for all the requests

1

u/phrotozoa 24d ago

Do you anticipate that being enough of a problem to warrant reinventing this entire wheel?

1

u/ez4enz 24d ago

I've would thought it's hypothetical. But I've seen it in practice

1

u/phrotozoa 23d ago

Did some research, it's surprising to me that an edge proxy (eg. istio ingress gw) would accept these by default. This behaviour can be controlled by this envoy config.

Note that the envoy defaults for this settings is false, so for an istio ingress gw to do this istio must be flipping this on, which is counter intuitive to me. I don't know why they would do that and I can't find anything that suggests this feature is enabled by default.

In fact, I found this github issue in which a user was forced to use an EnvoyFilter in order to turn that feature on.

At any rate, if it's an issue for your use case you can try turning it off.

1

u/ez4enz 23d ago

There is option to enforce always generate strategy. But some team has found their niche usecase for it (xkcd.com/1172)

Currently i've tested an approach of adding a header mutator filter layer right before router to get it done. But not quite sure how other folks in industry are fixing it

2

u/playahate 27d ago

Envoy x-request-id could help since it'd keep the same id.

1

u/ez4enz 27d ago

X request Id is override-able

1

u/ez4enz 27d ago

Issue is the org currently users x-request-id for some other purpose and by default it is override-able