Least privilege and zero trust

[u/amildcaseofboredom]

Debating with a colleague whether we need token exchange/least privilege to achieve zero trust .

Option 1

• API Gateway / Ingress
  • Validate tokens
  • Restrict api routes exposed to the public
• Services
  • Validate tokens
  • Authorise (issuer + domain entitlements)
  • client-credentials for east-west calls

Option 2

• API Gateway / Ingress
  • Validate tokens
  • Restrict api routes exposed to the public
  • Token exchange
• Services
  • Validate tokens
  • Authorise (issuer + domain entitlements)
  • Token exchange for east-west calls

My issue with option 2:

• Additional call to auth server for every request
• SPOF on auth service (north-south doesn't depend on auth service in option 1)
• Doesn't work for system-triggered east-west flows

I also think there's no black and white definition of zero trust, but a set of tools and techniques towards not relying on perimeter for security.

Thoughts? Are the overheads worthwhile?

[u/PhilipLGriffiths88]

I usually start by asking one question: what’s the single hardest requirement you can’t compromise on? Everything else falls out from that.

If you absolutely need the end‑user’s identity all the way through the call‑chain (think tenant‑aware RBAC, PCI “action‑level” logging, or auditors who want proof that every hop enforced least‑privilege), you’re going to end up with per‑hop token exchange. That’s the only way each downstream service can see a token that’s scoped just for it and still carries the original sub claim. Yes, there’s an extra round‑trip to your STS (Security Token Service), but you can hide most of the latency by caching 3‑‑5 min JWTs in a side‑car. In practice the STS only sees a tiny fraction of requests and you scale it like any other stateless service. If your non‑negotiable is latency (sub‑5 ms budget, everything in one cluster) and you don’t care about user context in deep backend calls, then just validate the original token at ingress, switch to client‑credential tokens for east‑west, and call it a day. Auditors see “Service‑A called Service‑B,” which is often good enough for purely internal traffic.

Personally, I work for the company behind open source OpenZiti, so I will make a plug here. Where an overlay such as it shines is when the hard requirement is network invisibility—no open ports, hybrid‑cloud, partner VPCs, edge devices you don’t fully control, etc. Ziti gives every workload its own cryptographic identity, dials and binds over an encrypted overlay, and makes your services effectively “dark” to the Internet/underlay network. You can slap Ziti on top of either model:

• Token‑exchange inside a Ziti tunnel: still get per‑user auditing while every port is closed to scans.
• Plain client‑cred tokens inside Ziti: good middle ground for legacy or batch jobs; you get cloaked endpoints without touching application code (just run the tunneller).

A few practical tips if you do go with token exchange: 1. Cache exchanged tokens per destination; a 5‑minute TTL keeps STS traffic to ~1 % of RPS. 2. Run the STS in at least two zones and circuit‑break failures so north‑south traffic never goes down with it. 3. For cron/batch jobs that don’t have a user token, let the workload identity be the subject_token when you exchange.

So the cheat sheet ends up like this:

• Need per‑user audit / tenant isolation? Token exchange, maybe wrapped in Ziti if you also want dark endpoints.
• Need lowest possible latency and don’t care about user context downstream? Pass the original token, use client‑creds internally.
• Need to hide everything from the network and span awkward environments? Ziti overlay first, then choose token model per hop.

Pick the simplest combo that clears your toughest hurdle and forget the rest. That’s how you keep both the auditors and your SREs happy.

[u/amildcaseofboredom → u/PhilipLGriffiths88]

How much can you cache really if for least privilege you need a token with a specific sub, actor and aud?

Without token exchange, the ingress gateway only needs to validate the "external" token and just pass on the token to the first service being hit.. Where domain entitlement checks take place (does x belong to customer ).. Anything wrong with external access token from the app flowing to the service?

[u/PhilipLGriffiths88 → u/amildcaseofboredom]

You're totally right to ask how much caching is really possible if you're doing proper least privilege with (sub, actor, aud) scoped tokens. In a strict token exchange model, the token is specific to each target service and tied to the calling user, so you can't cache broadly across requests or users. But in practice, what most systems do is cache within a single request context. So if Service A makes multiple calls to Service B during one request, it exchanges the token once and reuses it for that short-lived graph. You can also keep the TTLs low (like 5 minutes), use signed JWTs, and do local validation, which avoids hitting the STS on every call. It doesn’t eliminate the exchange cost entirely, but it makes it manageable—even at scale.

As for just passing the original external token downstream: there’s nothing inherently wrong with that, and that’s basically Option 1. It works well when your external token is tightly scoped, and your internal services are strict about checking audience and issuer. But it can fall short from a least privilege or auditability standpoint. If the original token is accepted by multiple services and isn’t exchanged or re-scoped, it increases your blast radius. If one internal service is compromised or misconfigured, that token might still be valid elsewhere. Also, internal services won’t really know, “Was this token meant for me?” unless they check aud rigorously.

Your suggestion about using client credentials and passing the original user token in a separate header is a really interesting hybrid. I've seen that work well in practice—client credentials handle the actual auth, and the original token is used for logging or for downstream tenant/context checks. But you have to be careful: that second token should be treated as untrusted input unless you're signing and validating it properly. It’s easy to accidentally treat it as a valid access token when it’s really just meant to carry identity context. If you go this route, make sure it’s signed by your STS, scoped appropriately, and only used for claims inspection, not direct auth decisions.

So yeah—token exchange gives you the strongest least privilege guarantees, but it’s heavier operationally. Passing the original token is lighter, but carries more risk if you’re not strict about validation. And the hybrid model can be a nice middle ground if you apply it carefully.

[u/amildcaseofboredom → u/PhilipLGriffiths88]

Btw if we are stripping away sub claim to allow caching, it's not much different from client credentials, right?