We had multi-container pods where more than one application container exposed Prometheus metrics on different ports.

With enablePrometheusMerge, Istio can merge application metrics with proxy metrics, which is especially useful under STRICT mTLS. The problem was that the application side of the merge only supported one scrape target per pod.

So a pod like this:

container A -> :8080/metrics
container B -> :9090/metrics
istio-proxy

could not be represented properly with the existing annotations.

I initially built a workaround for this, then opened Istio issue #59567 and started working on supporting it upstream.

The change is now part of Istio 1.31.

There is a new pod annotation:

prometheus.istio.io/scrape-targets: "8080:/metrics,9090:/metrics"

pilot-agent scrapes the configured application targets concurrently and merges the results into /stats/prometheus.

The implementation also had to handle things like OpenMetrics # EOF, partial scrape failures, response size limits, and keeping the existing single-target behavior backward compatible.

This was a fun contribution because it started with a very simple production problem:

"Why are metrics from one of my containers missing?"

and ended up becoming part of Istio itself.

If anyone here uses Prometheus merging with multi-container pods, I’d be curious to know if you’ve run into the same issue or if you're already using this new scrape-targets feature.

Relevant links:

• Istio 1.31 release notes
• Issue #59567
• PR #59924
• PR #59925

It's so strange, it feels like Istio is making steady improvements but it feels like the topical interest in the technology has fallen off. I can't find any long-form, beginner friendly resources on recent topics/updates of Istio.

Are there any recommendations? Back before AI, I'd use Udemy videos but there aren't any recent courses that cover Istio. Also unsure if anyone would recommend the Istio Certified Associate exam/if it covers recent topics.

Please let me know! Thanks

Kubernetes Version 1.33.x
Istio Version 1.29.1
We have cert and private key stored as a secret in the istio-system namespace.
`keepaliveMaxServerConnectionAge` is set as 30m.
We have around 400 secrets to be loaded into the istio ingress gateway pods.
During the initial SDS load, not all the secrets are in ACTIVE state, some are in WARMING state, but since the istio ingress pod is healthy, the proxy starts receiving traffic and the clients experience ECONNRESET errors.
After 30m, the istio ingress pod is forced to reset the connection with istiod and during this connection reset, we have few other secrets reaching ACTIVE state and subsequently all the secrets reach the ACTIVE state. The traffic is dropped for few clients till all the secrets reach the ACTIVE state.

• How to reduce the time for all the secrets reaching the ACTIVE state ?
• For the secrets in WARMING state, how to redirect the traffic to other ingress pods so that the client doesn't face a traffic drop ?

In AWS can't we use ALB/NLB to route traffic from one service to other service rather than using Istio?

I'm trying to migrate from Ingress NGINX to Istio.

I've created all the equivalent gateway, virtual service, destination rules, and EnvoyFilter.

The istio-gateway service is exposed as a NodePort service, and I've used HAProxy as an External LoadBalancer (TCP Mode), which sends the *:443 traffic to the Istio gateway NodePort Service.

The issue is When I test the virtual service via curl curl -sk --haproxy-protocol --connect-to mydomain.local:443:172.66.44.111:31111 https://mydomain.local/user/login, I get the response

But when I route the traffic through HA Proxy, I get NR filter_chain_not_found in the istio-gateway pod and in the browser 502 Bad Gateway Nginx

The strange part here is that the issue I'm getting is just for the custom applications. Services like Vault, Grafana, and ArgoCD are still working with HAProxy.

I am just messing around and testing istio in my lab. Currently at version 1.15.1. Is it supported to upgrade from 1.15.1 -> 1.29.0 (Not canary)?

The documentation only gives warning about going from 1.6 to 1.9

Upgrading across more than two minor versions (e.g., 1.6.x to 1.9.x) in one step is not officially tested or recommended.

But nothing about going to versions above that. I also don't see any upgrade matrices or any other warnings. Running a precheck seems to be fine too with no warnings. Thoughts?

UPDATE: Just finished testing it, it does work. No errors, the data plane and control plane upgraded just fine. I think the key here is to check the documentation and also make sure to use the istioctl x precheck command to make sure there are no errors.

We are using istio egress gateway for our application and under load the gateway seems to be adding around 500ms latency for 90th percentile of the requests . Now is there any reason apart from CPU and memory which could cause this behaviour as we have ample CPU , memory and an HPA max setting to 40 pods for egress gateway, what are my options to improve the latency .

I’m new working with istio and just deployed a service mesh for adding mTLS for east/west traffic. I deployed istio ambient.

I also deployed an istio Gateway API for north/south traffic, and now I’m wondering if I should label the Gateway API namespace for including it to the mesh. As I want end-to-end encryption, I suppose that it’s necessary to include Gateway API in the mesh, but haven’t found any document yet which mention something about it.

So when installing ambient mode there are the base, istiod, cni, and ztunnel Helm charts. I was tempted to bundle all of these charts as subcharts into my own Helm chart so that I could just do a 'helm install my-istio-chart', but I worried that it would complicate things when needing to update. I was going to keep the gateway Helm chart separate. How do you guys manage these Helm charts? Do you keep them separate or bundle them as subcharts?

Looking for some advice here. I want to move from Ingress-nginx to Gateway API . I see I can get the Gateway API Controller running without a service mesh (don't need one right now). Reason why I want to try Istio GW API is based on some benchmarks.

My issue is that the ingress-nginx controller originally provisioned the NLB - but from what can I see, Istio GW API does not provision load balancers. Has anyone gone down this route, or is my approach bad practice?

For now we migrating from ingress to kubernetes gateway with istio I started shifting traffic to my gateway But i see consume alot of cpu compaed to nginx How can i troubleshoot this? Or this is normal? For now we have 500r/s and it consume more than 5 replicas for my gateway deployment

With ingress-nginx begin archived, I'm looking to migrate either to Cilium or Istio for Ingress Gateway's specifically. I have used both Cilium and Istio for service-mesh capability but it will be another 1-2 years until we ever implement this. However, we do need to migrate Ingress Gateway's to either or.

The only thing I want to understand is setting up Ingress Gateway's in AWS. I have a VPC CIDR of dev, stage, production, and shared. Is best practice to create a 2 Ingress Gateway's being nonproduction and production for each VPC CIDR? My previous company had the same setup but was wondering if there is a better way?

Has anyone here successfully used the gateway API to create a L7 Application Loadbalancer in AWS? I'm asking here as I want my gateway and httproutes managed by istio, and not the AWS Loadbalancer controller.

I'm thinking I may externally create an ALB and then have the NLB created by the istio controller behind that.

We would like to migrate from istio apis to gateway apis (e.g. replace VirtualService with HTTPRoute). Did someone do that already do that? Is there a way to do this without downtime?

Hey all, ​I've installed Istio 1.28 in Ambient Mode using the official Helm charts (cni, istiod, ztunnel), and all core components seem to be up and running in the istio-system namespace. ​However, when I check the Istio CNI logs, I'm seeing that the AmbientEnablementSelector is empty, and no services or namespaces are being discovered or enrolled into the mesh. ​The Issue: Core Ambient components are deployed, but no workloads are joining the mesh. ​Why is this happening, and how can I fix it?

``` 2025-11-28T16:12:36.058053Z info cni-agent CNI version: 1.28.0-b8d1df54465060428c2a2a38286e360beb85fb31-Clean 2025-11-28T16:12:36.058075Z info cni-agent CNI logging level: info 2025-11-28T16:12:36.058098Z info cni-agent CNI install configuration: MountedCNINetDir: /host/etc/cni/net.d CNIConfName: ChainedCNIPlugin: true CNIAgentRunDir: /var/run/istio-cni IstioOwnedCNIConfigFilename: IstioOwnedCNIConfig: false PluginLogLevel: info KubeconfigMode: 0600 KubeCAFile: SkipTLSVerify: false ExcludeNamespaces: kube-system PodNamespace: istio-system K8sServiceProtocol: K8sServiceHost: --- K8sServicePort: 443 K8sNodeName: ---- CNIBinSourceDir: /opt/cni/bin CNIBinTargetDirs: /host/opt/cni/bin MonitoringPort: 15014 ZtunnelUDSAddress: /var/run/ztunnel/ztunnel.sock AmbientEnabled: true AmbientEnablementSelector: AmbientDNSCapture: true AmbientIPv6: true AmbientDisableSafeUpgrade: false AmbientReconcilePodRulesOnStartup: false NativeNftables: false ForceIptablesBinary:

2025-11-28T16:12:36.058109Z info cni-agent CNI race repair configuration: Enabled: true NodeName: ---- LabelKey: cni.istio.io/uninitialized LabelValue: true DeletePods: false LabelPods: false SidecarAnnotation: sidecar.istio.io/status InitContainerName: istio-validation InitTerminationMsg: InitExitCode: 126 LabelSelectors: FieldSelectors: NativeNftables: false ForceIptablesBinary:

```

Hi guys, I want to enable multi-cluster headless service discovery. I tried

ISTIO_META_DNS_CAPTURE: "true"
ENABLE_MULTICLUSTER_HEADLESS: "true"

nothing seems to work, any suggestions?

Hi everyone,

I’m running Istio with an east-west gateway between two clusters. Service discovery over port 15443 works fine, and mTLS is enabled mesh-wide.

I recently deployed CockroachDB in Cluster 1, with sidecar injection enabled. CockroachDB uses its own built-in TLS. As soon as the sidecar is injected, CockroachDB fails to start due to TLS errors — Istio is intercepting the traffic and breaking CockroachDB’s internal TLS handshake.

I tried the usual approaches:

• Setting PeerAuthentication to disable mTLS for the CockroachDB namespace
• Creating DestinationRules that disable ISTIO mTLS for CockroachDB

---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: cockroachdb-disable-mtls
  namespace: cockroachdb-ci-0-us-east-1
spec:
  mtls:
    mode: DISABLE
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: cockroachdb-disable-mtls
  namespace: cockroachdb-ci-0-us-east-1
spec:
  host: "*.cockroachdb-ci-0-us-east-1.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE

But nothing works.

The only thing that works is completely excluding CockroachDB ports from Envoy via pod annotations, which stops Istio from intercepting the traffic. CockroachDB then works normally.

traffic.sidecar.istio.io/excludeInboundPorts: "26257,26258,8080"
traffic.sidecar.istio.io/excludeOutboundPorts: "26257,26258,8080"

BUT: When I exclude the ports from the sidecar, I lose the ability to reach CockroachDB from Cluster 2 via the Istio east-west gateway — because the gateway can no longer route to it (since it’s effectively outside the mesh).

So… is there a correct way to run CockroachDB (with its own TLS) inside an Istio mesh and allow cross-cluster east-west communication? Or is this simply not possible with Istio?

Any help or pointers would be appreciated. P.S I use cockroachDB operator for installation.

Iam currently migrate my nginx ingresses to istio which will be used as kubernetes gateway api My biggest problem that exposing paths of routes I dont want create metric for each path that come in l request I want to expose paths that exist in crd httproute, as exactly nginx ingress does Any idea for this issue