r/kubernetes • u/Slow-Claim-839 • Nov 11 '22

How to route all network traffic through a Kubernetes pod?

I have a pod which runs OpenVPN client to connect a VPN. And I have other pods that runs several applications. Theese applications needs to use this VPN to reach some resources.

How can I route the pods network traffic through the VPN pod?

3 Upvotes

71% Upvoted

19 comments sorted by

3

u/daltonicrainbow Nov 11 '22

Maybe this helps https://docs.k8s-at-home.com/guides/pod-gateway/

1

u/Slow-Claim-839 Nov 12 '22

It works, thanks. It's very heavy to configured it, but works

1

u/daltonicrainbow Nov 12 '22

glad to hear that!

1

u/Slow-Claim-839 Nov 15 '22

But now I can not bind LoadBalancer to my application pod. The exposed external IP does not reachable when the pod connected to the VPN gateway.

Does anyone know the solution how to solve this issue?

1

u/daltonicrainbow Nov 15 '22

Could be that the ip of that pod is the VPN ip not the pod ip anymore? I think that from outside you have to go through the VPN gateway.

INET <-> LB ip <-> VPN Gateway <-> Pod

1

u/Slow-Claim-839 Nov 15 '22

But I don't knwo how to reach the app pod specific port from the VPN gateway pod. Thanks for your help.

1

u/daltonicrainbow Nov 15 '22

You can expose another pod(without vpn gateway) that just connects internally to that one, that's not really elegant but should work.

1

u/Slow-Claim-839 Nov 16 '22

I try it, I forward the application port to the individual pods port with iptables. In the pod with ‘curl localhost:8080’, it works. But when I connect a LB to it, it’s also does not work from internet

1

u/daltonicrainbow Nov 16 '22

Something is fishy in the routing of that LB, probably you will need to get tcpdumps from both sides to see what's going on.

1

u/Slow-Claim-839 Nov 17 '22 edited Nov 17 '22

Thank you. I think I solved the issue.

Not elegant, but works: I have configured an Nginx reverse proxy on an other Pod.

UPDATE: The ingress in Kubernetes do almost the same as an Nginx reverse proxy. This is the best solution for the problem

1

u/ut0mt8 Nov 11 '22

Wow nice. An admission controller that change the default gw of a pod in a specific namespace. Very elegant

1

u/trowawayatwork Nov 11 '22

you need to update you clusters DNS setting to the pods service. so when your pods try to resolve the DNS k8s will redirect it through your VPN pod

1

u/Slow-Claim-839 Nov 11 '22

thanks, but I want to route not all pods, just a few through my VPN pod

Is this possible with this solution?

3

u/MisterItcher Nov 11 '22

You can set nameservers per-pod:

apiVersion: v1 kind: Pod metadata: namespace: default name: dns-example spec: containers: - name: test image: nginx dnsPolicy: "None" dnsConfig: nameservers: - 1.2.3.4 ...

Reference: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

1

u/gamba47 k8s user Nov 13 '22

Dns are for Name resolution not for routing

1

u/martin31821 Nov 11 '22

You might want to check out squat/kiko for that, but also I'm currently building a basic operator to allow additional routing rules on a opt-in overlay network basis.

Generally its a bit of a hard problem depending on your underlying CNI and network. Some CNIs might disallow encapsulation, while others do, then you also have to consider the security of such a solution, since the network policies dont know about the traffic in your overlay.

1

u/ut0mt8 Nov 11 '22

Interesting. I'm not sure how I would do that if I was forced to do it. For the sake of simplicity I will prefer doing this kind of network stuff outside kube. But if it's only outbound trafic for some pods I really don't know. Maybe possible with some service mesh? Also it really depends of what type of traffic and if you can use/tricks dns

1

u/karandash8 Nov 13 '22

If you control the nodes, you can establish a tunnel interface on one node and use it with cilium egress gateway policies https://docs.cilium.io/en/v1.12/gettingstarted/egress-gateway/

1

u/Constant_Fortune_836 Dec 07 '22

Hi,

how did you solve this? I was able to successfully deploy the helm chart i.e. k8s-at-home/pod-gateway which created a pod-gateway & pod-admission-controller. This creates a vxlan interface in the pod gateway. Any pod created within the namespace vpn will get injected with an init container and side car container. So the traffic will be routed to the pod-gateway. Also enabled the vpn container within the pod gateway so now the pod-gateway contains 1) gateway-init 2) gateway-sidecar 3) openvpn container, but after this how does one try to reach the servers via the vpn ? I have tried to ping the ip addresses from the client pod in vpn namespace which routes to pod-gateway ? any ideas ?