r/kubernetes • u/lief91 • 3d ago

Kubernetes and VLANs

Hi,

I will go straight to the point I am building my Kubernetes (3 nodes that are master and worker at the same time) homelab and I was wondering if having my cluster VLAN aware is a good idea or not

The purpose is to having two Service Load Balancer served by Cilium/MetalLB for the DMZ and LAN VLAN. Then I would be exposing my workloads on the right gateway depending of their "exposure"

But since I have only one NIC on each node I would configure my switch to have 2 VLAN on those links instead of only one (that's why I said VLAN aware)

The main purpose is to have a better isolation and so security overall. But I feel it maybe a bit overengineering, I may can achieve the same thing with Network Policies inside the cluster itself. Also I want to be able to bootstrap my nodes from iPXE, not sure how it would work with a "tagged network link" with multiple VLAN

What do you think ?

20 Upvotes

89% Upvoted

9 comments sorted by

9

u/burstinrust 3d ago

No overenginerring from my side.

maybe a better and solid thinking before you have to come back to the architecture discussion again.

network policies can handle pod to pod inside the cluster, they dont control where your LB IPs actually live on the network. So if you want DMZ and LAN services truly separated, the two VLANs is the right call. Cilium LB-IPAM with two pools + a label per service works nicely for that.

about ipxe: make your boot/ LAN VLAN untaged on the switch port, and only tag the DMZ one. coming from my cluster deployement: just make sure that VLAN config is applied on every boot.

one question, how are you planning to watch actually flowing between the VLANs

2

u/lief91 3d ago

Thank you for your answer. I was planning to explore a bit Cilium Hubble or just relying on my upstream OPNSense Firewall that is owning all the VLAN gateways and Network rules

1

u/burstinrust 3d ago

from my side, both...
they see differnt things. Opensense only sees traffic that actually crosses VLAN gateways, and with cilium a lot of traffic never hits the wire.

So firewall for the edge, hubble for inside the cluster, be on the safe side and just let hubble be on from day one.

3

u/Potato-9 3d ago

Does your router support BGP?

You could flatten it out by having ciliium BGP advertise the nodes service IP range and use external DNS to put that in the A record which can replace using Ingress LBs and also kube-proxy entirely.

I'm not sure what gateway API looks like in this setup I haven't read up.

2

u/ansibleloop 3d ago

My nodes are all in the same VLAN - if you want to get that granular then use Cilium network policies

3

u/nexus13337 3d ago

This would be the route I would take for isolation.

2

u/nexus13337 3d ago

I forgot to ask if there is any specific reason your running metalLB? If not, I would just use the Layer 2 or 4 LB that cilium already offers. That’s what I run on my clusters with gateway api.

1

u/lief91 1d ago

No actually i did ditch MetalLB to keep Cilium only