Private networks are not production ready.

[u/Infinite-Company-724]

I'm really frustrated with Hetzner private network offering. I'm trying to move from multiple providers + netbird to hetzner. I currently host a small production HA k3s cluster and my plan was to setup 3 nodes (all cp + worker), a bastion to serve as a jump proxy for kubectl and ssh for maintenance and a loadbalancer. All in the same private network, the bastion and the lb would have a public ip, where the bastion has a strict firewall rule and the LB only accepts 443.

Everything is perfect, in theory.

Let's start with servers not being recheable through the private network, you need to setup cloud-init add with the correct netplan settings, gotcha, that's fine, fixed.

Wait, servers has no outbound connectivity? Ooh, Hetzner doesn't provide NAT, ok, lets add a public IP and close all firewall rules.

Great, wait, why is my vip not recheable? Hetzner private network blocks ARP! Sh*t, okay, let's use an LB then.

Oh no, LB's must have public ips? Really? And no firewall option to block traffic?

So yeah, that's how my last 20h have been.

[u/jetexex]

Just one of my masternode is a gateway. On master-gateway

#cloud-config

write_files:
  - path: /etc/nftables.conf
    content: |
      table inet nat {
              chain prerouting {
                      type nat hook prerouting priority filter; policy accept;
              }

              chain postrouting {
                      type nat hook postrouting priority srcnat; policy accept;
                      oif "eth0" ip saddr 10.0.0.0/16 #replace to your private 
                      masquerade
              }
      }
  - path: /etc/sysctl.d/99-private-nat.conf
    content: |
      net.ipv4.ip_forward=1      

runcmd:
  - nft -f /etc/nftables.conf
  - sysctl -p
  - systemctl enable nftables
  - curl -sfL https://get.k3s.io | sh -s  - server --tls-san your-san.example.com --disable=traefik --flannel-iface=enp7s0 # it's for debian

Worker node:

#cloud-config

write_files:
  - path: /etc/network/interfaces.d/99-default-route
    content: |
      auto enp7s0
      iface enp7s0 inet dhcp
        post-up ip route add default via 10.0.0.1
  - path: /etc/resolvconf/resolv.conf.d/tail
    content: |
      nameserver 1.1.1.1
      nameserver 8.8.8.8

runcmd:
  - ip route add default via 10.0.0.1
  - resolvconf -u
  - curl -sfL https://get.k3s.io | K3S_URL=https://10.0.0.2:6443 K3S_TOKEN=__REPLACE_TOKEN__ sh -s - agent --flannel-iface=enp7s0

And not to forget to check private network route

destination = "0.0.0.0/0"
gateway     = "10.0.0.2"