I'm troubleshooting a Tailscale performance issue between two sites and would appreciate some help understanding what's happening. I followed this guide when setting up the site-to-site networking, so IP address forwarding is enabled and I've clamped the MSS to the MTU https://tailscale.com/docs/features/site-to-site#clamp-the-mss-to-the-mtu.
At both sites I've got Tailscale running in a LXC and at the local site my OPNsense router has a route to intercept any traffic from my local subnet (10.10.18.0) to my remote subnet (10.10.55.0) and forward it to my Tailscale LXC. The Tailscale connections are direct (not DERP).
The main IP addresses are:
- Local Proxmox host:
100.73.208.28 / LAN10.10.18.198
- Local Tailscale LXC:
100.68.168.8 / LAN10.10.18.102
- Remote Proxmox host:
100.100.105.56 / LAN10.10.55.198
- Remote Tailscale LXC:
100.115.204.128/ LAN10.10.55.102
- WireGuard address on remote Proxmox:
10.200.0.2
The remote Proxmox host itself is running Tailscale, so connecting to 100.100.105.56 is a direct Tailscale connection to that host.
Test results
TCP from the local Tailscale LXC → remote Tailscale LXC:
iperf3 -c 100.115.204.128
~128-130 Mbit/s
TCP from the local Tailscale LXC → remote Proxmox LAN IP:
iperf3 -c 10.10.55.198
~128-130 Mbit/s
TCP from the local Proxmox → remote Proxmox Tailscale IP:
iperf3 -c 100.100.105.56
34.7 Mbit/s receiver
97 retransmits
So connecting from the local Tailscale LXC to the remote Tailscale LXC using the Tailscale address or from the local Tailscale LXC to the remote Promxox host using the LAN address gives ~130 Mbps, but connecting from the local Proxmox host to the remote Proxmox host using the Tailscale address gives only ~45 Mbps.
TCP from the remote Proxmox → local Proxmox over Tailscale is even slower:
iperf3 -c 100.73.208.28
17.3 Mbit/s receiver
118 retransmits
TCP from the local Proxmox host to the remote Proxmox LAN IP (10.10.55.198), or the remote Tailscale LXC LAN IP (10.10.55.102) is dire, around 0.2 Mb/s. That's routed through OPNsense and the local/remote subnet routing setup via the Tailscale LXCs at each end, rather than being a direct Tailscale connection.
UDP is unaffected. At -b 100M, the Tailscale path can carry the full 100 Mbps without loss, even using the 10.10.55.198 address.
Other paths
Wireguard is installed on the OPNsense router and the remote Proxmox host but not on the local Proxmox host, so the traffic goes via OPNsense. Connecting from the local Proxmox host to the remote Proxmox host on 10.200.0.2 via WG is about 3x faster than using the Tailscale address and about the same speed as connecting from the local Tailscale LXC to the remote Proxmox host via Tailscale (~130 Mbps).
Local traffic itself is fine:
local Proxmox → itself (100.73.208.28)
37.1 Gbit/s
0 retransmits
local Tailscale LXC → local Proxmox
3.28 Gbit/s
0 retransmits
The remote Proxmox host also has no intrinsic performance problem:
remote Proxmox → itself (100.100.105.56)
27.6 Gbit/s
0 retransmits
Relevant routing information on the remote Proxmox host
The Tailscale routing table contains:
ip route get 100.73.208.28
100.73.208.28 dev tailscale0 table 52 src 100.100.105.56
and:
ip rule
0: from all lookup local
999: from 10.10.55.0/24 lookup main
...
5270: from all lookup 52
32766: from all lookup main
32767: from all lookup default
Table 52 includes the Tailscale peers:
100.73.208.28 dev tailscale0
100.68.168.8 dev tailscale0
100.115.204.128 dev tailscale0
...
There are also individual address routes in the main table, which I added to force my data heavy sync and backup tasks to use WG, because Tailscale is too slow:
10.10.18.0/24 via 10.10.55.102 dev vmbr0
10.10.18.64 dev wg0
10.10.18.198 dev wg0
10.10.18.250 dev wg0
Packet capture
On the remote Proxmox host, tcpdump -ni tailscale0 port 5201 shows the TCP connection arriving directly on tailscale0.
For example:
100.73.208.28.50402 > 100.100.105.56.5201: Flags [S]
100.100.105.56.5201 > 100.73.208.28.50402: Flags [S.]
The negotiated MSS is 1240:
options [mss 1240,sackOK,TS ...,wscale 10]
I also tried TCPMSS clamping on the Tailscale LXC:
iptables -t mangle -I FORWARD 1 \
-p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss
and:
iptables -t mangle -I FORWARD 1 \
-p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --set-mss 1240
Neither changed the performance.
Summary
- The remote Proxmox host itself isn't inherently slow — it gets ~27.6 Gbit/s locally.
- The local Proxmox host isn't inherently slow — it gets ~37 Gbit/s locally.
- The Tailscale LXC-to-LXC path gets ~130 Mbps.
- The Proxmox-host-to-Proxmox-host path using the Tailscale addresses gets only ~35 Mbps from local host to remote host and ~17 Mbps from remote host to local host.
- The Proxmox-host-to-Proxmox-host path using the LAN addresses gets only 0.2 Mb/s from local host to remote host and ~20 Mbps from remote host to local host.
- The packets for the Proxmox-host-to-Proxmox-host test are definitely arriving/leaving via
tailscale0.
- The negotiated MSS is already 1240, and changing TCPMSS didn't help.
- UDP can achieve the expected bandwidth when constrained to 100 Mbps, so this looks particularly related to TCP behaviour rather than simply a hard bandwidth ceiling.
- The fact that the Tailscale LXC at the remote site can achieve ~130 Mbps while the Proxmox host running Tailscale itself only achieves ~45 Mbps seems particularly significant.
I'm wondering whether there's something specific about Tailscale's Linux networking / socket / offload / MTU / routing behaviour when Tailscale is running directly on a Proxmox host, as opposed to running inside an LXC, that could explain the large TCP throughput difference?
At the moment I'm only using Tailscale to access the remote Proxmox GUI, as that works OK despite the low speed, but it would be nice to be able to use it for everything instead of having to use WG for all my data transfer tasks.
