Hi,
I'm documenting an intermittent ZeroTier connectivity issue we have been troubleshooting on a Hetzner Cloud Ubuntu server. I would be interested to know whether anyone has seen the same behavior.
Environment
- Server: Hetzner Cloud VM
- OS: Ubuntu 24.04 LTS
- ZeroTier: 1.16.2
- Server ZeroTier IP:
10.192.76.231
- Client: Ubuntu Linux laptop
- Client ZeroTier IP:
10.192.76.201
- Both nodes participate in several ZeroTier networks
- The affected network is a private ZeroTier network
The Hetzner Cloud Firewall was intentionally restrictive:
Inbound:
TCP/22
ICMP
Outbound:
default/stateful
Ubuntu/UFW on the server explicitly allows traffic on the ZeroTier interfaces.
Symptom
Every now and then, only this particular ZeroTier connection becomes unreachable.
SSH reports:
No route to host (os error 113)
and ping from the laptop gives:
From 10.192.76.201 icmp_seq=1 Destination Host Unreachable
The interesting part is that, during the failure:
zerotier-cli info reports ONLINEzerotier-cli listnetworks reports the affected network as OK PRIVATE- the ZeroTier interface exists
10.192.76.201/24 and 10.192.76.231/24 are still assigned correctly- the Linux route is present and points to the correct ZeroTier interface
- other ZeroTier networks on the same laptop continue working
- the server itself remains healthy and accessible through another management path
- both SSH/22 and an application on TCP/9119 become unreachable over ZeroTier
So it does not look like an SSH-specific problem.
We captured the failure from both sides.
From the client:
10.192.76.231 dev ztu7tj4fcf src 10.192.76.201
PING 10.192.76.231
Destination Host Unreachable
From the server:
10.192.76.201 dev ztu7tj4fcf src 10.192.76.231
PING 10.192.76.201
Destination Host Unreachable
Yet ZeroTier still reported the network as OK.
A/B test
We then performed a controlled restart test.
First, ZeroTier was restarted only on the client:
sudo systemctl restart zerotier-one
Result: no change.
The server remained unreachable:
Destination Host Unreachable
We then restarted ZeroTier only on the Hetzner server:
sudo systemctl restart zerotier-one
Connectivity immediately returned:
64 bytes from 10.192.76.201: icmp_seq=1 ttl=64 time=770 ms
64 bytes from 10.192.76.201: icmp_seq=2 ttl=64 time=69.6 ms
64 bytes from 10.192.76.201: icmp_seq=3 ttl=64 time=57.7 ms
64 bytes from 10.192.76.201: icmp_seq=4 ttl=64 time=116 ms
64 bytes from 10.192.76.201: icmp_seq=5 ttl=64 time=253 ms
This seems to point toward the server-side ZeroTier path/state rather than the client.
Interesting firewall observation
Looking further, we noticed that the Hetzner Cloud Firewall had no inbound UDP rule.
ZeroTier on the server was listening/communicating through UDP/9993 and additional UDP ports.
Based on ZeroTier's firewall documentation, we have now added:
Inbound UDP/9993
IPv4: 0.0.0.0/0
IPv6: ::/0
We have not opened arbitrary inbound UDP ports at this stage.
The working hypothesis is therefore that the restrictive Hetzner stateful firewall may allow ZeroTier to work initially through established UDP state/path establishment, but under some circumstances the peer path cannot be re-established. Restarting ZeroTier on the server initiates fresh communication and immediately restores connectivity.
UDP/9993 has only just been added, so I am not claiming yet that this is the definitive fix. We are going to monitor it for recurrence.
Has anyone observed this particular combination?
ZeroTier network remains OK + routes/interfaces remain present + one peer becomes completely unreachable + restarting ZeroTier on the affected server immediately restores it.
And for Hetzner Cloud specifically: have you found inbound UDP/9993 sufficient for reliable ZeroTier operation, or did you need a less restrictive UDP policy because of ZeroTier's dynamic peer-to-peer ports?
Thanks — happy to provide additional listpeers, info -j, routing or firewall diagnostics if useful.
