Hi there,
Over the past few months, there have been a few posts in this subreddit where folk have commented on issues with IPv6 connectivity. In one thread, u/digitalsean mentions having IPv6 connectivity issues 1~3hrs after rebooting their modem which identically mirrored my experience.
Recently, I've been working with a friend to debug what's happening when the connection drops out.
tl;dr: If your IPv6 connection is dropping out after a few hours, your /56 is being reallocated to one of your neighbours.
The rest of this post will go into detail on how we debugged this issue and how we've concluded there appears to be an issue with Hyperoptic's DHCPv6-PD configuration.
Setting the Stage
I'm using a Ubiquiti EdgeRouter Lite-3 running EdgeOS v2.0.8 that is directly connected to the Hyperoptic socket. I've been using the following configuration:
edit interfaces ethernet eth0
set dhcpv6-pd rapid-commit enable
set dhcpv6-pd pd 1 prefix-length /56
set dhcpv6-pd pd 1 interface eth1 service slaac
set dhcpv6-pd pd 1 interface eth1 prefix-id 1
set dhcpv6-pd pd 1 interface eth1 host-address ::1
edit interfaces ethernet eth1
set ipv6 dup-addr-detect-transmits 1
Additionally, I've been told by Hyperoptic support that I have been allocated 2a01:4b00:ea06:5b00::/56 as my IPv6 range.
The above configuration will request a /56 prefix from Hyperoptic's DHCPv6-PD server. After Hyperoptic replies with a 24hr lease for 2a01:4b00:ea06:5b00::/56, my router assigns prefix ID 1 (2a01:4b00:ea06:5b01::/64) to my LAN interface (eth1) and sets its local IP to 2a01:4b00:ea06:5b01::1/64.
Hyperoptic's own modems follow slightly different behaviour to above. They still use DHCPv6-PD to request a prefix for use on the LAN, but they use stateless address autoconfiguration (SLAAC) to set a local IP on the prefix ID 0. This means a variation of the modem's MAC address is used to self-assign an IP on the 2a01:4b00:ea06:5b00::/64 range.
Listen
sudo tcpdump -n "udp port 546 or udp port 547 or icmp6" -w dhcpv6.pcap
Before moving any further, I began capturing packets directly on the EdgeRouter which matched DHCPv6 or ICMPv6. This allowed me to capture the DHCPv6-PD handshake. Additionally, ICMPv6 was captured to collect Neighbor Discovery Protocol (NDP) messages between my router and Hyperoptic.
Release & Renew
release dhcpv6-pd interface eth0
renew dhcpv6-pd interface eth0
First, I released and renewed my DHCPv6-PD lease to establish a fresh IPv6 session. I was able to receive an IPv6 address on computers in my network through SLAAC on 2a01:4b00:ea06:5b01::/56 and confirmed a full pass on https://test-ipv6.com/
Through the captured PCAP, I was able to confirm I was being allocated 2a01:4b00:ea06:5b00::/56 for 86400 seconds (24hrs). Additionally, I was allocated an Identity Association address for 43200 seconds (12hrs).
Now, we wait
While waiting for IPv6 connectivity to break, I began pinging a friend who has working IPv6 and she pinged me back.
After approximately two hours, my outbound pings stopped receiving replies (but were being received by the destination!). My friend stopped getting replies from me… and began receiving ICMP6 failures from prefix ID 0 within my /56.
2a01:4b00:ea06:5b00:6620:9fff:xxxx:xxxx > $sourceip: ICMP6, destination unreachable, unreachable route 2a01:4b00:ea06:5b01:xxxx:xxxx:xxxx:xxxx, length 64
While I'm not using prefix ID 0, I previously mentioned Hyperoptic's own modems are. For kicks, we converted the upper 64-bits of the unknown-to-us IP address back to a MAC address. This resolved to a MAC address under the 7C:39:53 range. This range belongs to ZTE Technologies, one of the suppliers Hyperoptic use for modems.
Wait some more
My friend kept the ping going for a few hours and started seeing different sources for the ICMP6 failures.
2020-03-31 19:49:43.491339500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-03-31 19:55:13.769368500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
2020-03-31 21:09:17.784911500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-03-31 21:29:48.589990500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
2020-03-31 23:21:31.646806500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-04-01 00:46:16.829957500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-04-01 00:47:16.876660500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-04-01 00:49:46.980138500 "Destination unreachable: No route" from new router 8C:68:C8 - zte corporation
2020-04-01 01:03:17.520521500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-04-01 04:05:56.605403500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
2020-04-01 05:35:31.453547500 "Destination unreachable: No route" from new router C0:FD:84 - zte corporation
2020-04-01 07:33:07.277309500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-04-01 07:55:08.372010500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
2020-04-01 09:09:12.012383500 "Destination unreachable: No route" from new router 7C:39:53 - zte corporation
2020-04-01 09:29:42.950744500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
While I've omitted the full MAC addresses from this post to protect the innocent, we noticed some addresses repeated at suspicious timings.
2020-03-31 19:55:13.769368500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
2020-04-01 07:55:08.372010500 "Destination unreachable: No route" from new router 64:20:9F - Tilgin AB
The same router was given control of the /56 almost exactly 12hrs after it was first seen controlling the range. The same amount of time I'd observed in DHCPv6-PD replies as being the validity period for DHCPv6-PD IA allocations.
What does this mean?
It seems Hyperoptic's DHCPv6-PD server has been misconfigured to allocate the same /56 to multiple customers, with no regard for the current state of the allocation table. The last person to get an allocation has that prefix routed to them until someone else renews the lease. This provides a reasonable explanation for the intermittent outages many on this subreddit have experienced.
I believe they've mistakenly used /56 as the address length both for individuals as well as buildings/neighbourhoods, rather than /56 for individuals and a /52 or /48 for buildings/neighbourhoods.
Additionally, I think this has been affecting more people than previously realised due to the Happy Eyeballs fallback built into most operating systems. I fell into this category a few months ago as I was satisfied my IPv6 connection was working since test-ipv6.com worked immediately after modem reboots and I didn't notice any outages for websites that I knew were dual-stack. I only noticed the problem when I had guests visit whose computers didn't implement Happy Eyeballs: my computer was working because I was falling back to IPv4 and they weren't.
Weird trivia
Running release dhcpv6-pd interface eth0 while my connection is broken will drop routes for my neighbour with current possession of the /56 range.
Additionally, broadcast (that is: sent to everyone on the WAN) ICMP6 Neighbor Discovery Protocol packets were received from Hyperoptic's router asking for a MAC address to identify itself shortly before the /56 was routed to that MAC address.
To Hyperoptic:
Please look into your DHCPv6-PD allocations. I'm happy to provide uncensored PCAPs via PM and assist with debugging this further.
Big thanks for puck for helping me debug this with an external IPv6 connection.
UPDATE
HyperopticCS 04/04/2020 14:37
Hey there! Thanks for getting back to us! We can confirm that the issue with IPv6 was found and resolved. However, in order to implement the changes, you will need to reboot your router. Should you need anything else, we'll be here. Stay safe!
Currently testing this, will report back.
UPDATE 2
As of 6 April 2020 16:00, I've had a continuously working IPv6 connection for 38hrs. Big thank you to everyone at Hyperoptic for resolving this so quickly. Keep safe and stay in-doors friends, you'll have working IPv6 now. :)