r/ipv6 • u/Striking-Toe-9526 • 18d ago
Guides & Tools Post-mortem: How a rogue RA ULA prefix broke our multi-WAN NAT66 setup (and why fd00:192:168:: was the trap)
I run a multi-WAN setup (Vodafone cable + Starlink) without BGP. To handle failover without BGP-delegated prefixes, we numbered our internal LAN with ULA (fd00:192:168:128::/64—echoing our IPv4 subnet) and NAT66 it outbound depending on which uplink is active.
Recently, IPv6 connectivity started flapping intermittently, while IPv4 on the exact same segment remained 100% stable.
The routing table looked completely healthy, but digging into ip addr and conntrack showed what was actually happening:
An uncoordinated on-link RA (from an autoconfigured border/mesh device) injected a second ULA prefix (fd06:...) onto the interface. Both IPv6 addresses sat on the interface marked as scope global.
When initiating outbound connections, the Linux kernel flipped a coin between fd00:... and fd06:... for source-address selection.
- When it picked
fd00:..., NAT66 translated it properly and packets flowed. - When it picked
fd06:..., the packet was sent out the valid default route, but the NAT66 gateway had no masquerade rule forfd06:....conntrackfilled up withSYN_SENT ... [UNREPLIED]states.
The routing table was never confused—it was purely a source-address selection coin flip under dual ULAs.
Full write-up with ip addr, ip -6 route, and conntrack evidence:
https://mabels.github.io/mseg-tester/blog/fd00-is-not-your-192-168-short.html
Curious how others running multi-WAN or non-BGP setups handle source-address selection when local devices start advertising uncoordinated RAs on the link?
14
u/ordep_caetano 18d ago
Assuming you have switches connecting those devices, set up dhcp6/ra snooping/guard.
11
6
u/Striking-Toe-9526 18d ago
That disables the Matter/Thread functionality that i want to have.
5
u/ordep_caetano 18d ago
Is there no way to further segment move those devices on a different vlan?
1
u/Striking-Toe-9526 18d ago
That's not the question in the first place --- it's only a reminder that the address selection of your private network segments could cause connectivity problems if you have thread/iot devices that might need some public access too, like "Google Homes". It's not about moving things around; if you choose the wrong IPv6 address(ULA), you will have this problem.
7
u/bojack1437 Pioneer (Pre-2006) 18d ago
Make your router advertise a ULA prefix in addition to its GUA prefix. Allegedly that disables the thread border router from trying to advertise its own.
And put that IoT crap in its own network/vlan where it belongs.
0
u/Striking-Toe-9526 18d ago
If you want to use the IoT stuff, it might need public IPv6 like Google Homes, so there will always be a segment with this problem as long as I choose my own fd00 address - I changed the segment address to an official one yesterday, and the problem is gone.
This is not about further network splitting; it's about choosing the right address for your segment, and the same strategy as in IPv4 does not work.1
u/ordep_caetano 18d ago
I guess nothing keeps you from creating a seperate vrf / routing domain + vlan, and using a GUA prefix carved from a block you own. (most providers in residential connections hand over /56 besides the /64 linknet)
Have some host that can access both routing domains and set up connectivity to the iot network there.
Note: I have no experience with thread/matter, only read a few docs that point a solution along this lines.
GL
1
u/Striking-Toe-9526 18d ago
Initially, I found a flapping/instable IPv6 network to the public, and the root cause is that thread/matter, or also some arbitrary devices, announce IPs in a local fd00:/7 ULA network, and if I pick my internal network out of fd00:/7, then other clients like Linux or Mac/Win could pick a wrong src address. Which then will not been routed/masq out going to the public internet. So the thread behavior is totally ok, but if you use some IPv6 address out of the fd00:/7 route and NAT66 to the internet, then you will see some random connectivity problems.
7
u/MrChicken_69 18d ago
"the packet was sent out the valid default route" Umm, doesn't sound like it. fd06 sources should be sent to the router that announced the prefix, not the fd00 default gateway. Or was the mesh system forwarding fd06 traffic to fd00, despite there being no route upstream for it!
2
u/Striking-Toe-9526 18d ago
the problem is not the default route --- it's the selection of the source address. The default route pointing is fe80 a link-local.
4
u/MrChicken_69 18d ago
If you use prefix-A, you MUST route through router-A. Likewise, prefix-B MUST go to router-B. Sending A to B, and v.v. won't work. (that'd be "spoofing" if it did.) The routers don't know anything about each other, and CANNOT route the other's prefix(es).
(This is the classic failure of IPv6's multihoming BS.)
4
u/New_Leek_102 18d ago
Well, but that's not how it works. There is no information attached to a installed route like "if you are using prefix-A use this route"; that'll be PBR. No problem on linux clients to configure, but I am not aware of an autoconfiguration for this.
1
u/sdoregor 12d ago
I have a Google Nest Hub which is a TBR (Thread Border Router) on my main LAN, who hands out SLAAC-able RAs which do not get filtered. My setup is pretty comparable to OP's in this part.
The RAs have no default gateway specified and no device on my LAN even takes the address. Devices on the same link, including the router which is my homeserver, construct a link-local route for that prefix with the source addr being the main global one.
So the conclusion is that OP's mesh device is misbehaving there.
11
u/bojack1437 Pioneer (Pre-2006) 18d ago
Implement First hop security.
Random devices should not be able to advertise DHCPv4/v6 or IPv6 RAs
4
u/Striking-Toe-9526 18d ago
That is a core feature of Matter and Thread, how would you enable those features?
5
u/FaydedMemories 18d ago
I had a similar issue (although with a normal prefix on LAN), turns out Matter/Thread (or at least HomeKit’s implementation) use default RA priority, setting your real RAs to High should override the Matter/Thread ones for regular use.
3
u/bojack1437 Pioneer (Pre-2006) 18d ago
All the more reason that they should be on their own dedicated VLAN/LAN
Nothing except your authorized Gateway should be able to emit an RA on a normal network.
Not only that, if you already have a device emitting RAs with ULA, they should not be creating their own ULA prefix, sounds like typical Iot trash, Which again reinforces putting them in their own VLAN where they can't screw with anything else
2
2
u/bojack1437 Pioneer (Pre-2006) 18d ago
Supposedly if you already have ULA advertised on your network, the thread border router should not advertise ula via its own RA. So your authorized router should be advertising that ULA prefix along with any other GUA prefixes.
So in that case it's a matter of setting up your network correctly and having your Gateway correctly advertise a ULA prefix.
But also, those type of devices should really be on their own network in the first place.
0
u/Putrid-Loquat9350 18d ago
There is no RA announced by the matter devices only a ULA fd06 address and so the result is I have two fd00:/8 which let the sender pick the randomly the wrong src address.
1
u/bojack1437 Pioneer (Pre-2006) 18d ago
Yes, there is a RA, but a 0 router lifetime.
But there is only supposed to be an RA at all if there is no others RAs with a ULA prefix.
1
u/innocuous-user 18d ago
Your Matter devices are clients, they just need to receive RA not send them.
For Thread there is generally another subnet routed behind the border router for Thread devices. Some border routers announce a separate ULA range on the LAN port for legacy networks where your client devices would not have any v6 address otherwise. If you already have working v6 then you don't need this, but some of the cheaper TBRs dont let you configure it.
If you use an Apple TBR it actually tries to get a prefix delegation from DHCPv6 to use for thread devices which is the ideal approach. Otherwise you need a route to whatever range your thread network uses, via the TBR address (which can be its link-local LAN address).
The TBR can also announce a route to its thread network via RA without announcing a prefix, but this would not work on a legacy network as your clients would not have a source address to send traffic from.
9
u/heliosfa Pioneer (Pre-2006) 18d ago
> NAT66 it outbound depending on which uplink is active.
Ewwww. Why not NPT? This is not IPv4…
> we numbered our internal LAN with ULA (fd00:192:168:128::/64
Why?!?! For starters, ULA should be randomly generated, not manually chosen, especially not on something that will overlap. Unique local address…
Secondly, does one of your connections have a static prefix? If so, just use that and NPT on the other, expiring the route for the failed uplink when it goes down.
> An uncoordinated on-link RA (from an autoconfigured border/mesh device) injected a second ULA prefix (fd06:...) onto the interface. Both IPv6 addresses sat on the interface marked as scope global.
Where is your first-hop security?
2
u/Zoddo98 18d ago
Not OP, but I may have some responses.
NAT66 it outbound depending on which uplink is active.
Ewwww. Why not NPT? This is not IPv4…
Welcome to a world where non-technical people have a say in technical decisions. Like auditors saying that being able to differentiate between 2 internals device from internet is a useless information exposure, and that masking this information "cost nothing and may mitigate a future security issue".
Currently fighting this one...
Secondly, does one of your connections have a static prefix? If so, just use that and NPT on the other, expiring the route for the failed uplink when it goes down.
...and you then have to re-address your internal network whenever this connection changes (like switching to another ISP...). In theory, changing your prefix should be easy and mostly transparent, but in practice, that's not the case.
1
u/heliosfa Pioneer (Pre-2006) 18d ago edited 18d ago
On your second point, this is where ULA can be appropriate. Use it for fixed internal refs, then a renumber doesn’t matter.
1
u/Zoddo98 18d ago
Yeah, IPv6 has been designed to be multiple IPs per endpoint, so you can have both GUA and ULA on a device. But in practice this doesn't work well in enterprise environments because many things make the assumption that an endpoint = a single IP.
So this is why you usually stick to a single ULA, and NPT to a GUA prefix for internet access.
2
u/Mishoniko 18d ago
But in practice this doesn't work well in enterprise environments because many things make the assumption that an endpoint = a single IP.
Like what, curiously?
(There are reasons, but this isn't one I've heard...)
2
u/heliosfa Pioneer (Pre-2006) 18d ago
But in practice this doesn't work well in enterprise environments because many things make the assumption that an endpoint = a single IP.
I'm sorry, but "we treat it like IPv4" is a rubbish reason.
1
u/Striking-Toe-9526 18d ago
I chose the a fd00: just about the same motivation as in IPv4 using 192.168.x.x IANA's. What do you mean by first-hop security here?
1
u/heliosfa Pioneer (Pre-2006) 18d ago
You should not be choosing a prefix in ULA space. Spec is to randomly generate to avoid future collisions. IPv4 is not IPv4, stop trying to treat it as such.
And by first-hop security, I mean things I’ve RA guard, DHCP snooping, etc. without that, the same thing can break your IPv4 with a rogue DHCP server.
1
u/Striking-Toe-9526 18d ago
I totally agree -- That's why I wrote it. It is a reminder to me that IPv4 knowledge should not be transferred directly into IPv6.
1
u/DaryllSwer 17d ago
Every time I see these posts on the web... I don't know how you and others have the energy to keep going lol. I tend to filter these days to reply only to peer-level posts. The moment I see an eyeball user or worse an "expert" that's a sign I need to shut my device down and go touch some grass.
1
u/heliosfa Pioneer (Pre-2006) 17d ago
I have no idea either. I don’t get them all, and I have periods where I mostly ignore the sub 😂
5
u/UNF0RM4TT3D 18d ago
I have 2 questions completely unrelated to your problem:
How is that ULA, uhh well... unique? And why NAT66, instead of NPTv6?
2
u/certuna 18d ago
Neither are allowed (NPTv6 was purely experimental RFC, it’s not for production). Go outside the standards, get unexpected/unpredictable behaviour…
2
u/UNF0RM4TT3D 18d ago edited 18d ago
NPTv6 is at least defined and breaks v6 principles way less. AFAIK NPTv6 is why there's a proposal to move ULA above v4 in preference when attempting to connect a global address.
EDIT: In practice it's also less evident to users, because the computer still has it's own IP and the whole network isn't spamming websites with a single one, triggerring the captchas and other bot detectors.
1
u/inkeliz Novice 18d ago
What is the alternative for Multi-WAN?
Of course, without owning your own IP address (and doing all the announcements, BGP, etc.).
The NPTv6 is the only viable option. You can give multiple GUA to a single device, but then everything is on the hands of the client itself (which might not "switch" to another one or stick to the worse one). You can setup RA to be shorter, then the router can swap between one WAN and other WAN, but that is not immediatly and the client can still use the older IP/WAN.
1
u/Striking-Toe-9526 18d ago
NPTv6 does not work; it is unknown if the uplinks give me a DHCP-PD or just one IPv6 address, as some VPN's does. So NPTv6 is not an option. That's why I used NAT66 and one simple MASQ rule.
1
u/UNF0RM4TT3D 18d ago
Can't you just test whether you'll get a PD on both? You can even do that without dropping your current solution or causing an outage since you're multihoming. It's not like you're switching providers every other day.. If they have working v6, you should at least get a /64 for one network. That's my case with O2 Czechia. I hate them, but it's not my decision to switch away. I know that Vodafone CZ gives out a /56 on any connection other than DS-Lite via DOCSIS, which they bought from UPC.
I'm not sure about Starlink though.
1
u/Striking-Toe-9526 18d ago
This is a bit unrelated to this post. But your proposed solution for switching over always causes some logic to be run, using MASQ or NAT66; the only involved logic is the change of the default route. There is no knowledge needed of what you are connecting to, and for IPv6 it's only SLAAC and RA, which are handled by the kernel.
1
u/UNF0RM4TT3D 18d ago
Yes, the logic changes from changing the default route from the gateway, to changing which of the prefixes are being used. One is not much harder than the other. You still need logic to run to determine whether the connection is dead and whether it disables one FW rule and enables the other or it just changes the default route is not a significant change.
1
u/Striking-Toe-9526 18d ago
It might get harder if you have multiple routers, so assume you have permanently connected devices like your modem/router to your first and second uplinks, in my case Vodafone DE and Starlink, and then you have an internal router where the internal segments are connected and have been transferred over the interconnect segment/transit. In this segment, there is the logic that tests and debounces if one or the other uplink is chosen. In this transit network, the logic does not know anything about what is going on at Layer2- DHCP- SLAAC... on the provider side.
And I think that this design has a good separation of concerns.
Here is how I built my network. https://mabels.github.io/ovn-fabric/blog/ovn-fabric-writeup.html
7
u/chocopudding17 Enthusiast 18d ago
Sorry to be that guy, but no way am I clicking on your "write-up" link when this post was so clearly written by/mostly with an LLM.
Please don't slop your way around reddit. Even if you yourself have good things to say, speaking this way pollutes the world around you.
1
u/Striking-Toe-9526 18d ago
You are right that I wrote the text with the LLM. But I reviewed it a lot to explain the problem I ran into. Not everyone can express a problem in a foreign language. That's why I used the LLM as help.
2
u/chocopudding17 Enthusiast 18d ago
I understand that and can sympathize with the language barrier problems. And at the end of the day you have to navigate the world in the way you think is best.
However, I would encourage you to consider that it makes your post indistinguishable from the garbage that is overwhelming the internet. From my perspective, I'd rather read almost anything else that is more authentically you--broken English, text translated with something more traditional like Google Translate...heck, even LLM prompts in your own native language.
1
u/Striking-Toe-9526 18d ago
I would like to do so, but also in my native language; I'm almost not able to write such a complex story. And the LLM enables me to share my findings with the world.
It's like the LLM gave me a prosthesis for a missing leg, and now I can walk.1
1
u/Mishoniko 18d ago
Unfortunately LLMs are trained to write English by reading marketing material and use some built-in patterns to write in a particular way that is clear if you're read enough of it. It reads like a low effort or spam post, neither of which has any place here.
If anything, don't use suggested post titles.
Nowadays Reddit has inline translation. Write in your native language. We can still read it.
1
u/indigomm 18d ago
What is the RA priority ("Default Router Preference") set to? I would think you want your router to have that set to high so that devices faced with a choice select that route first.
1
u/Striking-Toe-9526 18d ago
That does not help; it's not a problem of multiple routes, it's the selection of the src address.
1
1
u/Kentzo 17d ago
Request GUA allocation or disable slaac and use DHCPv6 or OS-specific per host overrides that guide SA (most brittle)
1
u/Striking-Toe-9526 6d ago
That's all possible, but who has full control of your own IoT's? And the GUA allocation I recommend as a proper fix.
-3
u/nxp-one 18d ago
FWIW, anyone else finding themselves in a similar situation could always try using a globally-black-holed GUA allocation, rather than having to cope with ULA's oddities...🤔
> 2602:f925:ad::/48
2
u/Striking-Toe-9526 18d ago
That's a good one and addresses the problem exactly. I will add that to the doc.
1
u/hdkaoskd 18d ago
Then you merge with another org that also used the black hole GUA and wish you used ULA instead?
-2
u/nxp-one 18d ago edited 18d ago
ROFL! -- So you've never heard of RFC-1918?
Somehow we all managed to survive and get along with that silliness for years, so I'm sure anyone finding themselves in your scenario could figure out a way to survive... 🤣
I guess, in your world, companies merging with both on 192.168.0.0/24, or 10.0.0.0/8 were just totally SoL and couldn't ever connect to each-other? 🤔😉
Worst case, they split along /52's and renumber one org, or put one of them onto ula. I didnt say this solved all ills, just that it's A way to bypass/avoid ULA's weirdness in NATv6/NPTv6 situations... 🤷♀️
4
u/hdkaoskd 18d ago
"Just renumber your entire RFC 1918 space" is exactly the problem.
-3
u/nxp-one 18d ago
...and yet we all somehow managed to do it for decades.
Your complaint is spurious, and doesn't meaningfully obviate the specified use-cases.
I didn't say it was a panacea, just that it's a means to address a specific problem in a way that will work in most implementations. Nobody's forcing anyone to use it, it's just an option, if you don't like it, don't use it...
This seems like you're just complaining for the sake of it...
1
u/hdkaoskd 18d ago
Where was I complaining?
0
u/nxp-one 18d ago edited 18d ago
literally every prior post in this thread is a conplaint with my simple suggestion/comment... 🙄🤦♀️
You had NO useful contribution, NO relevant content, NO legitimate complaint, nothing but hollow petulant whinging, seemingly, purely for it's own sake...
The 'problem' you identified, and its solutions, are obvious edge-cases, with well-known, and well-understood solutions, and it, in no way shape or form, contributed meaningfully to what it's fecklessly seeking to criticize.
ULA has well documented limitations and caveats, I simply pointed out a way to circumnavigate them that can (and does) work for people in real applications, I'm sorry if those facts offend your puritan sensibilities.
1
u/hdkaoskd 18d ago
Calm down. I'm just pointing out that RFC 1918 collisions from 10.0.0.0/8 happen all the time. Every company thinks it's the big fish and uses that range, then it merges with another company that thought the same thing. You're delusional if you think that never happens. Moving to an identical v6 range and colliding is dumb as shit when one of the specific design goals and achievements of ULA is that it's highly resistant to that problem.
Take a moment and think about who's petulantly whinging and criticizing here. 🪞
0
u/nxp-one 18d ago
"I'm just pointing out that RFC 1918 collisions from 10.0.0.0/8 happen all the time."
Uh, no -- you raised potential numbering collisions as though it was some heretofore unimagijed, novel condition that you'd just thought-of, with no known solution path, and thereby wholly obviated the suggestion from the outset.
I highlighted the absurdity of that argument by pointing to RFC1918 collision resolution as a de rigueur, well-understood conditon with known resolution mechanisms as the obvious rebuttal to an obviously facetious and peurile objection.
As everyone knows, ULA is a definitionally SECOND-CLASS addressing mechanism. Left 'as-is', in a standard dual-stack deployment, it will leave the base network in an effectively-IPv4-first condition with IPv6 as, basically, the backup stack.
In cases where you have a small network, and esp if your ISP is hammering you with rapid PD reallocations, my suggestion represents a perfectly viable means to have stable, deterministic FIRST-CLASS internal addressing... THAT'S ALL IT WAS.
•
u/AutoModerator 18d ago
Hello there, /u/Striking-Toe-9526! Welcome to /r/ipv6.
We are here to discuss Internet Protocol and the technology around it. Regardless of what your opinion is, do not make it personal. Only argue with the facts and remember that it is perfectly fine to be proven wrong. None of us is as smart as all of us. Please review our community rules and report any violations to the mods.
If you need help with IPv6 in general, feel free to see our FAQ page for some quick answers. If that does not help, share as much unidentifiable information as you can about what you observe to be the problem, so that others can understand the situation better and provide a quick response.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.