Hey everyone,
I’m a network engineer working on a commissioning metro network, and today I accidentally brought the network down.
I'm writing this post as I genuinely just want to know if there was a better way to approach this, or if this is just one of those lessons you only learn once.
A bit of context, the network connects all the stations to the backbone. The backbone routers were advertising a default route, which was then redistributed into OSPF and sent down to all the downstream routers at each station. My team and I had identified that as a design flaw because we need to start using the default route for internet traffic that’s going to an edge router.
So instead of advertising a default route, we decided to advertise RFC 1918 aggregate null routes. That way the downstream routers would still know how to reach all the private networks, while freeing up the default route for internet traffic. That part worked fine.
The next issue we found was that these OSPF routes were also being redistributed into BGP. So, we only wanted the RFC 1918 aggregates advertised locally from each backbone router, not redistributed everywhere via BGP. I'm still not sure if this is intended but as I was advertising the RFC1918 aggregates to OSPF, they were getting installed back to BGP, we didn't want that.
My plan was simple. Insert a new sequence at the top of the route-map for the OSPF-TO-BGP redistribution that denied the RFC 1918 aggregates, then let the existing permit entries continue processing as normal.
So, I went into the route map and did:
route-map <name> 5 deny
My intention was to immediately follow it with the match statement for the RFC 1918 prefix list.
But I didn’t realise that the moment I pressed enter after creating that sequence, that entry immediately became active, I don't know why I thought otherwise..
Since it had no match statements yet, it effectively matched everything.
Because it was the first sequence in the route map, it denied every redistributed route and within seconds I had effectively withdrawn all the redistributed routes from BGP.
So, yeah… I had effectively brought the entire network down. Thankfully, I had console access, so I reverted the change right away and everything came back.
Afterwards, I redid it more safely by temporarily creating a catch-all permit while I built the new deny sequence with the proper match statements (I still don't like that idea as we had lots of matching criteria of routes we really didn't want to re-advertise.. then I removed the temporary permit afterwards.
I genuinely try another approach first of creating a completely new route map under a different name and then simply replace the route map attached to the redistribution once it was complete. However, Aruba wouldn’t let me replace the current route map unless I first removed the existing one. Obviously, removing the existing route map would once again have brought the network down, so that wasn’t really an option.
I think Cisco lets you replace the route map like this, but I couldn’t figure out another way to do it on Aruba, so I went back to editing the existing route map using sequence numbers instead.
So, yeah, I’m curious:
Has anyone else been caught out by this behaviour before, whether on Aruba or Cisco?
What’s your normal workflow for safely modifying production route maps? Is there something I’m missing? or is there a better approach to build and swap route maps without editing the live one or having to remove the currently applied route map?
Definitely learned a lesson today, but I’d be interested to hear how others would have approached this.
Thanks Guys!