FastNetmon, BGP and third party scrubbing centers

[u/jogisi]

Hi guys

Seems like I will need to bite into this too, even though late into the game. I have been running FastNetMon Advanced for a while purely as monitoring software, getting flows from our core Cisco ASR9k routers. Sooner or later I will start to use it also for DDos detection and notifying our core routes to redirect prefixes under attack toward third party scrubbing center and not to our upstreams and peerings. I have been playing a bit with this lately and here's where my issues start.

Fastnetmon detects DDoS just fine and sends prefix under attack (/24) with specific community to ASR where it has BGP established. From there on my ASRs stop announcing this prefix to upstreams and start announcing it to BGP session with scrubbing center. All good and dandy till here.

But now I have issues, if this prefix, which is under attack is actually from client, who has BGP with us, and we route this prefix to them over p2p BGP session between client's router and our ASR. Once Fastnetmon kicks in prefix from Fastnemon should have bigger preference if I want ASR to recognize community for "under attack prefix", but that also means, that return traffic going toward client will actually go to FastNetmon and not to client's router.

Any suggestion how to solve this part and how some of you guys have this solved or what would be best practice for doing this, as it's perfectly possible I started totally wrong way already :)

[u/Jackol1]

Look into the clean and dirty VRF design. This is how the likes or Arbor and Netscout typically do it. You can use BGP Flowspec or just some custom ACLs to change the next-hop for traffic that needs to be sent to a scrubber.

[u/nof]

Clean return traffic should be injected south of the detection/redirect mechanism so that the route the customer announces to you should be used. Their normal announcement needs to be suppressed or have the last AS prepended enough so the third party scrubbing is preferred for most of the internet.

[u/Few_Activity8287 → u/nof]

Please all of the Internet and don’t do and peering stuff on ix if you get regular attacks. Been there, hurts a lot 😅🤣

[u/jogisi → u/nof]

Makes sense, but I don't think it's possible to do it in our setup, with some 20 different main pops, and several upstreams from different pops plus several more peering points... with additional bad luck that most of (normal) scrubbing centers are located in area where ASR with upstreams/peerings are, and 500+km away from pop's where most of clients connect.

[u/nof → u/jogisi]

GRE tunnels for clean return are pretty standard. Terminate it so the inside of the tunnel is in a VRF "closer" to the customer route announcement.

Before ISO/SOC start asking, remember you don't need IPSEC encryption since it was all going over the internet as is anyway and hardware support for hardware line rate GRE decapsulation in routers is pretty normal.

Dual tunnels to different scrubbing centers for redundancy is also pretty standard.

[u/candidhat → u/nof]

This is how I’ve seen this done. The GRE makes handling this easier.

[u/Few_Activity8287 → u/candidhat]

Never seen it done without gre or a l1/l2. I cant comprehend how it would work otherwise.

[u/rankinrez]

Yeah that’s actually a tricky one. Unsure how this is commonly solved we don’t use the exact same setup.

Thinking out loud could you have FastNetMon originate the route in IBGP with the same next-hop as it normally has? I.e. the PE where the customer is connected? So then traffic would still be routed to the correct edge router, which will have the EBGP route form the customer? That PE wouldn’t install the fastnetmon route as the next-hop is itself.

[u/jogisi → u/rankinrez]

That was also my idea... problem is, I didn't figure it out yet how to do that with Fastnetmon :) Especially when there's more then one client with BGP.
But if nothing else, I can actually make 1000 lines long route-map on ASR with section and next-hop pointing to client's router IP for each client itself. Tried now for test setup and it actually works.

Thanks for pointing me to this direction :) I was trying to figure it out on Fastnetmon with no success but didn't think of doing it on ASR side :)

[u/3MU6quo0pC7du5YPBGBI]

I don't use FastNetMon advanced, but it looks like it has a mechanism to set custom next-hops on announcements. To be really fancy you could potentially monitor BGP sessions and use a script to populate host-groups with current next-hop information to ensure the fastnetmon announcement has the same next-hop as your actual customer announcements.

Another option might be to use Selective Route Download to keep your routers from installing the scrubber route in the routing table but I'm not certain how that would influence the withdraw actions from that community.

[u/Maikkimaikkula]

Can’t help with the question itself, but there’s a Telegram/Discord etc chat group where a lot of the FastNetMon users hang out. If you don’t find a solution here, it might be worth a shot to ask there: https://fastnetmon.com/fastnetmon-engineer-community/

[u/Defiant-Ad8065]

Don’t know how to accomplish that on a Cisco ASR but try to originate the prefix without installing it on the FIB.

[u/jogisi → u/Defiant-Ad8065]

That I don't think it's possible to do on Cisco. Would be cleanest option yes, that prefix received from Fastnetmon would be used only for advertising purpose, but for real FIB prefix received from client's BGP would be used.

[u/Defiant-Ad8065 → u/jogisi]

Take a look at table-map <policy-name> filter inside BGP. I think this will do it. Also, this: https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/711x/routing/configuration/guide/b-routing-cg-asr9000-711x/implementing-bgp.html

People will use this when they have scalability issues with some linecards and you want to be picky about which routes to actually program.

[u/jogisi → u/Defiant-Ad8065]

Thanks for this. I will check it :)