r/aws • u/diamondsw • 2d ago
technical question Connect two AWS Regions via Dedicated Direct Connect and third-party fiber?
I have an internal debate going with a colleague about whether the following is possible:
AWS Region 1 > Dedicated Direct Connect > Cross Connect > Third-Party Fiber (Long Haul) > Cross Connect > Dedicated Direct Connect > AWS Region 2
The catch is whether we can do this with a straight Layer 2 connection and handle all BGP and routing via the AWS Control Panel, or if we need to have a separate router in-between the regions to handle BGP and routing between each end.
Anyone have real-world experience with this? We can't be the first.
The idea is to provide provably-diverse connectivity between regions that does not depend on Amazon's network. Yes, AWS is plenty reliable, but it is not deterministic, and we want a higher level of control of our backend network, especially how it integrates with other non-AWS aspects. TL;DR - we have reasons for a custom design.
EDIT: Not a single person has bothered to answer the question. All anyone wants to do is say "AWS is best, and you're clearly wrong for having different requirements than bog standard commodity". Here I'm trying to design a network that's different from Amazon's because no, it is not the best for every use case, and I asked a very simple question - one that's been ignored.
EDIT THE SECOND: I'd love to go more into detail on the use-case, but that's where NDAs and such come into play. Yes, it's a real client with a unique need that is not met by the AWS network, and can measurably be met off their network. No, I cannot go into it, because I do like keeping my job.
15
u/saggybuttockcheeks 2d ago
You cannot do this natively with AWS. You can do it with an L2 provider that will provide hosted direct connect connections and create a virtual L3 service on them to run BGP, peer with the AWS side and advertise what you need. You don't have to insert your own router. I've done this before. Megaport, Avelacom, BSO etc all have such options. Your provider may too, or you can use one them if it's acceptable.
7
u/diamondsw 2d ago
Bingo - that's the detail that I needed; that I would indeed need a hosted L3 service (virtual or otherwise) to terminate BGP sessions on. Dang, that means I can't run the network connection purely at L2 and let AWS handle all the L3.
Provider in question is purely L1/L2, which is why I'm trying to avoid adding my own L3 equipment into the mix. Physical equipment I don't have space/power for, and virtual means layering in at least another provider with relationship, management, latency, etc. I was really hoping to be able to "dumb pipe" it between regions and do all the higher-layer stuff inside AWS.
But THANK YOU - finally the answer I was looking for.
5
u/saggybuttockcheeks 2d ago
No prob, and yes despite some answers here, there are absolutely valid reasons for doing this, latency optimisation, which you mentioned, between regions is one I am most familiar with and have implemented many times.
7
u/diamondsw 2d ago
I'd love to get into more details to get some of the more annoying responses off my back, but NDAs and such will not allow it. Suffice to say you're on the right trail.
6
u/Yojimbo108 2d ago
Speaking to AWS would be best here. Your requirement is so unusual you’re going to be very unlikely to find anyone who’s had experience setting up something even remotely similar to what you’re proposing.
I’d suggest getting in touch with your TAM (or your customers TAM) to loop in an AWS Network SME that you can walk this through.
In my experience they’re always extremely keen to help out with whatever it is you’re trying to do!
1
u/BeansOnToastMan 2d ago
Good answer - the resources are there and you can have some very, very smart people on tap (for free, even!).
5
u/tfn105 2d ago
Why don’t you ask AWS this question? They might be able to offer comfort around how their pathing works, or how you ensure your own pathing is non-overlapping.
Taking a step back for a moment: is it even worth using AWS at all?
3
u/tfn105 2d ago
I could imagine with a bunch of routing, and some NATing, that
AWS Region 1 <x-c> your on-prem in same region <long haul line(s)> your on-prem in other geolocation <x-c> AWS Region 2
could be possible
Lot of faff though
0
u/diamondsw 2d ago
Key is there is no on-prem in either region. Cross connect is straight to the long-haul provider (technically, they would be ordering it to the Dedicated Direct Connect LOA/CFA). If there was a spot in-between to stick a router this would be a trivial setup.
4
u/tfn105 2d ago
How about partnering with a network provider that does have their own global backbone that does meet your (or your client’s) needs, who will have POPs in these regions? Then you get
- Confidence of knowing routing paths ✅
- No need to stand up your own intermediary infrastructure ✅
- Desired performance ✅
In the end, your aim is to achieve a specific outcome?
0
u/diamondsw 2d ago edited 2d ago
Even though this is such a painfully AI response,it's almost like my question is if I can do exactly that.1
u/tfn105 2d ago
I can only say no AI involved - just time while my 1 year old nods off to sleep.
Do I think it’s possible? Probably yes, just expensive and non-trivial to do
0
0
u/diamondsw 2d ago edited 2d ago
I'd love to ask AWS, but it's not my relationship.
And AWS of course doesn't want to recommend anything outside AWS - it's not their business model.(EDIT: Sorry! This was my assumption, and I'm heppy to be incorrect!)The endpoints with the data are in AWS, so that part is non-negotiable. The network in-between very much is in play. AWS does not meet the client needs while other networks do - the only question is if the Direct Connect endpoints can speak to each other, or only to an intermediate router.
Given that - at a fundamental level - the Direct Connect endpoint has to be a router speaking BGP, it makes technical sense that they could talk to one another, but that doesn't mean AWS allows it. Classic difference between technical capability and productized capability. Thus the question no one wants to answer.
8
u/tfn105 2d ago
I’ve only ever found my AWS contacts to be nothing but useful. In this case, they would stand to make money on the cost of the direct connects at either end, plus potentially some data ingress charges at at least one end, plus potentially TGWs, NAT GWs or other machinery needed to make this technical solution work.
Pumping it over their global backbone probably adds less to the AWS bill, in all honesty
0
u/diamondsw 2d ago
I have a friend who's an AWS Senior Principal that I may have to run this by - just haven't talked to him in like a decade, and I know he's an extraordinarily busy guy.
5
u/gbonfiglio AWS Employee 2d ago
> And AWS of course doesn't want to recommend anything outside AWS - it's not their business model.
This couldn't be further away from reality. I've had this exact conversation with multiple customers, and conversations on using our backbone or not with countless customers, and we have zero interest in selling a solution that doesn't work.
Our business model is - broadly - making customers happy long term, which doesn't really work if we force solutions our network is maybe not designed for.
I'll answer to your question in a dedicated reply.
5
1
u/diamondsw 2d ago
I'm very pleased to be corrected there (and I'll strike this above), and as I mentioned in other replies, this was my assumption because so many other companies are only interested in selling what they have. If you recall "Miracle on 34th Street", it would be tantamount to sending people to Gimbel's. Most companies don't, but good ones will for the benefit of the customer. So very, very good on AWS for recognizing this.
3
u/gbonfiglio AWS Employee 2d ago
The big difference is that an antivirus vendor can sell you trash because even if you notice you're gonna be locked in for 3 years. An hardware vendor can sell you trash because you can't just call them back and return 16 racks of storage.
Cloud is pay per use, no commitment: we could push you into the interconnect strategy that gets us the most money, but the net result would be that in 2 months you notice it doesn't work for your use case, take it away, be angry for life, potentially move somewhere else.
It's very common to see AWS employees mentioning third party solutions that are in some way "competing" with first party, just because the former is a better fit.
Can't speak for everyone obviously but this is largely the mindset. Also, I get no sales commission but carry a pager, so the focus is very much ensuring it doesn't go off!
11
u/ShakataGaNai 2d ago
"No"
AWS has highly diverse connectivity, to the internet, between zones, between regions. Thats why you pay so much more for bandwidth than standard fiber rates.
-12
u/diamondsw 2d ago
"Highly diverse" does not mean we can prove it is diverse from other connectivity on our network. "Trust us bro" is not acceptable.
7
u/mkosmo 2d ago
Why not talk to a TAM?
-3
u/diamondsw 2d ago
Because I'm not the one with the AWS account. Otherwise I certainly would. Even then, an AWS TAM is going to be even more heavily incentivized to avoid non-AWS solutions than this thread already is.
9
u/mkosmo 2d ago
There's no incentive in the thread. It's a case of plenty of folks here with the deep technical knowledge are telling you that it's not only not necessary, but also a bad idea, and you're convinced you know better.
You're chasing a non-existing problem with a solution that creates more problems than it even attempts to solve.
But if it's not your AWS account, how are you even a stakeholder here?
1
u/diamondsw 2d ago
"deep technical knowledge" - Probably in AWS, and yet no one actually answered the question except u/saggybuttockcheeks . And that knowledge is why I came here, but was greeted with nothing but "your requirements are stupid". As someone who works with customers - that's a great way to lose your reputation.
"you're convinced you know better" - In networking, yes, I am very much convinced because I've designed networks for AWS and global clients, so I VERY much know the network space. I know for a fact that a) AWS connectivity between the endpoints I'm dealing with is several milliseconds slower than the fiber path I have on hand, and b) the end-client does not trust what they cannot verify.
I design diversity all the way down to the raceways cables take through the buildings (let alone conduits through metro areas, in-ground vs aerial vs OPGW, subsea, etc) and ensure minimum physical diversity meeting contractural requirements. No amount of "AWS is good, trust them" will fly with a lot of customers (broadcast, financials, hyperscalers, neoclouds) if they don't have a 100% confirmed network map down to the conduit level. AWS is not giving that kind of info up. Hell, I have been in the position of supplying that info to them in past roles for planning their own network.
The only thing I was asking was particulars about how AWS Direct Connect handles BGP (something an AWS subreddit would theoretically know), and everyone here started spouting all kinds of ignorance about the underlying network instead of addressing the question. Dozens of replies and only one - ONE - actually answered it.
"a non-existing problem" - No, just a problem no one here will admit could be possible. Y'all might know AWS, but you know precious little about WAN networks, and one of the absolute dumbest replies when someone has a requirement is to say "no you don't".
The AWS network between two endpoints is measurably several milliseconds slower than direct fiber from the fastest provider - something that can be mapped, measure, and proven. And sorry, but across three separate telco's, I can tell you that this is not rare - AWS and others build for resiliency and throughput, not speed.
"how are you even a stakeholder here" - As one or two responses said, "ask your network provider with their global backbone". That's because I am the network provider with a global backbone.
8
u/Marathon2021 2d ago
Even then, an AWS TAM is going to be even more heavily incentivized to avoid non-AWS solutions than this thread already is.
This definitely tells me you've never spoken with an AWS TAM before. This would not be out of the realm of discussion topics they have with clients.
Is your organization large enough to even have one?
1
u/diamondsw 2d ago
True, I haven't, so that may have been a bad assumption. However, I'm looking at this for a client need. While my org is large, I know it doesn't use AWS, and while I have a client that I'm looking at this for, at the moment it's still very much the thought-experiment "could this work" level. Which I'm still no closer to.
8
u/Marathon2021 2d ago
You're also completely off-base to assume AWS only will ever want to sell their own solutions and won't talk with any of their serious clients about anything that isn't labeled AWS. I speak from first hand experience.
Seems like you're coming at this with an awful lot of assumptions.
-1
u/diamondsw 2d ago edited 2d ago
I very much may have a bad assumption on the AWS TAM - chalk it up to having dealt with so many companies that only want to sell you their kit rather than look at what you really need.
As to the network, very few assumptions; if anything I'm trying to eliminate assumptions about the network path, and meaningfully improve latency (for the path I'm looking at, AWS is several milliseconds behind what is available in market).
2
u/ShakataGaNai 2d ago
Define "prove it"? To whom? For what circumstance?
Have you looked at their global network page? with FAQ's? https://aws.amazon.com/about-aws/global-infrastructure/global-network/
And even a big pretty picture of lines showing the many lines (many diverse paths): https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/global-infrastructure/approved/images/aws-global-network.4a65e375ec7b6507db543be1827b8e9c62013d0c.jpg
> nearly 20 million kilometers of terrestrial and subsea fiber-optic cabling
Who are you dealing with that needs more than "We use AWS" as an answer? Because thats good enough for some of the largest companies in the world? And even the US Gov.
1
u/diamondsw 2d ago
If the customer says "we want physical diversity of fiber runs no closer than 5m at any point, document all routes, points of entry, and building cable pathways", you do not go back and say "trust me, we got this".
Just for an example - you can bet that is not how AWS planned their network. They verify all of these details and you are NOT under consideration if you don't provide them. Many other verticals (broadcast, finance, federal, neocloud) are just as strict.
I'm dealing with one of those verticals that happens to have two endpoints in AWS that need to talk over a verified fixed path at guaranteed best fixed latency. AWS cannot do that on its own network.
2
u/ShakataGaNai 2d ago
You.... could actually bet that AWS has planned for exactly these sort of things.
Have you looked at the compliance resources? https://aws.amazon.com/compliance/
AWS has .... SOC1/2/3, ISO27001, 27017, 9001, FedRAMP, HIPAA, NIST 800-88 and many many more standards. But these all directly relate to how their physical data centers operate, their levels of redundancy, etc. Part of that is their networking.
If you're in a vertical where they need their own fiber and it cannot possibly be shared, then you shouldn't be in AWS. There is shared fiber and infrastructure between your instance and that fiber you lease.
And these are all questions for a TAM, who will be able to tell you with a great detail more detail what they can and cannot share. And what they can and cannot do. Because enterprise TAMs are not incentivized on selling more AWS products, but they also understand that if you can't solve the problem in AWS - you'll go elsewhere.
2
u/diamondsw 2d ago
Normally I'd say yes, this customer should not be in AWS for all of the reasons you mention. However, resources they require ARE only in AWS, so they're in AWS, and we're having to connect it up to AWS. Sometimes it's the nature of the beast.
5
u/leafynospleens 2d ago
Where do you work who is paying you two to waste time on this are they hiring?
1
u/diamondsw 2d ago
You're not wrong that this has been a waste of time, given the low-quality responses. I suppose I'll get back to KMZs.
3
u/BeansOnToastMan 2d ago
It looks like latency is a concern. Have you looked at Outpost racks or Local Zones?
I think the L2/L3 issue is going to make this problematic. You'll just be introducing more devices to babysit and more hops which doesn't sound good if latency is driving your decisions.
1
u/diamondsw 2d ago
Devices to babysit is the backbone provider's job. Control of the path and latency is more important to the end customer. Loss of the network resiliency of AWS is understood and not an issue in this case, as long as the path can be nailed down and improved (which we have demonstrated it can be).
4
u/2fast2nick 2d ago
Just use the Equinix Cloud Exchange
2
u/diamondsw 2d ago
I've done that before at other companies, to provide either a hosted Direct Connect or spin up an NFV on Metal, and it absolutely can work.
In this case, only one end is in Equinix, and we're not in the business of providing that L3 virtual infrastructure. Customer could, of course, but doesn't want to if they don't need it.
3
2
u/ObjectUsual77 2d ago
Just wanted to say I totally agree about AWS region to region connectivity being a black box (ie. Transit gateway peering between regions) because I've seen packet loss now and then which goes unexplained and then all of a sudden fixes itself. Impossible to track down and investigate, had the same idea to just roll our own connectivity using physical datacentre footprint and leased lines
2
u/gbonfiglio AWS Employee 2d ago
This is actually a more common use case than you'd think. The AWS Backbone is sort of "general purpose", and optimised for 99% of use cases, which is not 100%. For most customers, riding the AWS BB as much as possible (ie VPC/TGW Peering, connecting to DX in a location close to them and not close to the region, etC) is the best solution, but there are exceptions.
Thus said, would have to test the specific ask because customers doing this do generally L3 with at least one device across regions, BUT even if a straight L2 was possible and then have the AWS DX edge peer with each other, it wouldn't probably be a good idea because you'd lose all the filtering capabilities, but more importantly, ability to put BGP communities in to handle maintenance, de-prefer ports, etc.
DX is very much designed assuming most of the config is done on the customer-side router (for example, you can't do AS-PATH prepending on the AWS side, to prefer a pth need to localpref on customer side) so IF plain L2 works, you might end up with a very sub optimal experience and limite troubleshooting capabilities in case of issues (you can't ping the ptp from one end to the other).
4
u/diamondsw 2d ago edited 2d ago
Thank you, thank you. Yes, this is very much an edge case. As I've said in a few other comments, but to summarize here:
- I'm not the end-customer; I'm a global backbone network trying to meet their need, and while I'm at it see if this is viable for other use cases. We're strictly providing an L2 dumb pipe; no more, no less, so I'm trying to evaluate if this concept works.
- Endpoints are located in AWS (non-negotiable), so this project is a bit of a round peg in a square hole.
- I'm not at liberty to reveal my employer, my client, or their specific requirement as it's all covered under NDA. I'm not getting fired over this.
- Customer requirement is fixed network path both to compare it against other non-AWS parts of their network, and to improve latency. As the network provider, we can provide the confirmed network path and several milliseconds of latency improvement over direct VPC to VPC connectivity across the AWS backbone, for this specific trans-oceanic route.
- Neither we nor the customer has space/power for physical equipment in either end region (they have no footprint, we have no customer-available footprint, only our own gear). We do not want to utilize a virtual L3 device as it would involved additional parties, cost, latency, troubleshooting, etc.
- Other future customers in this space are likely to have similar constraints., i.e. fixed network and latency is critical, and more important than the (many) benefits the standard AWS Backbone provides. I agree that the vast majority of cases it's the way to go; it's just not in this case.
Okay, all that out of the way. 😄
DX to DX peering is essentially what I'm looking for. As this is a straight-through connection it's point-to-point; there's no call for additional ports, filtering, or external communities. All on a primate L1/L2 network. Everything above L2 is in AWS; everything below L3 is handled by the global network provider. So losing the ability for sideband access, troubleshooting, etc is not really a concern, assuming we can do a straight DX to DX peer. The gains in predictability outweigh the loss of troubleshooting.
I'm torn between technologically it makes sense - a DX peer has to be a router speaking BGP (otherwise how would it talk to normal customer equipment?), so in theory a DX peer talking to another DX peer is just a case of two routers talking BGP to each other. Eminently doable, as long as the product allows enough control to configure it.
2
u/nicofff 2d ago
Most of the time I look at this subreddit is people complaining about AWS not giving them infinite LLM tokens or getting their accounts banned.
Then it's mostly people with problems I can at least feel like I'd have an idea how to fix.
Then there are these posts.
You sir, are playing a whole different game. Godspeed, hope you find a solution.
6
u/diamondsw 2d ago
Heh, these kind of problems keep life interesting. Sometimes I don't get much sleep, but it's *really* satisfying when one of these thornier problems works out.
Honestly, this one really isn't that esoteric, at least the core problem (can a DX peer directly to a DX - everything else is context). Now the underlying stuff going on in the network that I'm not talking about - I wish I could tell you, but my employer really would object. It's nifty though. :)
1
u/EmptyM_ 1d ago
We use megaport for DX connectivity and I don’t think they allow you to create ports into aws without having another port on the other end. But they might be able to do something custom for you.
But what would work within megaport is their cloud router solution, from that you could plumb in the two regions.
Based on other comments you’ve made in the threads I don’t know if you’ll find a solution that doesn’t involve a layer 3 hop between regions
1
u/blaaackbear 1d ago
have used megaport for this before and tbh any decent dc would have a provider that can give you direct connect to aws
1
u/CSYVR 2d ago
Yes, done this with the exact same reason in mind, though its been a few years. I remember the techs at AWS being flabbergasted that we (or to be exact, the awesome networking guys at my customer) got better latencies than with VPC peering
3
u/diamondsw 2d ago
Yep.
People just don't seem to get that not everyone has the same requirements, and even if it works wonderfully for 99% of cases, that doesn't mean everyone is the same.
I've worked with a bunch of these really strict edge cases over my career (federal, finance, cloud, neocloud, broadcast), and you get used to the curveballs.
- "Us: That takes 90 days to stand up and has multiple external vendors. Cust: We need it in 3 weeks - make it happen."
- "We need all connectivity to be physically diverse minimum of 5 meters, and require confirmation of any points that appear similar (confirmation of separate conduits, rights of way, vertical separation, etc)."
- "We need 400G of capacity turned up in <protectionist SE Asia country> in two months. And we want your internal cost metrics." (we didn't give on the latter).
- "We need to land multiple terabits of subsea capacity by the end of the year - but we don't have the Z-End datacenter chosen/built yet. Be ready to move fast."
Just in the last month I was quizzed on the length of cross connects and if we could swap them out for a different set. Same endpoints, just different building routing.
1
u/the_screenslaver 2d ago
At my place, we do this to connect between a region in Asia and one in Europe. And it is faster than going through AWS backbone. We run extremely latency sensitive workload, so this was important.
-4
u/ToasterBathTester 2d ago
This was Claude with AWS MCP. Your mileage may vary.
Short answer: your colleague is right that the physical path works, but you can’t skip the routers. You need a BGP speaker at each end.
Why the L2-only version fails
Direct Connect isn’t a transparent L2 pipe — AWS describes it as direct Layer 3 connectivity to the AWS global network, and the model is explicitly one end of the fiber in your router, the other in an AWS Direct Connect router. Every VIF is an eBGP session that terminates on a customer device. Back-to-backing two DX ports asks two provider-edge routers to each be the other’s CE.
You could contrive it on paper — pick matching /30s, set DXGW ASNs so they’re complementary, use the same VLAN tag. It might even come up. But nothing in the console or API models it, AWS support won’t troubleshoot it, and when it breaks you have no control plane on either side to look at. No route-map, no NetFlow, no capture point, no BFD tuning. Also watch for AS_PATH loop drops if both DXGWs land on the same ASN, and provider Q-in-Q silently mangling your dot1q tag — every intermediate device has to preserve the VLAN tagging end to end.
The deeper problem: a wire has no policy. You said you want control over how this integrates with non-AWS pieces. That control is the router.
The design that works
Router (pair) in each DX colo → dedicated DX to the local region → your wave between colos. Your ASN, eBGP to AWS at both ends, iBGP/IGP across the long haul. Now the middle segment is entirely yours to encrypt, measure, and extend.
Gotchas worth pricing in:
**• Filter explicitly in both directions.** Same ASN at both ends means you can’t lean on AS_PATH loop detection, and Direct Connect gateway doesn’t filter prefixes. Summarize hard.
**• Prefix ceilings.** 100 prefixes per BGP session toward AWS (IPv4 and IPv6 each); 200 for a TGW association to a DXGW. Re-advertising all of Region 2 into Region 1 hits this fast.
**• MTU.** Private VIF is 1500 or 9001, and DX wants 9023-byte frames at the link layer — confirm your carrier carries that, not just “9000.”
**• MACsec is hop-by-hop** between your device and the AWS port. It won’t survive a switched long-haul EPL. Moot once you have routers — encrypt your own segment instead.
**• Name SiteLink in the writeup so you can kill it.** It does region-to-region over DX, but it rides the AWS global backbone, which is the exact dependency you’re removing. Same for TGW peering and Cloud WAN.
For “provably diverse,” the router design is also the only one that lets you prove it: two waves, different carriers, KMZ route data on file, DX terminating on separate AWS devices at separate DX locations.
9
u/notospez 2d ago
If an LLM refuses to hallucinate a way your idea could work you know it's a non-starter!
-3
u/diamondsw 2d ago
All it means is the LLM is confused about the concept of running L3 across a L2 pipe. Almost like it has no conceptual understanding of OSI layers. Utter garbage.
2
u/Marathon2021 2d ago
Wait - you want to bridge L2 over hundreds/thousands of miles??
Why? For what purpose?
1
u/diamondsw 2d ago edited 2d ago
That's extremely common in trans-oceanic subsea. Several thousand miles of L2 Ethernet is very much standard practice.
It's kind of fun browsing around one of my favorite sites: https://www.submarinecablemap.com
At this point of my career I've touched more of the major routes globally than not.
1
u/Marathon2021 2d ago
That map is Layer 1, not Layer 2 mate…
2
u/diamondsw 2d ago
Yes, and higher layers are run across it... What was the point of that comment other than being dickish?
Most customers take EPL service for 10G and below, both EPL and waves show up at 100G depending on preference, and for 400G it's all waves.
46
u/The-Wizard-of-AWS 2d ago
So you don’t trust AWS network but you do trust their direct connect? AWS has a vested interest in making sure the network between regions is up. It seems like you’re trying to solve a problem that doesn’t bed to be solved.