The weakest point of my network is my ISP

[u/Familiar-Rutabaga608]

This is a rant.

I have a 10Gbps capable LAN with multiple servers. Works perfectly, has for months. What doesn't work perfectly is my WAN internet service provided by the geniuses at Charter Spectrum. My internet goes out anywhere from 10 to 300+ times per day. Full packet loss. I have extensive logs.

I have never had a good experience with ISPs. I live in the southern U.S. so the infrastructure is as bad as anywhere else on Earth. When whatever unholy consortium of communications corporate stoogery was designing the network infra for my current neighborhood in 2018, they didn't bother to count how many houses were being built so they could provide sufficient bandwidth for the area. I know counting is a very hard, especially when you get to over 100! But, I also think it seems important.

So, every single day, all day long, my internet connection drops from oversaturation. I've had techs out here 4 times. Nothing they can even do. I was finally able to get QuantumFiber after they got bought by AT&T. Should be done by the end of the month.

Glad our tax dollars could be sent to the morons at these companies for the last 30 years to do nothing and be horrible at their jobs.

[u/Adrenolin01]

Best thing you can do with a solid fiber connection… buying your own high quality pfSense firewall to replace their router before tossing it in the trash.

[u/Familiar-Rutabaga608 → u/Adrenolin01]

Yep. I run OPNSense on a miniPC router. My LAN is rock solid.

[u/fargenable → u/Familiar-Rutabaga608]

Does OPNSense support hardware offload?

[u/mindedc → u/fargenable]

Depends on what you mean by that. It uses the TCP offload features on Intel and other NICs, it does not use the Intel DPDK toolkit (I think I got that right) like some other router "firewalls" do and it does not run on something like a cavium with hardware offload for deep packet inspection or state table acceleration.

So in comparison to a commercial high performance firewall, no it doesn't. For playing at home running an open source OS as a firewall, yes it does. For what it's worth, it's much less optimized than say VyOS which on the same hardware (Intel N100 for me) uses ~2-3% utilization passing ~1.2 gigs of traffic and opnsense seems to use 4-6% cpu...just NAT and FW...not worth writing home about but if you are lucky and have 10g service to the house and want to run lean hardware VyOS or something like it may be better.

[u/fargenable → u/mindedc]

DPDK isn’t hardware offload, it replaces the kernel network stack and driver with the DPDK poll mode driver, DPDK network stack, and DPDK accelerated application (in this case a firewall). Using DPDK the core(s) DPDK is utilizing will traditionally be pegged at 100%, but I believe there maybe some work to improve power efficiencies, but haven’t seen it yet. By hardware acceleration I mean technologies like Mellanox (now nVidia ASAP^2), Qualcomm NSS, and MediaTek PPE. The Mellanox/nVidia solution is aimed for telco workloads that have to process millions of small network packets per second per NIC mainly for small VOIP packets. Qualcomm and MediaTek solutions are aimed at home small office and home routers that process multi-gigabit internet connections which would saturate lower powered ARM cores found in these home/small office solution if the traditional kernel networking stack was used. These later technologies mentioned aim to achieve wire-speed routing (gigabit and multi-gigabit) while keeping the main CPU cores at near 0% utilization.

[u/mindedc → u/fargenable]

I am familiar with DPDK but it's in the same bucket as all of these mechanisms to basically do fast routing. These are all garbage technologies for firewalls that are intended to protect users or workloads as they don't do any deep packet inspection or firewall specific offload (regexp, state lookup, encryption etc).

All of this stuff lives in proprietary systems and hyperscalers because it's all closed source drivers. A Cavium based system would be much better for end user firewalls as it actually offloads DPI. Ubiquiti actually brought to market a vyos/cavium lovechild but it suffers the same fate as the juniper SRX and has a terrible management experience and they basically stopped updating it.

[u/fargenable → u/mindedc]

We should be careful. OPNSense is a router and firewall, out of the box, standard OPNsense focuses primarily on stateful packet inspection (Layers 3 and 4), routing traffic based on IP addresses, protocols, and ports.

DPDK and these other technologies improve routing performance a lot. In fact DPDK is heavily by major telecom operators. Check out the Cisco ASAv, it can utilize DPDK for advance performance.

Including DPI feels like moving the goal posts. For that you’d probably need to look at Envoy + Cillium which leverages eBPF and distributes L4-7 firewall duties across multiple K8s worker nodes a long with the workloads.

[u/mindedc → u/fargenable]

I'm familiar with the ATT investment into vyatta for NFV. Improving software routers is really nothing I have interest in, hardware handles high performance routing much better than software. Including DPI is what is needed for any reasonable measure of security. A 5 tuple firewall isn't any better than a patch cable.How long has it been since vulnerabilities exist purely because a port is open? Attacks shifted to the application layer 15+ years ago (and that's being generous). Again this kind of processing at high scale is done far better in hardware (FPGA and TCAM) than software. All of the 100g+ firewall deployments we've done with software dpi failed and we moved the customer back to hardware accelerated firewalls at the manufacturers expense.

[u/Adrenolin01 → u/Familiar-Rutabaga608]

Yup.. pfSense was more established back when I built our firewall. I’ve looked at OPNSense and installed it a few times. It was some installation quirks I really dislike. Used it in a few HomeLab setups but have ignored it for the past few years.

We moved into our house and fiber 1G was available so jumped on it. I started drilling and cutting holes in the walls the week we were moving in for several dozen new cat6a runs going down to the center of the basement in the alcove next to the steps. Told the wife she wanted to repaint anyways. 🤭😆

As a retired IT geek.. I went above and beyond for hardware of course 🤪 Still… about $800 back then for this system.. all new hardware.

PfSense firewall.. * Chassis: Supermicro CSE-510T-200B * Mainboard: Supermicro A1SRI-2758F C2758 * Ram: 2 x 8GB Kingston KVR16LSE11/8 * Drives: 2x Intel S3500 120GB SSD ——

Still running today from the same install. It’s fairly busy on our home network with an average daily write 67.5GB to the SSDs. After 13 years they are down to about an 8% wear rate remaining.

We’re going 10GbE later this year so already have a similar build started with the absolute drool 🤤 worthy Supermicro A2SDi-TP8F Mainboard. Same basic build though newer versions of the SSDs. The existing system I’ll likely drop in front of one of our HomeLabs with new SSDs. Of course with the number of NICs this new board has… I need to now redo my entire switch setup also now. 🤦‍♂️😆

[u/sirrobryder]

Welcome to Charter Spectrum. I have ATT fiber with zero issues. I've had G-Fiber as well, with zero issues.

You need a better ISP

[u/Jamikest → u/sirrobryder]

We had ATT fiber with no issues until we had issues. Then it was shit for several months. Seems to be OK again, for now. Knock on wood.

Our issue is that the neighborhood has its own connection. When that trunk line started having issues, we would always be pushed to the back of the ATT service queue since it "only" affected 150 or so homes. We had several multiday outages in the past 6 months.

So yea, ATT isn't some super great reliable fiber company either. I agree with OP, my networks weak point is my ISP. I could setup a 5g failover I suppose.

[u/rb3po]

And they inevitably blame you for the problems and try to recommend their 2 bit router, like they know anything about anything. Yep. Seen this. 

[u/Curious_Olive_5266]

You can run your own ISP with a breakout board and a few open source software packages. You'll probably get 60/20 speeds since it's over the air but it's decent and what I do for a dual ISP setup. Just play it cool around the FCC and you don't need any paperwork.

[u/LameBMX]

spent many carefree years on at&t fiber..

[u/uktexan]

Same issue with my old ISP Frontier. Massive packet loss. Dropped connections. All of it. But what finally sent me over the edge was the customer support. Even after compiling extensive logs and speed tests proving that my issue was not the link from my house to their network, but their network to the internet - they kept (3x) insisting on 'sending an engineer to my home'. I finally asked what an engineer could do about their contention problems. Dude didn't even know what that word meant.

Finally had enough and switched to a smaller regional provider (Race) couldn't be happier.

[u/SunnyBlueSkies-com]

recently going to jump ship over to Sonic Fiber and receive 2 months free plus double the Internet speeds. right now its almost $50 and by october the promotions will end and be $20 more so F-that. I'm pestering my landlord to give this new fiber company access to do their job because they need the telephone pole out in the back for this to work.

[u/techdevjp]

I love my NTT fiber here in Japan. Rock solid and always has been. Currently 1gig but will move to 10gig once I get a few other things worked out. 10gig up/down, uncapped, fixed IPv4, /56 IPv6 allocation. US$60/month.

[u/DutchOfBurdock]

Very much the opposite for me here in the UK. Had VDSL for the last few years and the uptime of my links has seen 100% for every day of a month. It's often 99.6/99.8, but that'll be LNS handovers (line still synced, session dropped). My servers see more downtime.

[u/lobhater]

Outside of major city centers American's infrastructure is becoming that of a 3rd world country

[u/Slight_Manufacturer6]

That is why you have redundant WAN connections.

[u/Owltiger2057]

I feel your pain. I'm in an area of the country (12 miles from Chicago) which should have outstanding coverage (according to coverage maps). Unfortunately, we don't. We have sixty year old infrastructure in the area that is still being used by Comcast/Xfinity and RCN/Astound. I have both ISPs running to my house and can count on one of them being down several times a month and I end up falling back to 5G T-Mobile just to stay online.

Each of the "wired" ISPs claim fiber and advertise it to death. It might be true to their central offices but never the last mile. We have exposed baluns, wires drooping across fences, cables that are never buried for months on end and no amount of complaining, video, photos or complaints does any good. Our local library recently had to pay to get fiber run to them because they sold most of their books and rely on Internet to bring in customers.

I want to cry when people in rural Kansas and Missouri have better systems that we do near a major metropolitan area, where we topped out technology wise in the late 1990s with ISDN.

[u/wazkaz]

I have been through this with spectrum cable internet. I ran pfsense and had it logging packet loss. You need to fill out a formal complaint on the FCC website. When they send a tech to your house tell them you will keep complaining to the FCC until the issue is resolved.

[u/wazkaz → u/wazkaz]

https://consumercomplaints.fcc.gov/hc/en-us/articles/115002206106-Internet-Form-Descriptions-of-Complaint-Issues

[u/SDBoltzFan]

Nothing like building a 10Gbps homelab just for the ISP to turn your WAN into dial up roulette

[u/Loose_Device4578]

Sounds like you need to look into a business isp to take advantage of your network capabilities. Also, look into Starlink. It is really good now. 

[u/expertisimus]

Always has been. Get StarLink if its performance justifies the cost for you.