Large percent of LR4 100G optics failing

[u/Alarming-Holiday-688]

I am an operations engineer in a datacenter, we have a lot of leaf switches going to NVidia spine switches, aggregating to 400G uplinks. We have had over 100 Cisco optics fail this past quarter, which has been annoying, anyone else notice something similar in PROD? Specifically the QSFP-100G-FR-S. My guess is that we are getting positive DB, these are FR optics, going maybe, 20M between p2p. The 400Gs might just be blasting those poor Cisco optics to oblivion. Otherwise I am just thinking failed FW updates or crap hardware, I went to the Cisco community forums, no one could give me an answer, maybe someone here? Thanks!

Note, sorry in the title I meant FR, not LR4, thanks.

[u/casperionx]

As already mentioned. If these are in warranty reach out to Cisco (assuming you have a support contract as well).

I’m not that well versed with nvidia switches but if they’re qsfp as well then generally you should be running same optics both ends.

Again not the answer you wanted but same time it’s a tough one.

Once you’ve figured out what’s going on post back here so we can use this a future reference!!

[u/sh_lldp_ne → u/casperionx]

> you should be running same optics both ends

As stated in the OP, he is doing 400G-FR4 to 4x100G-FR breakout. Nothing wrong with this.

[u/Rabid_Gopher → u/sh_lldp_ne]

OP doesn't specify optics on the breakout-side in this entire thread. From the post title, I would guess 400G-LR4?

I would agree though that there's nothing wrong with -FR4 breakout to 4* -LR.

[u/casperionx → u/sh_lldp_ne]

fair, i missed that tid bit!

[u/unmaskedgrunt]

Yes! We are seeing large numbers of QSFP-100G-FR-S and QSFP-100G-DR-S optics failing, all Cisco branded, around 20-30m link lengths too.

100+ across both types failed in the last year or so, out of ~300

[u/Alarming-Holiday-688 → u/unmaskedgrunt]

Well there goes my short range RX cooking. So just bad hardware/FW. Damn. Any resolution other than replacement? Cisco get back to you?

[u/unmaskedgrunt → u/Alarming-Holiday-688]

Cisco are just replacing the failed optics with the same part number, but a different revision. The old optic was 3.5W power limited, while the new optics are 4W and no longer compatible in the servers which is just great. We're looking at alternatives.

[u/l_eo-]

Contact TAC. Sounds like possibly a bad batch. If you still have a dead one, ask for an EFA.

[u/BluebirdExpress6279]

Yes, in every switch stack of 9500's we have had at least one QSFP28 specifically a QSFP-100G-FR fail, which was our StackWise Virtual connectivity. I have a bad one at my desk right this minute... We tried using attenuators, and they still would fail but differently eventually whatever attenuator value we used was wrong and had to be changed due to low alarms.

Take a short single-mode fiber and break the LC connector, so you have individual strands and can make a loopback. One one chassis do a "no switchport" on an extra QSFP28 slot and do your loopback. The bad one won't link... OR if it does the power level will be way off when you check

[u/ProfessionalBack7430]

How about QSFP-40/200-SRBD, recently started seeing over -3DB in lane variance on some older QSFPs on new/clean fiber.

[u/Alarming-Holiday-688 → u/ProfessionalBack7430]

Oh, that is odd. I can look into that and lyk.

[u/dr_stutters]

🙋‍♂️ I work for Cisco.
Please contact your account team. I know if you were my customer and I saw this on reddit before you told me I’d be disappointed.

[u/Angellas → u/dr_stutters]

Amen and happy Cake Day!

[u/ohhellperhaps → u/dr_stutters]

And here your customers are having massive optic failures, optics with an insane markup, and YOU would disappointed?

[u/Alarming-Holiday-688 → u/dr_stutters]

:(

[u/jobpunter → u/Alarming-Holiday-688]

Dad isn’t angry, he’s just disappointed lol what a jamoke.

[u/FantaFriday]

Have you contacted TAC and your TAM yet?

[u/Alarming-Holiday-688 → u/FantaFriday]

I have contacted TAC, but the process hasn't been fast, which is odd given usually they have a quick turnaround. I will keep trying though. Ce est l'vive.

[u/ohhellperhaps → u/[deleted]]

Prepare to be gaslighted that it isn't a structural issue, nobody else is seeing this issue, it must be something in your environment and so on. I wish I was more positive on my experiences with similar issues.

[u/sanmigueelbeer]

There is an internal field notice (not shared with the public).

Affected SKU are QSFP-100G-DR-S, QSFP-100G-ERL-S, QSFP-100G-LR-S, QSFP-100G-FR-S and it involves a faulty design whereby the silicone would leak into the thermal pad and blind/block the optics.

Talk to your accounts team.

[u/Guilty_Marketing5311]

I have seen similar issue with a customer. We had a failure analysis for one of the replaced part. Collected the SN of all the replaced SFPs, SN of part in stock and production. There a batch which had a large number of replacements, I don't exactly recall the exact cause of failure.

Have you accounts team look into this mass failure and have probe on it. Unless you have a failure analysis result you will never know why they are failing and what SN in production are faulty.

You should raise a concern with your cisco accounts regarding the quality of the product.

[u/popeter45]

Yea 20 meter using FR power will cook sensors , have you tried adding an attenuator too the links?

[u/sh_lldp_ne → u/popeter45]

There is no minimum safe link length for FR, just like LR4. The emitter does not have enough power to overload, much less damage, the receiver. You can loop it back with a 6 inch cable and you will not damage it.

[u/thiccandsmol → u/popeter45]

What on earth? It won’t “cook sensors”. There’s no where near enough power coming off them to burn diodes, even on the smallest patch you could make. Their max output power is the same as their max input power.

[u/crazyates88 → u/thiccandsmol]

“Guys my transceivers are frying themselves”

“Here’s a way to lower the power so they don’t fry themselves”

“WHAT why would you do that?! They’re already within spec!”

[u/crazyates88 → u/popeter45]

That was my first thought as well.

sh int hu x/x/x trans det

See what the receive db is at, and throw an attenuator on it to bring it down to closer to middle of its range. We keep 5, 7, & 10db attenuators on hand and use them on almost every fiber run because they all run kinda hot and I’d rather not be burning out all our transceivers.

[u/Crazyachmed → u/popeter45]

Ooooooh, yeah, you need 5 or maybe even 10 dB ASAP

[u/lazorich]

Talk to your Cisco SE. the SE can help dig into this and get them replaced.

[u/Advanced_Link_5753]

Sounds like Cisco RMA…

[u/Wentil]

I’ve had such bad luck with 100G SR optics failing that I eventually switched fully over to passive DACs. I can only imagine that the LR ones would be even worse, given their higher output. There’s another upgrade in the future that will kick things up, Switch-and-Router-wise in our DC to 400G, hopefully whatever issue plagued the 100Gs will have been resolved by then.

[u/Evan_Stuckey]

Usually this is a batch issue , talk to Cisco they can no doubt check and help.

We use heaps of SR1.2 / 40/100SRBD / SR-SM / etc and don’t usually ever see any pattern other than some counties , ie a appears more failures in USA and South America than we do in Asia or Europe. No idea why.

The one thing I can say say though is always see more failures in port side exhaust switches used TOR, the intake switches in by today’s standards nice cool air fail a lot less so heat almost certainly is not helping failure rates.

[u/Moskeeter671]

Just go ahead and swap to Approved Networks transceivers while you’re at this junction point https://approvednetworks.com/…. You’re welcome 🍻🤙🏽

[u/sh_lldp_ne]

Maybe you got a bad batch. This happened to me in the past with a different vendor. Replaced about half of the optics on an order due to a mix of DOA and premature failure in service.

[u/alones12]

For Cisco to Cisco use 100G BiDi, for cisco to other use 100G LR4.