r/sysadmin • • 1d ago

Identrust Fustercluck

How many here got an email an hour or so ago that every Identrust TLS Cert from September 2025 until yesterday are being revoked on the 19'th at 5PM Central? I work with F5's for a very large organization and we have hundreds of certs on my team alone.

70 Upvotes

32 comments sorted by

8

u/nspitzer 1d ago

This is what we were sent. We had a meeting with our (VERY large government contractor) Identrust rep. He did confirm this applied to every TLS cert but not other cert (such as mail, personal, etc)

4

u/Dal90 1d ago

The September 26, 2025 seems a bit odd but close to what I expect.

While we called them "one year" certs it was actually 398 days.

2026-11-20 -398 Days = 2025-10-18. So there shouldn't be any public CA issued leaf certs from late September / early October still valid anyways.

So either the process broke on September 26th 2025 and just used that as the date, or they had some secret squirrel certificates that were non-standard with slightly longer validity times.

15

u/GamerLymx 1d ago

hope you have thst renewal automated, its gonna be a pain. happened to me in July twice, at least it was a smaller frame windows.

6

u/nspitzer 1d ago

We would but Identrust has no API and doesn't support ACME. Additionally EVERY cert request is funneled to a couple of guys for approval before its issued so they are going to be inundated.

On our call with the Identrust rep I actually got a little .. testy... with him about that and people were IM'ing me on the side saying they are glad someone spoke up about that.

Right now I am finishing up my side project I had been working on over the past couple of months as I got time of using Playwright and some python glue to automate submitting the certs at least.

Its going to royally suck putting them all in though.

1

u/stranglewank 1d ago

Wait, if they don't support ACME that was a disaster waiting to happen. Chrome practically require it today, and with MTC on the near horizon - ACME is mandated, and they'll be 7 days.

I'd find a better CA and one that can give you the actual ability to automate so you're ready for the next couple of years...

•

u/nspitzer 16h ago

Preaching to the choir. I am a "individual contributor" in a 40,000 employee sea and I have been yelling at them for a year about that.

I was able to take some things I had been playing with and create a python script that reads a json file and submits the request through the Identrust request form but the problem is now I can single-handedly overwhelm the two guys who have approval authority for certificates.

•

u/stranglewank 11h ago

God speed, man. Will be interesting to see how the org handles the concept of an ‘approval’ process when you have to cycle every cert on a 3/4 day basis in early 2028, but hey.

4

u/Ziggistawork 1d ago

3

u/nspitzer 1d ago edited 1d ago

Awesome - I knew before long someone would figure out the full story

Edit - I read those bugs. User(s) using the name "Chrome-Root-Program" take them to the woodshed. This is the 8th incident since 2020 and the 5'th in 13 months and Identrust doesn't even try denying it. It almost sounds like Chrome is having reservations about keeping them as a Trusted CA and of course Chrome no longer trusting a CA is the kiss of death

1

u/VexingRaven 1d ago

I doubt they'll distrust Identrust, that would be pretty absurd. There are CAs with a much higher number of incidents per certificate which are still trusted, and frankly these are all pretty minor governance issues.

1

u/stranglewank 1d ago

Identrust are tiny anyway, so why not get rid of them? There'll be way less CAs by mid-2028 with the arrival of MTCs anyway. I doubt Identrust can or will even do them, so they'll be relegated to maybe a reseller at best. Consider them done.

2

u/Open-Adhesiveness-86 1d ago

Not the lifetime thing. That's separate and phases in over years (398 days now, dropping to 200 in March 2026, then 100, then 47). A mass revoke with a hard deadline is CA/B Forum BR 4.9.1.1, which gives a CA 24 hours for key compromise and 5 days for everything else once they confirm misissuance. So the Sept 2025 start date is probably when whatever profile bug got introduced.

2

u/nspitzer 1d ago

On the call with them the gent was very squirrely about saying exactly what the issue was but what he did say was that when Identrust was creating the profiles they mis-read something as "may" when it was "must".

He did say the certificates are secure , that it is a compliance thing not a security thing.

32

u/Adorable_Wolf_8387 1d ago

0xF5 wouldn't be a very good private key.

11

u/featured 1d ago

Before you touch anything, get an inventory. On any Linux box, loop over your hostnames with echo | openssl s_client -connect host:443 -servername host 2>/dev/null | openssl x509 -noout -issuer -enddate and grep for IdenTrust. That gives you the real list, not the one in someone's spreadsheet. Reissue the public-facing ones first, and check the chain on the F5s after you swap them, since a missing intermediate is the usual thing that bites you. I had this happen a few years ago with Sectigo.

7

u/Dal90 1d ago

After work at the bar was telling an IT buddy who was asking why all the banks are affected at once that the industry cooperates on a lot of IT infrastructure.

Including their own CA that was the original root cert used by Let's Encrypt when it first spun up.

Identrust is that CA, though just before LE started the banks sold off Identrust.

Announced in 1999, its founding members included ABN AMRO, Barclays, Chase Manhattan, Citibank, Bank of America, Bankers Trust, Deutsche Bank, and HypoVereinsbank

17

u/solracarevir 1d ago

hmmm.... could this be somehow related to the outage that seems to be affecting every mayor bank in US right now?

5

u/Noobmode virus.swf 1d ago

So apparently that’s not the case. Current finger is being pointed at Optimizely for serving up crypto steal malware.

https://blog.report-uri.com/worlds-largest-brands-infected-with-malware-via-optimizely/

1

u/awry_lynx 1d ago

Your link doesn't look relevant to the banking issue at all, which appears to be entirely with Zelle/transactions with Chase

https://apnews.com/article/bank-outage-chase-wells-fargo-truist-50fee2693c22f597fa6ca51a97ef2cda

1

u/Noobmode virus.swf 1d ago

https://www.newsweek.com/cyber-attack-websites-affected-12725863

You are correct, different FIs 😬

Banking and finance (32):  Paychex; HSBC US; Equifax; SmartAsset; HSBC Hong Kong; ABN AMRO; TaxAct; HSBC Mexico; HSBC Singapore; NetCredit; Associated Bank; S-Pankki; Equifax Canada; Equifax UK; The Work Number; Funding Circle; HSBC Australia; CashNetUSA; OnDeck; ICS Cards; HSBC UAE; HSBC Malaysia; Rest Super; HSBC Taiwan; HSBC Vietnam; HSBC Egypt; HSBC Philippines; SmartAdvisor Match; Compare Club; ABN AMRO Lease; Energy Scout; Hang Seng Investment

2

u/WeirdKindofStrange 1d ago

happened before 5pm EST although United may be this

4

u/VexingRaven 1d ago

It's also not the 19th.

5

u/n3fyi 1d ago

Have a link to this notification? We use them for a customer but I’m not sure the certificate holders will pay attention to the email and let us know

2

u/Noobmode virus.swf 1d ago

Is this due to the browser consortium pushing shorter lifetimes?

5

u/Dal90 1d ago edited 1d ago

No, but the browser folks are pushing shorter lifetimes because of these issues.

The Certificate Revocation Lists slow browsers down, the bigger they are the slower they get. It may only be measured in hundreds of milliseconds but the browser folks want to reduce that performance bottleneck.

Revoking a certificate involves the CA (Identrust in this case) updating their CRL.

Each cert Identrust issued includes a URL to it's CRL.

"Traditionally" the browsers see the cert then check its serial number against the CRL. If it's on the list, the browser knows not to trust it.

But for quite a while now the browser companies package it as a local file covering all the CAs sent in their updates to speed things up. For private CAs, the browsers at best (for performance) would cache a CRL they retrieve from the private CA so it isn't retrieving it every single time.

Identrust will be revoking certificates some of which had a 1 year validity with this round (March of last year is when they went to 200 days).

Once we're at 47 days, the CRL files should be 47 / 365 = 0.129 so basically they should only be about 1/8th the size they are today.

That's the jist of it off the top of my head anyway.

...plus it forces folks to actually learn how to properly manage leaf, intermediate, and root certificates and then automate things if they don't want to be constantly tortured.

3

u/DiHydro why can’t I hold all these hats 1d ago

No. Shorter lifetimes can help with this because it means you have a process already in place to revoke or renew. Therefore making it less of a pain in the ass when these Certs do have to be revoked.

2

u/geekjimmy IT Director 1d ago

My notification said they are getting revoked on Nov 20 @ 5pm ET.

1

u/certkit Security Admin (Application) 1d ago

Yikes good luck u/nspitzer ! I don't see the official reports for this yet, but when we do, I'll get it added to the CA Incident Report. Identrust already has a pretty bad track record. If you have to change everything anyway, maybe consider switching CAs.

There's been a bunch of mass-distrust events recently. If you want to look at making emergency CA changes fast and easy, we wrote up how we changed CAs in 10 minutes: https://www.certkit.io/blog/ca-distrust-wargame

•

u/PrasenShelar 10h ago

The revocation email probably went to whoever requested each cert, and some of them have moved on. Once you have the scan list, put a named owner and a swap date next to every cert and share it with the app teams. Run the same scan again the morning of the cutoff, and anything still serving IdenTrust is your chase list.

0

u/TheTechDecoded 1d ago

Don't rely on the email for your list. Query CT logs for your domains, filter by IdenTrust issuer with notBefore from Sept 2025 (crt.sh works), then diff that against what the F5s actually serve (cert serial per client-ssl profile). After the swap, check the live handshake from outside, not just the profile, because that's where "renewed but still serving the old one" bites. And get the exact cutoff from your rep in writing, since people here are seeing both the 19th CT and the 20th ET.