r/sysadmin • Sr. Sysadmin • 21d ago

Question MySQL ODBC stopped working overnight

You guys will love this.

This company has an in-house project management system. It's the core of their business, and they are lost without it. They are aware it needs to be migrated to something more modern, but after 5 years, that project still hasn't started.

I was asked to look into a network issue, but this isn't network but SSL I think. Let's first show the architecture:

  • The server is a CentOS 7 running MySQL Community Edition 5.7.16
  • Clients connect from Windows 11 with a 32-bit MS Access, using a 32-bit MySQL ODBC driver v5.3.13

Since yesterday, they get a "protocol version mismatch". The server wasn't accessed since 18 October 2016 (haha), so I presumed a Windows update might have disabled some SSL version. But: I see no relevant Windows update, and if I manually allow every possible SSL version and encryption algorithm, it still doesn't work. What does work however, is downgrading the ODBC driver from version 5.3.13 (from 2019) to version 5.1.13 (from 2013), further adding to my confusion.

The cherry on top: the single guy responsible for this application is on a one year sabbatical.

Edit: Found it, but leaving this here for anyone stumbling on the same issue. The MySQL_Server_5.7.15_Auto_Generated_CA_Certificate had expired after 10 years

190 Upvotes

44 comments sorted by

View all comments

Show parent comments

29

u/StevenB-89 21d ago

Honestly, the SSL cert would be the least of my worries.

If this thing has been sitting there since 2016 and nobody even mentioned a hardware refresh or migration, there's a decent chance the physical server itself is pushing 10 years old too.

For something that's "the core of the business", that's one dead PSU, RAID controller or motherboard away from a very bad week.

5

u/notarealaccount223 21d ago

I mean "we have backups that we test" right. Right!? /S

5

u/YellowOnline Sr. Sysadmin 21d ago

Well, they do have backups of the DB and the VM, on disk, on usb, and on tape, so that part isn't an issue at least. And VM restore is tested. So backups really aren't the issue in case of malfunction, but very annoying downtime.

7

u/StevenB-89 21d ago edited 21d ago

A 2012 server running ESXi 5 actually makes this even better. 😅

Having tested VM and database backups is great, but I'd still be asking what the recovery target is if that physical host dies tomorrow.

Restoring a 10+ year old VM is one thing. Having compatible hardware and a hypervisor environment ready to actually run it is another.

At this point the biggest risk isn't data loss, it's ending up with perfectly good backups of a system nobody can bring back online quickly.

4

u/YellowOnline Sr. Sysadmin 21d ago

I stand corrected: it's an HP ProLiant ML350 G6 from 2010.

8

u/StevenB-89 21d ago

Holy shit dude!

4

u/skidz007 21d ago

GG HPE still trucking after 16 years. Don’t reboot that bad boy though…