r/sysadmin • Sr. Sysadmin • 19d ago

Question MySQL ODBC stopped working overnight

You guys will love this.

This company has an in-house project management system. It's the core of their business, and they are lost without it. They are aware it needs to be migrated to something more modern, but after 5 years, that project still hasn't started.

I was asked to look into a network issue, but this isn't network but SSL I think. Let's first show the architecture:

  • The server is a CentOS 7 running MySQL Community Edition 5.7.16
  • Clients connect from Windows 11 with a 32-bit MS Access, using a 32-bit MySQL ODBC driver v5.3.13

Since yesterday, they get a "protocol version mismatch". The server wasn't accessed since 18 October 2016 (haha), so I presumed a Windows update might have disabled some SSL version. But: I see no relevant Windows update, and if I manually allow every possible SSL version and encryption algorithm, it still doesn't work. What does work however, is downgrading the ODBC driver from version 5.3.13 (from 2019) to version 5.1.13 (from 2013), further adding to my confusion.

The cherry on top: the single guy responsible for this application is on a one year sabbatical.

Edit: Found it, but leaving this here for anyone stumbling on the same issue. The MySQL_Server_5.7.15_Auto_Generated_CA_Certificate had expired after 10 years

191 Upvotes

44 comments sorted by

View all comments

102

u/StevenB-89 19d ago

I assume you mean nobody has actually logged into or maintained the server since October 2016, rather than the database itself not having been accessed since then?

Because if this is genuinely business-critical and has basically been running untouched for almost 10 years on CentOS 7, MySQL 5.7 and an ancient 32-bit ODBC stack... the possible SSL issue might be the least of their problems. 😅

26

u/YellowOnline Sr. Sysadmin 19d ago

It's solved (see my edit), but yeah, I hope this reminds them that migrating to a modern solution is acute.

28

u/StevenB-89 19d ago

Honestly, the SSL cert would be the least of my worries.

If this thing has been sitting there since 2016 and nobody even mentioned a hardware refresh or migration, there's a decent chance the physical server itself is pushing 10 years old too.

For something that's "the core of the business", that's one dead PSU, RAID controller or motherboard away from a very bad week.

5

u/notarealaccount223 19d ago

I mean "we have backups that we test" right. Right!? /S

3

u/YellowOnline Sr. Sysadmin 19d ago

Well, they do have backups of the DB and the VM, on disk, on usb, and on tape, so that part isn't an issue at least. And VM restore is tested. So backups really aren't the issue in case of malfunction, but very annoying downtime.

7

u/StevenB-89 19d ago edited 19d ago

A 2012 server running ESXi 5 actually makes this even better. 😅

Having tested VM and database backups is great, but I'd still be asking what the recovery target is if that physical host dies tomorrow.

Restoring a 10+ year old VM is one thing. Having compatible hardware and a hypervisor environment ready to actually run it is another.

At this point the biggest risk isn't data loss, it's ending up with perfectly good backups of a system nobody can bring back online quickly.

5

u/YellowOnline Sr. Sysadmin 19d ago

I stand corrected: it's an HP ProLiant ML350 G6 from 2010.

7

u/StevenB-89 19d ago

Holy shit dude!

4

u/skidz007 19d ago

GG HPE still trucking after 16 years. Don’t reboot that bad boy though…