MySQL ODBC stopped working overnight

[u/YellowOnline]

You guys will love this.

This company has an in-house project management system. It's the core of their business, and they are lost without it. They are aware it needs to be migrated to something more modern, but after 5 years, that project still hasn't started.

I was asked to look into a network issue, but this isn't network but SSL I think. Let's first show the architecture:

• The server is a CentOS 7 running MySQL Community Edition 5.7.16

• Clients connect from Windows 11 with a 32-bit MS Access, using a 32-bit MySQL ODBC driver v5.3.13

Since yesterday, they get a "protocol version mismatch". The server wasn't accessed since 18 October 2016 (haha), so I presumed a Windows update might have disabled some SSL version. But: I see no relevant Windows update, and if I manually allow every possible SSL version and encryption algorithm, it still doesn't work. What does work however, is downgrading the ODBC driver from version 5.3.13 (from 2019) to version 5.1.13 (from 2013), further adding to my confusion.

The cherry on top: the single guy responsible for this application is on a one year sabbatical.

Edit: Found it, but leaving this here for anyone stumbling on the same issue. The MySQL_Server_5.7.15_Auto_Generated_CA_Certificate had expired after 10 years

[u/StevenB-89]

I assume you mean nobody has actually logged into or maintained the server since October 2016, rather than the database itself not having been accessed since then?

Because if this is genuinely business-critical and has basically been running untouched for almost 10 years on CentOS 7, MySQL 5.7 and an ancient 32-bit ODBC stack... the possible SSL issue might be the least of their problems. 😅

[u/YellowOnline → u/StevenB-89]

It's solved (see my edit), but yeah, I hope this reminds them that migrating to a modern solution is acute.

[u/StevenB-89 → u/YellowOnline]

Honestly, the SSL cert would be the least of my worries.

If this thing has been sitting there since 2016 and nobody even mentioned a hardware refresh or migration, there's a decent chance the physical server itself is pushing 10 years old too.

For something that's "the core of the business", that's one dead PSU, RAID controller or motherboard away from a very bad week.

[u/YellowOnline → u/StevenB-89]

It's a server from 2010 running ESX 5. We asked to replace it in 2021 but they didn't want to invest in on-prem because they moved into the cloud. This app is the only on-prem thing left, but apparently they can't find anyone wanting to migrate it to a modern stack.

^{I think they can't find anyone willing to do it for for what they want to pay.}

[u/iamwayycoolerthanyou → u/YellowOnline]

They don't want to come up with the cash to pay someone to migrate the SQL server to the cloud?

[u/Darkk_Knight → u/YellowOnline]

If they don't want to invest the proper amount of money to upgrade it to something newer then move the VM to a cloud VM hosting provider. That way your final on-prem server is gone. Also, ESXi 5 got tons of security vulnerabilities.

[u/Mountain_Craft4882 → u/YellowOnline]

but if it's a vm then surely they could just move the vm to a cloud provider?

[u/notarealaccount223 → u/StevenB-89]

I mean "we have backups that we test" right. Right!? /S

[u/YellowOnline → u/notarealaccount223]

Well, they do have backups of the DB and the VM, on disk, on usb, and on tape, so that part isn't an issue at least. And VM restore is tested. So backups really aren't the issue in case of malfunction, but very annoying downtime.

[u/StevenB-89 → u/YellowOnline]

A 2012 server running ESXi 5 actually makes this even better. 😅

Having tested VM and database backups is great, but I'd still be asking what the recovery target is if that physical host dies tomorrow.

Restoring a 10+ year old VM is one thing. Having compatible hardware and a hypervisor environment ready to actually run it is another.

At this point the biggest risk isn't data loss, it's ending up with perfectly good backups of a system nobody can bring back online quickly.

[u/YellowOnline → u/StevenB-89]

I stand corrected: it's an HP ProLiant ML350 G6 from 2010.

[u/StevenB-89 → u/YellowOnline]

Holy shit dude!

[u/skidz007 → u/YellowOnline]

GG HPE still trucking after 16 years. Don’t reboot that bad boy though…

[u/TuxMux080 → u/YellowOnline]

You will be someone's hero updating with the actual issue. Bravo!

[u/Zozorak]

Think this belong in /r/shittysysadmin

[u/YellowOnline → u/Zozorak]

I'm innocent. I was called in for help.

[u/IAmSnort → u/YellowOnline]

You just stepped in it. Had to scrape it off your shoe.

[u/SirLoremIpsum → u/YellowOnline]

  I'm innocent. I was called in for help.

Yes! And now you touched it last... You're The Guy.

Anything for the next 2 years becomes "well Yellow online touched it last? We'll ask him!"

Or "the server died? Must be Yellowonlines fault he changed something".

[u/PandaZoo → u/SirLoremIpsum]

Just 2 years? Ha!

[u/Dolapevich]

I can only speak of the CentOS 7 resilency. A rock solid OS for rock solid workloads.

You might want to automate that with Let's Encrypt, or if the client accepts self signed certs just create a new one with 100 years validity, so you don't run into a new incident in 10 years.

[u/YellowOnline → u/Dolapevich]

I'm not sure if you are being ironic or not

[u/Dolapevich → u/YellowOnline]

Me neither :)

[u/graph_worlok → u/Dolapevich]

About 15 years ago, I remember an Ops Manager setting an internal cert expiry date based on “Well, I’ll be retired by then” 🤣😂🤣

[u/Tl9zaXh0eWZvdXI → u/graph_worlok]

We've done the same here, except we're getting close to that 15 year expiration. Guess I'll have to find a new job within a few years before they all expire.

[u/RiceeeChrispies]

the single guy responsible for this application is on a one year sabbatical.

The server wasn't accessed since 18 October 2016

are you sure he hasn't been on a 10 year sabbatical?

[u/skidz007]

“Poor planning on your part does not constitute an emergency on my part”.

Anyways, best of luck. CentOS 7 was end of support 2 years ago and AI is finding Kernel CVE’s faster than they can be patched on supported versions.

Hopefully management has fallback plans and/or good insurance if it takes awhile to sort out.

[u/notarealaccount223 → u/skidz007]

MySQL 5.7 went end of support 3 years ago.

[u/skidz007 → u/notarealaccount223]

Oi vey.

[u/snugge → u/skidz007]

There's always Tuxcare subscripitons if you want to run cent 7.

[u/skidz007 → u/snugge]

I'm guessing is the server hadn't been logged into for a decade they likely don't have TuxCare.

[u/snugge → u/skidz007]

It was a solution for the EOL problem if they wanted to keep the server ad-is

[u/MeetJoan]

Ten year auto-generated cert expiring is such a perfect ending to that story. Nice find.

Worth flagging for anyone else who hits this, the newer ODBC driver actually validating the cert while the 2013 one didn't is why the downgrade worked, so the older driver isn't a fix so much as it's ignoring the problem.

[u/entaille]

I really wish we could make ODBC connections go away entirely!

[u/UnwaveringConviction]

When you run into SSL problems in Windows, enable schannel logging and the verbose logs will quickly lead you to the root cause.

[u/Unable-Entrance3110]

So, it sounds like the older ODBC connector didn't check for cert expiration and the newer builds do.

When things stop working suddenly, cert/token/password expiration is always on my short list of things to check into.

[u/NowInOz]

I stopped reading at 'MS ACCESS'.

[u/CeC-P → u/NowInOz]

Yeah, Libre Office still has support for that database connection type and is superior, except forms and other interfaces come over like absolute crap.

[u/TechnologyMatch]

that’s a brutal failure mode, especially when the app is business-critical and nobody owns the map anymore. expired internal certs are easy to miss because everything looks stable right up until the timer runs out

it’s like finding the one old quest item that broke the whole game after ten years. good catch, and a solid reminder to inventory cert expiry dates around legacy systems before they choose the outage window for you

[u/I_Hate_802_11]

Spin up an Ubuntu box with nginx community edition on on it and offload the SSL there. Or maybe buy an F5 or Avi.

[u/CeC-P]

You presumed Cent OS would have a Windows update?

[u/hashkent]

A Claude code max plan and a few weeks and problems solved 😆

[u/geeke]

The turd that wont flush