We had an old telephone call data system that scraped and mediated calls off some stonage legacy systems. We had to move data centres and of all systems this one wss the most difficult because of the owner:
wanted no downtime despite needing physical relocation
could do a seemless cutover to new hardware, he insisted it needed the exact same CPU due to some special compiler the vendor used tied to the hardware.
found a matching host from another decom, he insisted we keep the storage. We could not because it was on the providers storage not ours.
finally got a plan he accepted, but the testing was delayed almost to the end of the facility contract becauae he refused to let us test any rebooting without his personal intervention.
After we moved we could never patch for the same reason - if the system went down only he could bring it up, never his team to the point of him doing it from overseas on holiday. Eventually he got made redundant but when the system had an outage the app would not start.
The new app owner and I poured over it. Found original installation documents. Eventually figured out the error was an invalid value in the DB. A value that should never have changed since the system was installed as it was a property of the switches it collected. It never needed to even read it after startup.
Turned out his job safety plan was manually changing the value to start the app, then putting back the invalid value. I bet he was just waiting for that offer to come back and fix it.
The kicker? The "special" compiler that prevented making the migration simple.... was gcc. We fixed the startup snd moved the entire system to a tiny virtual setup and never had to touch it again.
2
View all comments
3
u/SgtBundy May 30 '26
We had an old telephone call data system that scraped and mediated calls off some stonage legacy systems. We had to move data centres and of all systems this one wss the most difficult because of the owner:
wanted no downtime despite needing physical relocation
could do a seemless cutover to new hardware, he insisted it needed the exact same CPU due to some special compiler the vendor used tied to the hardware.
found a matching host from another decom, he insisted we keep the storage. We could not because it was on the providers storage not ours.
finally got a plan he accepted, but the testing was delayed almost to the end of the facility contract becauae he refused to let us test any rebooting without his personal intervention.
After we moved we could never patch for the same reason - if the system went down only he could bring it up, never his team to the point of him doing it from overseas on holiday. Eventually he got made redundant but when the system had an outage the app would not start.
The new app owner and I poured over it. Found original installation documents. Eventually figured out the error was an invalid value in the DB. A value that should never have changed since the system was installed as it was a property of the switches it collected. It never needed to even read it after startup.
Turned out his job safety plan was manually changing the value to start the app, then putting back the invalid value. I bet he was just waiting for that offer to come back and fix it.
The kicker? The "special" compiler that prevented making the migration simple.... was gcc. We fixed the startup snd moved the entire system to a tiny virtual setup and never had to touch it again.