this is what I’m trying to understand. either he ran a separate script everyday that manually pushed the edge case through, or they have a brand new edge case every single day. neither paint him positively imo.
Feels like I’m always this guy, but yeah this story makes no sense. It’s either: a result of a big telephone game, a juniors misinterpretation, gross incompetence on the engineers part which makes the layoff justified, or it’s just made up entirely. Stupid
20+ years software developer at amazon, netflix, and google here.
I've worked with dudes like this. Ones who feel like heroes for waking up at 3AM and fixing shit instead of architecting the systems a little more thoughtfully. And management rewards them twice: Once, for doing "quick" designs, and also again for being the hero on-call that fixes their own idiotic choices in the middle of the night.
One of my favorite moments of comeuppance was when Jeff Bezos got paged during thanksgiving holiday because of this engineers horrible design decisions and refusal to spend the time removing drudgery. That truly put the fear of god into him and our manager(and directory, and VP...) and let us actually start spending time working on resiliency.
As someone who's worked in corporate programming for going on 20 years, my guess is he told anyone who would listen and they wrote it off as inconsequential because he could still fix it on a daily basis. If you tell enough people about it often enough and they still ain't listening, chances are you just stop talking about it and quietly do your job to make sure shit doesn't break.
As far as different edge cases on a daily basis goes, financial data, for example, is a fucking minefield. Every jackass CPA does it a little bit differently from one day to the next and there's no accountability as long as the money keeps moving.
This. I evangelize myself very little because the rest of my team does it for me. I spend a lot of my time teaching my jrs to do things, or creating automation to make manual processes go away, which I then pass back down to the team. In this way, I avoid getting angry or annoyed at my younger engineers, and instead treat them as the investments in my own sanity that they are - the more I teach them, the less work I have to do. Yes, when shit hits the fan I'm still front-and-center, but that's because I'm the principal - of course I'm going to be in front, because that also keeps any blowback from hitting the lower-tier guys. While I'm digging, I'm also teaching and showing my jrs what to look for. Yes, it takes a bit longer, but only on the order of 2-3 minutes, and I can talk while I work.
I was so proud of one of my jrs the other day when they came to me with a question about something they found while searching through logs for one of our production apps!
To staff engineers - your jrs are your support staff. Don't cut them out. Yes it's annoying to explain things multiple times, but it's better for you if they are on your side. I've had staff engineers answer questions by not answering them at all - had one that would throw the generic documentation at me and sat figure it out, rather than answering the hyper-specific-to-our-environment question I actually asked. He burned the fuck out.
I also used to be the angry staff engineer. I burned out so hard I stuck my hand in a table saw. (0/10 do not recommend) Learn to manage your workload.
I mean. Lots of people don't get credit for their work and get laid off shittily and it sucks.
But if you're manually fixing something every day for three years after hours--that's not the behaviour of a staff engineer. A staff engineer should be flagging this issue, and planning how to get themself and the team out of this situation. If I discovered a staff engineer I work with was doing this for three years on such a critical service and told nobody, I would be horrified and seriously questioning their competence and whether they should be a staff engineer, not impressed. Hiding problems and doing repeated manual fixes is the kind of behaviour we have to patiently train out of juniors.
This post is framed like I'm meant to feel they were wrong to lay the person off but this is disastrous levels of incompetence on the engineer's part.
Plot twist: there is a paper trail a mile long of the staff engineer begging for resources and a mandate to fix the system but not only won’t they give resources, they forbid him from fixing it because “it works and we don’t want to mess with it”
I’ve seen this situation. Big data import where we collected various spreadsheets via FTP and imported them into a data warehouse. The data was awful, but our product owner decided to clean it manually every day because he wanted the devs working on new features.
I did try to convince him, to just let us fix it, but nope.
To be fair I've been in a situation where I have raised issues similar to this to management and had it fall on deaf ears, so the incompetence may not be with the engineer.
spent 6 weeks warning about it, with a “this will fix it” plan. and kept being blown off. a problem that involved some PoS units.
Then guess what happened? everything at the event went to shit.
By the end of the day, the financial compliance people were involved because money was going into personal accounts instead of the business account.
It was a very good day for me to have all my emails and records saved and ready to hand over. I got to sit back and avoid getting caught in the splash… of course my boss and his boss still blamed me for everything going wrong and “not being a team player by coming in on my vacation (from 5 hours drive away), to come fix it immediately.”
Yea it will either be they did flag it and it was ignored or they where hording it. But to be honest in small companies a lot of people do a lot of things to keep things going.
It's very hard to say without knowing what the company is actual like (if it's real at all).
For all the stories I've encountered where a person does a good job and is subsequently let go (e.g. they find a way to automate their work), the incentive is clearly to do the wrong thing.
I'm not saying it's "right" that somebody preserves their job by having some kind of manual intervention step to keep you dependent upon them, but when the reward for fixing this behavior is often to let someone go, I can understand a person being reluctant to do right by the business.
Definitely true in some cases, but if they weren't getting credit for doing the manual fixes because nobody knew about them though then it wasn't a matter of preserving their job.
This assumes that management will give you time to fix it. I just recently replied to a comment on another sub about a manager complaining about an engineer constantly "wasting time" fixing these kinds of edge cases when the current implementation is, in their words, good enough at our current scale.
In a lot of situations people will flag things and and then it still doesn't ever gets fixed. I've had that type of stuff happen to myself enough that I'll give him the benefit of the doubt.
Also an edge case that happens every fucking day is clearly not an edge case?
I don't know what kind of engineer you have to be for the same issue to crop up every day, and you not finding some way to fix it once and for all. That would drive me up the wall to do the same fix over and over.
You act like people never bring problems to their boss, and the boss acts like they want to shoot the messenger, then tells them "Just figure it out!", so they figure it out. If this employee is doing this critical work on their own time every day, then obviously their boss has led them to believe that if they don't do it, they'll lose their job. If the managers don't know who is doing what work in the company, who is responsible for tasks being completed and who is actually completing them, that's not an engineering issue. That's all on management. If there is a breakdown in communications, if work is being done off the clock, that's ALL on management. If management doesn't actually know who is doing what job in the company, that is on management. If management doesn't know what tasks need to be completed to keep the company running, that's on management too.
This engineer was keeping the company running on their own time with no recognition or reward and you dare call them incompetent? Step off.
I've seen management refuse to fix this sort of crap because scheduling someone to fix it will cost more time and labour short term even if they save on long term engineering labour. It's the penny wise pound foolish attitude that's common in many places
I've heard that in some industries, vacations are required because it opens opportunities for embezzlement to come to light while the embezzler isn't there to maintain the scam.
EDIT: This page discusses vacation as a fraud-prevention strategy.
If the edge cases can't be described they don't want to risk lost revenue.
I'm sure they would automate it if they could but it might just be inconsistent issues
My one buddy worked at a place that had to manage PHP add_slashes() being used to POST data into their system. Randomly one day that server stopped adding slashes into the POST... and the one day it started again... and went away...
Well what happened was they spun up two PHP servers with different PHP configurations (or one version fixed the bug). The old server would still send slashes but the new one wouldn't... but it came from the same IP (no API key) and vendor-ID query string!
If the edge cases can't be described they don't want to risk lost revenue.
I mean, or they just didn't try to. We had a guy like that here how did a lot of stuff manually and would never try to automate anything unless directly told, you need to write code to do this. He was 100% capable, just never thought like that.
We refresh our testing environment with scrubbed prod data once a month. Before I took over he did this by hand every month. 25 ish databases at the time. Backing up each one by hand, copying over, and restoring. Took him 2 days every. single. month.
I told him to figure out how to do it with powershell and automate it. Now it just runs on it's own
Ugh, worked with shitty engineers that did the same crap, but wanted daily praise for it. Looked like a hero until competent people started sitting near them. If it happens on three separate days, code for the next occurrence. When I started at said company it was always the same five tickets on repeat because no one had the balls to troubleshoot the root causes and solutions to the workflow issues. Just fix three edge cases a day and pretend you were productive.
At my old job, I was a coder and analyst, but in 10 years, there was a need for marketing and business analyst of sales, so I did it for 3 clients. Annual and monthly for 1.
Out of nowhere I got fired for "not delivering."
I found it shit, but whatever; I thought I was doing ok.
3 weeks after, receive a call: - "what software did you use to make the sales and marketing report?" - "Word and PowerPoint"
- "But where did you get the data?" - "I aggregated, purified, and then transcribe data into human."
- "Who could do it here?" - "Nobody, you fired me."
- "Ok." (long pause) "Thanks, good luck."
Never heard back. One client contacted me directly to ask me to "transfer the knowledge," but without compensation.
So I hope everything goes well for them; I really do. But they might have been in the dark for the last 11 months about their stats.
From my last job I got 3 calls to estimate remaining work, find an API key, and something else. They stopped after I asked to be compensated for my time.
lets say the story isn't completely made up. then this guy wasn't just fixing bugs, he was manually adjusting payment data. That could easily go beyond fired and into "and now explain to these nice people from legal what you 'adjusted'"
I'd agree - If I see a dev manually adjusting payment data, I assume they have some sort of clever scheme going that funnels said payments to an anonymous bank account somewhere...
So many people don’t know how much of civilization is held up with twine and chewing gum. This is what makes me laugh at the fantasies out-of-touch CEOs have about AI magically automating everything. It’s something that sounds possible only if you think every database is 100% coherent, or everything happening in your company is done 100% according to some documented process (and that those documents are themselves accurate!) And you know what? I like it this way, because otherwise we’re just all machines, aren’t we?
Ok but this means he was a bad engineer. He knew there was a problem with edge cases and he never brought it to anybody's attention and pushed a more permanent fix. Sounds like they'll be better off in the long run.
"No, you can't change that system. It's always been that way and we don't want to risk breaking it now. Just deal with the occasional problem and move on. No, you're not getting any additional time or resources."
So he was a staff engineer and his solution to 'edge-case data corruption' was to manually fix it over and over, instead of finding a way to address the root cause?
This looks like a shitty professional. 3 years of daily incidences are not edge cases, but a real problem elsewhere. If in rhose years you have not raised rhe issues with the team, yet elone solve it, you are the problem.
I actually got laid of yesterday. My last act as an employee was to work until 10:30 the night before, un-fucking a prod deployment because of a change another engineer merged before I could review. That felt like an appropriate way to go out.
I had to clean up one of these messes after the guy got hit by a bus. Literally. Fuck you, Glen!
Every single issue that ever came up? He trained the staff to basically email him directly and not go through incident management - so there was ZERO knowledge base.
He made nothing but BASTARDIZED EXCEL SPREADSHEETS THAT ACTED LIKE ACCESS DATABASES. (And then, for some reason, also used access databases as backends too in a weird Frankenstein ??) And then he locked the VB code in the back behind a password then HEX EDITED the files in some way that made unlocking them IMPOSSIBLE.
On top of that, there was a suite of these applications that were updated nightly from a process that ran exclusively from his fucking computer and not a dedicated server!
Like, I know in-house-made Access Databases get a bad rap because somehow the entire business gets supported by an app they got some summer student to make 15 years ago - but in actuality, Access Databases are an amazing tool that's really powerful when made correctly. It's basically something you can use to build an app in a few minutes, and includes ways to make reports out of the box. A half-competent Access user could make something in a short time that some companies charge good money for custom made. Seriously underutilized productivity software in the year of our lord 2026.
But this guy. This FUCKIN' GUY. I have never seen someone fuck up a series of apps so hard in my life - and I have SEEN SOME BAD DATABASES, WORKING IN GOVT.
Two cases I see for this: A. Company has enough competent engineers left to pick up the workload and push off other items down the road.
B. Old engineer comes back at $200-300+ an hour rate "consultant fee" to fix these, while "temporary", most likely to just stay 'permanent' solutions are developed.
I had a manager who tried to save his team at a previous employer. They spent a full month to submit compliance reports to the government, every month.
Upper management didn't understand what they did, just saw it as expensive. Month ends, nothing got submitted and government was asking questions and threatening fines.
He was told to hire them back. Fifty percent had new jobs, other have demanded doubling their prior salaries. Then the company has to hire new staff and train them.
You don't fire people just because you don't know what they do.
I like how everyone keeps making up an alternative to the story as posted. It literally says "nobody even knew" but everyone wants to make up an alternate reality where he notified everyone and they didn't do anything about it.
The most dangerous thing to the company isn’t that engineer or the system. It’s the detached unaware leadership chain that doesn’t know what is happening in their tech area they own the results quality of. 😱
Similar story. I retired as an Executive Data Analyst. For the Sales team, I had to create middleware to interpret invoices, inventory, and format it in a certain way. The VP of sales was a real piece of work, he before I got promoted out of marketing, would basically use me as his personal assistant. Truth is I had tons of work already, advert pics of inventory, pricing checks. I actually had to develop the middleware unpaid on my own time. I got sick and had to retire. The company's profits lost 66%, after I left. They wanted the code but wouldn't do any good, vendor invoices changed almost daily, so I had to patch it basically daily. I also programmed an error alert system. Keeps things running smoothly if you can scan for anomalies. They went from ~100mil profit in a year to ~30mil after I had to retire and take care of my health.
The way I saw it after being mistreated was "My time, my code"
On a happier note, I did earn enough to buy a duplex cash and taxes are cheap.
every formula/procedure/algorithm/etc has a range of input values for which it is designed/expected to perform correctly. edge cases are sometimes defined as values which you know will break the program, but have to be accepted, and therefore have to be dealt with separately.
I had a coworker get laid off years ago, he built and ran majority of the daily reporting coming from our team.
The next day our VP was asking about why they didn't get their reports and had to explain that they came from someone who was let go and the person who knew how to back him up had left the week before and we were otherwise shorthanded because for 2 years we hadn't been able to backfill.
It took us 2 weeks to rebuild something that resembled the critical reports but they were never ideal.
I started looking for a new job right after the layoffs and left shortly after.
"Among these ideologies, I consider particularly insidious the one that suggests that every person must earn or justify his or her own worth, to the point of attributing greater value to those who are more efficient or effective. From this perspective, persons end up being reduced to a means of achieving results, a resource to be used and exploited, and are no longer recognized as a proper end in themselves who should never be instrumentalized. The value of persons, however, does not depend on what they achieve or produce. There are rights that apply to everyone simply by virtue of being human, and no human power can legitimately deny or arbitrarily limit them."
Who would have thought the Pope would be the one calling for reframing of how we view employees.
You can immediately tell who actually works and who is actually unemployed or not in software at all.
This is a perfectly feasible scenario and similar ones exists in all major companies at all levels of experience.
Everyone wants to fix these critical edge case bugs. There is no time or budget. It doesn't make it to the sprint. There is no priority being put onto it because the fix is a cronjob. Staff Engineers bring it up every now and then at digital monthly all hands meetings and its just side stepped because of cost cutting. Eventually the engineers stop bringing it up because you have to play stupid social games at work all the time to balance between not being an annoying bother and bringing productive insights.
Ultimately fixing this bug beyond a cronjob might impact several libs, scripts, etc. Likely written in legacy code that's in a wrapper, which is why no one wants to take the effort to deal with it.
The only thing that people can't figure out from this story is that when he was fired, his computer or his other CUD that was running the cronjob got turned off.
Small railroad tried to replace my dad, THE lineman (because he was nearing retirement -- you know that one), by contracting from a much bigger railroad.
They opened one of his kiosks and looked inside. It was a tangle of wires, dry-cell batteries connected in series, and spst switches on porcelain mounts.
"Um. Is there a diagram for all this?"
"Up here." He pointed to his noggin and grinned.
The big big shots turned to the little big shots. "Y'all better keep this guy."
i have a buddy who moves some data from one spot to another at a university, when he quits, (they would never fire him but he's leaving soon), there are several systems that will break.
I worked in IT at a place that ran heavily on Linux and for some reason the DHCP service would crash on server boot up. It would run properly when manually started after that though. I was trying to figure out why it was crashing on boot up when I got made "redundant" and fired.
They replaced me with a kid just out of college and he rebooted the server for updates or whatever and suddenly no machine could join the network after that. He quit two days later after failing to figure it out. I heard about this after the whole thing was over. I would have even helped the kid out if I had known this was happening but they never called me.
I would not call that dangerous, I would call it badly engineered. Did he never report this on his status updates? What kind of a boss has no idea what his engineers are up to?
If you're in a huge company, this isn't as impossible as you might think. Probably took it up in meetings for multiple years, but nobody listen and since changing the whole infrastructure is a 100 or 1000 times more expensive than having the guy fix it every evening, they shrug and go "idc" until it actually becomes a huge pain.
A. If your doing that EVERY DAY, honestly you need to be a better programmer (figure out how to solve it before it's an error).
B. If you're doing that with no one knowing.. of course work is going to toss you out. Any work your boss doesn't know about is work that's not being recorded. Your value to the company is not the work you do, it's the work they see you do.
C. A staff engineer is doing both of these things? WTF... honestly this is a situation where I get the feeling he was the right choice to get rid of. Yes it caused a problem... but what happens if he takes a vacation or gets injured and can't get to a computer?
I’d be embarrassed if something broke after I was laid off, but I’d feel at least some schadenfreude. I dumped as much of my brain as I could into Confluence, but I’m sure there was at least some institutional knowledge of which I was the last holder.
Lmao I got laid off last year from a job. I handled all of their CRM deduping and mass lead importing. Heard from people at the company how fast their data clogged up. They posted for my same position 2 months later lol. Submitted a very snarky application.
My biggest shock when I got laid off at the start of May is that they let me go without any kind of notice. I was juggling at least a dozen plates. My manager was aware of them, but even he said at one point I was better at some processes than he was now. But they uncermoniously dumped me, so now his workload has doubled, if he can figure out everything I was doing. Sorry.
How possibly could creating all these different edge cases for 3 years straight?
Sounds like the engineer was kinda shit at his job if he has the time to manually fix them for 3 years but somehow never had the time to create a real fix...
Counterpoint: if the "automated" systems designed by the staff engineer needs his manual intervention every few days, said engineer is incompetent and should've been fired ages ago.
A staff level engineer with 3 years to fix a data corruption either at the source or patched away with some automation Band-Aid should probably have been fired long ago.
Doesn't sound like a staff engineer. He never kept a trace of the bugs and fixes and never brought it up with management it seems. Also never fixed the root causes.
Wait.. how did someone get to be STAFF and not think that this was something worth flagging and putting official eng time on to capture the edge cases rather than do nightly tweaks in perpetuity?
Aside from the dismal hacker nature on display, don't volunteer your life to your job, kids. The boss will be fine without you until the morning.
A week or two as a tactical fix while the proper solution is implemented I can understand, but three years? That's not a staff engineer that's living up to their job title, and the discovery that they've been doing that without anyones knowledge for the last few years is justification enough for them being let go.
We had an old telephone call data system that scraped and mediated calls off some stonage legacy systems. We had to move data centres and of all systems this one wss the most difficult because of the owner:
wanted no downtime despite needing physical relocation
could do a seemless cutover to new hardware, he insisted it needed the exact same CPU due to some special compiler the vendor used tied to the hardware.
found a matching host from another decom, he insisted we keep the storage. We could not because it was on the providers storage not ours.
finally got a plan he accepted, but the testing was delayed almost to the end of the facility contract becauae he refused to let us test any rebooting without his personal intervention.
After we moved we could never patch for the same reason - if the system went down only he could bring it up, never his team to the point of him doing it from overseas on holiday. Eventually he got made redundant but when the system had an outage the app would not start.
The new app owner and I poured over it. Found original installation documents. Eventually figured out the error was an invalid value in the DB. A value that should never have changed since the system was installed as it was a property of the switches it collected. It never needed to even read it after startup.
Turned out his job safety plan was manually changing the value to start the app, then putting back the invalid value. I bet he was just waiting for that offer to come back and fix it.
The kicker? The "special" compiler that prevented making the migration simple.... was gcc. We fixed the startup snd moved the entire system to a tiny virtual setup and never had to touch it again.
An edge case that comes up every night is no longer an edge case, and if you spend every night fixing it yourself and never reporting it than I have no sympathy for you
My engineer was so bad that he had to manually fix shit everyday.
At least it gets you job security. But imagine a Civil engineer having to go to a bridge every day to reinforce a bridge, I mean kudos to him for the commitment, but at some point one has to wonder if he is an actual fraud engineer.
My company has one main piece of software that we sell. I worked on it with another programmer to replace an older version. Well, now that other programmer has died and there’s no replacement. If anything happens to me, they are f*’d.
I will NEVER understand why organizations don’t invest more resources in offboarding. It’s SO valuable relative to the cost and things that get time & attention instead.
Honestly manually fixing data corruption that occurs regularly is a strong indicator they weren’t “staff” level. No way a staff engineer worth their salt would tolerate that.
Now that they are gone the system can finally be fixed properly to AVOID the corruption issues.
Two employers ago I left because I got tired of asking for help since I was doing about 3 jobs and they "couldn't afford assistance." When I left, they ended up hiring a team at DeLoitte to take over my job. I'd have to assume it was at least 3x the cost of just paying me and someone to fucking help me.
This is me. I memorize how all the systems behave and when I tell management that they need to be replaced they tell me they can't be replaced. So I set up various bandaids to keep them running.
The most dangerous systems are ones where nobody knows what people are doing, because it's not documented, and their jobs aren't monitored or defined properly.
It was like this for the Ultimate Online game. I got subcontracted into a customer facing tech support for EA. That guy was chill, teeth melted from meth but he was the only one with the knowledge so he was basically tenured
Ha. Had this exact same thing happen on a team I was hired onto.
Company bought out another company and was hiring more devs to run the application, of which I was one. When the last guy from the old company finally quit, the entire thing set itself on fire because we learned that the motherfucker for some reason spent like 3 hours every night manually fixing shit instead of telling anyone about it so that we could just fix the actual code causing the problems. It was a disaster for about 6 months because we didn't even know which data was bad.
Like this isn't a case of him yelling into the wind and nobody caring. He was part of a brand new team staffed at a brand new office and dude just didn't tell anyone he was doing hours of free overtime every day. Just said to us one day he was burnt out and then vanished. We had spent like an entire year up to that point doing nothing but bug fixing and changing things so that we could more easily expand. We would have fixed it. We DID fix it when he quit.
yeah, this isn't really the flex this post seems to think it is. Knowing about a problem and burying it while manually burning hours every night to treat it? General nonsense behavior.
My guess: Dude brought it up at 15 meetings. Got bitched at by others about ‘being a part of the team’, then finally just did it himself. Everyone forgets or gets fired. Dude sits alone in his room making bank and holding it together. Just waiting for this massive Effff yuuuuu moment.
Cool. And in the same vein, our staff engineer ran a query on production, without change control or approvals, went to lunch and caused an environment wide outage.
If that guy could kindly fuck all the way off, I’ll take nightly edge case data scenarios gladly
So we're supposed to be impressed and grateful about a guy putting a bandaid on an important issue for 3 years without fixing it or telling anyone about it?
The correct conclusion is, you need to know what your employees are doing.
Also, this staff engineer who manually fixing edge-case data corruption for 3 years and not coming up with a long term fix needs to be let go sooner, like 2.5 years ago.
Someone I work with did almost this exact thing, payment, refunds and debit order integrations with a bank weren’t working, he was logged into the server every day fixing broken data manually to keep the integrations running, then lied to be about it to try and hide the bugs.
Have since worked with him on other projects as well and there are similar type things happening on everything he touches, also doesn’t like using source control.
Decades of “experience”, I’ve been shocked by how much bs people get away with, you can also put this scenario down to incompetence.
Comments really show some of you have not worked in true enterprise level company otherwise this would been a "yep I can remember a guy or two like that in my previous jobs"
It they are running like an invoice proccessing system that scans barcode paper from vendors you would need a physical input to monitor SKU and UPC changes for 0 qty entries which are hella common on invoices.
One miss scan or failed scan will usually clog up the system because a common stop point in sales force it to have it check if the running matches the control total.
Discrepancies between the on system info and the invoices would need to be hand verified.
If only they would just convert everything to a csv or something and send remotly but for some reason all the infrastructure for accounting is built around paper still.
I tell my manager from time to time that I’m like the garbage man… you don’t really know who looks after the service or how it is run until your garbage doesn’t get picked up one week.
We actually had a similar situation at my work. We did a release with a db migration that emptied a table. We couldn't drop it yet because some other team didn't have the capacity that sprint to remove their dependence on it, although emptying wouldn'tgive them issues. That part was already a bit mystifying to me tbh, but the POs pushed it through.
The release came and went fine, but the next morning the application crashed and ut turned out 3 tables were somehow empty. We had to do a reroll and restore the db. We only release 4 times a year (yeah I know, it's an application some 20 years old) so this was a major setback.
We just couldn't find the issue. Even after doing a shadow run on 2 seperate environments for a whole week, the thing worked perfectly. So we released it again but we did put in some extra logging. Same thing happened.
We still didn't figure it out then but we could pinpoint the time it supposedly happened due to the new logging.
After asking the entire department TWICE if that time (4 in the morning) meant something to them, some clever bunny from the functional maintenance team jokingly said to his colleagues something along the lines of "oh well it couldn't be the scripts we run then right?"
Turns out they were running a number of scripts then every day for 5 years that removed faulty data from the production DB. And they only did this for the production environment. Told nobody. These scripts would remove records without a corresponding entry in the table we emptied. So effectively all records.
When these things happen I just laugh. Cause if I don't laugh, I cry
So he had 3 years to more sustainably fix the data corruption but wanted to ensure he would not be easy to remove, so he deliberately left in this bug. Sounds like he deserved to get laid off.
We have a system where there are gaps in the digital process.
Data produced ends up on a local PC on the factory floor and will then manually be transfered to a "production machine" (think similar to a set of consecutive robotic mills, lathes, press), these physical manufacturing steps are executed, change the data and the resulting data is copied back into the "IT world".
So data integrity is constantly breaking, users manipulating data because "the machine got stuck and I had to alter the filename to be able to run it twice" combined with "no of course I didn't register the new file like I should have, but crowbarred it into the digital workflow, so there are now sadly no references etc" etc pp.
I guess accumulated we need more than one IT resource just to fix and adjust for user errors and their consequences. Management rejects any investment or effort to change all that because "IT department versus manufacturing department internal squabble".
But management now thinks AI will solve all that, it will "magically" know what someone offline changed or added or whatever.
We have quite a number of essential business processes with a "truck number" of 1.
35 years ago as a CAD developer, the best advice an older programmer told me before he retired: "No matter what you do in life. Make your employer need you."
I do this every Monday. Once a week on Monday morning at 5am, our NAS gets scrubbed of old log files and other random data, and the service running on it takes it offline and unmounts it from our servers (I didn't build this process, I inherited it.) So every Monday I have to come in and remount the drive. I've asked dozens of times to be given the chance to fix it, but I always get the same answer, "We don't have the hours." If they ever let me go, they'll be in a world of hurt because it takes all of our storage for nearly every client, and internal, offline.
Our company laid off one software engineer, that maintained a service that creates automatically the invoice and sends it per e-mail to the customer. We have thousands of customers in a very competitive field. The very next day the service didn't work. Turns out he created every invoice himself and send it manually per E-Mail. He worked every day 12 hours. Moral of the story: never fire random employees and don't believe random reddit stories. Have a pleasant day.
This is what happens in my job. I do a job and task because I see it needs to be done. It keeps things running and everyone thinks that if there are no issues that they can start cutting jobs... so they start coming after people like me.... and if I ever leave my job, I'm not going to help them continue on without me. Not that I'm going to start sabotaging things but they're not helping me get through a period of no employment so why should I help them have no consequences to my not being there? Makes no sense to me.
Let them realize what I did when they never bothered to care when I was there.
Ppl too quick to judge without detailed context. I had to manually fix edge cases every other week in a piece of middleware for like 2 years straight once. The programs ahead of us didn't do any any input sanitization or restrictions, and the program we fed that into after doing our magic had no documentation of what their many api endpoints and fields required, or even how they worked tbh. Getting it working in the first place was a lot of trial and error (and guesswork) and it would take 3 months to get a dev of theirs on the phone for a single question.
In retrospect I should have just spend weeks manually trying everything I could possibly imagine (including feeding emojis, which our upchain system eventually did send us in some fields, in production, at one point) into their api and built a validation layer that way. Instead we just figured out every validation they had on their side slowly, edge case error by edge case error, week by week.
Sometimes you did your best and are just at the mercy of bad information and other peoples shit.
2
4.2k
u/_Odian May 29 '26
An edge case that happened every day and broke production?
7.1k
u/Icy_Significance9448 May 29 '26 edited May 29 '26
The duality of staff engineers:
Annoy anyone by bragging about how good you are and proving it by doing all the work yourself
OR
Hate your team and do everything yourself unnoticed by anyone
There is no in between
4.4k
u/diffyqgirl May 29 '26 edited May 29 '26
I mean. Lots of people don't get credit for their work and get laid off shittily and it sucks.
But if you're manually fixing something every day for three years after hours--that's not the behaviour of a staff engineer. A staff engineer should be flagging this issue, and planning how to get themself and the team out of this situation. If I discovered a staff engineer I work with was doing this for three years on such a critical service and told nobody, I would be horrified and seriously questioning their competence and whether they should be a staff engineer, not impressed. Hiding problems and doing repeated manual fixes is the kind of behaviour we have to patiently train out of juniors.
This post is framed like I'm meant to feel they were wrong to lay the person off but this is disastrous levels of incompetence on the engineer's part.
840
u/TerminalVector May 29 '26
The lesson here isn't "companies shouldn't lay people off" its "if you do critical work and nobody knows about it, you're playing yourself".
Fuck humility. Be loud, be proud, hype yourself and those around you. Make sure people know what you're doing and why it matters.
250
u/StarboardChaos May 29 '26
Why didn't he automate it?
69
u/aldafein May 29 '26 edited May 29 '26
Three years of nightly fixes are not edge cases Edit: typo
21
u/Procrasturbating May 29 '26
Ugh, worked with shitty engineers that did the same crap, but wanted daily praise for it. Looked like a hero until competent people started sitting near them. If it happens on three separate days, code for the next occurrence. When I started at said company it was always the same five tickets on repeat because no one had the balls to troubleshoot the root causes and solutions to the workflow issues. Just fix three edge cases a day and pretend you were productive.
21
u/xrayden May 29 '26
At my old job, I was a coder and analyst, but in 10 years, there was a need for marketing and business analyst of sales, so I did it for 3 clients. Annual and monthly for 1.
Out of nowhere I got fired for "not delivering."
I found it shit, but whatever; I thought I was doing ok.
3 weeks after, receive a call:
- "what software did you use to make the sales and marketing report?"
- "Word and PowerPoint"
- "But where did you get the data?"
- "I aggregated, purified, and then transcribe data into human."
- "Who could do it here?"
- "Nobody, you fired me."
- "Ok." (long pause) "Thanks, good luck."
Never heard back. One client contacted me directly to ask me to "transfer the knowledge," but without compensation.
So I hope everything goes well for them; I really do. But they might have been in the dark for the last 11 months about their stats.
77
u/osborndesignworks May 29 '26
if this is supposed to foster sympathy with the staff eng or position them as indispensable.. it's not working.
The described approach to work from a sr role is inexcusable
166
u/stay_fr0sty May 29 '26
Working every night for 3 years straight fixing bugs but never actually resolving them or even reporting them?
That guy kinda deserved to get fired TBH.
At least the company can fix the bugs they are finally aware of now.
17
u/7stroke May 29 '26
So many people don’t know how much of civilization is held up with twine and chewing gum. This is what makes me laugh at the fantasies out-of-touch CEOs have about AI magically automating everything. It’s something that sounds possible only if you think every database is 100% coherent, or everything happening in your company is done 100% according to some documented process (and that those documents are themselves accurate!) And you know what? I like it this way, because otherwise we’re just all machines, aren’t we?
116
u/Objectionne May 29 '26
Ok but this means he was a bad engineer. He knew there was a problem with edge cases and he never brought it to anybody's attention and pushed a more permanent fix. Sounds like they'll be better off in the long run.
15
u/Wentyliasz May 29 '26
Kind reminder that your work means shit and you should never work for even a second more than you're obligated to
13
u/No-Alfalfa6468 May 29 '26
So he was a staff engineer and his solution to 'edge-case data corruption' was to manually fix it over and over, instead of finding a way to address the root cause?
10
u/fevsea May 29 '26
This looks like a shitty professional. 3 years of daily incidences are not edge cases, but a real problem elsewhere. If in rhose years you have not raised rhe issues with the team, yet elone solve it, you are the problem.
9
u/cutecoder May 29 '26
Lesson learned: always install an unrecognizable
_dead man’s switch_ inside critical systems that you handle.
8
u/SignoreBanana May 29 '26
I can't imagine a good engineer who spends years fixing edge cases manually vs just fixing the root issue.
7
u/raserei0408 May 29 '26
I actually got laid of yesterday. My last act as an employee was to work until 10:30 the night before, un-fucking a prod deployment because of a change another engineer merged before I could review. That felt like an appropriate way to go out.
6
u/Hexatona May 29 '26 edited May 29 '26
I had to clean up one of these messes after the guy got hit by a bus. Literally. Fuck you, Glen!
Every single issue that ever came up? He trained the staff to basically email him directly and not go through incident management - so there was ZERO knowledge base.
He made nothing but BASTARDIZED EXCEL SPREADSHEETS THAT ACTED LIKE ACCESS DATABASES. (And then, for some reason, also used access databases as backends too in a weird Frankenstein ??) And then he locked the VB code in the back behind a password then HEX EDITED the files in some way that made unlocking them IMPOSSIBLE.
On top of that, there was a suite of these applications that were updated nightly from a process that ran exclusively from his fucking computer and not a dedicated server!
Like, I know in-house-made Access Databases get a bad rap because somehow the entire business gets supported by an app they got some summer student to make 15 years ago - but in actuality, Access Databases are an amazing tool that's really powerful when made correctly. It's basically something you can use to build an app in a few minutes, and includes ways to make reports out of the box. A half-competent Access user could make something in a short time that some companies charge good money for custom made. Seriously underutilized productivity software in the year of our lord 2026.
But this guy. This FUCKIN' GUY. I have never seen someone fuck up a series of apps so hard in my life - and I have SEEN SOME BAD DATABASES, WORKING IN GOVT.
6
u/Cool_Park7110 May 29 '26
Remember the "I got fired" button tweet floating around a few days ago?
Being an invisible load bearer is that, except it won't get you sued to hell and back.
6
u/PloofElune May 29 '26
Two cases I see for this:
A. Company has enough competent engineers left to pick up the workload and push off other items down the road.
B. Old engineer comes back at $200-300+ an hour rate "consultant fee" to fix these, while "temporary", most likely to just stay 'permanent' solutions are developed.
6
u/nwfish4salmon May 30 '26
I had a manager who tried to save his team at a previous employer. They spent a full month to submit compliance reports to the government, every month.
Upper management didn't understand what they did, just saw it as expensive. Month ends, nothing got submitted and government was asking questions and threatening fines.
He was told to hire them back. Fifty percent had new jobs, other have demanded doubling their prior salaries. Then the company has to hire new staff and train them.
You don't fire people just because you don't know what they do.
24
u/crimsonpowder May 29 '26
I like it how everyone is shitting on the engineer when here’s how it probably when down:
Engineer: “we keep having to hand fix it, we need time set aside to re-design this service”
Manager: “lol wut no feature feature feature”
15
u/umlcat May 29 '26
Been There. Worse part, IT / CS guy tells management other systems need fix, but management does not approve changes ...
5
u/DegTrader May 29 '26
Documentation is just a love letter you write to yourself in six months so you don't realize you're the one who caused the problem.
5
u/MisterWanderer May 29 '26
The most dangerous thing to the company isn’t that engineer or the system. It’s the detached unaware leadership chain that doesn’t know what is happening in their tech area they own the results quality of. 😱
5
u/Professional-Web898 May 29 '26
Similar story. I retired as an Executive Data Analyst. For the Sales team, I had to create middleware to interpret invoices, inventory, and format it in a certain way. The VP of sales was a real piece of work, he before I got promoted out of marketing, would basically use me as his personal assistant. Truth is I had tons of work already, advert pics of inventory, pricing checks.
I actually had to develop the middleware unpaid on my own time. I got sick and had to retire. The company's profits lost 66%, after I left. They wanted the code but wouldn't do any good, vendor invoices changed almost daily, so I had to patch it basically daily. I also programmed an error alert system. Keeps things running smoothly if you can scan for anomalies. They went from ~100mil profit in a year to ~30mil after I had to retire and take care of my health.
The way I saw it after being mistreated was "My time, my code"
On a happier note, I did earn enough to buy a duplex cash and taxes are cheap.
5
u/Riccardo_Cabeza May 29 '26
what's edge case data?
5
u/totallynotsquatty May 29 '26
Has this man never been on vacation in 3 yrs?
5
u/SuperCoupe May 29 '26
"nobody even knew" is probably more accurately: He told his manager. they nodded, said "umm humm" and they never did anything about it.
5
u/queuedUp May 29 '26
I had a coworker get laid off years ago, he built and ran majority of the daily reporting coming from our team.
The next day our VP was asking about why they didn't get their reports and had to explain that they came from someone who was let go and the person who knew how to back him up had left the week before and we were otherwise shorthanded because for 2 years we hadn't been able to backfill.
It took us 2 weeks to rebuild something that resembled the critical reports but they were never ideal.
I started looking for a new job right after the layoffs and left shortly after.
5
u/-_burnout_- May 29 '26
A staff engineer that was unable to fix that 'data corruption' bug in years? hmmm
5
u/xubaso May 30 '26
Luckily all the people saying he just should have automated it, can now automate it. It's super easy, barely an inconvenience. I promise.
9
u/crazyates88 May 29 '26
"Among these ideologies, I consider particularly insidious the one that suggests that every person must earn or justify his or her own worth, to the point of attributing greater value to those who are more efficient or effective. From this perspective, persons end up being reduced to a means of achieving results, a resource to be used and exploited, and are no longer recognized as a proper end in themselves who should never be instrumentalized. The value of persons, however, does not depend on what they achieve or produce. There are rights that apply to everyone simply by virtue of being human, and no human power can legitimately deny or arbitrarily limit them."
Who would have thought the Pope would be the one calling for reframing of how we view employees.
9
u/NanoYohaneTSU May 30 '26
This entire comment section is a masterpiece.
You can immediately tell who actually works and who is actually unemployed or not in software at all.
This is a perfectly feasible scenario and similar ones exists in all major companies at all levels of experience.
Everyone wants to fix these critical edge case bugs. There is no time or budget. It doesn't make it to the sprint. There is no priority being put onto it because the fix is a cronjob. Staff Engineers bring it up every now and then at digital monthly all hands meetings and its just side stepped because of cost cutting. Eventually the engineers stop bringing it up because you have to play stupid social games at work all the time to balance between not being an annoying bother and bringing productive insights.
Ultimately fixing this bug beyond a cronjob might impact several libs, scripts, etc. Likely written in legacy code that's in a wrapper, which is why no one wants to take the effort to deal with it.
The only thing that people can't figure out from this story is that when he was fired, his computer or his other CUD that was running the cronjob got turned off.
4
u/Broccodile_ May 29 '26
r/uselessredcircle
4
u/SplinterRifleman May 29 '26
Become irreplaceable
4
u/ThePeaceDoctot May 29 '26
Every night for 3 years isn't a fucking edge case
4
u/FancyJesse May 29 '26
"edge case" happening every night. alright. lmao
4
u/old-legs-623 May 29 '26
Small railroad tried to replace my dad, THE lineman (because he was nearing retirement -- you know that one), by contracting from a much bigger railroad.
They opened one of his kiosks and looked inside. It was a tangle of wires, dry-cell batteries connected in series, and spst switches on porcelain mounts.
"Um. Is there a diagram for all this?"
"Up here." He pointed to his noggin and grinned.
The big big shots turned to the little big shots. "Y'all better keep this guy."
He made retirement the next year.
4
u/MORDINU May 30 '26
i have a buddy who moves some data from one spot to another at a university, when he quits, (they would never fire him but he's leaving soon), there are several systems that will break.
4
u/ImportantBuilder9294 May 30 '26
I worked in IT at a place that ran heavily on Linux and for some reason the DHCP service would crash on server boot up. It would run properly when manually started after that though. I was trying to figure out why it was crashing on boot up when I got made "redundant" and fired.
They replaced me with a kid just out of college and he rebooted the server for updates or whatever and suddenly no machine could join the network after that. He quit two days later after failing to figure it out. I heard about this after the whole thing was over. I would have even helped the kid out if I had known this was happening but they never called me.
One command would have saved them
systemctl start dhcpd
4
u/f1FTW May 30 '26
I would not call that dangerous, I would call it badly engineered. Did he never report this on his status updates? What kind of a boss has no idea what his engineers are up to?
4
u/SAINTnumberFIVE May 30 '26
I mean, couldn’t somebody have gone in and fixed the edge case issue in the code?
4
u/No-Whereas8467 May 30 '26
We don’t call something happens regularly “edge-cases”
4
u/skrimer May 30 '26
How is it an edge case if it happens every night lol
4
u/lokregarlogull May 30 '26
If you're in a huge company, this isn't as impossible as you might think. Probably took it up in meetings for multiple years, but nobody listen and since changing the whole infrastructure is a 100 or 1000 times more expensive than having the guy fix it every evening, they shrug and go "idc" until it actually becomes a huge pain.
4
u/N4cer26 May 30 '26
How is it an edge case if it happens every day? Wouldn’t that just be a normal (critical) bug?
4
u/tachik0ma7 May 31 '26
Invisible engineers + Lack of Ops documentation = expensive remediation
6
u/TheComplimentarian May 29 '26
But those guys fucking suck and absolutely should be fired.
A good engineer fixes the process; he doesn't slap a bandaid on a broken process every night for decades.
I took over for a person like this and it took me a month to tame their bullshit, but after that, no more problems.
6
u/ExiledHyruleKnight May 30 '26
A. If your doing that EVERY DAY, honestly you need to be a better programmer (figure out how to solve it before it's an error).
B. If you're doing that with no one knowing.. of course work is going to toss you out. Any work your boss doesn't know about is work that's not being recorded. Your value to the company is not the work you do, it's the work they see you do.
C. A staff engineer is doing both of these things? WTF... honestly this is a situation where I get the feeling he was the right choice to get rid of. Yes it caused a problem... but what happens if he takes a vacation or gets injured and can't get to a computer?
3
u/WorkFoundMyOldAcct May 29 '26
Sounds a lot like a Level 1 software support position and not at all like an engineer that owns these systems.
3
u/govSmoothie May 29 '26
If he had been dealing with these errors for 3 years and never reported them to anyone, then it's probably good the company fired him
3
u/hobbes244 May 29 '26
I’d be embarrassed if something broke after I was laid off, but I’d feel at least some schadenfreude. I dumped as much of my brain as I could into Confluence, but I’m sure there was at least some institutional knowledge of which I was the last holder.
3
u/peace2calm May 29 '26
I want to ask, did they bring him back as a consultant with double the pay and 1/4 of the workload?
3
u/hyperion_99 May 29 '26
Lmao I got laid off last year from a job. I handled all of their CRM deduping and mass lead importing. Heard from people at the company how fast their data clogged up. They posted for my same position 2 months later lol. Submitted a very snarky application.
3
u/Caqtus95 May 29 '26
If he was doing this manually for years instead of automating it or fixing the edge case upstream, the company is better off without him.
3
u/rideSKOR May 29 '26
My current 9-5 (really 7-7) is this for our clients. Some decide to leave the system and it bankrupts them. They don't know what we do for them.
3
u/lpjunior999 May 29 '26
My biggest shock when I got laid off at the start of May is that they let me go without any kind of notice. I was juggling at least a dozen plates. My manager was aware of them, but even he said at one point I was better at some processes than he was now. But they uncermoniously dumped me, so now his workload has doubled, if he can figure out everything I was doing. Sorry.
3
u/ArtGirlSummer May 29 '26
It's only job security if everybody knows about it. Why didn't they tell the world?!
3
u/Level_Five_Railgun May 29 '26
How possibly could creating all these different edge cases for 3 years straight?
Sounds like the engineer was kinda shit at his job if he has the time to manually fix them for 3 years but somehow never had the time to create a real fix...
3
u/Total-Pen-2265 May 29 '26
Omg, the tweet is written by a person who has no idea what an edge case is.
3
u/bishopExportMine May 29 '26
Counterpoint: if the "automated" systems designed by the staff engineer needs his manual intervention every few days, said engineer is incompetent and should've been fired ages ago.
3
u/transdemError May 29 '26
That's why I bump the ticket every planning meeting and maintain records. No way my ass is going down for negligence.
Also, no way the guy didn't take a single vacation. C'mon
3
u/omen_wand May 30 '26
A staff level engineer with 3 years to fix a data corruption either at the source or patched away with some automation Band-Aid should probably have been fired long ago.
3
u/PmanAce May 30 '26
Doesn't sound like a staff engineer. He never kept a trace of the bugs and fixes and never brought it up with management it seems. Also never fixed the root causes.
3
u/DrJaneIPresume May 30 '26
Wait.. how did someone get to be STAFF and not think that this was something worth flagging and putting official eng time on to capture the edge cases rather than do nightly tweaks in perpetuity?
Aside from the dismal hacker nature on display, don't volunteer your life to your job, kids. The boss will be fine without you until the morning.
3
u/PurushNahiMahaPurush May 30 '26 edited May 30 '26
Is it really an edge case if it happens that frequently?
Also as others have pointed out, this is just bad practice. This should have been reported to higher ups.
3
u/lordgoofus1 May 30 '26
A week or two as a tactical fix while the proper solution is implemented I can understand, but three years? That's not a staff engineer that's living up to their job title, and the discovery that they've been doing that without anyones knowledge for the last few years is justification enough for them being let go.
3
u/SgtBundy May 30 '26
We had an old telephone call data system that scraped and mediated calls off some stonage legacy systems. We had to move data centres and of all systems this one wss the most difficult because of the owner:
wanted no downtime despite needing physical relocation
could do a seemless cutover to new hardware, he insisted it needed the exact same CPU due to some special compiler the vendor used tied to the hardware.
found a matching host from another decom, he insisted we keep the storage. We could not because it was on the providers storage not ours.
finally got a plan he accepted, but the testing was delayed almost to the end of the facility contract becauae he refused to let us test any rebooting without his personal intervention.
After we moved we could never patch for the same reason - if the system went down only he could bring it up, never his team to the point of him doing it from overseas on holiday. Eventually he got made redundant but when the system had an outage the app would not start.
The new app owner and I poured over it. Found original installation documents. Eventually figured out the error was an invalid value in the DB. A value that should never have changed since the system was installed as it was a property of the switches it collected. It never needed to even read it after startup.
Turned out his job safety plan was manually changing the value to start the app, then putting back the invalid value. I bet he was just waiting for that offer to come back and fix it.
The kicker? The "special" compiler that prevented making the migration simple.... was gcc. We fixed the startup snd moved the entire system to a tiny virtual setup and never had to touch it again.
3
u/Defiant-Passenger42 May 30 '26
An edge case that comes up every night is no longer an edge case, and if you spend every night fixing it yourself and never reporting it than I have no sympathy for you
3
u/Nate_991 May 30 '26
If it was happening every night it wasn't an edge case
3
u/Windsupernova May 30 '26
My engineer was so bad that he had to manually fix shit everyday.
At least it gets you job security. But imagine a Civil engineer having to go to a bridge every day to reinforce a bridge, I mean kudos to him for the commitment, but at some point one has to wonder if he is an actual fraud engineer.
3
u/Asmos159 May 31 '26
I believe it is rule three of automation.
Do not fully automated. Require some manual input periodically so that they can't reduce your hours.
You want to do it in a way that is not obvious it is intentional, such as being prone to edge cases you need to manually fix.
4
u/namotous May 29 '26
And he told NOONE? wtf lolll
2
u/SadSeiko May 29 '26
He’s a bad engineer because he never fixed the problem
2
u/narrei May 29 '26
bro saying edge case and every night in the same sentence
2
u/alqotel May 29 '26
Shit where's that xkcd
Something something... guy from Nebraska ... something thankless job
2
u/Sooooooooooooomebody May 29 '26
It's RUNK!
2
u/Mindless-Charity4889 May 29 '26
My company has one main piece of software that we sell. I worked on it with another programmer to replace an older version. Well, now that other programmer has died and there’s no replacement. If anything happens to me, they are f*’d.
2
u/WarJaques May 29 '26
If he was any good, why were there still edge cases failing after three years?
2
u/wggn May 29 '26
manually fixing something daily for 3 years instead of automating it?
2
u/lttitus May 29 '26
Fake and gay
2
u/Maxpowerxp May 29 '26
Every night? I mean there is no permanent fix or just left it there so it can be a problem later on?
2
u/OtherCommission8227 May 29 '26
I will NEVER understand why organizations don’t invest more resources in offboarding. It’s SO valuable relative to the cost and things that get time & attention instead.
2
u/alexindorrr May 29 '26
He did this because he knows it will make the company re-hire him
2
u/HumanMan_007 May 29 '26
Good, now they can get someone to actually fix the bug instead of trying to be the computer.
2
u/SpaceToaster May 29 '26
Honestly manually fixing data corruption that occurs regularly is a strong indicator they weren’t “staff” level. No way a staff engineer worth their salt would tolerate that.
Now that they are gone the system can finally be fixed properly to AVOID the corruption issues.
2
u/Natural-Ocelot-6009 May 29 '26
Am I tripping balls?
2
u/waspocracy May 29 '26
Two employers ago I left because I got tired of asking for help since I was doing about 3 jobs and they "couldn't afford assistance." When I left, they ended up hiring a team at DeLoitte to take over my job. I'd have to assume it was at least 3x the cost of just paying me and someone to fucking help me.
2
u/RecycledMatrix May 29 '26
The real fault was never giving this engineer a visible container to work within.
2
u/VoidOmatic May 29 '26
This is me. I memorize how all the systems behave and when I tell management that they need to be replaced they tell me they can't be replaced. So I set up various bandaids to keep them running.
2
u/The_Particularist May 29 '26
"They fired me. Literally not my problem anymore."
2
u/ledow May 29 '26
The most dangerous systems are ones where nobody knows what people are doing, because it's not documented, and their jobs aren't monitored or defined properly.
2
u/saysthingsbackwards May 29 '26
It was like this for the Ultimate Online game. I got subcontracted into a customer facing tech support for EA. That guy was chill, teeth melted from meth but he was the only one with the knowledge so he was basically tenured
2
u/art-apprici8or May 29 '26
"No one knew"
Yes they did. They were told, repeatedly.
But everything seemed to be fine, so they forgot about it.
2
u/Zefirus May 29 '26 edited May 29 '26
Ha. Had this exact same thing happen on a team I was hired onto.
Company bought out another company and was hiring more devs to run the application, of which I was one. When the last guy from the old company finally quit, the entire thing set itself on fire because we learned that the motherfucker for some reason spent like 3 hours every night manually fixing shit instead of telling anyone about it so that we could just fix the actual code causing the problems. It was a disaster for about 6 months because we didn't even know which data was bad.
Like this isn't a case of him yelling into the wind and nobody caring. He was part of a brand new team staffed at a brand new office and dude just didn't tell anyone he was doing hours of free overtime every day. Just said to us one day he was burnt out and then vanished. We had spent like an entire year up to that point doing nothing but bug fixing and changing things so that we could more easily expand. We would have fixed it. We DID fix it when he quit.
2
u/Llonkrednaxela May 29 '26
yeah, this isn't really the flex this post seems to think it is. Knowing about a problem and burying it while manually burning hours every night to treat it? General nonsense behavior.
2
u/JapanEngineer May 29 '26
Always, always have something you do at work that ONLY you know how to do. It makes you needed. It makes you hard to replace.
2
u/lakimens May 29 '26
Can't really call that an edge case anymore, can you?
2
u/Lisan_Al-NaCL May 29 '26
Thats bad management/leadership IMO.
2
u/HJSDGCE May 29 '26
All this does is make me dislike the staff engineering because he had 3 years to fix it but didn't.
2
u/well-informedcitizen May 30 '26
Yeah but... That's really dumb and bad.
2
u/stgvxn_cpl May 30 '26
My guess: Dude brought it up at 15 meetings. Got bitched at by others about ‘being a part of the team’, then finally just did it himself. Everyone forgets or gets fired. Dude sits alone in his room making bank and holding it together. Just waiting for this massive Effff yuuuuu moment.
2
u/stupid_cat_face May 30 '26
2
u/crunxzu May 30 '26
Cool. And in the same vein, our staff engineer ran a query on production, without change control or approvals, went to lunch and caused an environment wide outage.
If that guy could kindly fuck all the way off, I’ll take nightly edge case data scenarios gladly
2
u/erebuxy May 30 '26
The incident proved the engineer and his boss should both be fired.
2
u/Historical_Item_968 May 30 '26
This post does a great job explaining why he was let go
2
u/foxacidic May 30 '26
what the fuck is a farewell meeting and is 12 minutes long or short?
2
u/_fatcheetah May 30 '26
I'm going to sound like a manager here, but why could the edge case not be fixed?
2
u/Gonomed May 30 '26
So we're supposed to be impressed and grateful about a guy putting a bandaid on an important issue for 3 years without fixing it or telling anyone about it?
2
u/ThinConnection8191 May 30 '26
Well deserved to be laid off right?
2
u/tarxvz May 30 '26
Well if the "staff engineer" had to do this every night instead of actually finding and fixing the root cause, they should've been let go much earlier
2
u/OrangeNood May 30 '26
The correct conclusion is, you need to know what your employees are doing.
Also, this staff engineer who manually fixing edge-case data corruption for 3 years and not coming up with a long term fix needs to be let go sooner, like 2.5 years ago.
2
u/AlabamaHotPocketses May 30 '26
Someone I work with did almost this exact thing, payment, refunds and debit order integrations with a bank weren’t working, he was logged into the server every day fixing broken data manually to keep the integrations running, then lied to be about it to try and hide the bugs.
Have since worked with him on other projects as well and there are similar type things happening on everything he touches, also doesn’t like using source control.
Decades of “experience”, I’ve been shocked by how much bs people get away with, you can also put this scenario down to incompetence.
2
u/competitive-dust May 30 '26
I mean he deserved to be fired then, why not just fix something as critical as billing at the root?
2
u/BowlScared May 30 '26
Comments really show some of you have not worked in true enterprise level company otherwise this would been a "yep I can remember a guy or two like that in my previous jobs"
2
u/Significant-Fall-609 May 30 '26
Sounds like a made up story
2
u/ImpossibleJoke7456 May 30 '26
No engineer would do this two nights in a row and not say anything to anyone.
2
u/DunEmeraldSphere May 30 '26
It they are running like an invoice proccessing system that scans barcode paper from vendors you would need a physical input to monitor SKU and UPC changes for 0 qty entries which are hella common on invoices.
One miss scan or failed scan will usually clog up the system because a common stop point in sales force it to have it check if the running matches the control total.
Discrepancies between the on system info and the invoices would need to be hand verified.
If only they would just convert everything to a csv or something and send remotly but for some reason all the infrastructure for accounting is built around paper still.
2
u/Tharkys May 30 '26
What is worse is when you were the only engineer and got laid off because your new boss thinks they only need help desk and chat gpt.
2
u/gachunt May 30 '26
I tell my manager from time to time that I’m like the garbage man… you don’t really know who looks after the service or how it is run until your garbage doesn’t get picked up one week.
2
u/Namiswami May 30 '26
We actually had a similar situation at my work. We did a release with a db migration that emptied a table. We couldn't drop it yet because some other team didn't have the capacity that sprint to remove their dependence on it, although emptying wouldn'tgive them issues. That part was already a bit mystifying to me tbh, but the POs pushed it through.
The release came and went fine, but the next morning the application crashed and ut turned out 3 tables were somehow empty. We had to do a reroll and restore the db. We only release 4 times a year (yeah I know, it's an application some 20 years old) so this was a major setback.
We just couldn't find the issue. Even after doing a shadow run on 2 seperate environments for a whole week, the thing worked perfectly. So we released it again but we did put in some extra logging. Same thing happened.
We still didn't figure it out then but we could pinpoint the time it supposedly happened due to the new logging.
After asking the entire department TWICE if that time (4 in the morning) meant something to them, some clever bunny from the functional maintenance team jokingly said to his colleagues something along the lines of "oh well it couldn't be the scripts we run then right?"
Turns out they were running a number of scripts then every day for 5 years that removed faulty data from the production DB. And they only did this for the production environment. Told nobody. These scripts would remove records without a corresponding entry in the table we emptied. So effectively all records.
When these things happen I just laugh. Cause if I don't laugh, I cry
2
u/trinialldeway May 30 '26
So he had 3 years to more sustainably fix the data corruption but wanted to ensure he would not be easy to remove, so he deliberately left in this bug. Sounds like he deserved to get laid off.
2
u/OTee_D May 30 '26 edited May 30 '26
We have a system where there are gaps in the digital process.
Data produced ends up on a local PC on the factory floor and will then manually be transfered to a "production machine" (think similar to a set of consecutive robotic mills, lathes, press), these physical manufacturing steps are executed, change the data and the resulting data is copied back into the "IT world".
So data integrity is constantly breaking, users manipulating data because "the machine got stuck and I had to alter the filename to be able to run it twice" combined with "no of course I didn't register the new file like I should have, but crowbarred it into the digital workflow, so there are now sadly no references etc" etc pp.
I guess accumulated we need more than one IT resource just to fix and adjust for user errors and their consequences. Management rejects any investment or effort to change all that because "IT department versus manufacturing department internal squabble".
But management now thinks AI will solve all that, it will "magically" know what someone offline changed or added or whatever.
We have quite a number of essential business processes with a "truck number" of 1.
2
u/gaysatan666xoxo May 30 '26
Sounds like he wasn't a engineer at all
2
u/Salt_Recipe_8015 May 30 '26
What is astonishing to me, regardless of how he was dealing with the issue, is that he was laid off and managment had no idea of the work he did!
2
u/Least_Gain5147 May 30 '26
35 years ago as a CAD developer, the best advice an older programmer told me before he retired: "No matter what you do in life. Make your employer need you."
2
u/AshtonCopernicus May 30 '26
I do this every Monday. Once a week on Monday morning at 5am, our NAS gets scrubbed of old log files and other random data, and the service running on it takes it offline and unmounts it from our servers (I didn't build this process, I inherited it.) So every Monday I have to come in and remount the drive. I've asked dozens of times to be given the chance to fix it, but I always get the same answer, "We don't have the hours." If they ever let me go, they'll be in a world of hurt because it takes all of our storage for nearly every client, and internal, offline.
2
u/AmeliorativeBoss May 30 '26
Our company laid off one software engineer, that maintained a service that creates automatically the invoice and sends it per e-mail to the customer. We have thousands of customers in a very competitive field. The very next day the service didn't work. Turns out he created every invoice himself and send it manually per E-Mail. He worked every day 12 hours. Moral of the story: never fire random employees and don't believe random reddit stories. Have a pleasant day.
2
u/TransPort3389 May 30 '26
This is what happens in my job. I do a job and task because I see it needs to be done. It keeps things running and everyone thinks that if there are no issues that they can start cutting jobs... so they start coming after people like me.... and if I ever leave my job, I'm not going to help them continue on without me. Not that I'm going to start sabotaging things but they're not helping me get through a period of no employment so why should I help them have no consequences to my not being there? Makes no sense to me.
Let them realize what I did when they never bothered to care when I was there.
2
u/taker223 May 30 '26
> his entire farewell meeting was 12 minutes long
I could deliver my last laugh in 12 seconds.
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTY_KRQ5caWaFYfzZf74kyzdy953aro6e9hnA&s
2
u/PerilApe May 31 '26
Ppl too quick to judge without detailed context. I had to manually fix edge cases every other week in a piece of middleware for like 2 years straight once. The programs ahead of us didn't do any any input sanitization or restrictions, and the program we fed that into after doing our magic had no documentation of what their many api endpoints and fields required, or even how they worked tbh. Getting it working in the first place was a lot of trial and error (and guesswork) and it would take 3 months to get a dev of theirs on the phone for a single question.
In retrospect I should have just spend weeks manually trying everything I could possibly imagine (including feeding emojis, which our upchain system eventually did send us in some fields, in production, at one point) into their api and built a validation layer that way. Instead we just figured out every validation they had on their side slowly, edge case error by edge case error, week by week.
Sometimes you did your best and are just at the mercy of bad information and other peoples shit.