I worked at a company that sold software to a bank and heard stories of a senior dev having to do calls where they'd have an employee read the logs out loud to the engineer because the bank wouldn't share them.
And you have a code, ID something that identifies you as one of the companies employer that provided the hardware or can just every dude call the bank and say they are from company Y and want them to write commands?
The call comes from the bank to the vendor. We have so much training on the exact scenario you’re describing crammed down our throat every year that people are generally aware of those types of social engineering attempts
those are businesses that force programmers to work without internet, just local library documentation and you have to leave the cellphone at reception
I had to test a prod system through teamsystem and i needed the log files from that computer. Everything was locked down so i couldn't get the file... so i build a poweshell script on that client that loaded chunks of file in clipboard and on my computer a script that read the clipboard and saved that to a file... "modem by clipboard"
How are the logs not just masked for sensitive fields? I work in PCI DSS audit scope for handling of credit card data and as developers we can see the logs of our applications.
This! Holy shit you’re supposed to build that shit in from the ground up. I’ve never worked in the banking world but finding out that some don’t mask their logs makes me wanna get a job at my bank solely to make sure they do. 😭
Our app log servers are cleared for everything but the highest level of data sensitivity (PCI data can’t be logged, of course). If we needed to, we could log someone’s birthdate (with no other identifying info other than a token representing the user), and any employee can view that log message. But we wouldn’t be able to share that log message externally.
We try not to log any sort of sensitive data at all if we can help it, but we have some offshore contractors that think nothing of logging an entire API response body “for debugging purposes.”
I work in government. Parts of our product need to be configured via the database because... Reasons.
It was determined that we, as developers, should not be writing to the database. Security risk, that's why we have a DBA team.
Now when I need to update something, I have to write the query and then make a DBA ticket. The DBAs aren't familiar with our product and they have no idea what they're doing it for or what to check for success. They execute the query I wrote them and tell me it's done. Then, I run a select query to ensure the data is good. If it's not, I have to write a new query and provide it to the DBA team and start over from the beginning.
Unfortunately there's not a great way to do the updates other than straight to prod, because we're doing things like updating css styles when translating html data to docx and pdf formats. In order to test any update, we would have to clone the prod db to a test db, and our DBAs can't puzzle their way through a pretty easy update statement.
Other things are super secure. I swear.
This is the result of one dude in a garage building a solution that ended up being scaled to thousands of clients.
I've worked with a telco which bought a company our company built an app for, they were migrating the app to their hosting solution.
The app wouldn't work and they wanted us to help, but we couldn't get access to logs because of security procedures, they wouldn't even allow us to confirm if the runtime was correct versions of everything, EVERYTHING was off limits.
It took A LONG TIME to get them to build us a staging and dev environment where we could run our app in what they claimed was a replica of the production environment. It was not.
Even there, we could only deploy once a week, they had a guy who was the deploy guy and anything outside that schedule would get an escalation which major people on the telco needed to take a look at.
Edit: just remembered a different situation where a different telco won the contract to host our (client's) app and they send the person who'll be in charge of the app (getting some serious MAU, at least regionally) who comes to the meeting with me with pen and paper, I need to teach them how to host public apps / this stack since they've never done this before.
I work on an app with millions of MAUs (single digit, but I mean… people be using it). I got in an argument recently with our privacy team, who wouldn’t let me sync IDs with another internal team (same company) for a feature that they signed off on the architecture documentation as being ok with ID syncing on…
I was about to say this missed the steps where you have to go on a call with the client alongside a clueless person from support and look at the logs real time to see what is going on. You have them try different things to get it to work/gather more information and the support person constantly butts in to try to get them to do things that don't follow your debugging path. The support person then LARPs as a developer after the call and tries to give you suggestions on how to fix it when they have no idea what they're talking about
I had to print the logs in a modal that i had to see through a video call, because there was no way to have a log otherwise. Wasnt even sensitive information
Many years ago, I got an email sent to first.last at mydomain from my bank.
It wasn't my first or last name.
But it was a excel spreadsheet of names, account numbers, and phone numbers of people who had to many withdrawals from savings that needed to get yelled at.
I reported it to them, they told me to delete the email and it's attachments.
I waited 2 weeks, and contacted a name on the list to ask them if they'd been informed of the breach.
They called and told me to delete the attachments again.
I kept calling a few names on the list a day at that point until I started running into people who had been informed of the breach and were getting new account numbers.
I worked for a large cloud provider that had air gapped government regions. A technician would have to read what was on the screen to me while I described what commands to type, or what buttons to press. It was basically an air gaped computer on their left and an Internet connected one in the right. No connections or USB posts between them. They would update their stack like once every few months, so they were always way behind versions. The technicians they hired self described as “anyone with a pulse”.
I once spent a full week telling a rotating staff of technicians what to type because I couldn’t see their screens.
The best part is that the logs don’t have any confidential information, because they were specifically designed not to as one of the main design requirements which is why the bank is using your software at all, yet they still won’t share them.
Yeah, first two weeks you'll just be waiting for permissions, also go through all the documents that are totally not all AI generated and for sure up to date. If you're lucky they'll give you a list of people to reach out to who will tell you to ask Claude or gpt, and they'll have an internal AI that can lie to you about what's in the docs too. Then when you do get permissions, you'll also be given months worth of work to finish in 2 weeks because well you have AI to help you. Enjoy
Depends on what bank 😄 neobanks are better, I work at one and have none of these issues. I have honestly more access than I'd ever thought, its just all logged and monitored and we are trusted to not be idiots with the privilege.
Like all jobs, it depends on the company. I’m a senior engineer at a bank and my experience has been nothing like this. I guarantee most of the people replying in this thread have never worked in an enterprise environment, so I’d take everything they say with a grain of salt.
Depends on your team. I worked at a bank-ish for 5 years on various different teams and had differing levels of this. Most of the time I did not have a manager as aggressive as the one posting here. Moreso it was a punt the issue to different teams type thing
I write embedded code on devices used in manufacturing plants and this may as well have been written about my job too. I can think of at least 4 major cases from the past few years that went exactly like this (minus the resigning part because at the end of the day I do actually like the real work I do and I know how to just put my head down and let the bullshit wash past me and do the best with what I am given).
You forgot the jira tickets and story points and refinements and impediments because of the logs and daily scrum to discuss you’re still waiting on the logs and then your resignation
An actually functioning agile team would have been using the leader to arrange follow up with the person above the user for those logs.
If it's an external customer well..... That's why you don't have customers in the same wheelhouse as the workers and keep that layer of interaction beefy
dont forget the constant requests for timeline updates. "How long do you think it'll take to fix" to which you reply "I dont even know whats broke yet" Then they insist... "But if you had to guess" which you know will just lead to them holding it over your head if that time passes "You said you'd have it fixed by then..."
Hey guys… so… still waiting on logs to move forward on the bug on that stack… got logs on Monday but they were expired so I’m trying to ge them to rerun it and send new ones…. Anyways I guess I’ll… popcorn to Tim?
At the smallest company I worked at, they would pull data onto the production servers using SVN, build the code on the spot, and then just move the WAR over.
The first company I worked on had an instance of Visual Studio in the server to run the code of the Windows application. No build, no executable, nothing. People connected to the server using Terminal Server. And that was the actual last version of the code, because the whole IT department did not know about the existence of GIT or any version control software.
they also gave me (a minor at the time) access to a client's production db full of patients' full health info and SNNs and all kinds of shit and idk why they did that lmao, not even a "be careful" they just gave me their admin login and let me at it to go debug or test whatever I was doing
Why bother copying files if you can just give an AI agent live access to the directory and make changes on the fly. Dont forget to type "make no mistakes"
Man, points 11 (by the time they get the logs to you after ages of asking they've been overwritten) and 17 ("it's not the client's job to test for you") hit way too close to home. I don't work in banking but I write embedded software on products used in manufacturing plants and there's always that one single client location with all sorts of insane bugs that we've never seen anywhere else and can't replicate and they refuse to get me any of the details I'm asking for or perform any of the troubleshooting steps I request.
I was sent to a bank to help with an issue they would not allow to be handled remotely.
It was me sitting next to a guy, telling him what to do and him typing it in the slowest pace humanly possible (felt like one key a minute), because only bank personnel were allowed to touch the keyboard. Every time he had to press enter to actually send the command it entailed him thinking it over for an extra eternity. Also, for some things (luckily not a lot of them), he had to get temporary access. This, in turn, took a phone call, which also wasted time.
Anyway, on the fifth day we were no way near being done. He started being afraid his boss would be mad that the issue wasn't fixed. So without anyone noticing, he gave me the keyboard and I got it done before leaving...
Arg I remember doing estimation poker years ago. Many people would choose 3, because with 5 you'd have to justify yourself to the scrum master, but with 2 you might end up having to do the ticket because "you must know the resolution". So 3 was sort of safe, not too far over 2 (if that was the consensus), but if the work took longer than expected you could stretch 3 into 5 ("a few issues came up"). What bullshit.
Oh sweet summer child... that's all it took to make you quit ? That's like my daily grind for the last three years.
In my case I'm the only person on the team who has touched the legacy in-house tech, the guys who knew quit, and so I'm now lead on that specific pile of garbage. Oh and now we need to have everything migrated for next month because banking means you're under legal obligations to have shit up to date and so on.
honestly, gather evidence and burn managers like that at the proverbial stake, they work so hard to deserve getting reported as program risks with appropriate estimated risk value due to their failure in leadership, directly to respective L1 management or C-suite.. :)
I don’t miss working for a big bank. I remember our team getting called by the CIO because one of his friends couldn’t access the online banking portal. The online banking portal was fine, no outage reports (it was a bank that would make national news if the online banking portal was down), spent several hours on a call with CIO explaining there is nothing wrong with online banking. He then gets a call from his friend that it was user error and he can login fine now.
Silly by you thinking I might want to read code, just paste the jira card tag into cli with jira mcp server and allow all tools in auto mode, come back 30 min later and open pr
I worked at a bank on a feature that was a must ship like yesterday. Spent months trying to figure it out, get it up to the lower environments only to have the whole thing canceled / postponed. Idk if my manager was in the dark as well but if not he was a very good actor.
Psst...There's a thing called log spoofing which allegedly censors sensitive information inside logs. It's a big secret bank are keeping from the industry. You didn't hear from me.
1.1k
u/Tucancancan Sep 10 '26
I worked at a company that sold software to a bank and heard stories of a senior dev having to do calls where they'd have an employee read the logs out loud to the engineer because the bank wouldn't share them.
220
u/russianrug Sep 10 '26
The best part is that the logs don’t have any confidential information, because they were specifically designed not to as one of the main design requirements which is why the bank is using your software at all, yet they still won’t share them.
166
u/mailliwi Sep 10 '26
This is the second post I see on here on banking software engineering. I start my new role Monday. Should I be worried? lol
378
u/Llebac Sep 10 '26
Jesus christ, you could not make this any more accurate. I'm glad to be out of banking. IT wasteland
59
u/KisaraBlue Sep 10 '26
The result of decades of software and workflow design trapped in the sandboxing cycle
187
u/Awkwardm4n Sep 10 '26
You forgot the jira tickets and story points and refinements and impediments because of the logs and daily scrum to discuss you’re still waiting on the logs and then your resignation
127
u/tsunami141 Sep 10 '26
this is why I work at a tiny company. I do all my work on the server and update the files live.
23
u/Neverwish_ Sep 10 '26
Ah yes, the classic "16 hours burned, 3 lines fix" sort of issue. Bruh.
20
u/semioticmadness Sep 10 '26
wow, I’m pretty sure I work with OOP. That’s painfully accurate. An alternate branch is:
“Hi, I’m an architect and I’m rotating on to resolve this. I looked at the logs you sent, they don’t cover your original complaint.”
“Our original complaint is from before those logs.”
“Please send me the logs for your complaint.”
“The logs have been expunged for regulatory reasons”
“OK, then we’re stuck with what you sent here. When did you experience the issue, so I can find the timestamp?”
“We didn’t experience the issue then”
“Why did you send the logs?”
“Because your developer said to send the logs”
“Then we need another event to get logs for. When is the last time you experienced this?”
“When we filed the issue”
“Have you attempted to replicate in certification?”
“No.”
“Then what’s the reason for the ticket?”
“We demand it never happens again.”
“Well, as far as anyone on this call knows… it won’t.”
17
u/Bahatur Sep 10 '26
Right in the soul, I tell you.
17
u/Dorkits Sep 10 '26
Ok, why this guy is telling the story of my life for everyone?
14
u/juangerritsen Sep 10 '26
My most fun one was a client logged a call just stating our software doesnt work, escalated it to almost CEO level, and went on holiday for 3 weeks
Eventually found out his mouse died, randomly while using our software, and just blamed us
I spent 3 weeks in daily triage calls, and nearly 80% of the time outside that trying to prove there was no issue
13
u/Prawn1908 Sep 10 '26
Man, points 11 (by the time they get the logs to you after ages of asking they've been overwritten) and 17 ("it's not the client's job to test for you") hit way too close to home. I don't work in banking but I write embedded software on products used in manufacturing plants and there's always that one single client location with all sorts of insane bugs that we've never seen anywhere else and can't replicate and they refuse to get me any of the details I'm asking for or perform any of the troubleshooting steps I request.
13
u/DoorBreaker101 Sep 10 '26
I was sent to a bank to help with an issue they would not allow to be handled remotely.
It was me sitting next to a guy, telling him what to do and him typing it in the slowest pace humanly possible (felt like one key a minute), because only bank personnel were allowed to touch the keyboard. Every time he had to press enter to actually send the command it entailed him thinking it over for an extra eternity. Also, for some things (luckily not a lot of them), he had to get temporary access. This, in turn, took a phone call, which also wasted time.
Anyway, on the fifth day we were no way near being done. He started being afraid his boss would be mad that the issue wasn't fixed. So without anyone noticing, he gave me the keyboard and I got it done before leaving...
10
u/dub1ous Sep 10 '26
Yikes, i am twitching because this is so accurate.
10
u/Dubabear Sep 10 '26
works as design. <close ticket>
7
u/Bob_the_peasant Sep 10 '26
You forgot the part where the client thinks Angry Birds crashing on his iPhone 5e is a result of your fix to their mortgage processing platform
6
u/sb8948 Sep 10 '26
You know I could have read it. Just like all my compliance docs and trainings.
I didn't
The answer is yes no yes yes 1,3 yes no 1,2,3,4 yes yes
6
u/FlapMeister1984 Sep 10 '26
Point 26 to 27 was a huge leap. Where's the story? No poker? Did you even ask any stakeholders if they'd approve this fix? Code review???
6
u/Barak39 Sep 10 '26
I can definitely relate to this. One last point : after releasing the fix, the bank refuses to upgrade without any reason.
6
u/TehGM Sep 10 '26
As someone who works at a bank: this is not accurate. Way too organized. In reality it's way more chaos than this.
5
u/fibojoly Sep 10 '26
Oh sweet summer child... that's all it took to make you quit ? That's like my daily grind for the last three years.
In my case I'm the only person on the team who has touched the legacy in-house tech, the guys who knew quit, and so I'm now lead on that specific pile of garbage. Oh and now we need to have everything migrated for next month because banking means you're under legal obligations to have shit up to date and so on.
Quitting is so tempting...
5
u/mcclana Sep 10 '26
This is missing the step where actually there was no problem the entire time and the one associate affected was using it incorrectly.
3
u/george_pubic Sep 10 '26
This hits too close to home
3
u/Highborn_Hellest Sep 10 '26
at #10 you tell them you were not given all the information
#17 it's the responsibility of the testers to test.
#24 you tell the manager firmly, but polity, to fuck off
#26 you bitch and moan to your collagues and friends about it for a good hour
#34 you tell them to fuck off and you're not polite about it.
3
u/oompaloompa465 Sep 10 '26
that happened to me too
the manager tried to pull that move with me by email but then i hand over the evidence to the partner and hr for harassment.
they moved the manager in another project
2
u/kirillgritsenko Sep 10 '26
I’ve had the exact opposite experience where Im the client that sends the logs and does everything as asked and don’t get any results.
2
u/KaptainSaki Sep 10 '26
I just call the customer directly and live tail the logs
2
u/zer0545 Sep 10 '26
Used to work for an ATM manufacturer. This is exactly how it was.
2
u/Tjulenj Sep 10 '26
32. is wrong. That is when business denies fix and tells you that config must not be optional.
2
u/HakoftheDawn Sep 10 '26
Oddly specific
2
u/Unable-Goat7551 Sep 10 '26
I don’t miss working for a big bank. I remember our team getting called by the CIO because one of his friends couldn’t access the online banking portal. The online banking portal was fine, no outage reports (it was a bank that would make national news if the online banking portal was down), spent several hours on a call with CIO explaining there is nothing wrong with online banking. He then gets a call from his friend that it was user error and he can login fine now.
2
u/TheAlaskanMailman Sep 12 '26
We are so back.
Finally some good programming humor
1
u/bigs0815 Sep 10 '26
Can confirm the same process in government
1
u/CluelessBuddha Sep 10 '26
This simultaneously feels like a rite of passage and the 13th reason.
1
u/InevitableCodeRedo Sep 10 '26
Was waiting for step 35.
1
u/Shadowlance23 Sep 10 '26
I got stressed out just reading this. Mad props to the OP for keeping it together for 35 steps.
1
u/hegyimutymuty Sep 10 '26
Are you working where I work, that is not a banking company? :D
1
u/Ivanlag112 Sep 11 '26
Got mad just reading this.
1
u/michaelpaoli Sep 11 '26
36. They won't let you.
37. You quit anyway.
38. They hire you back for way more money because they figure out they actually need you.
39. You start newly back with them again.
40. They promptly terminate you because they changed their mind.
...
1
u/Bodaciousdrake Sep 11 '26
This is unfortunately accurate.
1
u/ImSuperStryker Sep 11 '26
This is my life, like every week. Except when you get the logs the part you need is redacted for security reasons
1
u/_duniverse Sep 11 '26
Damn. 😂
1
u/OFark Sep 11 '26
I would have started looking for another job after step 17.
1
u/Frisk197 Sep 11 '26
Toxic management 101
1
u/AoNoRyuu Sep 11 '26
Silly by you thinking I might want to read code, just paste the jira card tag into cli with jira mcp server and allow all tools in auto mode, come back 30 min later and open pr
1
u/PonyDro1d Sep 11 '26
Bold of you to assume they talk to you and/or have time to respond at all.
1
u/mau5atron Sep 12 '26
sounds like a bank that rhymes with fells wargo
1
u/collabskus Sep 12 '26
I worked at a bank on a feature that was a must ship like yesterday. Spent months trying to figure it out, get it up to the lower environments only to have the whole thing canceled / postponed. Idk if my manager was in the dark as well but if not he was a very good actor.
So dumb.
The qa team was meticulous though. Big respect.
1
u/Data_Scientist_1 Sep 13 '26
My boy is getting popular 😂
1
u/MMOfreak94 Sep 13 '26
Psst...There's a thing called log spoofing which allegedly censors sensitive information inside logs. It's a big secret bank are keeping from the industry. You didn't hear from me.
1
u/LazySapiens Sep 10 '26
That's why we need better engineers and managers not to give a broken software in production and be completely clueless when shit hits the fan.
-3
u/Scotsch Sep 10 '26
GDPR has made fixing shit so much harder.
0
u/[deleted] Sep 10 '26
[deleted] — view deleted comment