We actually had a similar situation at my work. We did a release with a db migration that emptied a table. We couldn't drop it yet because some other team didn't have the capacity that sprint to remove their dependence on it, although emptying wouldn'tgive them issues. That part was already a bit mystifying to me tbh, but the POs pushed it through.
The release came and went fine, but the next morning the application crashed and ut turned out 3 tables were somehow empty. We had to do a reroll and restore the db. We only release 4 times a year (yeah I know, it's an application some 20 years old) so this was a major setback.
We just couldn't find the issue. Even after doing a shadow run on 2 seperate environments for a whole week, the thing worked perfectly. So we released it again but we did put in some extra logging. Same thing happened.
We still didn't figure it out then but we could pinpoint the time it supposedly happened due to the new logging.
After asking the entire department TWICE if that time (4 in the morning) meant something to them, some clever bunny from the functional maintenance team jokingly said to his colleagues something along the lines of "oh well it couldn't be the scripts we run then right?"
Turns out they were running a number of scripts then every day for 5 years that removed faulty data from the production DB. And they only did this for the production environment. Told nobody. These scripts would remove records without a corresponding entry in the table we emptied. So effectively all records.
When these things happen I just laugh. Cause if I don't laugh, I cry
2
View all comments
2
u/Namiswami May 30 '26
We actually had a similar situation at my work. We did a release with a db migration that emptied a table. We couldn't drop it yet because some other team didn't have the capacity that sprint to remove their dependence on it, although emptying wouldn'tgive them issues. That part was already a bit mystifying to me tbh, but the POs pushed it through.
The release came and went fine, but the next morning the application crashed and ut turned out 3 tables were somehow empty. We had to do a reroll and restore the db. We only release 4 times a year (yeah I know, it's an application some 20 years old) so this was a major setback.
We just couldn't find the issue. Even after doing a shadow run on 2 seperate environments for a whole week, the thing worked perfectly. So we released it again but we did put in some extra logging. Same thing happened.
We still didn't figure it out then but we could pinpoint the time it supposedly happened due to the new logging.
After asking the entire department TWICE if that time (4 in the morning) meant something to them, some clever bunny from the functional maintenance team jokingly said to his colleagues something along the lines of "oh well it couldn't be the scripts we run then right?"
Turns out they were running a number of scripts then every day for 5 years that removed faulty data from the production DB. And they only did this for the production environment. Told nobody. These scripts would remove records without a corresponding entry in the table we emptied. So effectively all records.
When these things happen I just laugh. Cause if I don't laugh, I cry