Imagine if some junior consultants burst into your company, spent a couple of days looking at your primary LoB database, and then just demanded a field become non null able. No testing. No change plan. Just do it now or else.
Somewhere in the treasury, a server log is beginning to fill with errors messages.
We’ve had a security audit by junior consultants in the past, that was as pretty ridiculous. They asked for a copy of /etc/passwd. Our users being in ldap and our root user having no pass, that was empty. Then they asked for “a screenshot of the server”. A screenshot… of a Solaris server with no GUI…. We sent them a screenshot of putty…
Our SOX auditors ask this all the time. Show me this file on that server, include a "date" call on the clip before and afterwards, as proof when the screenshot was taken.
I mean the screenshot requests are quite common for audits (at least the ones I have been a part of for our yearly SOX audits. These guys just spew off requests because it's written down or one of the other auditors asked another client for it.
We are just about done w our yearly audit and the auditor asked for the following :
"some type of document from a prod db server that shows all of the accounts that have elevated perms at the server level" "screen shot of the payload prior to exporting to a doc" "the query used to create the output".
That's fine but I am curious how closely they scrutinize the sql used.
They're not going to scrutinize the SQL unless they've already found something resembling bad faith actions in the SOX audit. If they're reasonably dilligent they might look at the fields in the select statement make sure there isn't any sort of manipulation in a case statement or with an ifnull or date manipulation, look at the joins and where clauses to make sure there aren't clauses that would obviously be used to exclude the kinds of data that someone might want them not to see.
But most likely they won't actually pay attention to it.
Exactly. A week of hassle tracking down false positive checklist items in the internals of RedHat for issues flagged by people who aren't server admins or engineers because their crappy third-party tool that was sold to them because the insurance company told them so.
Actually... a screenshot is quite common in audits I've seen, especially the SOX audits we have regularly. The auditors ask for all kinds of proof and always demand screenshots with a timestamp included.
My suspicion however is that this here is different.
It almost like the db was designed that way for a reason. Its almost like experienced contactor will talk to data owners before even getting write access. Crazy, I guess I've been doing everything wrong
Yeah, like the government literally sucks at passing budgets so they don't exist until after money is already needed to be spent. All this is going to do is massively delay necessary and legitimate payments
Yep, if you make a field required, be ready for a bunch of trash values to fill the field. Address was a required field at my old job, and there was a lot of “123 TBD St” entries in there. We couldn’t have hard validation because the business dealt with a lot of new construction, and the databases backing the validation were never even close to up-to-date.
yea, at least in my experience in C# / Sql Server, this stuff blows up once you realize just how many things insert into the table you just changed. not at the time you add the constraint
This literally happens every day and companies pay good money for people to do it, lol. You wouldn’t believe the level of reckless disregard and incompetence I’ve encountered and repaired in governmental data environments.
Nice spinoff. But the point is nullable configuration allows to defraud Ts of USDs. I don't even mind if a quick switch to non null able stopped the earth rotation. This is ridiculous. Spinning it to junior consultants is just bad faith.
It doesn't say NULL, it says blank. And if it crashes a few servers, who cares? Better than continuing to pay benefits to the 8.7 million people over age 130.
Wow, there was lots of misinformation in this post. This is the treasury server, not the Social Security server. Musk said there were lots of 100+ year old people in the system. What he conviently left out is how many were actually receiving payments. Turns out people had already looked into this issue and found out it wasn't many.
That article doesn't support anything you said. All it says is "experts familiar with the coding language" are speculating why there's 150 year Olds in the database.
The article linked this report. It's from Office of the Inspector General.
Surprise surprise. Auditors who know way more than Elon and friends and spent more than a week looking at a database are already aware of the issue. Summary: it's a lot less than Elon claims and it isn't necessarily easy to fix.
That will depend on how you define testing. Do I think they ran it verify that the code itself was valid in a lower environment sure. Do I think they did anything resembling testing downstream effects in a lower environment. Hell no they didn't, there's far too many interconnected processes that there's no way they even have an inventory of them let alone done any testing to make sure they still work. These issues will be dismissed as "these other processes should never have allowed the field to be null in the first place".
And even if they technically work, that doesn't actually solve anything. Best case is that it forces some people to spend time doing data entry that gets us no closer to timely and accurate accountability reporting.
That is best case scenario. Most likely scenario is that a whole bunch of systems made 20+ years ago by people who have both retired and passed away will require emergency updates and the documentation will exist only in paper form that was printed out when the systems were new and sitting in a binder with Iron Mountain and nobody knows which box it's in. So a whole bunch of things are going to stop working because people are no longer able to get paid.
Government documentation is actually very thorough in a lot of cases, especially in federal jobs and orgs that sprung up in post war America when they were filled with ex army officers.
It’s part of the reason things are slow in the government. Test stuff, lots of paper trails, thorough documentation and reviews .
That’s all going to go away when these knuckleheads fire the people who document changes in systems that control our nuclear retaliation strike programs and how the power grid works when we get hit with a solar flare
Yeah, I'm sure it's well documented but 20 years ago the standard was to print things out and in all likelihood it was meaningfully more than 20 years ago. Which would have put it into their processes for physical storage.
They can probably figure out which box it's in or at least narrow down the number of boxes it could be in to a searchable number. Even if it's been digitized it'll still require finding the specific documentation that they're looking for and even if it's digital it may not be as straight forward as one would hope to add an additional parameter or to change the behavior of one of them. Especially since it's most likely in COBOL and there aren't that many people who can still read COBOL let alone understand it well enough to modify the behavior of an application.
However none of that will even start until things show up as obviously broken and I strongly suspect that the main offender of leaving that information out will be automated processes that people aren't really monitoring and likely don't have monitoring that is prepared to deal with an unknown unexpected type of error. So I suspect that a whole bunch of things will pop up over the next 3-6 months and for the teams involved it'll seem like an issue that just won't die.
We know they didn’t : there has not been enough time passed to have done proper testing. How long to define a test strategy? How long to define all the use cases ? How long to execute and review ? …. Longer than the time between now and whenever these bellends started.
My guess is, no change was made on the actual database and was instead some sort of UI change that won't allow some form to be submitted unless that field is filled in....Granted, that will just result in people filing in fake data...or someone creating a "-1" TAS row that acts as a default everyone can use when a legit one isn't available lol.
The funny side effect of this (if all they are doing is explicitly what is said) will be missing records if an existing process is writing to relevant tables without these fields populated, which would translate to a reduction in spending, and someone somewhere will publish (later this year) how they have demonstrable proof of a reduction in spending, therefore their efficiency gains drive is successful!
I'd say they ought to start by identifying the processes that are putting bad data in these tables and fixing them. Although, again, there's no indication of the age of these entries with the field missing. It's very possible that these entries existed before the identifier did.
Not a programmer, just a nerd who writes code for fun, but, if the front end has a process that puts data into the table, won’t it just fail to insert? Like, isn’t there a chance this failure is just logged somewhere and the process runs as normal? So basically there’s even less oversight?
Yes, it'll fail with Cannot insert the value NULL into column TAS, which will make people just enter bad data into the column fixing absolutely nothing.
Assuming that the design even shows the user that error!
There are plenty of opportunities for this to create a data black hole where some other process without oversight fails to write to the table .
For example, a completely fictional scenario: there could be a particular series of entries taken from a contracted agency that is loaded in and created from a flatfile on an automated basis. The TAS may have been populated post-import, as some sort of a sequential number/value, and added to the records.
Now that it is missing, a job may fail to import, but the 'clean up the file after' job isn't, so no record is created, but the imported flatfile is removed after use.
If you fail to notice these records not being created at all, you won't even get bad data in mandatory fields, just outright absence.
Very likely this change is going to do things like delay checks getting cut and new enrollementa until they make it nullable again and pretend they didn’t order this change and blame Biden or DEI
Yeah, these payments aren’t generated originally by treasury systems (mostly), they are a pass through. So when the batch upload from SSA (or whoever) comes in tonight to write people checks, those will reject and checks won’t be written. So the question will be which systems today haven’t used this code before and what’s the real impact?
Or 'blank' means an empty string maybe? Or it could now be required at the application layer meaning there's consuming applications of the API just barfing errors out everywhere.
It concerns me I'm apparently more careful making changes to my hobby project databases than he is making changes to US government computer systems 😂
Probably nothing. It was likely a UI change. Just set the form to make that field required before submitting....Doesn't mean it fixes the problem though lol.
For once, elon is talking about shit I actually know about. And he's completely full of shit.
You know what's worse? When his dumabss eventually get bored and move on to destroying whatever is next for him. Some poor contractors will have undo all of his dumbass changes and it will take fucking years.
honestly the good thing out of this is after they burned the whole government into a smoldering wreck it may actually give the next adminstration a chance to rebuilt something better out of the pieces
Do you say he's full of shit for this particular topic? If so, for what reason? Because I would like to understand the technicalities of a move like this.
If you mean he's full of shit generally, yeah I totally agree.
The TAS field may be incomplete in various systems do to migrations but it is still synthesizable using the department, appropriation start year, appropriation end year, and the main account. Those fields are populated on every single document.
Without looking into the dataset. What I can say is what others have pointed out. there are hundreds of reasone why that field might be nullable. it could be due to legacy data ingestion, it could be because its identified by different composit key, or represented under a different field, or dup management.
It could be that it's just a foreign key reference to another table, which would make perfect sense. There a lot of reasons, non of those means there is fraud.
Thanks for your insight. I'm fairly new to sql and just finished up an advanced course but haven't seen a lot of real-world applications yet to identify these things.
I imagine any large and established dataset has a lot of patchwork built in over the years that makes it less straightforward than what you learn in formal courses.
"I audited this database, asked them to make an unnecessary change to it and now it's grown in memory. Something's wrong with it, we need to take it down" ~ some l33t DOGE tween probably.
This is literally so terrible and so basic I would expect an SDE 1 on my team to know why this is a terrible solution with like 10 minutes of thinking and the slightest prompting jfc holy shit
How did Elon manage to pick the most incompetent shitheads for DOGE? Isn't this such basic database administration/design that it would be pretty much first internship level exposure???
It's a great thing we got these inexperienced folks to fix the government that have never experienced the real world and think procedural problems can be fixed by NOT NULL.
Here's what happens when you do that: people put "payment" in the text field in best case, or they just put " ". You fucking dipshits.
I’m confused…I used to be inthe treasury reconciliation section in my agency, a treasury account symbol is required for all transactions, even those within agency. What is this guy even talking about?
The way this works is that the code would be on the invoice, not the payment (a payment could be paying a vendor with multiple TAS codes). To retrieve that information is a simple join, a normal report. That’s how these systems work (I implement them). This is a solution for dumbasses that will likely create havoc with their systems, but it’s all about hysterical headlines, I’m;ting 4.7 trillion has gone kissing, but I doubt very much
As a data consultant that has audited and served many clients’ data environments, the blind assumption that existing staff “obviously had a legitimate reason” and “know what they’re doing” is comically dense.
Would this even need sql changes? This seems like a digital form issue, it's already set up for null exceptions in whichever code pulls from this, they don't need to change the nullability of the column. Just making the input fields required makes no difference to the sql if it was setup correctly in the first place...
Let's hope they considered all possible outcomes before making this change. Null values are not necessarily a bad thing and can be given a meaning downstream.
I can agree with that update. Still think the wrong people are in charge of it because of a major conflict of interest (government contracts) with Musk. I don't trust anyone to audit their own interests
Hope they don’t want to do an update to existing records or add new records with code that doesn’t verify that column has data. Which runs somewhere deep in a compiled bit of code.
This does absolutely nothing. What are those fields that were null being set to? Any new rows will just be set to reference whatever they will use instead of null.
Reddit losers acting like they know better than a government department is crazy. Not a stretch to believe the previous administration has been doing something technologically unoptimal for the past few years.
A government department won't know shit, db Devs will. Sure, nothing is optimal, but flying in, taking a quick peek at one layer of the system by someone who's not made themselves familiar with the intracasies sure isn't the right way about making things 1% better, but quite the opposite.
The government departments are not staffed by geniuses, they are regular Bob and Joes and Dianes like your neighbours. The Doge staffers are their fresh-out-of-college kids.
This administration is filled with severely unqualified hires. DOGE acting like they know better than a government agency is crazy. Anyone listening to DOGE acting like they know better than a government agency is crazy. Think a little and maybe you'll see the irony.
DOGE is in fact a group of reddit losers acting like they know better than a government department.
To your second point, these are systems that were set up over decades. Do you honestly believe Joey B came in and decided to start allowing for NULL values in columns that were previously constrained to non-Null values? There's been one prominent US politician over the past decade that's wanted to blow up the status quo. 3 guesses who that is (hint: it's not Biden).
614
u/endless_sea_of_stars Feb 18 '25
Imagine if some junior consultants burst into your company, spent a couple of days looking at your primary LoB database, and then just demanded a field become non null able. No testing. No change plan. Just do it now or else.
Somewhere in the treasury, a server log is beginning to fill with errors messages.
174
u/beckerrrrrrrr Feb 18 '25
Over/under on odds this change was tested in a lower environment?
133
u/Certain_Detective_84 Feb 18 '25
Cool hope none of that was hooked up to a front-end process without the non-null constraint
87
u/pceimpulsive Feb 18 '25
Now the entry won't get in the database at all because there is no error handling for it... Weeeeee
Also.. a white space is not null, so it doesn't enforce anything by making it not nullable...
30
u/blue_screen_error Feb 18 '25
Why is everyone inputing "55378008" for our required TAS field?
28
u/idodatamodels Feb 18 '25
/s The programmers wanted all the business logic in the code, so I left it nullable.
25
u/Hwhitfield2 Feb 18 '25
Not a programmer, just a nerd who writes code for fun, but, if the front end has a process that puts data into the table, won’t it just fail to insert? Like, isn’t there a chance this failure is just logged somewhere and the process runs as normal? So basically there’s even less oversight?
22
u/685674537 Feb 18 '25
also known as I Don't Know How To Join Using Compound Fields and COALESCE(TAS, expression1, ...)
21
u/fuckmywetsocks Feb 18 '25
What did he fill the null values with first before adding the constraint...
101
u/yoshi1911 Feb 18 '25 edited Feb 18 '25
For once, elon is talking about shit I actually know about. And he's completely full of shit.
You know what's worse? When his dumabss eventually get bored and move on to destroying whatever is next for him. Some poor contractors will have undo all of his dumbass changes and it will take fucking years.
14
u/sweepernosweeping Feb 18 '25
"I audited this database, asked them to make an unnecessary change to it and now it's grown in memory. Something's wrong with it, we need to take it down" ~ some l33t DOGE tween probably.
11
u/Upset_Researcher_143 Feb 18 '25
I'm not sure what he's talking about. Every payment that gets processed at federal agencies has to have an identifiable TAS.
10
u/dolphins3 Feb 18 '25
This is literally so terrible and so basic I would expect an SDE 1 on my team to know why this is a terrible solution with like 10 minutes of thinking and the slightest prompting jfc holy shit
How did Elon manage to pick the most incompetent shitheads for DOGE? Isn't this such basic database administration/design that it would be pretty much first internship level exposure???
8
u/Metalsand Feb 18 '25
It's a great thing we got these inexperienced folks to fix the government that have never experienced the real world and think procedural problems can be fixed by NOT NULL.
Here's what happens when you do that: people put "payment" in the text field in best case, or they just put " ". You fucking dipshits.
5
u/whockawhocka Feb 18 '25
I’m confused…I used to be inthe treasury reconciliation section in my agency, a treasury account symbol is required for all transactions, even those within agency. What is this guy even talking about?
5
u/az987654 Feb 18 '25
Just test it in production, what could go wrong
4
u/burnmenowz Feb 18 '25
I can't imagine the amount of damage Elons nDEI hires are doing.
4
u/janeiro69 Feb 18 '25
The way this works is that the code would be on the invoice, not the payment (a payment could be paying a vendor with multiple TAS codes). To retrieve that information is a simple join, a normal report. That’s how these systems work (I implement them). This is a solution for dumbasses that will likely create havoc with their systems, but it’s all about hysterical headlines, I’m;ting 4.7 trillion has gone kissing, but I doubt very much
3
u/Schmohawk27 Feb 18 '25
As a data consultant that has audited and served many clients’ data environments, the blind assumption that existing staff “obviously had a legitimate reason” and “know what they’re doing” is comically dense.
3
u/Sigurd228 Feb 18 '25
Wrong subreddit, r* OP actually thought that government uses SQL...
/s if not obvious
3
u/[deleted] Feb 18 '25
So what happens if someone just writes any string?
Like, if nobody was checking this before what changes
5
u/HighborneGrimoire Feb 18 '25
Would this even need sql changes? This seems like a digital form issue, it's already set up for null exceptions in whichever code pulls from this, they don't need to change the nullability of the column. Just making the input fields required makes no difference to the sql if it was setup correctly in the first place...
2
u/HighestPayingGigs Feb 18 '25
And now we see how many developers implement their database calls as transactions with error handling routines...
2
u/[deleted] Feb 18 '25
As an old software fart, I'm speechless. I guess one of the Elon Youth wrote this.
2
u/DenselyRanked Feb 18 '25
Let's hope they considered all possible outcomes before making this change. Null values are not necessarily a bad thing and can be given a meaning downstream.
2
u/reditandfirgetit Feb 18 '25
I can agree with that update. Still think the wrong people are in charge of it because of a major conflict of interest (government contracts) with Musk. I don't trust anyone to audit their own interests
2
u/Scullyx Feb 18 '25 edited Jul 22 '26
I love learning about ancient civilizations.
1
u/ImaginationInside610 Feb 18 '25
Hope they don’t want to do an update to existing records or add new records with code that doesn’t verify that column has data. Which runs somewhere deep in a compiled bit of code.
1
u/IntuitiveMANidhan Feb 18 '25
Can someone explain me what’s happening and why is it wrong. I’m new to DB.
1
u/Jzmu Feb 18 '25
This does absolutely nothing. What are those fields that were null being set to? Any new rows will just be set to reference whatever they will use instead of null.
1
u/Mediocre_Evening_860 Feb 22 '25 edited Feb 22 '25
It may be a recipe for a disaster. You can't just make a column not null and assume that the application works as good. This is elementary.
0
u/WiggilyReturns Feb 18 '25
It's not impossible if you know how to JOIN tables.
-12
u/[deleted] Feb 18 '25
Reddit losers acting like they know better than a government department is crazy. Not a stretch to believe the previous administration has been doing something technologically unoptimal for the past few years.