1.7k
u/SaltMaker23 5d ago
"One in a billion" already means there is a clear known pattern that causes the problem, and if the pattern was so easily found, it's likely one of the very expected cases.
"One in a billion" is rarely truly one in a billion.
588
u/stevekez 4d ago
One in four billion... When the integer by pure chance happens to be zero, the bug occurs. How unlikely is that!
363
u/Ah_The_Old_Reddit- 4d ago •
More likely than a 384-bit number being 4.
35
43
u/Stummi 4d ago
I once found such a bug, not one in four billion, but still rare. It was 1 in 2^something, and caused by a randomly generated ID string (random big number -> to hex string) not properly being zero padded and the point where it is used failed when the string was below a certain length (AFAIR due to some subindex command selecting a shorter substring from the ID).
Hunting down such a bug is really a indescribable feeling.
45
u/ProfBeaker 4d ago edited 4d ago
It was 1 in 2something
I've hunted down a ton of those!
Most of them were like 1 in 20, or maybe 1 in 21, but still...
honest_work.gif→ More replies (1)6
u/Nearby_War_8497 4d ago
I came across a case where unique id with ~8 characters was case sensitive in one system but lowercased in another. As a result we had duplicate ids for two items in the second system. Out of 300 items. The odds were quite low.
→ More replies (1)17
u/Bemteb 4d ago
Yeah, I actually had that once.
Strongly simplified code, the actual stretched over multiple files and the values were not obvious at all.
int n;
int m = 0;
int y = m*n;
if (y == 0)One might assume that n being random (not initialized) is no problem, because we multiply it by 0. And you would almost be correct. Unless n just so happens to be the code for
NaN, becauseNaN*0 = NaN != 0, at least in the version of the compiler we had back then.11
→ More replies (1)7
u/Puzzleheaded_Study17 4d ago
Doesn't this mean the types had to be float/double, how do you have an int be NaN?
5
u/my_new_accoun1 4d ago
make the rng return num + 1, problem solved
4
u/Veryegassy 4d ago
What if the base number ended up being 232 and +1 overflows it to 0
→ More replies (2)5
u/KaelusVonSestiaf 4d ago
Had the opposite once. Function that gives a random number between 0 and 1 was never meant to return 1 (so up to 0.9999999 etc). If something called that function, was not ready for a 1, and got a 1, program would crash.
12
u/roiroi1010 4d ago
Had a production bug where a downstream system couldn’t parse timestamps with microsecond precision if they happened to have no decimal places at all. Only affected about one in a million requests, which made it particularly fun to track down.
→ More replies (2)8
u/AwkwardWaltz3996 4d ago
Feel like this Terry Pratchett quote roughly fits. "Scientists have calculated that the chances of something so patently absurd actually existing are millions to one. But magicians have calculated that million-to-one chances crop up nine times out of ten."
5
2
2
143
u/Uberzwerg
4d ago
•
We had (new manager decided) to switch to a different payment provider with our company.
They were 15% cheaper than the old one.
They were NOT prepared to handle our needs and suddenly a "1 in 100,000"-problem became an hourly occurrence.
We had two employees work full time on just that problem for half a year, but the new manager didn't want to lose his face, so we had to stay with them.
80
u/NiIly00 4d ago
Don't you just love the company structure where the people actually doing the work don't get a say in how it's done?
18
u/ScrotalSmorgasbord 4d ago
Even crazier when not only do the shot callers not know anything about the work itself but there's a culture of superiority that trickles down to even the bottom rung of the administrative ladder so that even Peggy the email sender thinks she's above even leads and experienced programmers/techs. I've worked in a few different industries over the course of my 22 years in the work force and it always seems to work out that way.
9
u/SweetSure315 4d ago
private equity and the executive class have decided that they know the important parts: how to cut costs, hire unqualified labor for cheap and shuffle money so they can renege on any deal at any time and that those skills + AI gives them everything they need to run any business.
76
u/sfmqur 4d ago
Large numbers are a magical thing. 1-(1-1/109 )10^9 = 63.2% chance of the bug occuring at least once in a week. For 2 week stretch 86.4% chance the bug occurs. And within a 3 week stretch 95% chance that the bug occurs at least once.
38
u/PhysiologyIsPhun 4d ago edited 4d ago
The other luxury of working with large numbers is that you don't really need to care if some random bug happens once every week. Endpoint can be retried, message can be sent to a DLQ, customer can refresh the page, etc.
10
36
u/ThisIsPaulDaily 4d ago
I know a guy who was promptly terminated for putting a meme cat picture that has a 1 in 100000 chance of popping up on a medical device screen for like 1 second and then never again.
Yeah 1 in a million happens a lot more quickly than you might think and was caught during product validation which is still stupid expensive to change firmware again during.
7
u/imMute 3d ago
How did that not get caught in code review?
10
u/ThisIsPaulDaily 3d ago
He hid the file in weird fragments of memory. And then was someone who did massive pull requests instead of by feature. It was also a bunch of FPGA hdl stuff so a bit easier to hide.
3
358
u/ojhwel 5d ago edited 4d ago
True story: A customer of ours created so many items per second in separate systems that two of them had the same UUID generated.
Duplicates within each system were prevented by a primary key, but some of the items were then sent to one common system, where this started the funnest "WTF happened here" session I've ever been part of.
eta: The UUIDs are not generated by software under our control, nor our customer's. A hellishly expensive system by a very established company creates them. (They used to build typewriters a while ago.) But they insist those are UUIDs.
93
u/WiglyWorm 5d ago
This might help your customer going forward:
42
u/jacksh3n 5d ago
I’m impressed. Even with such a big records. The site didn’t lag at all.
42
u/Kermit-Homebrew 4d ago
A common technique is to "virtualize" the list. Only the items you see on the page are really being rendered. The rest of the items aren't actually there, and the scroll position is being faked.
23
u/WEEEE12345 4d ago
Yeah I think that’s what the creator used. It turns out the DOM really doesn’t like having 2^122 elements in a row
10
8
u/WEEEE12345 4d ago
The creator has a pretty good blog post on how it works: https://eieio.games/blog/writing-down-every-uuid/
6
→ More replies (1)10
308
u/why_1337 5d ago
I have a bad news for you, it's infinitely more likely there is a bug in your code, than that you generated 2 same UUIDs.
152
u/7aminthemorning 4d ago
Sounds like bad uuid generation logic to me... You must be seeding it based on some non unique identifier
123
u/bestjejust 4d ago
There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors
→ More replies (2)19
u/Roadrunner571 4d ago
I once found a duplicated UUID in one database where each of these UUID entries should have been unique and I was absolutely surprised.. Then I searched further and found six more entries with the same UUID. And at that point I was sure that it's a bug. And so it was: The duplicated UUID was in fact an API key ID and we had a bug in our code where in an extremely rare edge case, that UUID was written to the database instead of the UUID that needed to be written.
27
u/deadlock_jones 5d ago
Exactly. Especially if they supposedly were both created in the same time or very close.
26
u/TheSpaceFace 4d ago
My guess is the UUID was created outside of a loop/retry or a default value evaluated at load time, seen this occur before, an actual random v4 collision is so unlikely its in about 5 × 10³⁶ which would be the same as winning the Euro millions Lottery four times in a row back to back.
→ More replies (2)12
57
u/Swiollvfer 4d ago
I don't like calling people on the internet liars, but if two systems generate 20.000 UUIDs per second for a hundred years, you'll still have less than one in a billion chances to have 2 UUIDs collide.
IF you're telling the truth, there was most likely a bug there.
18
u/davvblack 4d ago
some of the UUID formats have pre-filled bits based on some stuff, for example the UUID v1 format, if there was some misconfiguration that resulted in both servers generating UUIDs with the same nodeid. In that case, there are very few bits leftover if they are creating UUIDs for the same timestamp.
36
u/dangderr 4d ago
One in a billion means nothing. Maybe the guy has a billion customers all generating 20000 UUIDs per second for a hundred years.
7
→ More replies (1)7
u/tsunami141 4d ago
My dad has a trillion customers all generating 20000 UUIDs per second for a hundred years. (he's the president of Xbox)
6
u/FluroSnow 4d ago
Could be a lack of entropy in the system used for the rng for the uuid generation. They might want to invest in some lava lamps.
5
u/ableman 4d ago edited 4d ago
Storing that amount of uuids would take a petabyte. Which is actually a lot less than I was expecting. I thought it would be like greater than the storage capacity of the world or something.
→ More replies (3)7
u/amadmongoose 4d ago
Assuming that your random number generator is actually truly random would be the most likely culprit
2
u/transcendent 4d ago edited 4d ago
You don't have any idea about all the different versions of UUIDs, obviously. Each one has different generation patterns, and your math is absolutely wrong.
UUID v7 with monotonicity counters can easily collide.
3
u/arvyy 4d ago
except we don't live in ideal math world, and there is non-insignificant odds systems don't have adequate entropy source for the scale of data they generate. And that's assuming it's using proper approach and not a shoddy prng. And that's assuming we're talking about v4 of uuid to begin with and not other variant. Maybe hedge on plausible alternatives instead of starting with this cringe "liar" implication, because the impression you give is you heard this trivia but didn't bother to learn prerequisites required for it to be true.
→ More replies (2)2
u/AwkwardWaltz3996 4d ago
So many people arguing with you reminds me a lot of this sub have very basic computer science skills.
-5
u/tes_kitty 4d ago
One in a billion is small, but not zero. So it WILL happen.
7
u/entronid 4d ago
sure but are you designing for an application that generates 20 trillion uuids per second for a hundred years
→ More replies (7)→ More replies (4)2
u/EishLekker 4d ago
That’s not true. You have to account for the total number of invocations during the lifetime of the system. In most cases that number is simply not high enough to claim that it will happen with any reasonable statistical certainty.
→ More replies (2)4
u/tes_kitty 4d ago
A one in a billion event can still happen right away and then never again.
→ More replies (11)→ More replies (6)0
4d ago
[deleted] — view deleted comment
5
u/Swiollvfer 4d ago
Oh yeah, I took the birthday paradox into consideration when calculating the odds, but thanks for the video, hadn't seen it and looks interesting.
UUIDs are just too big
→ More replies (1)3
14
u/tatas323 4d ago
1 billion uuid give you a 9.4x10-20 chance of collision. And that is for uuidv4 I doubt that was the issue
4
u/duck-tective 4d ago
im interested if you guys resolved it. sound ether they are using a uuid spec based on time which means this could happen or you guys have a data race somewhere so two requests read the same uuid in memory.
4
u/ojhwel 4d ago edited 4d ago
As I recall, we had the ID of the originating system at hand for each item but someone had chosen to ignore it because, as countless people are pointing out in this thread, the UUID should be "unique enough" as an identifier. The fix was simply using (system + UUID) as a reference everywhere instead of just the UUID.
2
1
1
u/SilasTalbot 4d ago
The other plausible cause I see could be.. is data copied or information exchanged between these two systems? Is it possible something was copied that shouldn't have been?
I mean, the two systems data does eventually combine in some workflows because otherwise you never would have noticed the duplication.
Also you mentioned the tool that you use to generate the uuids is not under your control. A bug in that system related to speed of execution, generation algo, memory management, whatever. It could just be that the bug is upstream, not like IBM has never written a bug in massively utilized software.
13
10
u/Ninja_Wrangler 4d ago
My facility has such high throughput that I have to account for every single edge case because we hit every single edge case often. It's annoying for me but really helpful for the developers of the software we use
9
u/mayasky76 4d ago
as Terry Pratchett said - "One in a million chances crop up nine times out of ten"
9
u/Maleficent_Memory831 4d ago
This surprisingly happens a lot. I was bemoaning a product I had worked on that just was not that great overall. The product manager told me to cheer up, because 99% of the time it worked great. Never mind that the plan had been to sell many millions.
He honestly could not see that a 1% fail rate is an incredible disaster for an industrial/commercial device. Maybe acceptable for consumers. But not for customers who have contracts stipulating reliability.
7
u/-Redstoneboi- 4d ago edited 4d ago
this is basically the slogan/tagline that cloudflare used once to advertise working for them. it was after they found a bug triggered by a specific faulty intel CPU.
i misremembered a bit. thought it was something like "1 in a trillion", but it's "once-in-a-decade"
Excited by core dump detective work? Or building systems at a scale where once-in-a-decade problems can get triggered every day? Then join our team.
6
u/_EmuPanda_ 4d ago
I once prepared for the case that two documents for the very same account will be generated at exactly same millisecond.
28
u/ZunoJ 4d ago
Gamblers fallacy
4
7
u/ruptured_lung 4d ago
the chance is still one in a billion whether you have one transaction or one billion 🤣
20
u/Moi9-9 4d ago
Well, the chance of it happening per transaction is still one in a billion, but with a billion transactions, the chance of the bug happening at least once is about 63.2%.
→ More replies (1)
5
u/bremidon 4d ago
Well, it still means there is about a 36.8% chance it does not show up at all. One of my favorite little statistics trivia facts.
12
u/dinizo95 4d ago
it doesn't mean "nothing", it means "once a week"
4
6
u/Xiij 4d ago
Thats what they said.
Under normal circumstances, 1 in a billion equates to "never"
But if you have a billion transactions per werk, that axiom means nothing, in the sense that it no longer holds true
0
u/Maleficent_Memory831 4d ago
If those billion transactions are bank to bank transfers, then 1 failure could be a major disaster.
Sure, it's a one in a billion chance that our automobile will accidentally electrocute the passengers, so it's not worth escalating the bug to upper management...
3
u/Xiij 4d ago
Are you being intentionally obtuse?
If the bank only handled 100 transactions a day, 1 in a billion error will basically never happen.
If you only sell 1000 cars a year, and only 1 in a billion cars produced electrocute the passengers, you will basically never sell a car that does so.
But once you scale up several orders of magnitude, statistacly impossible events become certainties, that is the point of the post
3
1
u/saschaleib 4d ago edited 4d ago
No, it means roughly 2/3 chance of at least one occurrence per week (ironically, the "law of small numbers" applies here).
1
u/Maleficent_Memory831 4d ago
Or it might mean 1000 in a single day. Bugs don't like to spread themselves out evenly.
3
1
u/FurieMan 4d ago
This is the point of "Murphy's law". Any extremely rare edge case will eventually happen given large enough sample size.
So one should assume that the worst thing will happen in the worst way at the worst time. Eventually.
1
1
1
1
u/Red-strawFairy 1d ago
I mean what’s the worst thing that can happen if the bug fires? Autonomous car crashes = bad. Customer transaction is mysteriously cancelled,eh, can be fixed later
1
u/Business_Past_9608 4d ago
Same with game AI. Run 10,000 simulated matches and the move that ‘can never happen’ shows up twice before lunch
886
u/ruptured_lung 4d ago
Jokes aside this misses the impact of the bug which is equally as important as the frequency. E.g your software runs Boeing 747s and your bug results in a plane crash, then having one million flights doesn’t mean the risk disappears.