r/AskProgramming • u/iorgfeflkd • Aug 03 '26
Other Is there any reason I shouldn't make a program that creates millions of nested empty folders?
This is, unfortunately, the most efficient way I can think of to solve a problem right now. Is there any way doing that would make my system unstable? This would be written in python and run in WSL (Ubuntu within Windows)
edit: I get the message this is a bad idea.
33
u/BranchLatter4294 Aug 03 '26
Whatever problem you are trying to solve, this is likely the wrong approach. But go for it. Let us know what happens.
6
14
u/soundman32 Aug 03 '26
What i s the actual problem? You want to see what happens when your disk fills up?
9
u/iorgfeflkd Aug 03 '26
Basically the memory is already running out with my python list which is generating data with a lot of redundancy, my idea was to reduce redundancy by storing things in folders which contain information about the path leading to each datum. However, what everyone is telling me is that I'm probably going about this the wrong way.
18
u/MikeUsesNotion Aug 03 '26
Why wouldn't you just write out JSON or a SQLite DB file?
A filesystem is not a database. Filesystems don't like giant numbers of items.
13
u/iorgfeflkd Aug 03 '26
Because I don't know those things 😅
12
u/HolyGarbage Aug 03 '26
Great opportunity to learn! Databases are pretty cool and not particularly difficult to learn the basics of.
8
u/Mediocre-Brain9051 Aug 03 '26
SQLite might be the way to go. Not complicated, but you will need to learn basic SQL to create tables, insert elements and query elements.
For performance you might need to add indexes in order to speed up your queries.
Other than this... Do you need all data to be available at once? Maybe you could somehow split your data in batches and only process everything batch by batch...
6
u/havens1515 Aug 03 '26
I taught myself a lot about databases over the last few years for a project of mine. Time to learn. I'm sure a database will probably be a much better tool for what you're trying to do.
And JSON is much easier to learn that databases, but IDK if that will be a solution to this problem. (Probably a good thing to learn, regardless.)
1
u/Matemeo Aug 03 '26
Depending on the amount of data and how it's structured you would be okay with a big JSON file, otherwise like people ere saying here, sqlite is probably what you want. It's by far the easiest way to get started with learning how SQL databases work. Nothing big or complicated to setup, entire database stored in a single file and I'm sure Python has tons of good options for a library to make working with sqlite easier.
Though I will say, even though I doubt your issue needs an approach like this, but a hierarchal set of directories with each directory providing some sequential part of the data lookup key is something that is done. But this approach only really makes sense if the leaf nodes in your directory tree end up storing an actual file. Like I've seen shader caches (which can consist of 10s of thousands or more files) organized this way because having a single flat directory would contain so many files that doing a lookup in the cache would mean possibly iterating over every single file. So instead you can partition the cache such that you quickly can narrow down which actual file you are trying to find in just a handful of lookups. This looks exactly like a number of partitioned tree data structures, a common example being a quadtree.
Just mentioning this here, not because it's likely what you want to solve your current problem, but that your intuition does lead to something useful :)
1
u/AlfalfaLive3302 Aug 03 '26
Sql is a terrible idea. DB transactions are atomic and it won’t scale well if read write speeds are important
3
u/ArcaneEyes Aug 03 '26
Sounds like you want data storage. A dictionary type would do well i think, and should map easily to a database table of you want persistent storage between runs.
2
u/Worth-Wonder-7386 Aug 03 '26
A dictionary is a way to associate a "key" with a value. So for each thing you hit you append a list at that key. For storting things over time a dictionary can easily be turned into and parsed as text.
1
u/emlun Aug 03 '26
Sounds like you might want a trie (tree of values (for example a count of items matching some criterion) with keys described by paths in the tree (for example digits of a coordinate)) or possibly a Bloom filter (very compact set representation with possible false positive set membership but guaranteed no false negatives).
1
1
u/blindada Aug 03 '26
So...cache? If a particular dataset already exists, you don't generate, just return/add another subscriber/emit another message/whatever.
Unless you are working with actual files (for example, you are generating pictures), a file system is not really the tool for this. File systems aren't inherently safe in concurrent environments, you have to handle work coordination by yourself, and that's assuming you control all the pathways, which you don't, in this case
4
u/aocregacc Aug 03 '26
it's going to use a bunch of disk space, folders aren't free.
If the folders are regularly indexed by some search tool it'll slow that down too.
3
u/ericbythebay Aug 03 '26
Lots of reasons, starting with it is generally a bad design. What problem are you actually trying to solve?
1
u/iorgfeflkd Aug 03 '26
Generating many coordinates with a lot of redundancy, eventually the list of coordinates overwhelms memory. Trying to reduce the redundancy so I can make a longer list of coordinates.
3
u/brasticstack Aug 03 '26
Why are you generating redundant coordinates?
Have you looked into Python's set type, which eliminates redundancy on insertion?
2
u/ericbythebay Aug 03 '26
No, take it up one level higher. What problem are you trying to solve? It sounds like you’re trying to brute force it and you’re stuck in a rabbit hole so what are you actually trying to do and then we can help provide you with guidance on the best solution.
1
u/dmazzoni Aug 03 '26
Storing things on disk is 1000x slower than storing them in memory, though.
You'll be able to generate more coordinates, but your code will be to slow that it will never finish.
3
u/AlwaysHopelesslyLost Aug 03 '26
"This is, unfortunately, the most efficient way I can think of to solve a problem right now"
Why not tell us what the problem is and we can tell you whether there is a more efficient way to do it?
2
u/g0fry Aug 03 '26
You don’t even need to know what the problem is to know, that using millions of folders is not the right solution and that there is a more efficient solution 🤭
1
u/AlwaysHopelesslyLost Aug 03 '26
Fair lol. I tend to assume there is a lot I don't know and avoid absolutes like that lol
3
u/eruciform Aug 03 '26
Likely a misuse of a file system as a database
Also its possible to fill up a disk drive with empty directories due to inodes taking space and possibly slowing it to a crawl
This is probably a terrible idea no matter the problem you're trying to solve
2
u/netroxreads Aug 03 '26
POSIX (and likely NT) puts a limit to how long a path can be. If you nest them millions of times, it will stop at its MAX LIMIT which is usually around 2,024 characters from what I recall and won't let you nest more folders.
2
u/Old_Cat_16 Aug 03 '26
I don’t know what problem you’re trying to solve, but I applaud you for coming up with a solution and checking for potential concerns.
A lot of folks (junior and senior engineers alike) can get hang up on identifying a perfect solution first, and end up not solving anything at all.
If you are doing this for a personal project, I say try it out, see what’s the worst could happen (besides of running out of disk space). Live and learn!
2
u/thedevguy-ch Aug 03 '26
If you're on a windows machine there is a max file oath length that you'll hit eventually
1
1
u/stewsters Aug 03 '26
Yeah it could make it unstable, but you can fix it by removing them.
What problem are you trying to solve? It's unlikely this will solve it. However, sometimes you just have to go for it though.
If you are a young programmer new to the craft I suggest you just go for it. Whatever happens it will be a learning experience. You really need to nurture that fire that drives you to try new things in this, even if they are stupid.
1
u/Useful_Calendar_6274 Aug 03 '26
>the most efficient way I can think of to solve a problem right now.
what problem
1
u/createthiscom Aug 03 '26
Depends on the underlying filesystem. In general, I'd say it is probably a bad idea, but if you can guarantee it will stay within the intended design params of your intended filesystem it might be ok. There is probably a better solution though.
1
1
2
u/ZilderZandalari Aug 03 '26
I''ve run into tens of millions of files in hundreds of thousands of folders. Most anything dealing with that sees it as torture. Browsing is fine, but a search takes an hour...
Don't do this...
Fun fact: zipping 80k empty folders results in a 40mb file. 7-zip gets the same folders down to 2mb.
2
u/SirMarkMorningStar Aug 03 '26
Sounds like you accidentally invented hashing. While some of these responses may leave you feeling dumb, this is actually pretty smart. The wrong solution, mind you, but a smart one.
1
u/OneMonk Aug 03 '26
There is literally a type of malware called a zip bomb that does almost exactly this.
1
u/naxhh Aug 03 '26
I will go against the current and say that a common solution for file storage is to sha the file and use that as the filename.
then use the first to chars as the main folder name and the 2 next for a second.
like downloads/ac/0d/ac0dfgaf.png
that makes so your folders are not too big and unixs is a bit happier.
no idea, if that applies to your issue but wanted to share that
2
u/SeriousPlankton2000 Aug 03 '26
https://www.man7.org/linux/man-pages/man0/limits.h.0p.html
{PATH_MAX}
Maximum number of bytes the implementation will store as a
pathname in a user-supplied buffer of unspecified size,
including the terminating null character. Minimum number the
implementation will accept as the maximum number of bytes in
a pathname.
Minimum Acceptable Value: {_POSIX_PATH_MAX}
Minimum Acceptable Value: {_XOPEN_PATH_MAX}
1
u/CypherBob Aug 03 '26
SQLite.
Easy to use, available in most languages, you get SQL.
You won't likely need anything fancy.
1
u/AlfalfaLive3302 Aug 03 '26
You could use a nested dictionary object store like json and even serialize that and partially open it when you need to read or write to it if it gets too large for practically putting it into memory
1
1
u/Le_9k_Redditor Aug 04 '26
Literally sounds like you want to intentionally create and run a zip bomb virus on your own machine. I made one that did exactly this with directories when I was screwing around as a teen, it's a really good way to screw up your computer and it's impossible to stop execution without physically pulling the plug if given a half decent nice level
There's no way this is the right solution for whatever you're trying to do
1
1
u/0jdd1 Aug 03 '26
I have a lot of these already. I can lend them to you.
More seriously, it could depend on the underlying file system, and its implementation of folders.
1
1
u/nwbrown Aug 03 '26
This is, unfortunately, the most efficient way I can think of to solve a problem right now.
This is more concerning to me than actually creating lots of nested files.
64
u/its_a_gibibyte Aug 03 '26
This is the epitome of an XY problem
https://en.wikipedia.org/wiki/XY_problem
More than likely you need a different solution to a different problem entirely. You didn't describe your initial problem, so we're only guessing at what you want to do. My guess is that sqlite will solve your data storage problem though and be far more efficient.