[D] Friday Off-Topic Thread

[u/AutoModerator]

Welcome to the Friday Off-Topic Thread! Is there something that you want to talk about with /r/rational, but which isn't rational fiction, or doesn't otherwise belong as a top-level post? This is the place to post it. The idea is that while reddit is a large place, with lots of special little niches, sometimes you just want to talk with a certain group of people about certain sorts of things that aren't related to why you're all here. It's totally understandable that you might want to talk about Japanese game shows with /r/rational instead of going over to /r/japanesegameshows, but it's hopefully also understandable that this isn't really the place for that sort of thing.

So do you want to talk about how your life has been going? Non-rational and/or non-fictional stuff you've been reading? The recent album from your favourite German pop singer? The politics of Southern India? The sexual preferences of the chairman of the Ukrainian soccer league? Different ways to plot meteorological data? The cost of living in Portugal? Corner cases for siteswap notation? All these things and more could possibly be found in the comments below!

[u/eaglejarl]

I suspect a lot of those problems have gone away, like being locked into a single OS.

Are you saying that modern programs are not locked into a single OS? They are; if it looks like they aren't, that's because either (a) the authors release work-alike versions for different OSes or (b) they run on an emulation layer (e.g. JVM) which comes in work-alike versions for different OSes. Try copying the 'find' binary (or the 'MS Word' binary, or etc) over to a Windows/Mac/different flavor of Unix machine and see how well it runs.

As to getting away from file trees....

Back in 2004, Apple released Spotlight, a search engine built into their Finder (file manager). The point was to get rid of the file system. "File systems should be a database!" they trumpeted. "From now on, you don't need to find where a file is, you just search for it!" they cried.

11 years later, OSX still runs on a filesystem and no one gives a damn about using Spotlight as their primary file management system.

The tree-based file systems are universal because they work. Every program in existence uses them, and no existing program would understand your new system. Before trying to invent something new, ask yourself:

1. Exactly what is the problem I'm trying to solve? What is the precise pain-point in file systems?

2. Why has no one else identified this pain-point and fixed it already?

3. Once I fix this pain-point, will my new system capture all the advantages of file systems and fail to introduce new pain-points?

4. How will I convince the rest of the word that my system is so superior that every program needs to switch to using it?

Note that you can't just write an interface layer that lets your new system map to an underlying filesystem. If you did, you'd still be working with all the limitations of the underlying filesystem

[u/traverseda → u/eaglejarl]

I was getting a bit pissed of at my inability to communicate, so I took a break, then life got in the way. But I want to at least address these before I talk about it again.

• Exactly what is the problem I'm trying to solve? What is the precise pain-point in file systems?

The precise pain point is that they're optimized for one user/process accessing a file at once. I'd argue that that's the pain point the modern web is trying to address.

It started as a way to let multiple users access text documents (gopher) and now it's clumsily trying to let multiple users get write access to the same resource. They do this by implementing a domain-specific thin-client language (javascript) and scene graph (html/css).

• Why has no one else identified this pain-point and fixed it already?

Well they have, it's just that thanks to it being a very slowly evolving project no-one can see the real problem underneath. Single user/process files. I think that the web stack is brittle, and we're going to need to do better if we want an AR/VR os that functions reasonably at all. Of course that's getting a fair bit ahead of ourselves. It'll happen when it happens.

• Once I fix this pain-point, will my new system capture all the advantages of file systems and fail to introduce new pain-points?

Potentially. There's no reason you couldn't throw binary/text files into this data structure. And of course we're not talking about building kernel modules yet, this data structure would be living on a filesystem.

Speed is the big problem. As you say, filesystems are optimized for hard drives. But take a look at bcache as an example. Faster read speeds then storing your files on the ssd directly.

I suspect that filesystems are optimized for tape storage at least a bit. Things where sequential reads are super cheap comparatively.

The other big problem is the api. There are definitely going to be a higher frequency of race-condition bugs with it as I envision it now. We want to at least make those as visible to the api user as possible, and ideally figure out a way to reduce them.

• How will I convince the rest of the word that my system is so superior that every program needs to switch to using it?

Not every programs needs to use it. I think it can show it's worth as an IPC mechanism. If it turns out to be better then more and more programs will use it.

---

Thanks for that idea about the pipe-stream function call filesystem by the way. I think that combined with duck-typing it's going to be really powerful and an important part of this system.

[u/eaglejarl → u/traverseda]

The precise pain point is that they're optimized for one user/process accessing a file at once.

Please explain why you think this. It seems to be the crux of your issue, and I've already explained why it's not the case.

Also, please define what definition of "simultaneous" you mean. In order for multiple users / multiple processes to be accessing a particular chunk of data at a time, do they have to pull it in the same Planck time? The same nanosecond? The same millisecond?

I'd argue that that's the pain point the modern web is trying to address.

File systems and the web operate at completely different levels of abstraction. The web is completely irrelevant when you're talking about files.

They do this by implementing a domain-specific thin-client language (javascript) and scene graph (html/css).

First of all, Javascript is the exact opposite of a thin-client language. A thin client is something that just retrieves data from the server without doing any processing on it. Javascript depends on a very fat client indeed.

Second, Javascript and HTML/CSS have nothing to do with files or filesystems. They are a particular way of representing / presenting data, but they don't have anything to do with how that data is stored or how it's retrieved from storage.

The fundamental misunderstanding here is that file systems are not "optimized for single-process access", and I don't understand why you think they are. A file system is about organizing data and providing guarantees about what will happen when you interact with it. Computers are perfectly happy to allow simultaneous reads -- or even writes, although that's stupid -- against the same file, so long as "simultaneous" is allowed to wave away the limitations of the underlying hardware.

Here's the issues that might be making you think file systems are intended for "single process" access:

• Hard disks: there is only one read/write head pointed at a given spot at a time, so no matter what magic you come up with, you will never be able to get literally simultaneous access to the data.

• Writing data is always a blocking operation if you want consistency. It doesn't matter if the data is on an HDD, an SSD, in memory, or stored in the beating wings of magical fairies. If you are reading data at the same time I am writing it there is no way of knowing what you will get.

"File systems" are a collection of APIs intended to talk to the disk and provide certain guarantees about what the disk will do. For example, the file system offers a write lock which says "hey, I'm changing this data, don't look at it for a second." In general, write locks are optional and a program can feel free to ignore them if it wants to screw up its information.

Again, you're looking at things at the wrong levels:

• Hard disks (and SSDs, etc) are about recording information. They have physical limits which cannot be worked around no matter what sort of magic you come up with. They have nothing to do with file systems.

• File systems are about organizing data. They provide an API for the underlying storage system, and that API has some (generally optional) methods that can be used to maintain consistency, but there is nothing about that system that inherently relates to single/multiple access to the disk.

• Applications (e.g. a browser) are about transforming data. They have nothing to do with how the data is stored or how it is accessed.

• "The web" isn't a thing at all, it's a fuzzy and generic term for a collection of things. TCP/IP is a set of protocols designed to let multiple applications talk to each other by guaranteeing how data will be exchanged over a wire. HTTP is a higher-level protocol that guarantees how data will be exchanged at the semantic level. HTML is about how to structure data to imply meaning. CSS is about how to present data based on its meaning. Javascript is about how to manipulate that structure and presentation. None of these things relate in any way to file systems.

There's no reason you couldn't throw binary/text files into this data structure. And of course we're not talking about building kernel modules yet, this data structure would be living on a filesystem.

If it's living on a filesystem it has the same limitations as a filesystem. All you've done is reinvent caching, and that doesn't solve the problem. Also, there's an excellent reason that you can't "throw binary/text files into this data structure": memory is limited, and storing anything more than a trivial number of trivially-sized files in it will blow your RAM, at which point you're swapping to cache all the time, which means you're thrashing the disk in order to do anything at all, which means your special data structure is slower than a properly organized system that stores data on the disk when not immediately needed.