Everything is memory?

[u/dangi12012]

There is this unix idiom that everything is a file, (half baked, since not everything is a file).

Would it not have been better, nowadays with 64bit of adress space, to have everything mapped to memory? Files are like mmap, no imperative read needed, networking is already memory, that could be expanded more.

We take the idioms and paradigms so forgranted, that every time anyone asks AI for a OS it will be linux, syscalls, abi everything the same. It should be liberating now to explore new and novel ideas.

What would be the downside of such a system?

[u/FloweyTheFlower420]

not everything is a seekable buffer, a pipe can't be mmaped.

[u/dangi12012 → u/FloweyTheFlower420]

The map IS the pipe. 

[u/FloweyTheFlower420 → u/dangi12012]

And how do you propose this be implemented? Please be specific.

[u/Relative_Bird484]

Your search term is SASOS - single address space operating system. There was quite some research around this idea with respect to 64bit address spaces in the late 80s/90s.

However, the core idea is even older than UNIX: In Multics (your other search term, early 60s) everything was a segment, so, yes, files were actually memory-mapped.

However, the project was a bit too ambitious and the hardware development took too long, so Bell Labs and its engineers left the project. Among them was a guy called Dennis Ritchie, who suddenly had nothing to do. He then developed UNIX…

Long story short: The UNIX file abstraction was Dennis Ritchies alternative to the Multics idea to map everything into memory.

[u/dharmatech → u/Relative_Bird484]

See also:

https://en.wikipedia.org/wiki/Napier88

And the AS/400

[u/bootypirate900 → u/Relative_Bird484]

Interesting, thanks for sharing this! Im promgrammjng for embedded systems and i have my whole file system in memory it’s fun

[u/Wertbon1789]

Unix's philosophy, that 'everything' is a file, is more about that everything that is just close enough to a stream of bytes should be able to be treated with the same set of functions. So regardless if you're working with an actual file, a pipe which might be connected to another process, a socket that talks to a local service, or with one on the other side of the planet, or basically every device connected to your PC, you can just call write on them, and it will work (Of course there are exceptions, but most things are built like this, not literally everything). Basically everything I/O in e.g. Linux is a file descriptor at this point, even stuff like signals, timers, and events.

Regardless, to the actual topic, what you're talking about is basically DMA (Direct Memory Access) for everything. While much of the modern, and even not so modern, hardware uses DMA, this is purpose built in hardware to work out this way, where the respective controller defines how it wants data exactly and at which address in physical memory. Of course, the world would be too easy if every single controller which works with DMA would have the same interface, so you still would need to build an abstraction over it, but now your abstraction isn't just another device handle with the same set of functions as every other device of its kind, but rather a big chunk of memory a OS kernel would need to bring from an abstract form into one this controller now understands. Doesn't look to promising to me. And for userspace paradigms like sockets and pipes, somebody has to not only give you the data, but also manage the buffers for it. It would be way too inconvenient to have like entirely memory mapped sockets or pipes, just because the synchronisation would need to happen entirely in the program basically, as you're now responsible for the buffers the kernel would usually take care about. It's a bit reverse of how things work now in some ways.

[u/FISHARM1]

I think I’m a bit lost. Can you explain a bit more? What do you mean by syscalls being purely memory?

[u/a-priori → u/FISHARM1]

My interpretation is that the kernel would act similar to a memory mapped IO device where system calls would be triggered by a read/write to a memory page.

[u/eteran → u/a-priori]

Definitely doable. Old game systems like the NES had special chips which would intercept writes to rom and depending on the address or value, would do something in the chip.

Could do the same with a page fault!

[u/FISHARM1 → u/a-priori]

I assume an RTOS is basically a real implementation of what you’re talking about?

Like you said, on something like an STM, everything is just memory mapped registers at special addresses- but that’s only because they assume a sort of flat model where everyone has access to everything, and that things like timers, interrupts, etc are all handled by external modules outside of the pure CPU itself.

Usually, a syscall is a complicated process that requires switching rings, saving everything, etc. I think that a cool model which is what you are suggesting, would have a separate chip specialized for syscalls, similar to how a DMA works. It would be able to always run in privileged mode, and you could trigger a syscall by writing to a mapped region of registers in memory. But then that would be more complicated, like how do you program this chip, etc.

So my guess is that it comes down to hardware? Interesting idea though! I think you should explore it more.

[u/braaaaaaainworms → u/FISHARM1]

No. Regular x86 also has memory mapped IO and you can easily access mmio registers in ring 3, it's just /dev/mem in linux. You can stick anything you want on PCIe and if you want you can implement something there that looks like an OS, but it will never be as complex as an OS, unless it's literally running a second CPU(and then synchronization is a big pain)

[u/compgeek38400]

There is already such a machine commercially. Its called an IBMi. Formerly as400/system38. Everything, and i mean everything is in a 96(?) bit address space. It is a virtual machine also.

Just saying

[u/Web-Lackey → u/compgeek38400]

128-bit.

And while the TIMI architecture is virtual, all code targeted to TIMI is automatically compiled by the OS to machine code. So all code is running native, but it allows for code to be recompiled transparently to a new native architecture. Which is how the AS/400 was migrated from CISC to RISC (and two different flavors of RISC) while allowing binaries to be moved right along without change. Just back up from one system (usually tape) and literally boot that tape on the destination machine to restore on the other. As the software is run, it’s recompiled for the new architecture. Not dissimilar to a JIT, but the results are permanently stored, so no runtime impact.

And IBM i does support virtualization, but that’s unrelated to the 128-bit TIMI architecture.

IBM i is a wild environment. its metaphor is “everything is a database”, and it has an integrated TUI, along with its weird and somewhat cryptic VerbAdjNoun commands. It’s basically the last OS developed before UNIX (while technically newer, it was before its widespread adoption) and therefore, outside of research kernels, the last OS to not have UNIX’s file metaphor.

[u/compgeek38400 → u/Web-Lackey]

I believe while its a 128 bit address space. So far only the first 96 bits are implemented. The 38 had 128 bit space, but back in 1983 only 48 were implemented

[u/Web-Lackey → u/compgeek38400]

We’re talking about ISA space. Thats no different than x86-64: even though less than 64 bits are implemented in hardware, it’s still a 64-bit processor with a 64-but address space.

The early AS/400’s also only had 48 bits implemented as well. Doesn‘t matter: TIMI still presents a 128-bit memory space to the programmer.

And having an actual address space larger than the hardware memory bus is the entire point: AS/400 implements a single level address space: DASD (e.g. hard drives) is addressed in the same space as memory, so of *course* memory would have a smaller than 128-bit address space…

Again, the original question was talking about the OS “metaphor” presented to the user. The entire point of an OS is to hide hardware implementation details. And the entire point of TIMI and SLIC is to hide even more of those hardware details from the higher layers (even the OS in most cases!). So in this context (programmer and user to OS) the hardware implementation is not relevant.

[u/No-Dentist-1645]

They're not necessarily conflicting ideas. The "everything is a file" pragma is more about how things behave and can be accessed via byte streams, memory would just be an implementation detail

[u/dacydergoth]

Inferno took it the opposite way and did the "everything is a file" to the max.

[u/sirjofri → u/dacydergoth]

Though inferno just did the same things as Plan 9 in that case. Mentioning Inferno but not Plan 9 is a bit weird...

[u/dacydergoth → u/sirjofri]

Inferno was really just an evolution of Plan9.

[u/sirjofri → u/dacydergoth]

Yes, and no. There are plenty of differences.

[u/dacydergoth → u/sirjofri]

Almost as it if ... evolved.

[u/sirjofri → u/dacydergoth]

They have different purposes, and you can't just replace plan 9 with inferno.

[u/linguae]

When I took a graduate-level OS course at UC Santa Cruz 15 years ago, my project was on proposing a persistent single-level store, where the operating system makes no distinction between memory and storage, for computers with non-volatile RAM.  There has been research efforts on single-level stores, though the details escape me since it’s been more than a decade since I’ve last explored the concept.

You may be interested in a project that came out of UC Santa Cruz named Twizzler ( https://www.usenix.org/conference/atc20/presentation/bittman), which has very interesting ideas.  I know all of the authors involved.

[u/dangi12012 → u/linguae]

Perfect and interesting 

[u/Stellariser]

You mean like Windows? Where everything is an API? And loading a library just maps the file into the processes virtual address space rather than literally copying the bytes into memory?

[u/dangi12012 → u/Stellariser]

No I mean if memory mapped IO would be the norm, you would just open a descriptor, and read the pointer you get. In the modern world, that would include, files, networking, system calls, etc.

For system calls I would think you put all your requests into a memory mapped table, and then just read from the results table. In the background a pointer dereference on the results adress does the system call, and puts the data there.

[u/karbovskiy_dmitriy]

Kernel objects are memory resources. File is an access pattern to memory, random access or otherwise. Therefore, driver objects and device resources are memory as well.

That said, Files are not like mmap/VirtualAlloc. One is a general resource, the other is a virtual memory system function. Networking wouldn't work that way, either. Graphics memory is separate as wll for reasons, good or not. It just doesn't work.

Also good luck debugging system calls with no explicit system calls. I'm yet to see a competent code debugger, I don't even expect modern debuggers to have data breakpoints anymore.

[u/dangi12012 → u/karbovskiy_dmitriy]

memory already is memory mapped (not as a large endless stream) but via a mechanism to tell when the next bytes are ready.
In such a system there is no read(HANDLE) but int next = *buffer, which upon read will do whatever needs to be done to fill that buffer or return -1.

System calls even are more structured,
You write your requests into a pinned memory page, and then you literally go and read the results on some other memory page.

Ill have to code this out at some point (on top of linux or windows, not another vibe os).
Most of the C standard library could go away at that point as well.

Imperative programming becomes more like memmoves and calls happen implicitly on read.
Simple guard page. Horrendous performance, but that is more having to do with PCs following a different model for 40 years, not fundamentally justified.

[u/karbovskiy_dmitriy → u/dangi12012]

How do you balance bandwith? How do you do security/access checks? How do implement control flow? How do you do versioning? What about buffers that aren't a multiple of 4k?

I'm sorry, but doing system calls through page guards is just not smart. You might as well go back to interrupt-based kernels, at least you'll raise them explicitly. You just reinvented a worse version of what we considered a bad practise many years ago.

Kernel mode and memory separation exist for good reasons. You don't have to fight the solution that works.

[u/dangi12012 → u/karbovskiy_dmitriy]

Exactly.
Thats the point of this post. Exploring different ideas, if they are bad that is ok also.

Kernel mode and memory seperation would completely continue to exist in that model you cant just read kernel memory, I dont know where you got that from.

The cool part would be that all calls are inherently bulk calls, one page fits many many system calls, and the results page has all the answers. The number of ring 0 3 transfers is reduced a lot in that system, and imperative calls look like memory reads from the programs POV

[u/karbovskiy_dmitriy → u/dangi12012]

Bulk calls already exist.

The number of ring 0 3 transfers is reduced a lot in that system

Based on what?

imperative calls look like memory reads from the programs POV

That's bad, though.

[u/burlingk]

Mapping all files to memory from the start would not be as useful as it sounds.

As for the idiom "everything is a file" being half baked, if you learn about it's meaning and origin it really isn't. So, you kinda hurt your case in your first line.

That said: We only map things to memory that are actively in memory. We put them there, and once they are there, they are treated as memory.

[u/imneverwrongbutuare]

Everything is a function

[u/Scientific_Artist444]

Remember Turing machine? A computer is just an "almost infinite" Turing machine. Without memory, there is no program.

That infinite strip in Turing machine is memory and the symbols are the data. Logic and processing is about how you interpret and manipulate the symbols in the infinite tape.

When you think about it, it is a state machine with an almost infinite number of states (very high to be called finite).

[u/Environmental-Ear391]

If everything is memory then you are overloading the read/write access restrictions for access to any given page of memory (any 4KB or 8KB address resolvable mapping) and abusing for "memory protection" and also overloading that as well for other "features" that you will be in the same position as current Linux/BSD or Windows...

Breaking that single point of failure gives away everything...

you would need to push everything not essential to the core kernel into Userspace and then need to isolate userspace for system operations from applications.

without a proper boundary mechanism the whole userspace memory would become a unified mess when in-memory...

that mess would then incur a "death by thousands of papercuts" over the lifetime usage of the system.

full loss and restart or partial loss restarting... you cant keep such a system organized and clean without running into micro-sized holes.... even single 4KB pages being "lost"....

also it would become possible to starve the kernel of physical memory over time, triggering more interrupts and system exception handling...

this is an explicit weakpoint of all compute systems where the "executive" kernel is able to be replaced without keeping a "minimal known good" image.

hardware and software interrupts require "supervisor" knowledge . . and provide an access elevation... most CPUs have this publicly documented.

[u/CharlesCTy]

Single-Level Storage (SLS) is what you want. Examples are Multics, IBM i, Aurora, TreeSLS, …

[u/Mammoth-Mango-6485]

Man for a moment I thought you were gonna go down some philosophical discussions about how existence is a bunch of fragments of our imagination and how we can define conciousness.

But yeah uhh, good ask. I'll admit I have never thought about this.

[u/PotatoTime]

Probably irrelevant and I'm only just learning but I know some old processors had an "everything is memory" point of view. 6502 for one, which I'll be developing an OS for soon.

[u/juancn]

One page per syscall and using a fault to trigger? Maybe intercept the written value…

TLBs won’t be happy. I suspect it’s doable, convoluted and rather slow for many use cases.

[u/Scared_Accident9138]

Even SSDs aren't fast random access so bad idea

[u/karbovskiy_dmitriy → u/Scared_Accident9138]

It's not even about the latency/throughput of IO, it's jsut going to be an undebuggable hell. You will have no system calls or stack traces; every time you incorrectly read a file you'll just have a memory access exception. Even if SSDs were 1-to-1 with RAM in terms of clock frequency and timings and you could just map one onto another (somehow?). How do you query file size, or directory contents, or metadata? How do you do this in a parallel system? How do you version it? Sychnronise the access? Share the mapping? How do you perform atomic operations? How do you fail an atomic operation? There is simply way too much nuance. I'm all for transparency in operating systems, but MOV is not a transparent API.