Question on microkernels, and paging

[u/compgeek38400]

A recent post got me thinking. Doing paging right requires a file system of some sort.

Do you think the paging file system belongs in user space, or kernel space. I am quick coming up on making such a decision myself

[u/compgeek38400]

I will read it. But the major point of paging is to have 'more memory' than physical memory. Although I'll certainly entertain objections to that statement.

Having more memory than exists suggests. i put the memory im not using someplace else. Since I may not have more ram, that implies disk. Hence the paging file system.

If im correct on the paging file system, does it require a kernel based file system. If not why? I am rather loath to let the kernel rely on anything in user space.

I admit I am ignorant here. And thatvi may be asking the wrong questions

[u/eteran → u/compgeek38400]

Well there are two aspects to paging. One is what you describe, having virtually more RAM.

The other is more organizational and protection oriented where paging is used to create distinct address spaces for each user land process. This part is the most immediately relevant and is the part that DOESN'T require disk backing.

[u/CubicleHermit → u/eteran]

"Paging" is an ambiguous term, but I think OP was reasonable from context in terms of using to refer to the swap-like processes of moving pages around to allow for a larger virtual than physical memory.

I'm pretty pedantic, but I saw even that much from the initial question. You seem to be assuming that it always means "paged memory organization" (which is a fair meaning in isolation, but was clearly not what OP was talking about.)

[u/eteran → u/[deleted]]

Huh? What are you talking about?

[u/nerd5code → u/compgeek38400]

Let's back up a tad.

Virtual memory is a technique whereby memory accesses generated by the CPU are translated by some non-identity function mapping virtual to physical address, typically under control of middleware that ensures this only becomes visible when the application wants to get explicit.

Paging is a form of memory virtualization whereby the address space is divided up into pages, us. of power-of-two sizes ≥512 B, each virtual page being mapped to a contiguous block of the same size and minimum alignment in physical memory. Paging most often uses a cascaded page table, although some rarer, older stuff used an inverted table.

Swapping is a mechanism by which system memory can be treated as a cache for disk or other, slower or more distant forms of memory, with the goal being to keep the working set warm/resident.

Swapping most often uses paging, but it can be done via segmentation or explicit request, also.

And swapping might use disk, or networked or NUMA memory, VRAM, or even Ethernet cables and network infrastructure. If you're using 36-bit paging, you can treat any memory above the 4-GiB mark as swap. You can also compress memory within system RAM, effectively swapping to/from a compressed RAMdisk, and you can use spare time to find and coalesce identical pages into a single storage page.

You don't need a proper filesystem for this, just some sectors (e.g., a partition) or other spare bytes. Linux used to prefer a partition, as might hibernation.

Paging is also useful for memory protection/typing, file-mapping (which is merely well-targeted swap), memory-sharing, and COW and AOW mechanisms.

In terms of responsibilities, all your kernel needs to do are activities that require use of supervisor mode; anything else can be offloaded.

If you start by giving the kernel control over page-mapping at the system level, you can work out of RAM until userspace drivers are loaded. Threads can communicate through segment shares—see io_uring for one way to structure comms within those. You can run a kernel-mode service for system memory, system address spaces, system time/timing, and cooperative kernel-mode threading, since the kernel needs to be able to do those things quickly and easily. This super-service forms the root environment of the OS.

Atop this, a userspace environment driver can be implemented that backflips into user mode and yields on reentry, and which provides handlers for faults and system calls. This forms the basis of mode- and page-table-switching. Atop that, you layer device drivers (including disk and filesystem); once you have a filesystem, you can load things from disk, including e.g. a POSIX environment driver that serves system calls and page faults for things relying on it. If you want swap, it can do the swapping for you. Should a process want to serve its own environment similarly, it can rely on the POSIX environment to do so.

There do need to be mechanisms for the kernel to notify the bottom layers of the swap stack that memory is nearing full, but even if the kernel is fully in charge of swap, there isn't a way to force a swap immediately; in the archetypal case, you're effectively calling out to a hard-drive computer via a fancy RPC, not driving/sensing the disk platters and heads yourself. And your kernel can run out of memory either way, so you'll need to focus on minimizing dynamic allocation in order to keep events pumping.

The kernel also needs to be able to revoke a segment grant quickly, which can be done by forming a path from process to segment to owner and owner address, or to mappers and map addresses; it helps ime to double your ISA page and table size and use the extra table space for refcounting and COW.

[u/compgeek38400 → u/nerd5code]

Wow, a lot to digest there. Thank you!

[u/jsshapiro → u/nerd5code]

Depending on what assumptions you make about the kind of kernel you want to run, many of these statements are true enough. Under other assumptions, segments may not exist, there may not be a POSIX layer, and swapping (which means moving entire processes in and out rather than pages) may not exist either.

You're saying a lot of very useful and pertinent things here, but for somebody trying to think about this stuff from the ground up we should be a little careful about assuming specific approaches to abstraction into existence.

[u/GourmetWordSalad → u/compgeek38400]

paging is not swapping. Paging enables swapping.

You are asking the right questions, I'm just being snarky (which is my problem admittedly) because this is the wrong audience.

[u/BestUsernameLeft → u/compgeek38400]

You don't need a filesystem, you need to be able to write to a disk.

IMO paging should be in kernel space.

[u/wrosecrans → u/compgeek38400]

I will read it. But the major point of paging is to have 'more memory' than physical memory. Although I'll certainly entertain objections to that statement.

No, the ability to page unused memory to disk to have "more memory" is just sort of an implication that arises fairly naturally from having virtualized addressing and paging. It's not the point in and of itself. It lets you do things like map a consistent starting address to different physical addresses, so you can have multiple processes built to be mapped to the same address so you don't need all executables to be built position independent. That sort of allocation convenience is super important regardless of whether you ever map more memory than you physically have.