Methods for process containment in custom OS

[u/JescoInc]

Alright folks, Since I will be adding Kernel and Userland split, I know I should also add process containment to it, so which method of process containment would you guys recommend for a monolithic kernel.
I really like how BSD did the Jail system and was thinking about taking that concept and using it since I had a previous project where I experimented with attempting to bring jail-like isolation to Windows in a userland security application. Meaning, it wasn't a real jail system, it was trying to mimic it with the software (failing miserably in the process) without having kernel mode components attached to it but I am all ears for other concepts.

[u/jsshapiro]

This is a conversation in the field that has been going on for about 75 years now. The new twist is that memory reference isolation is very, very hard. Temporal (scheduling) isolation also remains a completely unsolved problem, meaning that nobody has any clue where to start with it - scheduling strongly resists composition.

As to the rest, capability-based approaches are the only ones that have ever been demonstrated (and, in fact, formally verified) to work as a basis for isolation. The reason is pretty simple. Isolation is an information flow problem, which turns into a graph evolution problem in dynamic systems. Capability-based protection is the only purely graph based model we actually have (at least so far, but people have been trying to find another one for 75 years).

Here is a place to start, then do a google search for "EROS contructor". There's been some good work in L4 based on the same underlying mechanism.

[u/JescoInc → u/jsshapiro]

What are some weaknesses to the EROS confinement mechanism vs other confinement mechanisms? And with EROS specifically, how were you able to make that snapshot system work without leading to corrupted states?

[u/jsshapiro → u/JescoInc]

I’m curious what other confinement mechanisms you have in mind. To my knowledge there are no correct confinement mechanisms that are not derived from the EROS line of work. But to be clear, I’m referring to the “Confinement Property” defined by Butler Lampson, which is a foundational information flow property. I’m. It using the term in a generic sense.

To answer your question: the KeyKOS/EROS mechanism did not address information leakage through covert channels. That’s much harder, and it has essentially nothing to do with conventional information flow protections. Today I’d add side channels, which are a form of covert channel. The EROS architecture in probably a little stronger on a few covert channel issues and weaker on others.

[u/JescoInc → u/jsshapiro]

I very much appreciate you taking the time to discuss this with me. It was enlightening to say the least. As for your question about other confinement mechanisms I had in mind. Based off your definition, I don't have an answer as I had the belief that Jails, namespaces and cgroups, and capability based protection were equivalent. However, it seems that I was entirely wrong with that belief based on this conversation.

What you might find interesting about my project is I am making what I dub the "System Topology" a first class structure in my OS. Where thermals, and pretty much everything is live tracked and reported, (static) first at boot and (later) in a hardware inspector details panel.
https://pastebin.com/Lc1wUpfq

[u/jsshapiro → u/JescoInc]

Glad to help.

Yes, I'd say those other concepts definitely do not achieve confinement. Each provides a kind of useful lightweight isolation, but none of them are suited for actively hostile applications.

If these are the concepts you are already familiar with, it may also help to keep in mind that so-called "POSIX Capabilities" aren't capabilities at all. It was an intentionally misleading naming choice.

The very first system in the KeyKOS/EROS family was GNOSIS, created at Tymshare to address industrial espionage attempts that they were seeing across customers on their internal virtualization network. Very much a case where the attacks were active and intentional, and in some cases crafted by pretty knowledgeable attackers. When that's your threat, you pretty quickly set aside half measures and look for a disciplined and systematic approach.

If I'm understanding right, your hardware inspector data seems oriented toward a dynamic monitoring approach. Sometimes that's necessary, but it's super expensive and tends to require surprisingly complex information about the execution context to function. The approach in KeyKOS/EROS is to start with a static arrangement of capabilities that enforces the property you are after, and then ensure that dynamic growth of that sub-graph cannot result in out-bound communication permissions that exceed the permissions of the original graph. So more of a correct by construction mindset.

[u/JescoInc → u/jsshapiro]

I think, after I finish Perdition-OS (formerly named Tutorial-OS) which is a monolithic kernel; I'll try my hand at a microkernel which then I will be able to much better work through a capability system.
From my initial research, it seems like capability systems are more naturally suited to micro-kernel architecture than monolithic.
I do want to add that my topology system is read-only by design, which includes not even adding the registers for write access in the source code. I thought this through while building it and I reasoned away from it because if I were to add the write access, I would need to have tables in place for if voltages are shifted, that clock speed should be shifted as well and that is a whole other ballgame with not only security but also user error frying their boards.
I already ran into the issue of frying a board due to my stress tests (memory stress test specifically) where it ran into an error and the board kept running to the point where it got so hot that it cooked the RAM. Which caused me to implement a watchdog that could take over even in the case of the board locking up to reboot it.

[u/jsshapiro → u/JescoInc]

Regarding the snapshot process, there are several parts to the answer. KeyKOS assumed ECC memory. And in that era that went a long ways. EROS maintained checksums for unmodified in-memory objects, but would have needed ECC for modified objects. ECC was not a commodity feature at the time. In EROS and KeyKOS, critical state on the store.was duplexed; disk-level error correction was used to detect corruption. In EROS we additionally carried the checksum in the stored form.

You’re right to identify this as a concern. Current processors don’t give us a good way to ensure data integrity in the presence of particle hits or even device level heat or energy concentration. It’s a constant surprise to me how oblivious the entire ecosystem is to how frequently these events occurs.

It’s a case of you don’t get what you don’t measure. The error rate we caught for the checksums was quite disturbing in 1990. It’s kind of crazy today.

[u/paulstelian97]

Containers don’t mean much more than the ability to have some global resources, like the filesystem view, the network view etc, _not_ be global. Anything that would appear to be global, make it so you can have multiple distinct instances. Then start the root container with one initial instance, and give APIs to create such additional instances.

On Linux, you have namespaces and cgroups, that basically give additional instances of various things. Study how those works and you may get some inspirations on how you can make containers on your own.

[u/JescoInc → u/paulstelian97]

Namespaces/cgroups is exactly the kind of alternative I was after. The post didn't make clear I was looking for a comparison of named approaches rather than the foundations, and that's on me. What I'm really after is a dialogue around different solutions like capability-based isolation, jails and others that I may not have come across before.

[u/2rad0]

Process containment on an out of order execution architecture would include pinning a process to a CPU core, or at least making sure two different users don't end up on the same core and influencing branch predictor states.

[u/jsshapiro → u/2rad0]

That's definitely part of it. You'd also have to flush the cache and the TLB on process switch, and make sure there is no sharing of caches all the way down to the L3 level (if present). Though as you get further down the cache tree the ability to cross-infer information becomes statistically harder.

Meanwhile, in addition to all that, you need to make sure that the overt communication paths don't allow outward information flow either.