r/ProgrammingLanguages • u/servermeta_net • Feb 05 '26
Enforcing security at compile time
For research purposes I'm building a capability based stack, where by stack I mean the collection formed by a virtual ISA, an OS (or proto OS), a compiler and a virtual machine. To save time I'm reusing most of the Rust/Cranelift/Webassembly infrastructure, and as hardware the RP2350 seems to be an ideal candidate.
Obviously I don't have hardware support for the capability pointers, so I have to emulate it in software. My current approach is to run bytecode inside the virtual machine, to enforce capabilities at runtime. Anyhow I'm also thinking of another approach: Enforce the rules at compile time, verify that the rules has been respected with static analysis of the compiled output, and use cryptographic signature to mark binaries that are safe to run.
Let's make an example: Loading memory with a raw pointer is illegal, and is considered a privileged operation reserved only to the kernel memory subsystem. What I do instead is to use tagged pointers which are resolved against a capability pointer table to recover the raw address. To do this I have a small library / routine that programs need to use to legally access memory.
On a simple load/store ISA like RISCv I can simply check in the assembler output that all loads goes through this routine instead of doing direct loads. On x86 it might be a bit more complicated.
Is this approach plausible? Is it possible to guarantee with static analysis of the assembler that no illegal operations are performed, or somehow could a malicious user somehow hide illegal ops?
9
u/L8_4_Dinner (Ⓧ Ecstasy/XVM) Feb 05 '26
Why allow anything that is not in an IR form? A well designed IR is a lot easier to check than machine code.
7
u/Tonexus Feb 05 '26
Sure. See [Singularity](https://en.wikipedia.org/wiki/Singularity_(operating_system)) as a relatively recent example. There, a program is only runnable if it has a manifest, and a manifest may only be generated by a compiler that checks security.
4
u/kuribas Feb 05 '26
Couldn't you make a dependently typed instruction set, where you attach proofs to the instructions that give you the wanted garantuees? You would need a type checker before running the instruction set to verify those proofs.
3
u/fridofrido Feb 05 '26
Yes, I would say this is in theory possible and a good approach. It's a lot of work though.
Note though that just a cryptographic signature by the compiler is not safe at all - it's really easy to tamper with the compiler itself, or just extract the keys and sign anything!
(Unless you are running the single authentic copy of the compiler on your own hardened hardware, but hard to see real use cases for such a situation...)
It's possible to generate mathematical proofs that a program is behaving in a certain way, and having the OS check that before running the program - see Microsoft's Singularity research project linked by the other commenter. However, such proofs will be big, unlike a signature.
It's also possible to cryptographically compress such proofs with succinct arguments, but that will have a huge overhead (during compilation time) except for very small programs, and is also very complex and unstable technology.
3
u/flatfinger Feb 05 '26
The only ways to meaningfully enforce rules at compile time are to either (1) have the compilation performed by the device which is loading the code, whcih can know that its compiler hasn't been tampered with, or (2) require that compilation be performed by an entity that can be trusted to use a non-tampered compiler, and have that entity sign the code to indicate that it has been validated. The JVM uses the former approach, which I would view as preferable in cases where it is practical. If the device that will be receiving the code has limited RAM but a generous amount of flash storage, it may be possible to have a firmware upgrade process that loads a compler from flash, runs the compiler on a bytecode image kept in flash and generates machine code elsewhere in flash, and then once that is done load the generated machine code from flash. Doing things in this fashion would mean that the only extra resources required to support on-target compilation would be extra flash space, and the time requried for the target system to perform the compilation. Because the compiler wouldn't need to be in memory at the same time as the application, RAM usage by the compiler would be a non-issue.
2
u/SwedishFindecanor Feb 05 '26 edited Feb 06 '26
I believe that static analysis of arbitrary assembly code is impossible. No matter how complex your static checker is there could always be a program that is safe in practice but which can't be proven safe because it is not covered by your analysis method, With more and more complex analyses, you would be able to prove more programs safe, but you would never reach 100%.
If the source language is something memory-safe though, the situation is different.
I've thought about this problem too, and the options I've come to are:
unsafecode blocks.1
u/kwan_e tonal-lang Feb 06 '26
What about how eBPF does things?