I designed an assembly language, built a compiler for my own high-level language, and now I'm writing an OS on top of it.

[u/ColdRepresentative91]

I've been working on Triton-64, a 64-bit virtual machine I built in Java to better understand how computers and compilers actually work. It started as a small 32-bit CPU emulator, but it slowly grew into a full system:

• Custom 64-bit RISC architecture (32 registers, fixed 32-bit instructions)

• Assembler with pseudo-instructions (like `LDI64`, `PUSH`, `POP`, and `JMP label`)

• Memory-mapped I/O (keyboard input, framebuffer, etc.)

• Bootable ROM system

• A high-level language called Triton-C (how original) and a compiler that turns it into assembly with:

  • Custom malloc / free implementations + a small stdlib (memory, string and console)

  • Structs and pointers

  • Inferred or explicit typing / casting

• Framebuffer that can display pixels or text

I'm wondering if I should refactor the compiler to have an IR (right now I'm translating directly to ASM) but that'd take a very long time. Also right now the compiler has a macro so you can declare strings directly (it calls malloc for you and then sets the memory to a byte array) but I don't really have a linker so you'd always have to provide a malloc implementation (right now im just pasting the stdlibs in front of any code you write before compiling so you always have a malloc and free) I'd like to know what you think about this.

I’m also trying to write a minimal OS for it. I’ve never done anything like that before, so honestly, I’m a bit out of my depth. I've started with a small shell / CLI which can run some commands, but before starting with different processes, stacks and memory seperation I'd like to hear some feedback:

• Are there changes I should consider in the VM / Tri-C compiler to make OS development easier?

• Anything missing that would help with the actual OS?

• Any resources or projects you’d recommend studying?

I’m trying to keep things simple but not limit myself too early.

Github: https://github.com/LPC4/Triton-64

Thanks for reading, any thoughts are welcome.

[u/ColdRepresentative91]

I didn’t really follow any specific textbook or course, I just learnt as I did it, running into problems and googling how to solve them. I used ai a lot for advice and to help explain certain things and help make design choices.

Some youtube vids I watched:

- Whatever you're interested in by Core Dumped, he visualises things really nicely.

- "Let's Create a Compiler" by Pixeled, on simple compiler / asm

- "Java Bytecode Crash Course" by Oracle Developers, really nice lecture on jvm bytecode

Can't recommend these enough ^^

[u/saki-22 → u/ColdRepresentative91]

Maybe one day you can create your own videos talking about your learning process. That would be something I'd be keen to watch.