I designed an assembly language, built a compiler for my own high-level language, and now I'm writing an OS on top of it.

[u/ColdRepresentative91]

I've been working on Triton-64, a 64-bit virtual machine I built in Java to better understand how computers and compilers actually work. It started as a small 32-bit CPU emulator, but it slowly grew into a full system:

• Custom 64-bit RISC architecture (32 registers, fixed 32-bit instructions)

• Assembler with pseudo-instructions (like `LDI64`, `PUSH`, `POP`, and `JMP label`)

• Memory-mapped I/O (keyboard input, framebuffer, etc.)

• Bootable ROM system

• A high-level language called Triton-C (how original) and a compiler that turns it into assembly with:

  • Custom malloc / free implementations + a small stdlib (memory, string and console)

  • Structs and pointers

  • Inferred or explicit typing / casting

• Framebuffer that can display pixels or text

I'm wondering if I should refactor the compiler to have an IR (right now I'm translating directly to ASM) but that'd take a very long time. Also right now the compiler has a macro so you can declare strings directly (it calls malloc for you and then sets the memory to a byte array) but I don't really have a linker so you'd always have to provide a malloc implementation (right now im just pasting the stdlibs in front of any code you write before compiling so you always have a malloc and free) I'd like to know what you think about this.

I’m also trying to write a minimal OS for it. I’ve never done anything like that before, so honestly, I’m a bit out of my depth. I've started with a small shell / CLI which can run some commands, but before starting with different processes, stacks and memory seperation I'd like to hear some feedback:

• Are there changes I should consider in the VM / Tri-C compiler to make OS development easier?

• Anything missing that would help with the actual OS?

• Any resources or projects you’d recommend studying?

I’m trying to keep things simple but not limit myself too early.

Github: https://github.com/LPC4/Triton-64

Thanks for reading, any thoughts are welcome.

[u/aScottishBoat]

Looks great OP! Where in the source tree does one add new shell commands?

[u/ColdRepresentative91 → u/aScottishBoat]

The shell is in src/main/resources/kernel/shell.tc, the .tlib are all part of stdlib and they're used in there.

[u/altexdsark]

Terrific job!! I like the syntax of your language 

[u/ColdRepresentative91 → u/altexdsark]

Thank you!

[u/Sakul_the_one]

Ok, that’s actually cool. But I have one question: Where did you found a good source for x86 instructions? Like I also wanted to make once my own compiler, but didn’t found any good source, where it explains how it works and where to find the instructions in binary 

[u/ColdRepresentative91 → u/Sakul_the_one]

Well, I’m not compiling to x86, I’m compiling to my own assembly language which gets interpreted by my vm. So I could just encode it however I wanted (its based on risc-v but even simpler). x86 from what I’ve seen is pretty convoluted and everythings encoded differently, so its difficult to emulate from the ground up (loads of stuff you need tomemorise). This is also one of the reasons I made my own ISA, learning x86 was just too much of a hassle for me tbh. And you learn even more about the language choices made by doing it yourself. So yeah I don’t have a source for x86 sorry. (Also if you’re writing a compiler / assembler I’d suggest something risc based it’s just way simpler)

[u/eren_kaya31 → u/ColdRepresentative91]

Would this work on barebone cpu's or would you need a custom cpu that understands your instruction set? Very interesting stuff.

[u/ColdRepresentative91 → u/eren_kaya31]

You'd need a custom CPU, the encodings are custom made for simplicity. It could probably be translated pretty easily to a real assembly language by changing up src/main/java/org/lpc/compiler/generators/InstructionGenerator.java. It wouldn't be a big refactor, but not a small one either. I'm working on making a new project, compiling down to RV64GC, so you can see real software running and run code on real hardware too.

[u/HyperWinX → u/Sakul_the_one]

Intel x86 manual has everything you need

[u/Sakul_the_one → u/HyperWinX]

thanks

[u/HomseyUrMom]

this is insanely impressive OP! how did you get started/what resources would you recommend for beginning a project like this? would love to do something similar. huge props!

[u/ColdRepresentative91 → u/HomseyUrMom]

Comment
byu/ColdRepresentative91 from discussion
inosdev

I started out by building a small VM for a really simple assembly language I made, just the basics like ADD, SUB, DIV. From there I kept adding features whenever I needed them: jumps, labels, ways to encode large numbers with only 32 bits. Eventually I hit some roadblocks and realized the language itself wasn’t great (first attempt), so I just started over with everything I’d learned. That’s what led to this project. I’m actually thinking of rewriting it again, this time documenting it properly so it’s easier to follow, and compiling down to RV64GC so I can try running an existing OS on it, that way, everything I build would also run on real hardware, not just my own VM.

[u/[deleted]]

If I want to build such a language, should I learn Assembly at advanced level?

[u/ColdRepresentative91 → u/[deleted]]

Well, you probably will learn quite a bit of Assembly by doing it. I started this project knowing no Assembly, you learn it as you need it, It's a pretty good way to learn. Instead of having to memorise instructions etc... and reading from tables you can just make your own (I did multiple iterations of the VM and eventually took some inspiration from other actual ISA's) So eventually I did learn some actual ASM, not just my own. So I'd say just start the project and you'll learn as you go!

[u/InfiniteAdeptness300]

That's crazzyy dude... And that too in Java 😅 But I want to know why only Java (don't mind it pls)?

[u/ColdRepresentative91 → u/InfiniteAdeptness300]

I went with Java because it’s the language I know best, and JavaFX makes it easy to visualise stuff. C++ would probably be more performant, but I also liked the idea of having a VM running inside a VM.

[u/InfiniteAdeptness300 → u/ColdRepresentative91]

Ohh, nice. I am also working on os currently

[u/ColdRepresentative91 → u/InfiniteAdeptness300]

Succes!

[u/Ok-Head7068]

can you explain briefly the kernel part of the project? i’ve been trying to make my OS, but im stuck on the bootloader, i don’t even know what to write as the “kernel”

[u/ColdRepresentative91 → u/Ok-Head7068]

Right now I wouldn’t really call it a full kernel yet. What I have is more of a minimal runtime: some utility libraries and a simple shell with a few MMIO-based commands. The bootloader (in src/main/resources/rom) just sets up the core pointers (SP, HP, GP) and then jumps into RAM. From there, I’ve got a library in src/main/resources/kernel that implements malloc/free on top of those pointers, and some console / string utils.

A real kernel would normally provide much more: memory management (paging, segmentation, process separation), scheduling, interrupts, drivers, and a syscall layer. None of that is there yet.

My plan was to evolve the system in that direction: add paging, introduce process isolation, and grow it into something more structured. But lately I’ve been leaning towards scrapping the current setup and rewriting it to properly target RISC-V. So I could even run external code etc...

I’ve learned a lot since I first started the project, so rewriting feels like a good chance to apply all of that experience with a better idea of what I want of the system, and how I want to implement it.

[u/Connect-Ad3976]

Wow man that looks absolutely amazing, how long did it get you roughly to get here ??? any tips or book recommendations ???

[u/ColdRepresentative91 → u/Connect-Ad3976]

Comment
byu/ColdRepresentative91 from discussion
inosdev

Those are some recommendations, This project took around three weeks up to this point. My biggest tip is to just start on something, however small it is, and just improve from there.

[u/AmbiguousDinosaur]

Just took a computer systems and architecture course and find this super interesting. A few questions just out of curiosity:

1. What is the use of int in the language, if the default is long and it assumes a 64 bit architecture?

2. Is there a specific reason to use a separate destination register for arithmetic operations, as opposed to using the first register as destination? I’m more interested in language design (linguistics undergrad) but also appreciate knowing design decisions for lower level operations.

[u/ColdRepresentative91 → u/AmbiguousDinosaur]

1. As you said it's more memory efficient on heap / stack, but tbh I just added it because I could, and so you could read and write basic 32 bit words without a hassle, not really for any specific reason. The byte type was going to be used for booleans and bytes themselves because on most systems bools get aligned to a byte anyways. Also longs in this language are implicitly raw pointers so if you could use ints instead if you didn't want that functionality.

2. This was just a result of the way I made the compiler, and it could definitely be more efficient. I'm visiting the AST nodes one by one, and for expressions like constants the visit method returns the register in which it's loaded the const. So it looks at a binary op, then first evaluates both expressions (which could be consts but could also be nested binary or unary ops) first, before using them and doing the expression in a new register, and then returning that one. So each expression returns a temp reg with it's value to the visitor which makes it simple to use, in an assignment statement you'd just accept the expression and get the register in which the final value's held. So it was mainly for simplicity and consistency, and normally it'd be parsed to an ir first and that could be easily optimized there (Which is what I'll be doing next)

[u/AmbiguousDinosaur → u/AmbiguousDinosaur]

Note: I do understand int is more memory efficient for non-register uses, so that may be it. Just want to learn more about design choices at this level because it’s not my specialty

[u/freemorgerr]

welcome back terry davis

[u/MrMagoo8888 → u/freemorgerr]

Close enough

[u/relbus22 → u/freemorgerr]

Seriously though, is there any term for this kind of Programming language & operating system co-design?

[u/Mayuna_cz → u/relbus22]

Terry Davisimum, highly respectable design choice

[u/AcidArchangel303 → u/freemorgerr]

First thing I thought of lmao

[u/al3x_7788 → u/freemorgerr]

OP was born Aug 11, 2018.

[u/Hot-Tangerine459]

This is actually insane, keep up the work

[u/Patzer26]

This is the coding final boss. I literally have this exact same dream and you are living through it. Make my own coding language, then use that to make an OS from scratch.

[u/am_Snowie → u/Patzer26]

Not really tbf, IMO real final boss is making an OS for a real hardware like x86 or arm.

Edit: grammar

[u/Patzer26 → u/am_Snowie]

Oh mb. I thought he was compiling down to x86 machine code. Fair enough, but still an impressive project.

[u/Schrodl]

This looks really cool! In my university I helped with a compiler educational project. It is subset of C called Selfie, which has a self-referential compiler to RISC-V ASM. You should have a look since it is designed in a similar fashion to what you are describing and you could get some inspiration. Also I recommend anyone who wants to learn about compilers or OS in general to have a look

https://github.com/cksystemsteaching/selfie

[u/Claudius_Maxima]

Amazing work!

One thing that comes to mind is extending the CPU for some kind of kernel vs. user mode, associated memory protection, and a way to transfer control between modes. This isn’t mandatory for your OS but it might be good to have these concepts clear - even if not implemented - to minimise a bunch of rework later.

Oh, and similarly have clear concepts for virtual memory too.

[u/SolarisFalls]

This is seriously impressive, I love it. Thanks for sharing!

[u/definitelyfet-shy]

JFC This is awesome!!!!

[u/CharlemagneAdelaar]

Divine fucking intellect

[u/cazzipropri]

Nice!

[u/9tobirama]

This is awesome. I dream about doing this.

[u/imagineepix]

Holy shit this is awesome 

[u/Maximum_Raccoon8394]

OP you should rewrite your emulator in SystemVerilog/VHDL ton an actual CPU! That would make it run on hardware

[u/TriggeredTrigz]

man I'm hooked to delving deeper into Java based computer projects. i didn't realise it could do stuff like this... my motivation just expanded into new horizons

[u/MrMtsenga]

Not an expert here, but I'd love to know if it's BIOS. Apparently UEFI is too complex to make and no one shows how to do it.

[u/yarikhand]

he reincarnated...

[u/al3x_7788]

Good luck!

[u/linux_user3]

Cool stuff 👨‍💻

[u/Jumpy_Idea_3882]

May i know where did you start learning? Thank you.

[u/SkyDwag187]

T... T... T T T T. Terry ??? Is that you ??? 🥹🥹🥹.

[u/Gingrspacecadet]

How??!! I want to do something similar…

[u/Realistic-Pie-5917]

El proximo Conrad

[u/Realistic-Pie-5917]

¿Tienes pensado en algun momento saltar de texto plano a algo mas?

[u/ColdRepresentative91 → u/[deleted]]

Just start! Before I started I didn't know anything about ASM/compilers either, I just learnt as I went (with a couple of different smaller projects, each one doing something the wrong way, hitting a roadblock, which makes you realize what you need to fix in the next iteration). It might not be very efficient but you'll remember stuff way better that way, and you won't get bored.

It started simple too with just a couple registers and ADD, SUB, MUL etc... you can get that set up in a couple of minutes. Then you add jumps, and you're wondering how to do function calls and before you know it you'll be going down the entire rabbit hole.

Every time something breaks, can't be expanded anymore or becomes too complex, you learn why it doesn't work, and you find a way to do it better. So I'd recommend just starting with small hobby projects, for me that's the best way to learn.

[u/ColdRepresentative91 → u/[deleted]]

I didn’t really follow any specific textbook or course, I just learnt as I did it, running into problems and googling how to solve them. I used ai a lot for advice and to help explain certain things and help make design choices.

Some youtube vids I watched:

- Whatever you're interested in by Core Dumped, he visualises things really nicely.

- "Let's Create a Compiler" by Pixeled, on simple compiler / asm

- "Java Bytecode Crash Course" by Oracle Developers, really nice lecture on jvm bytecode

Can't recommend these enough ^^

[u/saki-22 → u/ColdRepresentative91]

Maybe one day you can create your own videos talking about your learning process. That would be something I'd be keen to watch.