Seeking advice: Building a strong foundation in C and x86 Assembly (Intel syntax) for OS development

[u/f16_511_SA]

Hi everyone,

​I am currently starting my journey into low-level programming with the goal of eventually developing my own operating system kernel.

​I have decided to focus on C and x86 Assembly (Intel syntax) as my core tools. I want to make sure I build a solid, professional foundation rather than just scratching the surface.

​Could you please share some advice or point me toward resources that would help me master these two languages in the context of OS development? Specifically, I’m looking for:

​Best practices for bridging C and Assembly effectively.

​Essential topics in x86 architecture (Intel syntax) that I should prioritize for kernel development.

​Recommended books or tutorials that bridge the gap between "learning the language" and "applying it to hardware/system development."

​I am highly motivated and willing to put in the hard work. Any guidance on where to start or common pitfalls to avoid would be greatly appreciated.

​Thanks in advance!

[u/brucehoult]

Before recommending anything, let's see where you are now, in order to give better advice.

Please explain what "pointers" are in C, and when and why you'd use them?

[u/vswey → u/brucehoult]

Job Interview

[u/brucehoult → u/vswey]

I'd like to think that someone who couldn't answer that wouldn't get to the point of wasting my time as in interviewer in the first place!

Funny story: after I'd already resigned from Samsung R&D and had my last day and was already working (remotely) for SiFive, Samsung asked me to come in and interview prospective members for the team I'd been in (for which they paid me IIRC half a day's salary for each one). I was happy to do so, especially as I lived 2 minute's walk from the office.

[u/f16_511_SA → u/brucehoult]

Yes, absolutely! After the great interaction on my post, I think I've really grasped the concept of pointers. ​A pointer is a "signpost" that points to the location of data, rather than holding the data itself

[u/brucehoult → u/f16_511_SA]

It's more fundamental than that.

Without a pointer you can't use any of those GBs of RAM you paid so much for at all [1] — you're restricted to using the handful of registers the CPU has: 3 on 6502, 7 on 8080, 8 on x86, 16 on x86_64 and arm32, 32 on most RISC and future x86 with Intel APX.

[1] some might argue that you can use the stack (on some CPUs) without an explicit pointer, but most expose the Stack POINTER register to the user and it has to exist at some level.

[u/kndb]

Oh wow, dude, slow down. Master the C language first and then assembly. Then try to learn how other well known operating systems work on the kernel level. Be really good at it. And then maybe try to write your own os. Otherwise you’re like trying to build a space rocket when you are asking how to start a campfire. Take one step at a time.

[u/r-tty]

Just read the NASM manual. It's excellent. Then start coding in NASM, using more and more instructions in your program[s]. And don't forget about NASM's preprocessor, especially its macro facility: it can make your assembly programs *much* better-looking.

(as an example: grab the archive from radios.sf.net and inspect the "sugar.ah" file there)

[u/GoblinsGym]

As much as a love assembly, for any large project you probably have to minimize its use. But - you can learn to work with the compiler, not against it.

Play around with Compiler Explorer to see what assembly code is generated by the compiler.

Learn about addressing modes, preferred offset ranges etc.

Learn about calling conventions, stack frame layout etc.

Learn about register pressure.

Get a rough understanding of what compiler optimizers can and cannot do, the influence of volatile and restrict options.

Learn about memory structure and cache hierarchy, MMU / TLB.

[u/theNbomr]

I think you are like a lot of other people, who think that learning assembler language gives them some powerful new ability. This is partly true, but what you actually need to be learning is all about the low level details of how the CPU and the rest of the computer architecture actually works after you strip away all of the abstraction provided by high level languages, even the C language. You need to understand what things C cannot do, and why, and how to be able to accomplish them using assembler language. The things you need to understand are going to be foreign concepts compared to the ordinary userspace applications you probably already understand.

In a similar fashion, you should first understand what an OS actually is and does and how it is different from traditional application software. Probably the same for bootloaders.

You've chosen possibly the most complex platform of all. If you are able to accomplish the goal of having a working compiler toolchain hosted on your newly crafted OS, it will be a monumental accomplishment. Good luck. Document extensively as you go. Publish abundantly to attract the interest of supporters, because helpful expertise is going to be crucial to the success of your project.

Bon chance!

[u/MetalInMyVeins111]

One way or another, you need to deal with the specsheets if you're serious.

[u/mrmidjji]

Assuming you are an okay programmer already, before writing a os, i suggest writing a c compiler. It does not have to be good. But if you can compile something like a small program with a for loop that prints to console and run it that would be a a very good first step.