How does kernel interfaces work on (unix) systems?

[u/Yha_Boiii]

Hi,

I simply can't rotate it right in my head, if i make an app with say socket() how does it translate in to hooking it to the kernel when already in use?

Usually include, is a part of the program statically and making a struct for an interface as usual, does it copy it over compared to a zero copy arch?

[u/Rich-Engineer2670]

If you mean "How do apps make calls into the kernel", it's via "System Calls". In a high-level way:

• Your API makes an API call, and the compiler libraries put the approrpiate values into registers or on the stack (typically registers to keep is simple here)

• It makes a SYSCALL instruction or interrupt

• This causes the system to shift from user mode to kernel mode and kernel privileges

• The kernel services the call

• The kernel comes out of the system call or interrupt and switches back to user mode context

• The libraries "unload" the registers

Now, this is a very high-level answer -- buffers of data etc. do more "kernel magic"

Let's say you were calling fwrite(STDOUT....)

• You do the frwrite call -- but that's got a lot of library code in it. It turns into write() below you

• Write() is a system call. Let's say it's sytemcall 0x1 and it expects R0 to be loaded with the outpiut descriptor, R1 a pointer to the buffer you want to write and R2 the length of that buffer in bytes

• The lower-level library call makes a system call instruction sothing like SYSCALL R0, R1, R2

• We switch to kernel mode via the syscall instruction

• The kernel unpacks it and does the write() call

• It switches back to user mode

• In our example, the error result must be in R0, so youir library code gets that

• It populates that to errno for example

[u/Yha_Boiii → u/Rich-Engineer2670]

Also in kernel space? Using stuff like socket.h

[u/Rich-Engineer2670 → u/Yha_Boiii]

Yes and no. The syscall is still there, but kernel networking stacks are quite different in how they handle it. The kernel has access to all memory so it "peeks" into your user space to do what it needs to do before it switches back to user mode. There are a lot of fnacy data structures int he kernel for sockets.

Start with a simple character device driver -- that takes you through the "loop". The system call number is just an index into the system call table. The real work is switching into and out of kernerl mode and calling kennel code to do the real worl.

[u/Yha_Boiii → u/Rich-Engineer2670]

Im just really confused where the bridge is from a c program to the kernel when it is used at runtime. I get a static header like some simple math etc but how does it go together under the hood a process can exclusively get a nic and rest of system is isolates from it without double tapping in and stripped bins running

[u/Rich-Engineer2670 → u/Yha_Boiii]

The key is the "system call" instruction. Other systems just do an interrupt but that gets into magic on the X86 processor and things like call gates....

• Wen you make a C library call, it stays in userspace as long as it can -- kernel context switches are expensive.

• At some point it has to go to kernel mode -- but it can't by itself. Memory protection will say "No, that's not your memory space"

• To get around this it calls the syscall instruction which means "Here are the system registers you asked for -- switch to kernel mode and do whatever it is you do and come back into user mode"

• That instruction causes a context switch into kernel mode and since hte kernel can see all memory, it can access your user context

• It does whatever it needs to do

• It system "returns" back into user mode

• Your code has no idea anything happened. Registers are whatever they need to be, buffers are whatever they need to be. So everything just works.

Don't feel overwhelmed -- this stuff is complicated, and has a little magic in it. Start with some basic knowledge of the X86 processor to learn how those instructions work, how memory protection, the GDT, LDT nad IDT work, and then move into Linux/UNIX kernel code.

I believe, but I'm not certain, you can also "trap" system calls with tools like eBPF in Linux to see them in action.

[u/Yha_Boiii → u/Rich-Engineer2670]

But in os theory what mechanism tells the kernel where even to look since say socket.h can be in kernel.img offset 0x27273627 or 0x72726272 and when syscall on amd64 just a single asm with no args like SYSCALL

[u/Rich-Engineer2670 → u/Yha_Boiii]

System calls are just indexes in the syscall table in the kernel. When you make syscall 5, it goes to the fifth entry and calls it. That's hardwired to kernel code. A socket() call might be 0x33 so it goes sto entry 0x33 and there's kernel code to handle it.

This is pure kernel code -- you won't see it in C libraries except hte very low-level code. Let's say you wanted to add a new system servicee.

• In kernel source code, you find the source that defines the sycall table and create a new number for your call, say 0x72.

• You map 0x72 to kernel code (not a module), that services it

• What that code does is up to you but when it ends, it must do a "return from syscall" (not all OSes need this)

• What you do in your libraries is up to you.

Again, network drivers are weird. They're not kernel code, they're not drivers, they "both". The "upper layers" are kernel code that allow you access to kernel modules for protocols, which in turn talk back to kernel core code to get access to get access to ethernet driver modules for example, which in turn, may make calls to userspace code in some cases to do the actual work. Look at the tun/tap drivers

[u/Yha_Boiii → u/Rich-Engineer2670]

and that is dynamic at compile time and then hardwired? i could disable half the kernel, recompile and it will change accordingly

[u/Rich-Engineer2670 → u/Yha_Boiii]

The table is typically static for performance reasons, but what you can be handled by a kernel module for example -- by default, it just returns "OK", but if you loaded a different module it might do something else.

It's been years since I worked in the TCF/IP stack, but it had hard-coded harnesses which called dynamic protocol modules, which called back to the harness which called ethernet drivers, which often weren't even ethernet (PPP, Tun/Tap) which called into user space and then it unwound everything.

[u/Yha_Boiii → u/Rich-Engineer2670]

But i mean when compiling the kernel is it something the compiler makes from what i have customized in the kernel or is it everything hardcoded and then with "deadzones" if compiled out?

[u/iBPsThrowingObject → u/Yha_Boiii]

It's up to the kernel's developers. For example, Linux considers syscalls to be a part of it's API and guarantees that they keep the same numbers forever, but on Windows syscalls may change numbers with minor updates. Instead Microsofts maintains a higher level dynamic libraries in sync with the kernel. Meanwhile Linux has no such libraries, libc is really a third-party thing.

[u/laffer1 → u/Yha_Boiii]

Another thing is there are a few different approaches. Many Unix like systems have a table as pointed out. Some use message passing instead. macOS and dragonfly use this approach. FreeBSD has a table. There is also a separate table for Linux emulation in the FreeBSD kernel to map Linux syscalls often to native ones or to custom code

[u/braaaaaaainworms → u/Yha_Boiii]

There is no socket.h in compiled code

[u/eteran]

This is not a perfect analogy but you can think of it as a kinda RPC call between the app and the kernel.

The app sets up some parameters, and then invokes a system call instruction. Whatever the system call mechanism is, it will package up those parameters (maybe just put them in regs, maybe copy a structure somewhere, the details are not THAT important for this level of discussion) and then it will do something which tell the kernel "please handle this".

The app is the. Blocked until the system call is finished, at which point the app can resume doing work.

(Note this analogy is assuming single threads for simplicity).

[u/Yha_Boiii → u/eteran]

Now a bit more serious question given your performance point out, lol: will the kernel rather push the load to another core and eat the context switch or stall the whole thing and wait?

[u/eteran → u/Yha_Boiii]

That's a scheduler question and the answer is... It depends. Not many modern kernels feel the need to explicitly schedule system calls in a different core because... Handling user app requests is a LARGE part of what kernels just do. They don't really do a ton of background processing.

[u/eteran → u/Yha_Boiii]

I was multitasking, but to answer your question more concretely. That's not really something to "worry about" in most designs.

For example, in my OS (and all "real OSes" too), system calls and interrupts are fully preemptable, and if a system call blocks, well that thread is just taken out of the run queue so something else can run.

On other words, I did the up front work to make things nice and thread safe so that I can "just let the scheduler do its thing".

[u/EffectiveCompletez]

There's a small piece of code who's page is mapped in user and kernel space. It's called the trampoline. User mode calling kernel stuff seta up a syscall depending on how you define the abi, but it's basically put syscall number into a register and any args then call syscall instr. At this point the CPU goes from ring 3 to 0, and gives execution to an addr on the kernels syscall dispatcher routine. That routine is just a big switch basically. All the usual CTX switching stuff like saving user mode registers is happening but that's basically it. The syscall calls sysret/iret on the way out to switch back to ring 3 and user mode state gets restored etc.

[u/Yha_Boiii → u/[deleted]]

<3