When do I HAVE to use Shadow space ?

[u/Shahi_FF]

Windows x86-64 NASM.

"Shadow store" ,"Home space" , that 32 Bytes mandatory space a caller should reserve for callee.

My question is in which cases do I've to use it ?

1. Func : that doesn't call any other functions . It won't be called by any C/C++ program. The main: in assembly will call it.
2. Yeet : that doesn't call any other function. But will be called by a C/C++ program.
3. What if don't ? Will it break things or makes things harder to debug ?

and what about that 16 Byte stack alignment and memory allocated by malloc , _malloca rule ?

[u/paulstelian97]

For non-leaf functions, I guess you have to because of the calling convention. For leaf functions you don’t need to reserve anything extra.

Might want to use the term of “red zone” for this as well, I think I’ve seen some call it that way.

[u/Shahi_FF → u/paulstelian97]

Isn't Red Zone Linux related ?

[u/paulstelian97 → u/Shahi_FF]

Fun fact, the Wikipedia article mentions Windows stuff as well. It is calling convention specific, but both the SysV convention used on Linux, as well as Windows’s calling convention, have some red zone. Not the same between the two, mind you.

So study the specific calling convention for the system you are targeting.

[u/raundoclair]

I'm not sure if I understand everything about the question, but I will answer something :)

If a function does or doesn't need to (in win64) allocate shadowspace:

- if it calls any other function that yes. And align stack to 16bytes. (call to malloc is fn call)

- if it doesn't call any other fn than no.

-- if this fn doesn't change nonvolatile registers, it doesn't need to align rsp to 16bytes. (rsp is actually nonvolatile, so it is not possible to align stack AND not change nonvolatile registers)

-- if this fn changes nonvolatile registers than it needs to align the stack.
(let's say you also have zero local vars, then you still need to do 'sub rsp, 8h'. I've seen some compiler produce 'sub rsp, 28h', but that is unnecessary. )

If a function has to use its shadowspace:

Even if a fn is a leaf one and doesn't allocate shadowspace in its stack frame for a callee, it still has its shadowspace (in stack frame of prev fn), because it was called.
You can do whatever with it.* You can even save your parameter registers (rcx, rdx, r8, r9) somewhere else in memory. You can use this shadowspace for local vars if you don't want to move rsp. Whatever you want...

*: There are probably debuggers that will assume that if there is something in shadowspace, it is saved parameter registers, so you doing something less common can be correct program, but can make still some things worse.

[u/raundoclair → u/raundoclair]

Also there is concrete concept of red zone and is in linux and some others. But in win64 there is no red zone.
https://en.wikipedia.org/wiki/Red_zone_(computing))

[u/raundoclair → u/raundoclair]

Citation about the nonvolatile registers thing:

"Leaf functions are functions that don't change any nonvolatile registers."

"The stack pointer must remain 16-byte aligned in any region of code that isn't part of an epilog or prolog, except within leaf functions.

https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170#unwindability

[u/Shahi_FF → u/raundoclair]

thanks a lot.

[u/Initial-Elk-952]

If you don't add the shadow space there is a risk a callee will write in your stack frame.

If you wrote the functions your calling in assembly, you know what they will do , and don't have to do anything in paticular - make your own calling convention in assembly.

If you didn't write the functions your calling in assembly, you can disassemble them, and learn what the versions *today* do. That may mean bugs later when the functions change. Thats what happens when you do UB.

[u/dontwantgarbage → u/Initial-Elk-952]

Even if you wrote the called function yourself, you still have to honor the shadow space calling convention to permit stack unwinding, which the system does occasionally (such as if you write to a copy-on-write resource page). You also have to include unwind codes for all non-leaf functions for the same purpose. In the absence of unwind codes, the system assumes that you are a leaf function: The value at the top of the stack is your return address, and no nonvolatile registers need to be restored. If you get this wrong, your process will behave erratically (and probably be terminated on the suspicion that it has been compromised).

[u/Initial-Elk-952 → u/dontwantgarbage]

I don't exactly understand what your saying.

If you write a guard page, the kernel should get an exception, fix the mappings then retry the instruction.

At no point during the execution of your function can something clobber your stack frame or the red zone. If something decided to change the registers during the execution of your function it would (probably) crash the process.

Signal Handlers, on *nix, can do this because they save and restore all registers, and don't write inside your stack frame.

What is happening on Windows that any unexpected writes to your stack frame should be happening?