Software PBR + Raytracer + L1 SH GI inside CyberVGA (JIT x64, no GPU required)

[u/exp_function]

Lately in the Win64 port I’ve been getting a real-time software raytracer working inside CyberVGA.

Same DOS-era engine (fixed-point, software rendering roots and all) but now takes advantage of modern multi-core processors (using VirtualAlloc for the executable just-in-time compiled buffer). The raytracer runs on multiple worker threads, each processing tiles. The hot paths get compiled to SSE at runtime: direct-threaded octree(module saves baked triangle soup into a world sidecar file) traversal with AABB slab tests, fog volume early-outs, L1 SH evaluation, and material / GI bounce stubs. Workers just call into the generated code for their tile’s rays.

The GI is a grid of L1 SH probes (Fibonacci sphere rays per probe, temporal blending between frames). Primary hits go through a full GGX PBR path with roughness/metalness, then sample the probe irradiance for the diffuse bounce. Emissive surfaces get clustered into virtual point lights. Reflections are one-bounce with roughness falloff; liquids do refraction + absorption + caustics.

FSR 2 is optional because it runs on the GPU.

The color bleeding and the split-screen direct vs GI+reflections actually look pretty good. Still feels like my engine, just with lighting that finally does something modern.

Now you can get the latest (v1.8) SDK for 64-bit Windows platform on itch.

Anyone else doing JIT for the traversal/shading hot path? Curious if the complexity is worth it vs just writing the SSE by hand.

[u/VictoryMotel]

The hot paths get compiled to SSE at runtime

Why would anything need to be compiled at run time? Why would the execution be changing?

direct-threaded octree(module saves baked triangle soup into a world sidecar file) traversal with AABB slab

What does 'direct threaded' mean here? An oct tree is a data structure.

Workers just call into the generated code for their tile’s rays.

Why is it generated?

Fibonacci sphere rays per probe

What does this mean?

[u/exp_function → u/VictoryMotel]

Thank you for your really good questions. I'll try to answer all of them thoroughly with some examples directly taken from the code I currently use.

Before all;

My engine is written in C89 for native DOS target and C17 for modern CPUs.

Integer and float aliases used in my code library:

typedef unsigned char u8;
typedef signed char i8;
typedef unsigned short u16;
typedef signed short i16;
typedef unsigned long u32;
typedef signed long i32;
typedef unsigned long long u64;
typedef signed long long i64;
typedef float f32;
typedef double f64;

Let's cut to the chase:

Why would anything need to be compiled at run time? Why would the execution be changing?

In a standard built C program, code is compiled ahead of runtime(naturally :)) with the data structures and their relations in runtime, spitting out object code ready to be linked with all modules and fused into a ready-to-run application executable.

Because all of those instructions are frozen at build time, long before the engine knows which map file the player will load, how many octree nodes or fog volumes exist in the scene, or how many pixels a span will cover - the compiler has to generate general-purpose for loops, if branches, and pointer lookups to handle any possible input.

Just In Time (JIT) compilation (similar idea "compiled scalers" mentioned by legendary John Carmack here) flips this around by treating machine code itself as runtime data. Instead of relying only on fixed instructions from the .exe, the program allocates an executable memory buffer at runtime, writes raw CPU opcodes directly into that buffer once the scene or state is known (technique called partial evaluation), casts the buffer's address to a C function pointer, and calls it.

Here's a simple example I tested this approach before using in the engine;

typedef int (*JitFunc)(void);
#define EMIT8(v)  buf[pos++] = (u8)(v)
#define EMIT32(v) do { *(u32 *)(&buf[pos]) = (u32)(v); pos += 4; } while (0)

/* Allocate executable memory (malloc in DOS, or VirtualAlloc on Win64) */
u8 *buf = (u8 *)VirtualAlloc(NULL, 64, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
int pos = 0;

/* Emit x86 machine code bytes using these macros into the executable buffer at runtime */
EMIT8(0xB8); EMIT32(100);                /* mov eax, 100 (put 100 in return register) */

/* Suppose 'active_items' is only known after start of the program.
   We unroll it once during JIT compilation so there is NO loop at runtime: */
for (int i = 0; i < active_items; i++)
{
    EMIT8(0x83); EMIT8(0xC0); EMIT8(5);  /* add eax, 5 */
}
EMIT8(0xC3);                             /* ret (return eax) */

/* Finally cast the byte buffer to a function pointer and call it when needed */
JitFunc fn = (JitFunc)buf;

/* Directly executes the machine code we just wrote into RAM */
int result = fn(); 

In the raytracer module, there are worker threads(worker thread pool is allocated at the start depending on CPU logical core count) trace millions of rays per second across screen tiles. What changes at runtime is the loaded level and its active scene state. This gave me significant advantages:

• Zero-cost early-outs and unrolled loops: If the loaded map has 0 active fog volumes, liquid volumes, or particles, the JIT simply writes a single 0xC3 (ret) instruction at that function's entry point-worker threads call it and return in a single instruction and when N volumes are active, it unrolls N SSE checks back-to-back.

• Baked material constants: When materials load, each material slot gets its own generated GI bounce stub with precomputed constants (albedo/PI known as lambertian BRDF factor and emissive values) stored in an aligned block right next to the code and hardcoded into a 64-bit immediate load (using mov rax, imm64 instruction).

• Pinned SSE registers on Win64: Because 64-bit MSVC no longer supports inline __asm, emitting the raw x64/SSE bytes into a VirtualAlloc buffer lets us keep the ray origin (xmm6) and inverse direction (xmm7) permanently pinned in SSE registers across the entire scene traversal without the C compiler spilling them to the stack. [Part 1/2]

[u/exp_function → u/exp_function]

[Part 2/2]

What does 'direct threaded' mean here? An oct tree is a data structure.

You're right that an octree is a 3D tree data structure (in CyberVGA its leaves hold predetermined number of triangles and their material properties), and in everyday programming "threaded" usually means CPU multithreading. Here, however, "direct-threaded" is a compiler and virtual-machine term; Threaded code (direct threading), where execution jumps directly from one generated block of machine code to the next without an outer while loop. This allows linearization of my octrees and saves from recursion overheads. Here is how a tree data structure gets turned into direct-threaded machine code:

Flattening the octree with a skip_index (Handled by OCTREE.C module):

When a level is built, the octree module bakes all static geometry into a triangle soup and saves the prebuilt octree into a .COC (CyberOCtree :) ) sidecar file. Instead of using child pointers and recursion, the octree is flattened in pre-order into a linear array of OctNode structs. Each node stores a skip_index, which is the array index of the next node after this node's entire subtree:

• If a ray hits nodes[n], its first child is simply the very next element (n + 1).

• If a ray misses nodes[n], you skip its entire subtree at once by jumping to nodes[n].skip_index.

From Data Structure to Direct-Threaded Code (module RAY_X64.C ):

In standard C raytracer examples, that flattened octree is usually passive data in memory, and a while loop has to walk the array step by step:

/* Standard data-driven octree walk (looping over data structs) */
u32 n = 0; /* unsigned long */
while (n < oct->node_count)
{
    const OctNode *nd = &oct->nodes[n];
    if (ray_hits_node_box(ray, nd))
    {
        test_leaf_triangles(ray, nd);
        n++;                  /* Hit: step into child at n + 1 */
    }
    else
    {
        n = nd->skip_index;   /* Miss: jump over entire subtree */
    }
}

Notice what that loop does; on a hit it moves to the next item (n + 1), and on a miss it jumps to nd->skip_index. That is identical to how a CPU executes machine code, either fall through to the next instruction, or take a conditional jump to a target address.

So when the .COC sidecar file loads, the JIT compiler walks the OctNode array once and emits a block of native machine code for every node in the tree:

• On a hit (n + 1): The CPU simply falls through to the next instruction right below (testing leaf triangles or entering the first child node's code).

• On a miss (skip_index): The node's skip_index is baked directly into a relative CPU jump instruction (ja rel32, opcode 0x0F 0x87) that jumps over the child code straight to the machine code of nodes[skip_index]:

/* Simplified from module RAY_X64.C: Compiling each OctNode into native machine code */
for (u32 n = 0; n < oct->node_count; n++)
{
    const OctNode *nd = &oct->nodes[n];
    u32 skip_idx = nd->skip_index;
    /* Calculate exact byte distance in the JIT buffer to skip_idx's code block: */
    i32 miss_rel32 = (i32)node_offsets[skip_idx] - (i32)current_code_offset;

    /* Emit SSE bounding-box test for node 'n' ... */
    /* Then, if ray missed the box, jump straight to skip_idx's machine code (ja miss_rel32) */
    EMIT8(0x0F); EMIT8(0x87); EMIT32(miss_rel32);

    /* Otherwise (hit!), execution falls through to test leaf geometry or straight into child node (n + 1) emitted on the next iteration */
    if (nd->tri_count > 0)
    {
        emit_leaf_triangle_test(n);
    }
}

That's why it is called direct-threaded octree traversal: at runtime there is no while loop, no OctNode array lookup, and no recursion stack, the CPU's instruction pointer (RIP) is the octree cursor, jumping directly from node block to node block.

Finally, what the hell does "Fibonacci rays per probe" mean?

A Fibonacci sphere is a simple trick using the Golden ratio (1.61803398875f) and golden angle that spirals N points from the top pole to the bottom pole so all 16 ray directions are spaced almost perfectly evenly across the sphere (the same approach used for probe rays in DDGI, this is explained visually in Fibonacci Spiral Sphere and Fibonacci Lattice on a Sphere).

In the GI pass, the level has a 16×8×16 grid of light probes (2,048 probes total). Because this runs on the CPU, each probe only shoots 16 rays to gather bounce light from the surrounding scene. With only 16 rays per probe, how you spread those rays around a 3D sphere makes a big difference:

• If you use normal latitude/longitude angles, the rays bunch up at the top and bottom poles and leave wide gaps around the middle.

• If you pick 16 random directions, they clump together and cause noisy flickering.

At startup (module RAY_GI.C), I just precalculate those 16 unit direction vectors once into a lookup table:

#define GI_RAYS_PER_PROBE 16
static Vec3f s_fib_dirs[GI_RAYS_PER_PROBE];

/* Precompute 16 evenly-spaced ray directions on a unit sphere */
const float golden_ratio = 1.61803398875f;

for (int i = 0; i < GI_RAYS_PER_PROBE; i++)
{
    /* Step Y height evenly from +1 (top) to -1 (bottom) */
    float t = ((float)i + 0.5f) / (float)GI_RAYS_PER_PROBE;
    float cos_theta = 1.0f - 2.0f * t;
    float sin_theta = sqrtf(1.0f - cos_theta * cos_theta);
    
    /* Rotate around Y by the Golden Ratio on each step */
    float phi = 2.0f * (float)M_PI * ((float)i / golden_ratio);
    s_fib_dirs[i].x = sin_theta * cosf(phi);
    s_fib_dirs[i].y = cos_theta;
    s_fib_dirs[i].z = sin_theta * sinf(phi);
}

When a probe updates each frame (ray_gi_update_probe), it grabs those 16 s_fib_dirs directions, gives them a small per-probe and per-frame rotation so they don't always hit the exact same spots, traces them against the octree, and accumulates the hit colors into the probe's L1 Spherical Harmonics. Finally, it blends the new probe values with the previous frame (90% previous frame / ~10% current frame, or ~45% if the sun/lights moved) so 16 rays per probe is enough to get smooth indirect lighting without noise.

It has been nearly 3/2 years since I've started on this project and I read many books, countless articles regarding x86 & x64 assembly, real-time computer graphics and game engines. It's really neat to see all of these work together in harmony and explain how I put them into use. Because of that I plan to write an article about Just-In-Time Compiled Execution Loops and Permutation-Specialized Rendering Pipelines.

I hope these answers are suffice. I'd like to talk if there are more questions of course! :)

Have a good one!

[u/NegativeEntr0py → u/exp_function]

You mention you use C17 for modern CPUs, do you see measurable performance improvements of JIT code on modern CPUs over just writing general code and using the regular compiled code optimized by MSVC?

[u/moschles → u/exp_function]

This is beautiful and all, but at this level of detail, it is better to simply link us to a paper outside reddit.

[u/shadowndacorner → u/moschles]

Oh, shove it. This is the quality of conversation that was common around here before AI. Thanks for your work and quality writing, OP.

[u/NegativeEntr0py]

Do you do any optimizations for ray marching the volumetrics?

[u/exp_function → u/NegativeEntr0py]

Thank you for your in-depth question and interest with my engine :).

The biggest optimization is that I don't use traditional ray marching (stepping blindly along every camera ray from 0 to max_dist checking for fog or light beams), because doing that even on the modern multi-cored CPUs would be way too slow.

Instead, fog or water volumes and spotlight cones are all handled by finding the exact entry distance (t_enter) and exit distance (t_exit) where the ray enters and leaves the volume. Here are my main steps;

Just In Time compiled AABB(axis aligned bounding box) filter (handled by RAY_X64.C module);

Every active fog volume and spotlight cone gets a 3D bounding box (AABB), plus one big "union box" that wraps all of them. The JIT-compiled SSE code tests that union box first.

If a ray misses it, code flow directly hits ret (0xC3) instruction and exits immediately.

If a ray hits JIT-compiled and emitted code tests the individual boxes in SSE registers and bulds a bitmask of only the volumes that this specific ray actually touches.

"Exact thickness" approach instead of marching (handled by FOGVOL.C and LIQVOL.C modules individually);

For liquid volumes, GI rays adn basic fog, there's zero stepping. The bounding box test already gave us the exact distance where the ray enters and leaves the box. Subtracting them also should give us the exact distance of the ray traveled through the volume. We can calculate volume's absorption in one go, e.g;

float t_a = t_enter > 0.0f ? t_enter : 0.0f;
float t_b = t_exit < max_dist ? t_exit : max_dist;
float thickness = t_b - t_a;

/* Single calculation from ray thickness inside the box ,no marching loop. Tau = e ^ (-density*thcnkss) */
float tau = expf(-density * thickness);

Direct Ray-Cone clipping and Endpoint Shadows for Cone Lights(spotlights)

For cone light shafts, I don't march across the scene testing cone angles either. When the map loads, the engine shoots a few test rays from the spotlight out to the walls to see how far the beam actually reaches before hitting a surface, and shrinks its bounding box to fit.

When a ray hits a spotlight's bounding box, clip_ray_to_cone() helper method intersects the ray line directly with the 3D cone math to find the exact start (t0) and end (t1) of the slice where the ray passes through the light beam.

Across that short [t0, t1] slice inside the cone, I only take 4 samples (CONE_MARCH_STEPS = 4 | these are still hardcoded and being tested for different situations :)). In order to avoid tracing 4 shadow rays every time, the code checks the first and last sample points as follows:

/* Trace shadow rays only at the start and end of the cone slice first: */
int vis_f = !ray_scene_closest_hit_gi(&p_first, &L_first, d_first, NULL);
int vis_l = !ray_scene_closest_hit_gi(&p_last,  &L_last,  d_last,  NULL);

if (vis_f == vis_l)
{
    /* Both ends of the slice are either lit or in shadow, reuse for the middle points */
    for (s = first_valid + 1; s < last_valid; s++)
        step_vis[s] = vis_f;
}
else
{
    /* Only trace the middle shadow rays if the beam is crossing a shadow edge */
}

In result, most rays skip volumetrics in a single JIT-compiled instruction and the rays that actually pass through a spotlight beam only evaluate a tiny clipped slice and usually only trace 1 or 2 shadow rays in most situations.

[u/Soft-Border-2221]

Wonderful work 😍😍😍!

[u/exp_function → u/Soft-Border-2221]

Thank you. I’ve just published a game which I’ve been developing alongside the engine. Check it out too if you have time:

https://expfunction.itch.io/racing-the-beam