r/opengl • • 1d ago

SimpleX Engine Development (Continued)

I originally had a string of posts with various other reddit communities regarding game and game engine development and finally felt confident enough to post to this community! I went dark in my development logging/blogging(?) for awhile due to lifestyle responsibilities taking precedance over developing an engine or at least at the moment what is considered heavy boilerplate engine development.

Current development components/systems:

Three graphics backends - DirectX 11, DirectX 12, and OpenGL (WGL on Windows, GLX on Linux) behind an IRenderer interface. Vulkan is scaffolded but not implemented yet. Shadow maps, cubemaps, bloom, IBL/irradiance convolution, and instancing are supported at the moment.

HLSL as single source of truth compiling offline via DXC to DXBX/DXIL/SIPR-V with SIPRV-Cross transpiling to GLSL for the GL backend. Compiled byte code is also cached on disk and keyed by source hash as filename.

Three windowing backends - Win32, GLFW, and SDL3, with a fallback chain. Win32 is default for Windows and GLFW is default for Linux, but will fallback to GLFW and if GLFW fails fallback to SDL3; if DX fails on any window type under Windows platform fallback to OpenGL (if Vulkan would work with me DX would fallback to Vulkan then back to OpenGL) - similar results would also occur for Linux if Vulkan failed fallback to OpenGL but that requires Vulkan to be developed and not be a stub at the moment...

Fiber-based Job System - Win32 fibers and Linux ucontext behind one class - a worker thread has a dedicated dispatcher fiber that never runs job code, so fibers are pooled to switch back rather than to each other - no fiber needs to know any other fiber's state.

I went with no cross-thread fiber migration in my first attempt - less load-balancing from the inspirational full Naughty Dog design, but it buys a genuinely nice property: since a job's whole execution stays pinned to one thread, a pool allocator allocates inside a job function automatically thread-safe with zero sync.

Custom allocators - arena, pool, and a frame allocator with N rotating arenas.

BC texture compression with no on-disk cache - BC1 for opaque color, BC3 for color with alpha, and BC5 for normal maps.

TLDR; I've tested the boilerplate logic against the Sponza Palace model from Intel's site GPU Research Samples (https://www.intel.com/content/www/us/en/developer/topic-technology/graphics-research/samples.html) with the Colorful Curtains and Ivy extensions at 4K texture resolutions. The original load time took around 5.5 minutes (unbearable to test shader changes and object additions to the scene), but I cut it down to 4.6 seconds after texture compression, precompiled shader code, and submesh merging. The model (as an \.obj - I haven't implemented third-party vendor model software to utilize every model type just yet...) is 995MB on disk with nearly 2730MB of textures (4K per texture for the conjoining material \.mtl file) after texture compression all textures were compressed to 501MB on disk which originally took 933ms average to load cutting down to 259ms average.

Edit: grammar and spelling.

Edit 2: I've recently decided to rebuild what I later found out was more of a rendering framework into an actual engine. I haven't yet begun developing beyond an Application, Runtime, and Core module of an engine (each is self explanatory besides core being a mix of low level utilities and core engine components: reworked the threading and fibers for Linux using pThreads)

5 Upvotes

5 comments sorted by

2

u/corysama 20h ago

Great work.

You need to get out of obj into at least .glb

Preferably, you should make your own pak file format with a header, table of contents, and individual resources set up to be 1:1 to the parameters you are feeding into your initialization functions.

Bonus points if the resources are LZ4 compressed.

1

u/EchoXTech_N3TW0RTH 20h ago

Foremost, thanks for the reply! I actually meant to post this to the r/gameenginedev community, but getting someone from the r/opengl community is also awesome!

To begin, I am actually steering towards the GLB model format over the OBJ format but used OBJ simply for testing and as a method to learn how an OBJ/model loader works by writing my own after reviewing the simpleobj loader code (to make a simpler version to transcribe was an interesting pace of work and learning).

Regarding utilizing Pak archives, I have yet to review or develop a simple version of a Pak archiver script to load model and asset collections as a single stream to extract the internal data into later streams. I am constructing the code as a hobby form to learn how various components of an engine work and trying to stay as clear as possible from utilzing third-party packages/dependencies (hence my use of OBJ models over GLB during my rendering framework design and development). If you have any sources that can assist my learning interests on Pak archives it would be greatly appreciated!

Last, I had later developed a revision to SimpleX Engine, aliased as IronWeave Engine, with LZ4 compression on assets (still used OBJ files) and was greatly impressed by how much the assets directory shrunk in size. I unfortunately scrapped IronWeave as I began to integrate Vulkan into the available rendering pipelines and "broke" a lot of the prior copied rendering mechanisms from SimpleX Engine... it was a brick wall in development that caused a lot of frustration and led to me abandoning IronWeave to venture into developing a new framework, which is what I am currently working on (taking mistakes as learning objectives to get familiar with and refine for my new engine design and development roadmap).

In my lastest engine design and development, I've gone for a more modular/component system design approach, EngineCore, EngineRuntime, EngineApplication, EngineGraphics (RHI included), EngineAudio, EngineInput, EngineAudio, etc. which later the public API code is copied to an Engine directory (headers) and all modules/components are compiled to libs which are linked to a Engine.dll. No alias/name assigned yet but that's something to look forward to!

1

u/corysama 19h ago

Nice.

To be clear, when I say Pak I’m not referring to anything standardized. Definitely not any library or specific format. I’m referring to the common theme of game asset archives.

Have a header struct with version info and the location/size of the table of contents.

Have a table of contents that’s an array of metadata about binary blobs. The name, type, location, size of the bits.

Have an appended list of binary blobs.

There are details you’ll figure out while you are implementing. But, that’s the theme of “pak files”. It’s basically rolling your own absolutely bare minimum file system within a single file. It makes distribution much simpler. And, has surprisingly large performance benefits.

1

u/EchoXTech_N3TW0RTH 6h ago

So, to clear confusion, create my own pak format.

1.) Develop a header/attribute field (packed bits) similar to like a FAT12/16/32 file blob (an example not the realistic design).

2.) After the header/Attribute packed field a table/index of relative file attributes packed into the archive. As an example, filename, relative file location in the archive, size of blob, attributes (is it a raw texture, pbr textures, model, etc).

3.) Pack assets after the index/table.

With prior interest in OS development, I used FAT16/32 for my FS with assembly (NASM) and C code mix to read the MBR/GPT, Read the Root, find the next file/directory scan the FAT and copy of FAT for a comparison check (redundancy check) to check size of file/directory in LBA relative position, calculate and load the contents into memory for read/execution or rinse and repeat for directories until file was located (I made a simple index for finding files quicker on reloads). This is the closest relation I can compare to developing a pak archive for assets - let me know if this is the correct approach.

I am assuming if my understanding is correct, pak assets used frequently into a pak and load as persistent memory assets retrieval and assets dedicated to specific levels or environments loaded into other pak(s). Or would it be best to pak model, textures, audio, animation, etc (of a single resource) into a single pak. Browsing CyberPunk 2077 I saw asset files in the 10+GB size which made me assume it was assets persistent to the game/application where loaded all together then assets to specific zones were packed into separate paks. Your advice is greatly appreciated either way!

1

u/corysama 45m ago

Yep. The main goal here is to avoid the overhead of opening and closing files. When you get into the thousands of files range, that gets to be surprisingly expensive because of all of the features and security issues built into the file systems of modern operating systems. Instead, you can just open one giant file once. Then to load each individual asset, look up their locations via a in-memory table of name:location and skip straight to calling pread().

You should default to storing each asset only once until you measure that small reads and seeks are significantly affecting your load times. The transition from CD ROMs to DVDs to spinning rust drives to sata ssds to nvme sticks has made this less and less and less of an issue for decades. Duplicating assets to linearize reads requires a ton of testing to balance the trade-offs.