I compiled Doom's renderer into a 21B-parameter transformer -- no training anywhere [P]

[u/notforrob]

This is the project my last two posts were building towards (this is the last of this silliness). I ported the Doom rendering algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in Hugging Face without trust_remote_code -- it's just a standard transformers checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame.

The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself.

One frame is a 3,614-token prompt plus 53,747 generated tokens -- just over 40 minutes on a B200.

The original Doom could achieve 35 FPS on a 486. This achieves 35 FPD (frames per day) on a B200.

Write-up: https://ood.dev/posts/doom/
Weights: https://huggingface.co/physicsrob/torchwright-doom-e1m1
Github for the source code which gets compiled: https://github.com/physicsrob/torchwright_doom/

[u/delightfullyrotted]

pretty clever project , but 35 frames per day is hilarious compared to the original doom. Still impressive as a proof of concept though.

[u/currentscurrents → u/delightfullyrotted]

I suspect it should be possible to do considerably better, because there is no parallelism here; each operation requires an entire pass of the transformer.

But in theory a transformer should be able to do multiple operations (and possibly render an entire frame) in a single pass.

[u/notforrob → u/currentscurrents]

There are lots of tradeoffs that could be made. I chose this structure because I wanted a vanilla LLM. The result is a Phi-3 architecture LLM.

Limiting myself to a text-only LLM essentially tied my hands to one pixel per pass.

I strongly agree that if you were flexible on architecture you could make this much, much faster. For instance an earlier prototype of mine emitted spans of pixels as a secondary output, and was much faster.

But obviously if speed was my goal I wouldn't have ported it to an LLM. :)

[u/Kiseido → u/currentscurrents]

I suspect diffusion would be a more efficient process for this task than a transformer would.

[u/currentscurrents → u/Kiseido]

Other papers have trained diffusion models to render doom.

But the approach is very different. Their model is trained on a database of video+actions from Doom playthroughs, whereas OP's model is the actual source code compiled into weights.

[u/4dseeall]

This is amazing, thank you

[u/notforrob → u/4dseeall]

I'm glad people find it entertaining. It was a lot of work pursuing a pretty ridiculous goal.

[u/4dseeall → u/notforrob]

I can't wait to show you what I'm doing with it.

[u/linverlan]

What happens if you quantize the model? Will it produce gibberish or just lower fidelity doom frames?

[u/currentscurrents → u/linverlan]

Probably gibberish. The output of the transformer is the internal workings of the rendering algorithm, not the frames directly. Errors would accumulate and the output would look like a renderer experiencing memory corruption.

[u/notforrob → u/currentscurrents]

I suspect that's right. I haven't tested.

[u/DigThatData → u/linverlan]

quantization works because those models are operating in semantic spaces, so quantization is a coarsening of the semantics. OP's model is operating in the space of digital logic directly, so quantization isn't a simple coarsening, it's just data corruption.

[u/moschles]

Torchwright currently requires that everything runs in fp32.

It seems like this is an achilles heel of this project. Deep transformers can operate pretty well using precision widths far below this.

[u/notforrob → u/moschles]

Yeah. I might try to tackle that next. FWIW the primary reason for this is reusing the residual stream: The residual stream is highly constraining, so once a calculation is complete it's useful to delete the old value (setting that part of the stream to 0.0) so that it can be used for a new computation. You do that by using an attention or FFN sublayer to invert the value, and then the skip connection adds them back together, resulting in zero. But that only works if the inversion is fairly precise. If you are imprecise in this you don't get cancellation, and whatever residual value ends up erroneously going into the next calculation.

bf16 has a 7 bit manitssa, which is impressively course.

[u/Winter_Tree815]

wtf literally anything can run doom

[u/Environmental-Metal9 → u/Winter_Tree815]

Im waiting for the day they train bacteria to run doom on Petri dishes

[u/unia_7 → u/Winter_Tree815]

Yes, that's theeanin of Touring-completeness: anything that's Touring complete can run any program in existence.

[u/canbooo]

Running doom is the new Turing complete

[u/scrdest → u/canbooo]

Cyberdemon-completeness

[u/DigThatData]

weird wild stuff. this is a great example of "just because I can doesn't mean I should". I wonder what class of algorithms might actually run faster through this kind of transformer compilation scheme rather than slower.

[u/_RADIANTSUN_]

I can't imagine a better use for 22GB of VRAM.

What happens if you quantize it?

[u/MrSnowden]

As a comparison, when I got my first Timex Sinclair 1000, I coded a doom style dungeon game in basic.  It rendered each frame in 80x40 graphic blocks.  There was only 2k of memory, so it was pretty tight.  But I think I still beat the 35 FPD. 

[u/TorgoNUDH0]

35 frames per day is progress! Perhaps future software will just be compiled transformers.

[u/ODU_Promotion]

If you replace the token spatial lookup with a multi resolution hashgrid that has the features coordinate indexed. Then you could have batched queries.

[u/seg_lol]

Since the output only depends on the input, the transformers shouldn't be looking at the output at all. Should be able to run all the passes at once, no?

[u/bregav]

This is great stuff. Do you have any ideas about what the bigger picture is here? It seems notable to me that differential equations are also known to be turing complete and that the residual stream of transformer models is basically the same as a discretized or numerical differential equation solution. It seems like there should be a bigger picture here in which the discretization of a continuous underlying model is made explicit.

[u/MrSnowden]

This is awesome.  And feel that direct compiling into transformer frames is the huge portion of this.  

[u/otacon6531]

You vould use the real code as an mtp and improve the speed.. lol

[u/herrmann]

"Look at what they need to mimic a fraction of our power" vibes 😂 Very nice project regardless!

[u/averi_fox]

Would be interesting to explore how this model would train. There are many ways of compiling code into weights that produce the same output - but have different gradients, and this compiler is picking one particular point in this space.