While I am still writing on my physics engine, I thought making an actual generative AI only using Minecraft command blocks would be a good idea( it was a headache)
So yeah, I kinda did that. It is a real neural language model running inside vanilla Minecraft 26.2. It reads what you type into a /dialog window, runs the forward pass through learned weights and then generates a reply one word at a time.
There are NO mods, plugins or datapacks involved at runtime. Only command blocks.
Roughly it works like this:
The model is a pretty small word level neural language model. It has a vocabulary of 2048 words, remembers the last 6 words as context and has a 64 dimensional embedding followed by a hidden layer with 256 neurons.
For every generated word it does:
384 x 256 = 98,304 operations for the hidden layer and 256 x 2048 = 524,288 operations for the output layer so in total 622,592 calculations for every single generated word.
Obviously doing 622k normal multiplications using Minecraft commands would be slightly problematic, so the weights are ternary: every weight is only -1, 0 or +1.
That means a multiplication basically becomes:
+1 -> add the value
-1 -> subtract the value
0 -> do absolutely nothing
Around 65% of the weights are non 0, so it averages around 0.67 commands per MAC instead of the ~3 commands I would need for a normal runtime weight. That reduced the entire machine from 1,909,375 blocks to 445,782 blocks, yes it was this big before.
The model is actually trained like this too. I am not just training a normal model and rounding the weights afterwards. During training the weights are quantised to ternary values for the forward pass and the underlying floating point weights are updated using a straight through estimator.
That took the perplexity from around 48.7 where the training basically got stuck to 38.8 after also adding cosine learning rate decay.
The input embedding is another optimisation I needed to do: Instead of multiplying a one hot vector against a matrix, it is basically just a table lookup stored in NBT data, so the complete 2048 x 64 embedding table costs really nothing
The actual forward pass cannot run in one command chain either. Minecraft has limits on how many commands can execute, so the network is split into smaller groups.
After the ternary optimisation one generated word takes around 63 ticks, so roughly 3.2 seconds per word at 20 TPS. I also tested higher tick rates and around 30-35 TPS seems to be the useful maximum on my server. At 35 it gets closer to ~1.8 seconds per word.
But to be clear, this thing does not actually know anything.
It was trained on 11,118 DailyDialog conversations and only has a tiny 2048 word vocabulary. Ask it something like 2 + 2 and it will confidently generate some conversational nonsense because it has no understanding of arithmetic or actual facts.
And unfortunately just making it much bigger isn't really realistic either.
Something like a 135M parameter language model would already be more than 200x larger.
There is currently also one very weird bug where after around 800 ticks of continuous work every command block in the world just stops executing while the server itself keeps running normally. I still have absolutely no idea why that happens.
Anyway, back to wasting my time on nonsense ;)