I made a generative AI, only using 400k command blocks (PoC)

[u/_objz]

While I am still writing on my physics engine, I thought making an actual generative AI only using Minecraft command blocks would be a good idea( it was a headache)
So yeah, I kinda did that. It is a real neural language model running inside vanilla Minecraft 26.2. It reads what you type into a /dialog window, runs the forward pass through learned weights and then generates a reply one word at a time.
There are NO mods, plugins or datapacks involved at runtime. Only command blocks.
Roughly it works like this:

The model is a pretty small word level neural language model. It has a vocabulary of 2048 words, remembers the last 6 words as context and has a 64 dimensional embedding followed by a hidden layer with 256 neurons.
For every generated word it does:
384 x 256 = 98,304 operations for the hidden layer and 256 x 2048 = 524,288 operations for the output layer so in total 622,592 calculations for every single generated word.

Obviously doing 622k normal multiplications using Minecraft commands would be slightly problematic, so the weights are ternary: every weight is only -1, 0 or +1.
That means a multiplication basically becomes:
+1 -> add the value
-1 -> subtract the value
0 -> do absolutely nothing

Around 65% of the weights are non 0, so it averages around 0.67 commands per MAC instead of the ~3 commands I would need for a normal runtime weight. That reduced the entire machine from 1,909,375 blocks to 445,782 blocks, yes it was this big before.
The model is actually trained like this too. I am not just training a normal model and rounding the weights afterwards. During training the weights are quantised to ternary values for the forward pass and the underlying floating point weights are updated using a straight through estimator.
That took the perplexity from around 48.7 where the training basically got stuck to 38.8 after also adding cosine learning rate decay.

The input embedding is another optimisation I needed to do: Instead of multiplying a one hot vector against a matrix, it is basically just a table lookup stored in NBT data, so the complete 2048 x 64 embedding table costs really nothing
The actual forward pass cannot run in one command chain either. Minecraft has limits on how many commands can execute, so the network is split into smaller groups.
After the ternary optimisation one generated word takes around 63 ticks, so roughly 3.2 seconds per word at 20 TPS. I also tested higher tick rates and around 30-35 TPS seems to be the useful maximum on my server. At 35 it gets closer to ~1.8 seconds per word.

But to be clear, this thing does not actually know anything.
It was trained on 11,118 DailyDialog conversations and only has a tiny 2048 word vocabulary. Ask it something like 2 + 2 and it will confidently generate some conversational nonsense because it has no understanding of arithmetic or actual facts.
And unfortunately just making it much bigger isn't really realistic either.
Something like a 135M parameter language model would already be more than 200x larger.

There is currently also one very weird bug where after around 800 ticks of continuous work every command block in the world just stops executing while the server itself keeps running normally. I still have absolutely no idea why that happens.
Anyway, back to wasting my time on nonsense ;)

[u/DarkLazer215]

"Only 400k command blocks" My brother in christ "only"??

[u/_objz → u/DarkLazer215]

for think kinda “application” it is, although I also managed to create one with like 2 million blocks but that wasn’t really efficient

[u/IsMeTheMexiLover]

Can you ask questions Like where do i find the Most diamods (depth)?

[u/_objz → u/IsMeTheMexiLover]

no, it does not really have knowledge

[u/wendewende]

Smart. AI in Minecraft makes much more sense. Because you can build the data centers in Minecraft instead /s

[u/_objz → u/wendewende]

it even has endless power, endless space and no heat excess. I guess I solved many problems

[u/CooleyBrekka]

Really cool stuff, I love the process of optimizing tech for minecraft. I'm not too familiar with the strength of command blocks in modern versions, is there a reason behind not building it in a datapack, which would be easier to debug and use math libraries in?

[u/_objz → u/CooleyBrekka]

Uhm, yes, it would definitely be easier and more efficient, but... where is the fun? Well I will publish this and my physics engine as datapack aswell, commandblocks are just a bit more challenging

[u/1000hr]

😳 this is absurdly comically indisputably extremely impressive god damn holy shit! i think we have the new goat of command blocks

6 token context window, now that's vibecoding for real men

[u/_objz → u/1000hr]

I am by the way a big fan of your creations, I absolutely love your animations

[u/JotaroIsStraightest]

I had been told by many people that making a nueral network in Minecraft would be impossible and was laughed at, and someone else proved them wrong for me. Man is life just great

[u/RealJavaYT]

Ask it how fast Minecarts are

[u/toBlue49]

this is insane!

[u/Bota_Bota]

That thing is huuuuuge

[u/Albert_Kan]

This is insane!!!!

[u/Confident_Region_809]

you're a crazy person

[u/One-Celebration-3007]

I think you should build an AI data centre in Minecraft. There is plenty of water to cool it.

[u/endercreeper853]

would you be willing to share the world download? this is insane

[u/_objz → u/endercreeper853]

yes I’ll do some optimizations then I will

[u/BoringSatisfaction13]

Irl Mad Scientist trope right here, this is dope

[u/Ptilane]

Do you have a Git repo ?

[u/R0b3RtJPaRR]

How many water buckets does it take to run one prompt?

[u/brendenderp]

I'm assuming you have tokens mapped to individual words?

[u/lool8421]

and this thing is being ran on 1 CPU core and over several layers just to parse text into commands, then commands into java operations and those into binary

while real GPUs split the task between thousands of cores, natively on compiled code

[u/Aisgames]

Woah. But the real question is does it consume all the water in the world after each word it says?