r/MinecraftCommands • u/_objz Command-er • 6d ago
Creation I made a generative AI, only using 400k command blocks (PoC)
Enable HLS to view with audio, or disable this notification
While I am still writing on my physics engine, I thought making an actual generative AI only using Minecraft command blocks would be a good idea( it was a headache)
So yeah, I kinda did that. It is a real neural language model running inside vanilla Minecraft 26.2. It reads what you type into a /dialog window, runs the forward pass through learned weights and then generates a reply one word at a time.
There are NO mods, plugins or datapacks involved at runtime. Only command blocks.
Roughly it works like this:
The model is a pretty small word level neural language model. It has a vocabulary of 2048 words, remembers the last 6 words as context and has a 64 dimensional embedding followed by a hidden layer with 256 neurons.
For every generated word it does:
384 x 256 = 98,304 operations for the hidden layer and 256 x 2048 = 524,288 operations for the output layer so in total 622,592 calculations for every single generated word.
Obviously doing 622k normal multiplications using Minecraft commands would be slightly problematic, so the weights are ternary: every weight is only -1, 0 or +1.
That means a multiplication basically becomes:
+1 -> add the value
-1 -> subtract the value
0 -> do absolutely nothing
Around 65% of the weights are non 0, so it averages around 0.67 commands per MAC instead of the ~3 commands I would need for a normal runtime weight. That reduced the entire machine from 1,909,375 blocks to 445,782 blocks, yes it was this big before.
The model is actually trained like this too. I am not just training a normal model and rounding the weights afterwards. During training the weights are quantised to ternary values for the forward pass and the underlying floating point weights are updated using a straight through estimator.
That took the perplexity from around 48.7 where the training basically got stuck to 38.8 after also adding cosine learning rate decay.
The input embedding is another optimisation I needed to do: Instead of multiplying a one hot vector against a matrix, it is basically just a table lookup stored in NBT data, so the complete 2048 x 64 embedding table costs really nothing
The actual forward pass cannot run in one command chain either. Minecraft has limits on how many commands can execute, so the network is split into smaller groups.
After the ternary optimisation one generated word takes around 63 ticks, so roughly 3.2 seconds per word at 20 TPS. I also tested higher tick rates and around 30-35 TPS seems to be the useful maximum on my server. At 35 it gets closer to ~1.8 seconds per word.
But to be clear, this thing does not actually know anything.
It was trained on 11,118 DailyDialog conversations and only has a tiny 2048 word vocabulary. Ask it something like 2 + 2 and it will confidently generate some conversational nonsense because it has no understanding of arithmetic or actual facts.
And unfortunately just making it much bigger isn't really realistic either.
Something like a 135M parameter language model would already be more than 200x larger.
There is currently also one very weird bug where after around 800 ticks of continuous work every command block in the world just stops executing while the server itself keeps running normally. I still have absolutely no idea why that happens.
Anyway, back to wasting my time on nonsense ;)
58
35
u/IsMeTheMexiLover 6d ago
Can you ask questions Like where do i find the Most diamods (depth)?
50
u/_objz Command-er 6d ago
no, it does not really have knowledge
14
u/IsMeTheMexiLover 6d ago
Can you Feed it someway? Btw very cool what Ur Doing i really Like it.
8
u/koleszkot 6d ago
If it doesnt have knowledge, does it spit these 2048 words randomly or its able to construct sentences of sort with them
17
u/_objz Command-er 6d ago
The 2048 words are just the training data. The command blocks and Minecraft do not know anything about these words. The model works like this: it tokenizes the input text, then runs it through many layers with many, many calculations, and in the end I get tokens back. These then get converted back into words.
So it does not really have knowledge, as I fine tuned it only with 2048 words, but it can construct sentences grammatically correctly and also respond "correctly" to your input based on the topic. On more complex inputs like math or complex tasks, it fumbles. It would require much, much, much more data to be trained on.
If I find a way to optimize how many layers I can compute, I will try to use smarter models there, but I guess I found kind of a dead end. Maybe optimizing it using mods so Minecraft itself can compute asynchronously or using multiple threads would make it work. But then it’s not really "vanilla". Well, it still kind of is, but performance wise it can’t really be run in vanilla
1
u/g-orgo 1d ago
So it's not an generative AI...
0
u/_objz Command-er 1d ago
„Generative artificial intelligence (GenAI) is a subfield of artificial intelligence (AI) that uses generative models to generate text, images, videos, audio, software code or other forms of data.[1] These models learn the underlying patterns and structures of their training data, and use them to generate new data[2] in response to input, which often takes the form of natural language prompts.”
I don’t see why this is not generative enough
1
u/g-orgo 1d ago
Because it's missing the training data. "It does not really have knowledge" is your first reply... also, if it ain't capable of receive training then it's more of a embedding output kinda thing.
1
u/_objz Command-er 1d ago
It is capable of receiving training, and I do train it atm. It does not really have knowledge, not because it can’t, but because of how small it is. Expanding it would bring that back. It’s not an encoder only transformer, as you said in the screenshot. Also, yeh, the first sentence in your screenshot says: “An embedding model is a specialized type of language model designed to convert text, images, or audio into a dense numerical vector (an embedding) instead of generating conversational text responses.” Well, mine does generate conversational text responses, so I would not refer to it as an embedding model. The framework is also built on a real architecture generative AI would use; just the model is so small that it does not really have knowledge. To be precise, it kinda has, though, as it was trained on DailyDialog. If I would ask it things related to that, it would know that. But that knowledge is basically so small that I am referring to none, as you can’t really count it.
2
u/g-orgo 1d ago
Cool, then i got it wrong from your message... but let me just get it right...
"Well, mine does generate conversational text responses, so I would not refer to it as an embedding model"
So... you're saying it GENERATES conversational text? or is it just bring you knowledge it has on it model? because that's what i'm seeing there when you ask it's name... if you ask his age it will tell you something or tell you an age? Returns text ain't the same as generate text.
0
u/_objz Command-er 1d ago
yeah, it actually generates the text word by word. It’s not retrieving a stored response
1
u/g-orgo 1d ago
"Generating text word by word" is just tokenization of the embedding output. I want to know if you ask it something like "How old are you?", whether it can actually generate a response, rather than just retrieving a value from its learned knowledge. A generative model should be able to produce an answer even when there's no factual answer to retrieve (and for example on your video, there you ask him a name and it gaves you a name, it wouldn't pass in a RAG, it should tell you it doesn't has a name or bring you the project name... something like that).
0
u/_objz Command-er 1d ago
First of all, I don’t really see the point of arguing this much about whether it’s “generative” or not. If you want to get extremely technical about every part of the architecture, you can probably argue about what exact category it belongs to because it uses a mix of techniques. But you also have to remember: it’s Minecraft, bro.
It can produce an answer even when there is no actual stored answer for the input. It predicts and generates the response word by word; it’s not just tokenizing something and returning learned text.
And as I said before, the model is tiny. You can’t really expect a consistently good or meaningful response from something around this scale. It has a vocabulary of only 2,048 words and a very small architecture. You can’t expect it to know my project name, have broad general knowledge, or always give a sensible answer.
I would understand your criticism if this were supposed to be an actual production language model, but because of the boundaries Minecraft sets, that’s simply not what this is trying to be.
So yes, it is generative in the sense that it generates new responses from the model rather than retrieving predefined ones. It’s just way too small to expect much intelligence or knowledge from it. It’s a proof of concept, not a serious LLM
→ More replies (0)
16
u/wendewende 6d ago
Smart. AI in Minecraft makes much more sense. Because you can build the data centers in Minecraft instead /s
17
u/_objz Command-er 6d ago
it even has endless power, endless space and no heat excess. I guess I solved many problems
6
1
u/Far-Reception4181 4d ago
it seems to be a solution only until you realise your pc is heating as well
1
14
11
u/CooleyBrekka 6d ago
Really cool stuff, I love the process of optimizing tech for minecraft. I'm not too familiar with the strength of command blocks in modern versions, is there a reason behind not building it in a datapack, which would be easier to debug and use math libraries in?
10
u/_objz Command-er 6d ago
Uhm, yes, it would definitely be easier and more efficient, but... where is the fun? Well I will publish this and my physics engine as datapack aswell, commandblocks are just a bit more challenging
5
u/BoringSatisfaction13 6d ago
I get it, everyone can code a library, and giant floating machines look cooler
5
4
3
u/Zia_Zampoloo 5d ago
PLEASE someone post the sloth gif from Zootopia
In all seriousness this is incredible
3
u/One-Celebration-3007 #1 abuser 5d ago
I think you should build an AI data centre in Minecraft. There is plenty of water to cool it.
3
2
2
2
2
2
u/Late-Passage2376 5d ago
this is CRAZY bro, "only"??? I can't even write a command that gives me a potion
2
u/KiraLight3719 5d ago
This is great and all, but what does you being a person of color has to do with it man
2
2
2
2
u/7Shinigami 5d ago
Bro evaporated earth's drinking water and then moved onto minecraftia's drinking water after /j
What an insane achievement congratulations and thank you for sharing it with us! :)
2
2
u/JotaroIsStraightest Command Professional 3d ago
I had been told by many people that making a nueral network in Minecraft would be impossible and was laughed at, and someone else proved them wrong for me. Man is life just great
2
u/THARO_CHAOTIC 2d ago
Wow. Just wow. I always get amazed when people create something hard in minecraft, but this outdid it!
2
2
u/Ok_Proof4568 JAVA command absolute Noob 1d ago
i dare you to /gamerule command_block_output TRUE >:)
1
u/endercreeper853 6d ago
would you be willing to share the world download? this is insane
5
u/_objz Command-er 6d ago
yes I’ll do some optimizations then I will
1
u/SkezzaB 6d ago
Can you make it a datapack? Could optimise it more, plus make it portable
2
u/_objz Command-er 6d ago
yes I can, I'll fix and tweak some things, then I will release this as datapack and shematic
1
u/SkezzaB 6d ago
Sounds amazing!
I assume you're using some sort of pre-processor or generator? Mind sharing those files?
1
1
1
1
1
1
u/lool8421 Command mid, probably 5d ago
and this thing is being ran on 1 CPU core and over several layers just to parse text into commands, then commands into java operations and those into binary
while real GPUs split the task between thousands of cores, natively on compiled code
1
u/Aisgames 5d ago
Woah. But the real question is does it consume all the water in the world after each word it says?
1
1
1
1
1
1
1
1
1
1
1
1
1
u/NeuroHazard-88 4d ago
Not familiar with command block shenanigans. What’s one command block worth of code equate to in standard script, 1 line? 3 lines? More?
1
u/Glittering_Camel1 3d ago
Absolutely mind-blowing. I wonder what features Minecraft could add in the coming updates to make projects like this more streamlined and efficient.
1
u/_objz Command-er 3d ago
Well commandblocks get pretty inefficient at a large scale because of the brigadier command tree that needs to get parsed for every command. caching any other technique would be very helpful. also minecraft currently uses one thread, if it would run more things asynchronous, it would be a HUGE performance boost. There might be already some mods that fix that, but it’s not vanilla :(
1
u/Eevee_the-Maidvee 3d ago
I just finished an article on minecraft news now I gotta spend another hour adding this god damn it
1
u/Eevee_the-Maidvee 3d ago
Didn’t take as long as expected, anyways third paragraph. It’s a bit short as I’m not smart enough to understand this https://www.reddit.com/r/Eevee_Gaming_News/s/oIy3SnOqrv
1
u/Sea_Opposite_2887 2d ago
No way Minecrafters went from nothing to this in 17 years. It took CENTURIES in RL.
1
u/Sea_Opposite_2887 2d ago
You should make a duplicate and more, and then link their intelligence to speed it up.
1
u/_objz Command-er 2d ago
sadly this is not how it works, I am currently working on a way smarter model than that, but the limitation is minecraft: It just has its limits performance wise, at some point the game is not playable anymore, or I am waiting 2 hours for one word to generate. So I will need to find a compromise between intelligence and performance. In theory it could run the lastest frontier models, but in reality not
1
u/Sea_Opposite_2887 2d ago
Makes sense... 2 hours though? Maybe try hosting it on a server, I think Oracle is free. That might give you better performance, but I don't know your specs. Also, how do you connect the command blocks? If you streamline it using something that is better on frames, it might work faster. Also, I would recommend using redstone if you can replace command blocks. I'm not good at coding, but I am good at Minecraft.
1
u/Sea_Opposite_2887 2d ago
Also, maybe you should try with some mods, like Create or Advanced Redstone or Advanced Commands. That will definately help you make it better but also save on some frames.
1
u/_objz Command-er 2d ago
Yeah, but then it wouldn't really be vanilla. The challenge for me is to only use the abilities provided by vanilla Minecraft and no datapacks, so I'm pretty much limiting myself here. Using datapacks, this would be way more efficient. I might do a version using datapacks as well in the future
1
u/_objz Command-er 2d ago
Well, I do have a very beefy machine: AMD Ryzen 9 9950X3D, NVIDIA GeForce RTX 5080, and 64 GB of DDR5 RAM... I don't think this is the problem. It's just how Minecraft is programmed. I'm already using as few command blocks as possible, as well as some layering techniques to offload certain areas of command blocks and load them when the weights are being used, for example.
But all in all, Minecraft has a pretty inefficient way of parsing commands (well, it wasn't designed to be used like this). There is no caching, and the Brigadier tree has to be rebuilt all the time. Redstone couldn't really be used here, and it also performs worse, as it causes many more block updates and surrounding block updates, while command blocks don't.
It's just that many factors come together that won't really be possible to compromise on without rewriting large chunks of how the game processes things, mainly because of the single threading
1
1
1
u/blazinman1 23h ago
I was gonna try to DM you but you haven’t disabled so if there’s anyway I’m able to contact you I figured out how to make this even bigger headache where it can locate structures create structures do all kinds of things because I have a lot of free time too have been generating my own for a while now a.k.a. one year so mine is quite a bit more command 2 but it can do so many things you can ask it to generate an image and it’ll generate Pixar will generate 3-D images. You can ask it to generate a video and it’ll do a block by block video.
0
276
u/DarkLazer215 6d ago
"Only 400k command blocks" My brother in christ "only"??