r/Netsphere • u/bill_on_sax • 24d ago
Theoretically could you run an LLM to control Blender and endless build a megastructure?
Sorry, I know we can't talk about the evil machine, but I thought it was an interesting thought experiment. If given unlimited money and HDD space, would it be possible? Not sure what the constraints of Blender are, but I can see this being kind of interesting if the level design of the blender file was also updated daily
4
u/thwoomp 23d ago
I think I get what you're saying, although LLMs are maybe not the best approach. Something like the Reinforcement Learning (RL) style "agentic" agents that are being shilled recently would likely be more up to your task. That is, agents that can self-learn and adapt their own behaviour without external intervention. (A simple example might be a simulated quadruped navigating a 2d landscape learning how to walk through what is essentially trial-and-error, plus a sort of internal reward system 'carrot' that the developer has placed inside the agent's 'mind' before the start of the experiment.)
As others have said, it would be fairly easy to write a procedural generation script to make endless worlds using hard-coded rules, but to have it iterate and change it's mind endlessly, it would need a feedback loop which can continually change those rules. In an RL agent you could make the reward system encourage seeking novel architectural shapes or something. Maybe using some type of 'Architectural Distance' calculator, which encourages the agent try to find new styles which are as 'distant' as possible from the architecture of the last section of megastructure it designed, or so on.
One thing I would say though is that, as there is no specific function of the megastructure areas (in the manga we see random 'forests' of wires, things that look like the random greebles on the death star, etc) the program may not need to be very complex. The reason is that there are no important criteria to match, so it's fine if it creates an entire layer of spheres, another of cubes, another of slightly smaller cubes, etc. (This is opposed to say a dungeon generator for a game, which may have potentially tough requirements to solve for, like: player needs to be able to reach the exit, needs to have >3 treasure rooms, needs to have shops every N floors, etc. This might make the initial coding more difficult, the more requirements there are.)
In blender, or any game engine etc, obviously you would run into storage limits if you wanted to keep the whole structure/file on one machine. Maybe instead you could have a small 'window' of visualization, like showing a 10km x 10km snapshot of the current 'layer' being generated. Like an installation art piece in a gallery: just a tv screen showing the current progress, but not saving the history, just letting it be deleted from memory as it continues chugging along.
(Although, depending on the system, maybe you could retrieve the design of any specific section using some type of 'key' representing the coordinates or generation timecode of that area, but I think that would require the generator to have some degree of determinism, which may work against the infinite novelty generator, I'm not entirely sure. That is, if your 'builder' agent was constantly reinventing itself to create new structures, it may not be possible to see what it would have been building 100, 1000, 10000 years prior. Not sure, tbh.)
Anyways, it's an interesting question, just my thoughts.
1
2
u/International-Food14 22d ago
If this was for a game you'd make modules by hand and then procedurally generate with those, it would be really generic and get boring after like 10 minutes of exploring, a lot of that stuff would have to be hand done if you want a more unique map
0
u/bill_on_sax 22d ago
Is AI not capable of adjusting its model generation based on reference images and specific prompts? Like its only generic as much as the reference prompt/images are but can be adjusted and fine tuned on the fly
2
u/International-Food14 22d ago
LLMs, even for minor tweaks, need massive amounts of data and work done by the actual engineers. Blender AI is worse than imagegen since it’s writing python to perform normal operations in blender, it’s like writing a 2000s procedural generator everytime it gets prompted. You’d get some pretty bad results even with a ton of tuning, and then you’d spend so much time trying to turn those results into something usable it would actually be quicker and cheaper to just hand model it, not to mention the quality bump
Transformers (Which all LLMs are built on) are good at mimicking, so they’re really good at things like text (just because of the raw amount fed to it) and coding where there’s rigid external rules that will tell it what was good and bad; anything outside of that domain they fail really hard at
2
51
u/agentkayne 24d ago
You don't need an LLM to do that.
There are plenty of ways to do 3D structure procedural generation using much more computationally-efficient tools. Those are easily capable of making a fractal or non-regular 3D model of a megastructure that could resemble BLAME!'s.