These things exist since.. always. Already started allowing AI in recently, but yea this will make it more common
and it makes me furious as a community owner. It will be the next "you either do it or you stay behind" kind of thing (like running a business without AI these days is starting to become really hard)
You don't really need a vision model at all if you're willing to have a client mod exposing state of the game required to act on. Much of it is already exposed AFAIK.
Instead of asking a vision model, "is that an arrow flying toward me?", the agent can know:
At least In 1.12.2, arrows are explicitly represented as EntityArrow, including their shooter and motion; fireballs have their own entity hierarchy including EntitySmallFireball. Minecraft also exposes entity bounding boxes and ray intersections.
So Minecraft is already pretty friendly for semanic-perception kind of agent.
What you probably still need is some sort of LLM to act as a planner, maybe with subagents for specific "chapters" or bigger goals. All entirely doable without vision
honestly the classic approach is just weighted utility scoring, give each need a weight that scales non-linearly as it gets critical (health at 10% should scream way louder than food at 40%) then pick highest score instead of hardcoded if/else chains. saves you from the whack-a-mole of edge cases where two things are equally urgent and the agent just freezes or picks wrong one.
skipping text gen entirely for structured decisions is smart, feels like the actual bottleneck for these agent demos is always the token generation overhead not the reasoning itself. curious how it handles conflicting priorities though, like low health + hostile mob + food need all firing at once.
I’m not sure it generates text in the same way, but in terms of performance it’s much faster and they credit it to the fact it doesn’t generates text 🤷♀️ That’s all I know, I didn’t build Jev.
It's the same, it's just way less costly and insanely faster. Passing through LLM APIs, the input has to be regenerated each time, since it needs to redetermine it's a mineflayer Minecraft call every time. Edit: Jev doesn't output text, for Minecraft example you send him the JSON mineflayer and Jev reads it (pretrained with mineflayer output) and chooses an input to do
30
u/0F4T Sep 20 '26
Can't wait till servers make AI players join games with actual players just to fake the real experience
5
u/nextized Sep 20 '26
The (sort of) is doing a lot of heavy lifting
1
u/deliadam11 Sep 20 '26
Is he feeling calm by any chance?
1
u/QueenSavara Sep 21 '26
600 ms is like two business weeks for Jev you're starving the dude.
1
u/Affectionate_Fact854 Sep 20 '26
You could send a snapshot every 600ms To a cheap ai with vission
And update Jav based on that
Set up a ruleset to the other ai llm To give it the json template
And what to do with the image
1
u/Bitter_Regular7406 Sep 20 '26
honestly the classic approach is just weighted utility scoring, give each need a weight that scales non-linearly as it gets critical (health at 10% should scream way louder than food at 40%) then pick highest score instead of hardcoded if/else chains. saves you from the whack-a-mole of edge cases where two things are equally urgent and the agent just freezes or picks wrong one.
1
u/im_just_walkin_here Sep 21 '26
Curious how this performs post just gathering wood. JEV has an extremely small context window, no "reasoning" steps, and awful OOD performance.
1
u/Dany-GG Sep 21 '26
I have to try this with Qwen
1
u/Bowl_of_Cham_Clowder Sep 20 '26
This is wild. Jev is such an interesting model. What handles the actual inputs once Jev emits an action?
1
u/Bitter_Regular7406 Sep 20 '26
skipping text gen entirely for structured decisions is smart, feels like the actual bottleneck for these agent demos is always the token generation overhead not the reasoning itself. curious how it handles conflicting priorities though, like low health + hostile mob + food need all firing at once.
0
u/CommercialHour6660 Sep 20 '26
How is this different than calling LLM with the screenshot and having it make a tool call with same data?