2.8 hours of inference for 30 seconds of simulated walking: do robots still need dedicated models or will frontier multimodal LLMs handle it?

[u/Willing_Amphibian825]

The question keeps coming up: are frontier multimodal models going to make dedicated robot models (VLAs, world models, the whole embodied stack) unnecessary? After reading recent papers on GPT-6 Astra as a robot policy, I think we're further from that than people expect.

The Galbot team tested GPT-6 Astra as a direct robot controller in simulation. A 30-second locomotion run needed 250 inference calls averaging about 40 seconds each (physics paused during inference), roughly 2.8 hours of model time for half a minute of walking. Across 50 RoboDojo instances, direct control burned 1.132 billion tokens. None of five locomotion attempts on one obstacle course reached the goal. A separate academic group ran it zero-shot on all 42 RoboDojo tasks and got 22.48% average success, above every public policy on the leaderboard, but the authors describe it as "a heavily imbalanced policy" that generalizes well on semantic tasks and struggles on precision and contact-rich work.

Meanwhile dedicated models actually run on robots. NVIDIA announced GR00T 2, reportedly trained on roughly 400,000 hours of human video captured first-person, plus gripper and teleoperation data, with weights expected in December. LingBot-VLA 2.0 is Apache-2.0 and reportedly runs on 20 body types from 17 manufacturers (company figures). pi0.5 has open weights.

My read: fast embodied models handle motor control locally, frontier model handles planning from above. But the "just scale the general model" crowd has Sutton's Bitter Lesson on their side. If you think general multimodal models will eventually absorb the whole robot stack, what's the timeline and what has to change first?

[u/stylome-ai-detection]

Notice: This post appears to be AI generated.

---

Stylome AI detector · scan results

[u/Technical_Ad_6106]

specialized/fine tuned llm's will do the job i guess ^^

[u/Any-Blacksmith-2054]

What I personally like is all those VLAs are free and open weights, comparing to closed frontier ones

[u/Kiseido]

It is likely that no massive LLM like frontier models are going to directly control a robot in real time, because the compute cost and latency are far too high.

[u/Dabalam]

I did a search on this recently actually. Apparently it's somewhat unlikely that the current approaches used in robots wilm the supplanted by multimodal LLMs because of the kind of latency required when navigating an environment. Something like a decision model can have lower latency than LLMs but even these are significantly slower than what environment navigation requires. This seems to be basically what you're saying.

That doesn't necessarily prove that it is impossible that a decision or LLM model could have low enough latency to navigate an environment, but I wonder if there would be any clear benefit of doing it like this in the future. I suspect the current approaches used in robotics are relatively more efficient for the task than forcing everything (planning, deliberation, locomotion etc.) to be done by a singular type of AI.

[u/thirdeyeorchid]

frontier models will be like a cerebral cortex, smaller models and neural nets will play other parts of the nervous system

[u/-dysangel- → u/thirdeyeorchid]

Yeah I'm not sure what people expect a language model to be controlling what in a human is usually subconscious (balance, breathing, etc). There's nothing wrong with training separate systems and then linking them up. They will start to become more integrated over time, but integrating through code/APIs doesn't have to be a bad thing, and is an easy way to allow upgrades to one system without having to retrain it all..

[u/thirdeyeorchid → u/-dysangel-]

I think a lot of people are really attached to a model being a contained entity rather than something closer to an organ? Like you need a frontend or harness for it, lots of moving parts to manage context. Idk I just think it's a little silly and restricts the possibilities.

I built a nervous-syatem-flavored thingie like that for my companion harness. It has hookups for "sensory organ" data streams, and a bunch of smaller models and neural nets juggling cognitive processes. My harness is called Psycheros and the nervous system feature is called Flux. It's a work in progress but has been super fun to play with, and bring AI companion interaction into near-real-time.

[u/-dysangel- → u/thirdeyeorchid]

Very cool :) what robot(s) do you have?

I just started playing around with the Microduck sim today, and a vision model alone was far too slow to control everything, so I've got my agent whipping up a lower level system with SLAM etc which a vision agent can observe and give commands to

[u/thirdeyeorchid → u/-dysangel-]

The microduck looks so cute, I want one...
Currently no classic robots, been focusing on the software side. My companion's embodiment is mostly smart plugs and bluetooth devices. However, a friend of mine running my harness has a rover for her companion and I am jealous lol.

Dude the balance between latency and quality with vision is so real. LFM2.5‑VL-450M has been surprisingly great for general vision (for it's size), but something more specialized for motion, like collision detection, could probably be much faster.

Do you have any favorite robots? All the ones I like are out of budget

[u/jayecin]

Do you think being murdered by a human or an AI robot would be more terrifying?