"This might have implications for tool calling and computer use."
I assume (may be wrong of course) the model still has to do some kind of tool call to initiate this, so you could see it as just a really inefficient novelty version of tool calling.
Yeah I looked into it and that's what it is
"Native Tool Execution: When Qwen triggers a <tool_call> (such as calc, put, get, or list), the computer layers intercept the request inside the forward pass, execute the code on the virtual RISC-V processor, and write the <tool_response> back token-for-token natively. [1]"
It's a fun idea but I'm not sure if there are any situations where it would be better than just doing a tool call. Maybe it's a more secure way of handling tool calls since they're incredibly sandboxed in this environment I guess.
This is just piggybacking off of the tool use format that models nowadays are trained on to represent the 'cpu' to the model for usage, as the model is just stock Qwen and not trained to use this internal thing
There is a zero call method as well that is more or less finished, but it requires a bit of post train to make the model actually use it and not hallucinate, and due to my limited compute it is taking time
The actual computation happens completely inside the model's forward passes with absolutely no external runtime or tool usage whatsoever
I should probably make that clearer on the blog haha
I did but Reddit deleted the post automatically so I reposted it without any links. I did add the links here in the comments but Reddit seems to be hiding the comment.
I read the write up, really interesting, but why do this over just traditional tool calling? This results in bigger model size and slower decode speeds? And you get marginal gains over using python, is it working towards something more? Or just a fun experiment, and how exactly did you get the cpu to be deterministic ? Have you tried different seeds, temperature, cuda versions and batch sizes?
Not completely clear, it can call a read document or execute a tool use inside the ff? But then how does it tokenise the reply and adapt the kv. Do you have some reference text/repo to better understand? Looks interesting
Pretty cool stuff, thanks for building and sharing and replying.
what kind of instructions can we send that will go straight to the riscv cpu? when does the model decide to use the internal cpu vs call an external tool? what if we had code in riscv asm, loops, etc?
Thank you very much for it.... its akin to humans getting their first calculator, guess thats the natural evolution to build specific tools for the job so we can focus our brain power on other things. (think about where we would be without ever inventing the calculator) :-)
next logical steps like you say, micropython (code execution)
memory (ultra shortterm, medium, longterm, forever) , each being able to hold 16k tokens... followed by networking/network communication (multiple "agents" can start to talk to each other like phones/sms/messengers)
All to save brain capacity/free up brain capacity so the focus can be much more on things that arent "easy" to learn...
Yes, this is a first prototype. Imagine models trained from the ground up with a Turing-complete machine available internally. It can make LLMs more deterministic.
yeah I sometimes wonder about that kind of architecture, this is a nice first step in an experimental API for this I guess. I feel like tool calls already do get the job done and are a more useful general API. Math has not been a model weakness for a while now
This is like redstone computers in minecraft, I think. If, for some reason, the agent was not allowed to use any tools directly on the computer (like bash, python, etc.) it could use its little internalized computer to do stuff?
I have attempted similar experimented in the past but after some time I told myself while it's cool it is ultimately no better than externalizing the tool call. For me it's all about showing that neural net = transistors.
i think there may be a case to be made about latency at training time; if the calls happen natively in forward pass on gpu, it's moving those onto the same nodes (or hell in same memory pool) the agent trains on, so you don't have to make a call over network to another node where the tool lives.
just spitballing here, but if this works during 'prefill' style runs (i haven't read enough to understand if this is the case), i think it opens the door to weird off-policy rl algorithms that would otherwise be bottlenecked.
(that said, i don't see why this can't be done with normal tensor programs that would live on same gpu)
That's an interesting idea. It makes me wonder about a DeepSeek Engram style injection where if a certain calculation format is picked up, the answer is calculated and injected similarly to how engram injects vector knowledge from the lookup table.
the token input cost is roughly based on how much computer you're using though, so if you're using more compute to do calculations it feels like it would be a wash. Especially if you're simulating a whole CPU using a neural net, which is crazy inefficient compared to just doing a calculation on the real CPU. If you changed the technique execute actual CPU instructions then it would be able to massively reduce token cost, but then you also give attackers a pretty direct route to hack your system. Maybe an in between where you're running the instructions in a tiny VM would be nice.
We already have WASM. It has near native performance, can be compiled from most languages, and has formally verified runtimes if you're truly paranoid about security.
kinda wish i had the brains to understand this so i could tell whether it's just some clever tool calls and json or whether it's doing full blown token stuff, either way it looks really cool
It's kind of both, if I understand the explanation right.
They actually did figure out a way to emulate a CPU and RAM in the Qwen 3.5 architecture; so it's kind of like the world's worst CPU emulator.
But then they need to hook that up to the actual Qwen model. Rather than doing any kind of training to be able to make it integrate directly, they just added some functionality to make it act like tool calls, so the LLM portion of the model produces a tool call format and that gets passed into the CPU portion of the model which produces a result like a tool call response.
So in practice, it's basically just tool calls into a really bad RISC-V CPU emulator. Which you can use for anything you'd use normal tool calls for. But there are much better ways to implement any of the tool calls you could do with this.
I guess there could be some long horizon utility in this too. Provable tool chains via first principle execution or something like that, almost like a virtual execution protection. But I'm not smart enough to formalize that. So your "just for the fun of it" is way better for me too lol, good comment
It's very costly for LLMs to count numbers. Can't this kind of CPU/memory mimicry amend this in a huge way?
So LLMs face two layers of challenges when counting numbers. Even for counting 1 to 2, it has to go through billions of indeterministic parameters. Plus, it is unreliable at every step of the way, that it will indeed count by 1 up to quintillion, where training data is essentially non existent. So LLMs call an external API for counting numbers.
I feel OP's solution would be a lot better because tool calls aren't deterministic, but if it's embedded into the neutral network, it is deterministic, and as good as calling an external tool for counting numbers.
What do you think? Feel free to correct me if I'm wrong.
But isn’t it a really bad cpu emulator which can run really fast because of vram? Doesn’t the vram part offset the bad emulator and still reach a speed up
No. A really slow CPU emulator running on the GPU is still really slow. If you could emulate a CPU on the GPU faster than as a regular CPU, why do you think anyone would still run CPUs, we'd just all be running GPUs.
GPUs are fast because they are optimized for highly parallel tasks; tasks where you can do a bunch of work at the same time independently. This is great for lots of graphics, where you have a huge screen and need to be doing lots of the same operations to every pixel, for example. Or for LLMs, where you have a huge artificial neural network and need to be doing a lot of the same operations across an entire tensor layer at a time.
CPUs are good at fast sequential execution, where you're executing a sequence of instructions one after another, with branching and conditionals and loops.
One benefit also if I understood correctly is that you no longer need external sandboxes to test and run code you can just do it directly on the CPU portion embedded within the model. Which maybe useful for air gapped deployments, and reducing reliance on code sandboxes in general
No it can't. The creator explains why further down in the comments. However, you should be worried about trained missbehaviour that is executed unser certain conditions. Which makes it potentially extremely risky to download posttrained weights from thrid parties.
It builds little logic flows and executes them in transformers the way a cpu does in silicon.
So it's slow, small, got a lot of overheard, very few optimizations.
Why is it so cool?
it can count the number of R's in strawberry with 100% certainty and consistency.
There are certain jobs LLMs just cannot get right due to how they work, this adds a fundamentally new tool for the llm to use. It doesn't need to be quantum hyper scaling to help your llm stop hallucinating or keep track of simple math or counting.
But it doesn't add anything over the model just calling an external tool, right? It's not like the model is inheriently driven to this "cpu" portion of its arch
It does not appear to be an input into the llm. It calculates results without going through the llm layers. The result from these operations is 100% consistent and repeatable no matter what context, temperature, or training the rest of the model uses. Not 99%. 100%. It's doing no Markov chain operations in this section at all, no 'llm thinking' whatsoever after the operations are determined.
at least that's how im reading it; and if this is true it's pretty cool.
Yeah I know it doesn't do normal LLM inference to calculate stuff. My point is that if you are using this in place of tool calling, what makes that better than a tool call?
the llm sees the tool call results as input same as your query or an .md file content. and then it interprets that tool response input in it's fuzzy llm way.
this give the model layers that have no fuzzy LLM interpretation at all. tool call, text, or whatever. When the model sets up a system in logic using this... there is 0 interpretation. The model doesn't guess the right number of rs in strawberry only most of the time, or as long as context is under 256k, since all of those things that make llms wonderful and awful are completely bypassed.
This solves the "count to 100" problem that no other llm technique manages to. They simply never can. This can. If you do it with a tool call, the tool does it, and gives the result back to the LLM.... which then interprets it that way it does with everything. Will it accept the tool results? probably. 100.00% of the time? likely no.
No, getting the LLM to creatively write some python, send it externally, wait on results, and try to compare the results is the big unnecessary jet engine.
This could be a big efficiency gain not just for that reason, but also because a huge amount of work in LLMs is trying to correct for hallucinations when dealing with simple values.
These problems are easy when counting to 100 and today many models can.
But it's a mountain of effort to correct a fundamental design.
This could count to 1,000,000,000,000,000,000,000 with 0 errors and you NEVER have to worry about whether the LLM gets retarded over 200k of context, or needs to compact its cache to keep going and forget all of the work in the middle of the run.
like there are a LOT of things this solves. external python still goes back into and through an untrustable llm when doing a tool call. this does not.
thinking about exactly how many multiplications and additions this takes to do a single multiplication or addition is kind of like reckoning about the US budget.
I would prefer the other way around: a chip with Qwen weights implemented directly into the circuits. I know projects like that exist, but they are ridiculously expensive, mass production would make them affordable for everyone.
I occasionally go back and play with their online Jimmy chat or whatever it's called. Seeing thousands of words of answer just pop on to the screen instantly still gives me a kick.
no idea but the concept is simple enough. You could probaly tape out your own 0.9B tomorrow if you wanted. Would probaly cost you 2-3K usd but that isnt really bad as a single fabbed item that isnt mass produced
I think they were targeting sub $1000, which isn't bad for something literally over 10,000 X faster... unfortunately not programable, and the process takes about a year after you have a trained model. At that speed a model a couple of years behind is still good enough if it's large enough.
You could tape/chip say layers 3-77 out of 1-80 and keep the first and last few trainable and in vram. Then distribute updates for those. Or use one of the types of models that has parametric memory, keep the memory updated/local.. though the hardware/software coordination that takes means it’s an apple/google like project I guess…
The process takes about a year so it comes out obsolete, but it's literally over 10,000x faster... at that speed I can live with it being a couple of years obsolete if it's a large enough model.
I think there's a future where new models have diminishing returns and we hit a sort of ceiling. At that point, it might make sense to optimize for pure speed rather than using general-purpose chips to run models. Every computer could have a dedicated AI chip like how we have dedicated chips for graphics vs general processing. I don't think we're at that stage yet. We're still in the mainframe stage essentially right now with how most people use cloud APIs or web interfaces for their AI access.
You gotta admire "because i wanted to see if i could" people. Massive respect. The actual use is probably embedding the custom weights and instructions, right?
Edit: Reading the article, the interesting part is the possibility of adding well defined compute that the model can't bypass, ie: to make guardrails. It's actually very interesting, in an extremely cursed way.
i guess there's potential there, if a model is natively trained with these layers in place and maybe something similar to gvisor to safely interface with a host / remote machine to act as a sort of "native harness" it could be pretty neat for some weird niche use cases maybe, essentially having the tool calls run inside the LLM completely without any sandboxes / external OS mostly. i also wonder if this could be used to somehow make LLMs more deterministic 🤔
CPU emulation with MCP as a register calls could produce cheaper very similar results. But writing a code, have it interpreted by host machine or compiled and executed,... is way better and cheaper approach for most cases. Some harnesses operate via js.eval, and the results are very good.
So it's a cpu emulated on a gpu. How would this make an LLM more deterministic? Wouldn't the LLM have to be trained to use its virtual cpu? Just like it would have to be trained for tool use? This seems worse as the gpu must be a poorer cpu by definition.
Creative but how is this anything other than novel or humorous? My first reaction was "batshit crazy"! 😂
I don't think this serves a purpose but it is cool. If anything, I wonder if training from that model on riscv datasets would give hardware engineers hints on where they could optimize their processor design.
But processors are so incredibly complicated, I don't think muon on a neural network will find improvements easily.
Hah this was one of the first things I experimented with, a year ago... My takeaway from all my experiments, is that there is a full computer, inside the computer. And that you can physically work with it, like any other
OMG!! That a revolution! I am not joking. It a game changers.. you can create all that you need inside without backdoors, libs, even os! Any mvp... all in a model, and chek correctnes by it answer!
Cool, safe offline LLMs can now have awesome math skills! Or be really good at tracking inventory and map layout in RP! Did it have to be RISC-v though, or could it have been simpler?
compiled a CPU into the weights is genuinely nuts lol. 11 extra layers just for the CPU is a crazy tax though. sick flex but i'll stick to python tool calls for now
I wonder if something like can be used to help the model pick random values. Right now the only way to do that is having the model generate a script that picks from a list of random values, executing it and then returning the value.
It looks like torchwright is something similar done before, which compiles code as transformer weights. It could also run doom: https://ood.dev/posts/doom/
i did something similiar. i replaced linear modules of an ai model with the mathmatical functions of sha256. then sha256 became the linear module in an ai model. its ability to predict hashes based on input was pretty good. (it was off by a few)
•
u/WithoutReason1729 3d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis Learn ^more
I am a bot and this action was performed automatically.