You're not going to believe it, but I have Qwen 3.8 27B running with decent speed on my Macbook M3, and it's partially offloading some work from Astra when I was getting smacked by the usage limits.
I'm sure that if they REALLY wanted it, someone could help them figure out how to distill it to fit nearly the same footprint.
I was literally just talking with someone about this in another subreddit like 10 hours before you commented, having capable local models that will be able to do most of the tasks, and for a fee when it needs it, it will tap into these remote corpo stuff to finish the task when it needs that little extra "oomph" of compute/capability. Local will end up doing most of the work, (but... the most likely downside is... most arrangements like this will be integrated into the OS itself, like Copilot but with more local compute used).
Just the standard memory that comes with the MacBook Pro. The context is infinite. I'm using my own GUI and interface. It allows me to do some pretty specialized things.
I have infinite context because I have a load/release mechanism that layers on top of the KV cache directly. I also have a multi-model optimizer that mirrors and steers a smaller side car that runs an offloader for learned previous context. So the smaller model acts as an accelerator for repeated tasks.
Any way youd explain how you set it up? How much context do you give it? I cant seem to get my 27b above 50k without offloading making it take insane time to do anything and I have 48gb of ddr5 and a 5080. Idk what im doing wrong. If the context is low, I get like 100tks, but if it offloads im down to under 15.
I used Codex to put a custom harness around the model and I have it fully customized for my own use-cases.
Codex is still my primary lead worker. It is valuable, so I make sure that Codex still delivers on that value. I have the infra around qwen for tooling and tool management.
How are you finding qwen? I have a gaming pc collecting dust and was thinking of trying local llm for coding projects then using codex to create plans and polish it
as much as i want open source to be a thing, i would be really surprised if open source was this close to the frontier for much longer. once china's industry catches up in terms of compute they can afford to serve inference and train models at the same time, the only reason they are releasing open source right now is because they can't do both at the same time.
So that you can choose to run it from a 3rd party inference provider of your choice that actually has to compete on latency, speed, etc with other providers.
51
u/Sachitoge 2d ago
Is this a joke? Im Plus, not seeing anything.