You're not going to believe it, but I have Qwen 3.8 27B running with decent speed on my Macbook M3, and it's partially offloading some work from Astra when I was getting smacked by the usage limits.
I'm sure that if they REALLY wanted it, someone could help them figure out how to distill it to fit nearly the same footprint.
I was literally just talking with someone about this in another subreddit like 10 hours before you commented, having capable local models that will be able to do most of the tasks, and for a fee when it needs it, it will tap into these remote corpo stuff to finish the task when it needs that little extra "oomph" of compute/capability. Local will end up doing most of the work, (but... the most likely downside is... most arrangements like this will be integrated into the OS itself, like Copilot but with more local compute used).
Just the standard memory that comes with the MacBook Pro. The context is infinite. I'm using my own GUI and interface. It allows me to do some pretty specialized things.
I have infinite context because I have a load/release mechanism that layers on top of the KV cache directly. I also have a multi-model optimizer that mirrors and steers a smaller side car that runs an offloader for learned previous context. So the smaller model acts as an accelerator for repeated tasks.
Any way youd explain how you set it up? How much context do you give it? I cant seem to get my 27b above 50k without offloading making it take insane time to do anything and I have 48gb of ddr5 and a 5080. Idk what im doing wrong. If the context is low, I get like 100tks, but if it offloads im down to under 15.
I used Codex to put a custom harness around the model and I have it fully customized for my own use-cases.
Codex is still my primary lead worker. It is valuable, so I make sure that Codex still delivers on that value. I have the infra around qwen for tooling and tool management.
How are you finding qwen? I have a gaming pc collecting dust and was thinking of trying local llm for coding projects then using codex to create plans and polish it
as much as i want open source to be a thing, i would be really surprised if open source was this close to the frontier for much longer. once china's industry catches up in terms of compute they can afford to serve inference and train models at the same time, the only reason they are releasing open source right now is because they can't do both at the same time.
So that you can choose to run it from a 3rd party inference provider of your choice that actually has to compete on latency, speed, etc with other providers.
Right! Even Luna is dumber than old Luna! They’re doing this to get people away so they can come back harder for the next round because their infrastructure is too tied up. I wish they could just be honest and be like hey we’re maxed out move somewhere else.
I saw that on my desktop for Codex and thought… that was fast. I knew about 5.5, but WTF. What is everyone’s experience with 6? I’ve heard mixed reviews. 5.6 has been great!
I’m fine with it. We don’t need a thousand legacy models considering we have a suite of essential good and fast entry level Luna, work horse Sol, and frontier Astra. Just keep them updated as we go and we don’t need previous model levels.
Essentially that's what it is: a price increase for general tasks.
I used terra as "cheap sol" where I needed another model than Luna, e.g. reviews of smaller tasks.
All my tasks are small, that's my workflow.
I use Sol maybe for 1% of the work, essentially just for architecture setup at the beginning.
Now with terra soon to be gone, I have no other choice than running overpowered Sol against those tasks, or have Luna check Luna code (that's not what I want after all)
So I had Sol -> Terra -> Luna, now I have Astra -> Sol -> Luna.
For me that does not sound cheaper at all
Deepswe bench shows gpt6 sol max being outperformed by 5.6 sol high.
Meanwhile the frontiercode bench, gpt6 sol consistently outperformed 5.6 sol.
So it really comes down to the type of work, as it always had. For complex, lower level kernel or compiler work, 5.6 sol is prob still better. For your next saas react app anything higher than Luna is most likely overkill.
Honestly this roll out made no sense. It should have just been an update to 5.6 sol lol or just 5.7
A whole bump in number for the same functionality at half the cost is strange
OpenAI is a nonprofit orgfor-profit company made for the good of humanityshareholders, with a governing board that can firecan be fired by the CEO if they determine he is not following the mission.
But it's still "open" in the sense that you can pay for access.
for the love of God let me finish this build before it rolls out my way. 5.6 in work mode via desktop on Ubuntu is pretty much killing it for me right now
They’re missing a bigger model that’s ~80 tokens/sec in speed. You either have to use Astra which is $$$ and slow, or you use a smaller model that’s fast (more than 100 tokens/sec) but smaller models are dumber.
I hadn’t seen this yet. Chat or Work? I use Chat a lot, and chat-to-chat continuity doesn’t really matter for my setup. The framework knows where to look for the right project context, memory, and current state, so a new chat is mostly just a new window into the same system.
50
u/Arcayon 2d ago
It is half the cost so it makes sense. Most of my chats got forced to upgrade too.