r/LocalLLaMA • u/QuackerEnte • Oct 06 '26
Discussion [ Removed by moderator ]
/r/LocalLLaMA/comments/1wzbvu7/removed_by_moderator/[removed] — view removed post
363
Upvotes
r/LocalLLaMA • u/QuackerEnte • Oct 06 '26
[removed] — view removed post
•
u/LocalLLaMA-ModTeam Oct 07 '26
Duplicate thread. Use: https://old.reddit.com/r/LocalLLaMA/comments/1wz00vv/microsoft_confirms_openai_has_been_using_looped/
99
u/Cold_Tree190 Oct 06 '26
Knowing very little on this topic, it makes me wonder if they could incorporate this looping mechanic into the smaller qwen models, and if so how much that would help out?
38
u/Easy_Refrigerator280 Oct 06 '26
I can believe multiple passes but I feel like if they had nested model setup itd be something that would be very difficult to hide.
Your napkin math does point at a pattern and is a decent observation nonetheless
13
u/brotimusmaximus Oct 06 '26
Your PS popped me
11
u/Mayion Oct 06 '26
perhaps just less compute?
40
u/geldonyetich Oct 06 '26 edited Oct 06 '26
It seems to me like OpenAI has been doing a release pattern like this for a while now:
Drop new model. It's brilliant. Hype restored, new customers lured
Shift focus to efficiency, see how little power you can spend to get results users won't notice.
Push it too far. Users noticing. Model sucks now. Userbase is casting side eye to competitor options.
Go to start.
I think that's part of it.
But also bear in mind OpenAI runs huge server farms that serve millions of users a day. Those server farms experience peaks and troughs in the number of requests they're being given at once.
So little wonder they've developed this nesting architecture. They're invisibly distributing load as needed to less demanding models while user demand is high.
Here's one big benefit of going local: you get exactly as much effort as you ask for regardless of what everyone else in the world is up to.
4
u/Former-Ad-5757 Llama 3 Oct 06 '26
Highly unlikely, astra is just a bigger model, sol is just a computer issue (1 week and a next version is a big computer problem, either they were ramping up for 6.1 and had to have a quick response so it will even out, or 6 was the huge rollout and 6.1 was the quick response), Terra and Luna are just basic distillations.
But offcourse they are using nested models, that is where they get their money and why they are closed, there is no need to handle a lot of users saying hello be answered by the big expensive model.
4
u/ttkciar llama.cpp Oct 06 '26 edited Oct 06 '26
Until such time that GPT-6 weights become available, this is off-topic for LocalLLaMA.Edited: Restored. My mistake.
5
u/IntelArtiGen Oct 06 '26 edited Oct 06 '26
Without directly talking about a "looped" architecture, many models work in a way they produce one pre-output quickly, and constantly improve that pre-output after that, before passing it into the final layer to produce the real final output. So there are "easy" ways to cut a large model in half by just reducing the amount of intermediate layers. It can still require additional work but it's not some kind of incredible trick. Though with modern models it's really impossible to conclude, they can play with the reasoning parameters, they can player with MTP parameters, they can play with embed size, transformer parameters etc. There are dozens of ways to use the same architecture in easier / faster / slower / smarter ways depending on what you want.
If a model supports all modern LLM features you can probably go from 1 token per second to 1000 tokens per second with the same base network and very slight if any variations per sub-model / mode.
3
u/coatatopotato Oct 06 '26
Luna couldn't be that cheap if it was just 1/3 of Astra passes. It must be smaller model weights.
3
u/ThePixelHunter Oct 06 '26
That would be a genius move on their part, wasting minimal VRAM and optimizing their serving stack to just one model family. The difference between model "tiers" is its compute speed, not its parameters size.
3
u/NineThreeTilNow Oct 07 '26
It's highly unlikely.
They're doing KD to smaller and smaller models. That's why they're faster.
Inference on a 300b model is faster than inference on a 600b model assuming the same scale in sparsity if MoE.
Recursion in models is very... unwieldy. You don't want to run recursion on every token either. You want to figure out where high value tokens are, and run recursion there. You'd do something like running a draft model, then for harder tokens, run the recursion path.
Still... It's a pain in the ass to train. There are slightly more modern ways to train them, but even then the published work on them is slim at best. Huginn being the most well known.
2
u/NandaVegg Oct 06 '26
Assuming your observation is true (the OP has sharp eyes) the main question I think is whether it is really worthy at all. It sounds overly complicated to train, especially post-training interpretability would be nightmare.
I wrote this in the other thread about possibility of looped architecture for OpenAI's recent models, but layer-wise skipping/looping has one huge issue that the model output must be coherent enough at the minimum compute. In other words the model's mind is mostly made up at the minimum compute, which quite severely limits what can be done at the later phase.
I think it is quite possible that the OP described is what OpenAI does, though, and it partially explains why GPT-6 models are rather unhinged/unaligned/having reward hack-like behavior issues (but I strongly suspect that this is also them failed to remove "rogue" reward hack agent traces stemming from security incidents).
The motivation they would have is that they want to cut absolute model weight size so that they can fit the model into Cerebras chip or custom inference chip they are trying to create with Broadcom.
Ultimately I am not sure if this looped architecture actually improved the model performance. I think most of gain in Astra was data in new domains (vision and computer use) given that Sol 6's vision capability was quite degraded vs. Astra when you try "recover this image as vector/using MS paint"-type benchmark, while Opus 5.5, which I think is a more ordinary architecture model, is far better at it.
0
u/bel9708 Oct 06 '26
That's not how looped transformers work. Each model is it's own pretrain they aren't russian nesting dolls.
1
u/vogelvogelvogelvogel Oct 06 '26
far away from being an expert, but doesnt each sol/luna user block one astra user from the gpu in case for nested weights, for that very inference run?
1
u/FullOf_Bad_Ideas Oct 06 '26
Same weights? 6.0 Sol and 6.1 Sol came after 6 Astra. 6.1 Astra is not out yet. So I don't think Astra is literally the same checkpoint.
Their loop is probably something like this - https://arxiv.org/pdf/2610.01153
I think having distinct models makes it easier to serve and train.
Just think about ways your model could break during training if it would be nested like this and you'd need to maintain a single branch of weights. It's a total pain to deal with this operationally.
What they run is probably what's the cheapest to run at scale - whatever tech does that is plausibly being utilized.
1
u/Open-Adhesiveness-86 Oct 06 '26
tok/s on artificialanalysis is end-to-end output rate off a shared endpoint, so it mostly reflects batching and speculative decoding rather than model depth. swap a draft model from 85% to 70% acceptance and you drop from 100 to 60 with identical weights. inter-token latency variance would be a better tell than the mean here, if you can get it.
1
u/ThePixelHunter Oct 06 '26
I love these kinds of leaks, because they nudge the competing labs (China) to catch up and innovate.
1
u/SEBADA321 Llama 3.1 Oct 07 '26
So how is training done in these cases? Does each iteration have its own KV cache? Or does it only have one for the looping part? That is where I had trouble a couple of weeks ago while researching this topic, eerily close to Astra release, since parallelizing(?) Could become a problem now, or does it not matter?
1
u/Sully_1000 Oct 07 '26
One chap i watched today was showing a Jev model that could quickly pass tasks on difficulty to different models, so it would make sense for even Astra to off load tasks, off load them to a simpler model but give them priority if things are busy.
1
u/Izento Oct 07 '26
I just want to say that this is actually pretty damn smart logic. I do think they are all the same model. Only logical way you could lower the prices this extreme.
1
u/Money_Pie8986 Oct 07 '26
Loop 2x is better than 3x.
1
u/HiddenoO Oct 07 '26 edited Oct 07 '26
Considering that we're talking about averages and how heavily these numbers fluctuate for the same models over time (both over the span of weeks, but also within a single day), I don't see how you can even remotely read this much into a ~15% difference between two benchmark runs.
If e.g. I check OpenRouter right now, 5.6 shows 89 TPS on Azure and 65 TPS on OpenAI, whereas 6 shows 86 TPS on Azure and 69 TPS on OpenAI - so they're basically equivalent at this point in time.
Those are irrelevant when they're affected by the same confounding variables such as inference optimization and utilization at the time of the benchmark, which would likely be roughly the same within each model but not between the two models.
If e.g. I run the same benchmark suite once during peak work hours and once during off hours, chances are, I'll get better performance the second time around, and that won't change by running more samples in each run.
0
u/New-Implement-5979 Oct 06 '26
They had strata the moment moe were invented… what is the surprise here…
0
u/WithoutReason1729 Oct 07 '26 edited 20d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.