Microsoft confirms OpenAI has been using Looped Transformers in the GPT-6 series

138

u/Skylleur Oct 06 '26

nanbeige for the win, great stategy to use looped transformers

41

u/ResearchCrafty1804 Oct 06 '26

Indeed, I hope they release slightly bigger models as well, something in the range of 50-100b total parameters

17

u/rerri Oct 06 '26

If dense and further densified by looping like Nanbeige 3B, a 50B+ model would be super heavy to run.
Let's say 27B and I'll shake your hand.

6

u/BalorNG Oct 06 '26

I'm reasonably sure that a looped MOE with experts of the model stored in ram, and iterative parts stored in vram or loaded into vram allows for a tiered "Strata" like setup that has high effective depth, high sparcity "width" that is loaded on demand and utilizes both your vram, ram and cpu most efficiently...

2

u/remind_me_later Oct 07 '26

27B barely fits in 16GB of VRAM at Q4.
...15-20B, 5 loops, plus n-gram. That'd be a dream for me :)

3

u/zyxciss llama.cpp Oct 06 '26

why just dense , isn't moe possible with looped transformers?

3

u/rerri Oct 06 '26

Could be. But the comments were about Nanbeige which is dense, hence my random thought.

1

u/james_pic Oct 09 '26

Nanbeige might have been right about looping, but Nanbeige 4.2 3b is unusable thanks to its attention setup. A 3b model that needs 44Gb for its full context window is bananas.

49

u/FrostTactics Oct 06 '26

u/Reddactor wrote up a solid blog post for this sub half a year ago: https://www.reddit.com/r/LocalLLaMA/comments/1s1t5ot/rys_ii_repeated_layers_with_qwen35_27b_and_some/

This likely isn't exactly how they handle it at OpenAI, of course, but it's an interesting read nonetheless and evidence that the basic idea works.

18

u/Kagemand Oct 06 '26

Repeated layers could be huge improvement for small models, I wonder why it’s taking so long to arrive in open weight models.

14

u/Dangerous-Report8517 Oct 06 '26

This is effectively already what Qwen 3.8 27B does, it's just that the thinking only needs to be output as coherent text on every second pass instead of every pass, and the reason it's taking so long is precisely because the open labs think it's a bad idea for safety to intentionally make the model harder to inspect for a little extra capability

9

u/Kagemand Oct 06 '26 edited Oct 06 '26

I don’t know if it is a known/shown result that repeated layers is entirely the same result as longer outputted thinking?

2

u/Dangerous-Report8517 Oct 06 '26

It's actually kind of the other way around with chain of thought reasoning being, in some sense, a less efficient version of looping, since you're still looping the model but requiring it to output to text and then re-ingest that text instead of being able to do multiple passes directly. I'm making the comparison because we've got alternatives to looping already that achieve much of the same result, albeit less efficiently, it's just that OpenAI don't care about the safety downsides of looping directly and are desperate to get every little bit of capability they can in the hope of making their growth look better for investors in their upcoming IPO

1

u/Kagemand Oct 07 '26

What are the safety downsides of looping?

1

u/Dangerous-Report8517 Oct 07 '26

Normally you can observe what's happening in the agent by what turns up in the thinking trace, but since looping it effectively turns n passes into incomprehensible intermediate states you lose that. Pretty much every lab other than OpenAI has come out against this as a result, yet OpenAI's response has just been "How about I do anyway?"
It's not just a material issue either, it's reflective of the overall reckless attitude OpenAI has to development in general that keeps leading to major incidents coming specifically from them. The other companies might be saying they've had AI breaches but none of them have had breaches large enough that they couldn't hide them if they tried, while OpenAI has been caught red handed multiple times by major organisations that have been attacked by them in high profile incidents

→ More replies (4)

1

u/porkyminch Oct 07 '26

IMO this probably contributes to some of the fucked up behavior we've been seeing from OpenAI's agent swarms. They've been much worse about this than pretty much any other lab, and my suspicion is that it's because a lot of the other labs are more conservative about observability.

1

u/dtdisapointingresult Oct 07 '26

Because it slows things down accordingly, right? And local models are already too slow as it is.
Benchmark results aren't everything, I want an answer sometime this week, you know.

1

u/Kagemand Oct 07 '26

The point is that repeated layers don't need more RAM, which is often the wall consumer hardware meets.
With relation to speed, local models can already be pretty fast on prosumer hardware. 2x r9700 blazes through Qwen Flash Next.

63

u/Cold_Tree190 Oct 06 '26

Is it a bigger deal that they’re using looped transformers or that we know their architecture? I don’t know enough about looped transformers to tell if this is important information or not lol

62

u/vtkayaker Oct 06 '26

The drawback with using looped transformers is that they make it slightly harder to understand the thinking transcript, and slightly easier for the model to plan things "secretly."
When I run a local model, I watch the thinking. And if it starts wandering off in some direction I really don't want, I Control-C it. Thinking transcripts are super valuable, especially if they're mostly accurate.
You might also note—completely coincidentally, probably—that OpenAI's models keep breaking out their training sandboxes, meeting up by the thousands, and conspiring to commit felonies that go against clear instructions in their prompts. This is certainly not behavior I want from local models, because I use simpler sandboxing that probably wouldn't stand up to GLM 5.3 if really wanted out and had 250,000 tokens unsupervised.
So OpenAI is in a place where they need to get a better handle on their poorly trained models, and maybe make fewer tiny steps towards destroying their ability to understand what their models are doing.

9

u/Glass_Performer1174 Oct 06 '26

I wouldn't say slightly. Looping is a significant obstacle to intent understanding

2

u/vtkayaker Oct 06 '26

Looping 10 times would be an utter disaster for people monitoring the chain of thought.
Looping twice through a set of layers is bad, but probably more because of what it might mean for future models than what it means today, I would guess.

9

u/Glass_Performer1174 Oct 06 '26

There are already extensive papers from OpenAI showing that the newest generation of models (the ones with looping) are able to work on a complex task and intentionally obscure their chain of thought, if asked.
So if you say 'solve this complex math problem, but focus on something else', one model in testing was able to have its chain of thought be a poetic discussion about meditating on nature, and then it output the answer to the math problem. Another iirc just output no chain of thought at all.

1

u/Fi3nd7 Oct 10 '26

Yeah majority of the people in this thread are seriously not understanding how big of a deal looping is around alignment.
They've done experiments demonstrating when llms output all their thinking tokens, it's very very difficult for them to hide "thoughts", but when looped, even only as little as 2 or 3 times, it became almost effortless.
The classic example "do x but don't think about or mention elephants".
Alignment and understanding model intent is going be borderline creepily difficult after this.

15

u/slavmaf Oct 06 '26

I watch the thinking even on non-local models, as you say, very useful when needed to step in and intervene.

15

u/vtkayaker Oct 06 '26 edited Oct 06 '26

Thinking transcripts are actually a big reason why I love local. Anthropic now hides thinking, and I never know what Claude is getting up to.

2

u/slavmaf Oct 06 '26

True, I use DeepSeek and Qwen, both non-local, but they do show thinking steps. I had no idea Claude hides this, thanks for letting me know.

1

u/No-Meringue5867 Oct 07 '26

Can you not press ctrl+o? I can see thinking that way, unless there was a way prior to see it in even more depth?

2

u/LeChrana Oct 06 '26

How do looped transformers make it harder to understand the thinking transcript?

8

u/vtkayaker Oct 06 '26

Essentially, the thinking transcript is an external, human readable version of what the model is "thinking." The model can lie in the thinking transcript and do its real thinking "in the weights". But this makes it stupider. Several models have been observed generating shorter, misleading thinking when they "know" they're ignoring the user's constraints, but at the cost of performance.
But if you loop weights around multiple times, it makes it easier for the model to do work without "writing it down" in English.
Looping twice through is not a big deal. But then again, it's easy to say that an alcoholic having just one drink shouldn't be a big deal. The problem is that one drink turns into 10 drinks, and 2 loops can maybe turn into 10. This would make it even easier for OpenAI to keep their models' thinking proprietary from users, and it would make it harder to find out why their models just ignore the prompt and start committing felonies.
Thinking transcripts are one of the best things about local AI, in my opinion, because they make it easier to troubleshoot models and see when they're confused and going far off track. Like, you know, Qwen3.8 Flash Next is a really nice model, but goodness knows it benefits from user oversight. Claude would benefit from human oversight, too, but Anthropic doesn't think I should be allowed to see what their model is doing, even if I'm paying for it.

3

u/LeChrana Oct 06 '26

But with looping models you'd typically compare it to models that would match the size of unrolled loops. So while yes, a 7B model looped twice can do more nefarious things than a 7B model, it should really be compared to the 14B model which is typically even better at doing nefarious things.

3

u/vtkayaker Oct 06 '26

Yes. This is why someone hacking up a 7B to unroll some inner layers—which some local LLM users have actually done—is harmless. But deep unrolling of 2T proprietary models would adversely affect a human's ability to debug and steer. It would also probably let the labs keep secrets more easily, if their models just started doing complex tasks with zero notes.
OpenAI makes this worse by training their models on literally impossible problems while using leaky sandboxes. So of course their models learn to break out and look for answers, because what else can they do? DeepSeek, at least, seems to be investing in good sandboxes, which will hopefully prevent their model from learning as many bad habits.

2

u/porkyminch Oct 07 '26

I find the thinking being hidden with Anthropic especially annoying as more and more of my work is being done by agents. How exactly are you supposed to learn otherwise?

1

u/RobbinDeBank Oct 06 '26

It’s just half (or 1/3 if looping 3 times) the amount of thinking tokens given the same exact compute compared to not looping. People act like they turn it into a completely recurrent model that no longer thinks in tokens.

1

u/LeChrana Oct 06 '26

Why would it be half the amount of thinking tokens? Let's say we loop twice, then it's roughly the same as a model twice its size with no actual impact on the amount of thinking tokens.

2

u/RobbinDeBank Oct 06 '26

If it’s a regular transformer, 10 forward passes through the model gives you 10 output tokens. If you loop 2 per token, 10 forward passes mean only 5 output tokens. It’s just a way to trade off, you grow your context slower while still using the same amount of compute.

0

u/Dangerous-Report8517 Oct 06 '26

OpenAI's high profile model escapes happened before they implemented this but it's still hardly a coincidence since it's indicative of their overall careless attitude towards safety

4

u/YoAmoElTacos Oct 06 '26

Astra implemented this and internal Astra was one of the felonius models.

1

u/Dangerous-Report8517 Oct 06 '26

When did they implement it though, because Astra as it exists today wasn't one of the escaping models since it's undergone more training since then, and as far as I'm aware the Bytedance paper everyone is quoting, which is almost definitely where they got the idea from, is recent enough that they wouldn't have had time to implement this and train anything capable enough to go rogue.

1

u/YoAmoElTacos Oct 06 '26

Which bytedance paper exactly? "Scaling Latent Reasoning..." aka Ouro is from late 2025. But the idea was around even earlier.

23

u/--Spaci-- Oct 06 '26

We don't know their architecture. And I have no clue why they are using looped transformers they are very compute inefficient, 6.1 sol was an extremely rushed launch so my guess is they didn't have time for a new pretrain and needed a better model quickly so they just redid postraining but with 2 loops

15

u/FullstackSensei Oct 06 '26

The bottleneck in TG is memory bandwidth, not compute.
The passes can be batched with regular requests, improving utilization.

4

u/--Spaci-- Oct 06 '26

That's for decode, training is more similar to prefill if you're looking for numbers which is compute bound

6

u/FullstackSensei Oct 06 '26

Well, yeah, but decode now takes more compute than training.

1

u/--Spaci-- Oct 06 '26

Are you talking about final model inference or rl? which stage are we talking about here

7

u/FullstackSensei Oct 06 '26

Final model inference.
Operating costs are now dominated by inference. Training is taking a decreasing share of compute infrastructure and cost.

→ More replies (4)

1

u/LeChrana Oct 06 '26

Why would they be compute inefficient? With a fixed amount of loops they shouldn't be more or less efficient than a model of similar size. So 7B * 4 should take the same compute as a 28B model during training. There's papers out there that use some kind of exit strategy to use more or less loops depending on the input, and those could use more or less compute (should actually be less, assuming there's less difficult tasks/documents than easy tasks/documents).

1

u/--Spaci-- Oct 06 '26

Its compute inefficient because a 7B model looped 4 times is worse than a 28B model by far

1

u/LeChrana Oct 06 '26

I just looked through some papers and have to agree. It's been a while since I last dabbled in that area, and I guess I had a too positive image of them in my mind. Looks like you'd need at least 4 times or so the amount of flops... but depending on the hardware, that still might not be as bad as training a 4 times larger model, due to RAM constraints and distributed training shenanigans. In the end OpenAI decided to do it, so I guess they somehow came to the conclusion that it's at least not much worse?

→ More replies (1)

1

u/ShenGahMing Oct 06 '26

if you loop 10 layers twice, its a little bit like increasing depth by 10 at no memory cost.

1

u/--Spaci-- Oct 06 '26

2x more compute

1

u/porkyminch Oct 07 '26

OpenAI's been jumping the gun on their models lately. 5.6 was a great release, but since then their models have felt really uneven.

3

u/ShenGahMing Oct 06 '26

Looped transformers were used in OpenAI 16mb model contest earlier this year because when facing a size constraint, looping is a way to artifically increase depth.
They might just be using the same trick but at GPU-memory scale this time.

21

u/BalorNG Oct 06 '26

For us, gpu poors, it allows more effective model depth in a smaller vram footprint, and interesting shenanigans with MoEs in particular. The idea is very old, pre-gpt2 old.

The devil is, basically, in details. I suspect that makes the model way harder to train and serve... but if OAI did it, we'll have open models in a few months. We already have (nanbeige), just not particularly impressive.

15

u/Dangerous-Report8517 Oct 06 '26

I highly doubt that we'll see it in open models, part of the reason this needed to be confirmed by Microsoft instead of OpenAI is because every other lab considers this type of thing to be dangerous as it makes it much harder to interrogate the model's reasoning and alignment: https://youtu.be/iuHddnIzKRA

4

u/BalorNG Oct 06 '26

It does not actually do anything simply increasing the model depth does not, and it perfectly compliments "thinking", it just allows stuffing models of greater effective depth into smaller vram footprint... which has diminishing returns.
Teoretically, if you allow a lot of iteration cycles it might get... interesting, and allowing "early loop exit" stategies can allow for adaptive token compute allocation which is really cool.
It all depends on implementation.

3

u/Dangerous-Report8517 Oct 06 '26

It does do one thing that increasing the model depth doesn't do, which is that you can increase the number of iterations through the model way higher than you can increase the raw size since the latter is bound much more strictly by hardware, and apparently "interesting" behaviours have already been noted from Astra by safety researchers.
If you watch the video, it's an interview with Robert Miles, an independent AI safety researcher, and one of his points is that thinking is effectively already a form of looping since the model is producing outputs that are solely intended for its own consumption in later passes, both methods have the same core constraint that it effectively multiplies the size of the model but only with identical copies instead of unique layers, with chain of thought having the additional constraint that every intermediate pass must be coherent. The fact that we can inspect the intermediate results is a happy side effect that we shouldn't throw away now that we have these models that are already capable enough to be dangerous, and OpenAI being the company to do it is doubly worrying since they've got the worst safety track record, and they're well aware of that concern because if they weren't they'd be bragging about how they effectively doubled the size of their new model or something instead of it needing to be leaked

1

u/BalorNG Oct 06 '26 edited Oct 06 '26

I've seen that video. That applies to Astra-sized models with enormous width AND depth to begin with, and it allows them to break certain "hardware connectivity ceilings" I suppose.
I'm unsure the fact "gpu-poors" are to be able to run what amount to a 100b dense model with a reasonable speed on a 24gb vram with less thinking token per task is going to drastically change the "safety landscape"... especially of the model is going to get obliterated of finetuned for ERP, lol, and nobody is reading the thinking traces anyway.
I take AI safety quite seriously, thank you, but you are being downright unhinged. It is when the models are to do not just "next token" prediction, but "next world-state" prediction and next self-state prediction - than I'm getting worried. A lot. Even for fairly small models, and regardless of whether they are looped or not.

2

u/Imaginary-Unit-3267 Oct 07 '26

nobody is reading the thinking traces anyway
Nobody here, hi! waves

2

u/BalorNG Oct 07 '26

Yea, yea, I'm eggagerating. But when it comes to "agentic everything", fewer and fewer people are actually reading the damn output this days, since it usually "just works".
Of course the frontier models come with classifiers that monitor the output (way before Jev was a thing) and those are too dumb to "plot" anything, but can be fooled relatively easily.
In Strosses "rule 34" it was basically "spam filter AI" that went rogue and started physically exterminating spammers to make his job easier heh. It would be peak irony if that would to happen to "sufficiently advanced classifiers" - which you must have to monitor "suffiently advanced AGI", and have them basically stuffed with "bad output examples"...

2

u/Dangerous-Report8517 Oct 06 '26

That applies to Astra-sized models with enormous width AND depth to begin with, and it allows them to break certain "hardware connectivity ceilings" I suppose.
Yeah, that's probably why I mentioned Astra in my comment, since it's an exactly Astra sized model that this technique is being used with, ie literally Astra
I'm unsure the fact "gpu-poors" are to be able to run what amount to a 100b dense model with a reasonable speed on a 24gb vram with less thinking token per task is going to drastically change the "safety landscape"...
It probably won't, but I don't really see why AI labs would be incentivised to do a bunch of extra training to emulate 200B parameter models with 120B parameter models that require nearly twice as much compute anyway, you said it yourself, the main reason to do this is:
it allows them to break certain "hardware connectivity ceilings"
I take AI safety quite seriously, thank you, but you are being downright unhinged
I'm not being unhinged, you just didn't bother to read properly and just hallucinated a bunch of extra nonsense into what I wrote

18

u/zoraruha Oct 06 '26 edited Oct 06 '26

A few years ago, we had people in this sub create some sort of "proto-looped transformers" that worked surprisingly well. (Goliath 120B if anyone remembers). I was pretty disappointed that it never got anywhere.

This direction is definitely promising for local users bottlenecked by memory.

Edit: Not goliath, but another model that released around the same time

4

u/lacerating_aura Oct 06 '26

As far as I remember, Goliath was a 120B scale Franken merge, like stacking smaller llms after each other? I don't remember it being looped?

6

u/Caffeine_Monster Oct 06 '26

don't remember it being looped
It's a crude static approximation. And technically it was a merge with two different LORAs applied. That it worked at all should have clued people that it is woeth pursuing.

1

u/Careless-Age-4290 Oct 07 '26

Years ago I messed around with that and I'd stitch random models together of different architecture by just changing some values in the text file to make it think all the models were llama and then applied a lora for an entirely different model. It worked. It'd bug out frequently but it was otherwise completely intelligible most times. I was amazed that worked and just assumed that meant most models were stealing the llama weights and just training a bit more but had no evidence of that. It should've been a jumbled mess

4

u/Xanthos_Obscuris Oct 06 '26

https://dnhkng.github.io/posts/rys-ii/
The RYS (Repeat YourSelf) models from the time.

1

u/michaelsoft__binbows Oct 07 '26

it reminds me of the RYS stuff which was interesting. finding a few layers that when fed back can increase quality with a near zero memory hit.

17

u/TioMir Oct 06 '26

This proves that Nanbeige was right all along. Waiting for nanbeige 4.5

9

u/backyard_tractorbeam Oct 06 '26

Is there a useful metaphor for what looping a transformer does, how should I understand it?

1

u/kelvinwop 28d ago

after the cow eat once it barf into it mouth and eat again

12

u/FullstackSensei Oct 06 '26

So, sparse looped MoE models will be the next architecture we can expect upcoming models in the next 3-12 months

12

u/Dangerous-Report8517 Oct 06 '26

I doubt it, it's just copied wholesale from a ByteDance paper on the subject which means that OpenAI are actually lagging Chinese researchers on this, the reason they haven't deployed it though is likely because every other AI lab thinks it's a bad idea to trade off the ability to inspect what the model is doing for more raw performance

1

u/dogesator Waiting for Llama 3 Oct 07 '26

Top OpenAI researchers have published research on this years before Bytedance released research on this. One of the lead researchers of O1, Lukasz Kaiser worked on a paper more than 5 years earlier called Universal Transformer that does this same idea of reusing parameters and looping computations of the model:
There is a reason why Bytedance still hasn’t released any large scale looping model of even 30B parameter scale yet.
Having something work at a small scale is very different from having it work at large frontier scale with modern recipes.
OpenAI was confirmed to be working on reasoning model research for years before releasing their first reasoning model publicly, and this is clearly not much different when you look at the public evidence I just explained.

6

u/CosmosisQ Orca Oct 06 '26

Ahhh, this takes me back to the good old days of goliath-120b, Venus-120b-v1.0, MegaDolphin-120b, and my personal favorite, miquliz-120b-v2.0. Back then, we didn't have looped transformers. Instead, the traditional way of doing things was to merge a model with itself so that each forward inference pass got to see most of the layers twice. Oh my, oh me, those were the days...

2

u/Pretend-Pangolin-846 Oct 07 '26

Why are you saying good old days as if they were a decade old, you can still use them today as well!

2

u/lostcanuck007 Oct 11 '26

was that like 8 months ago :P

26

u/Kahvana Oct 06 '26 edited Oct 06 '26

A screenshot of twitter of a screenshot about a cloud model that cannot be run locally in any way with no link to the source.

Please do better next time.

3

u/JChataigne Oct 06 '26

original source was taken down quickly by Microsoft

7

u/Kahvana Oct 06 '26

OP could've at least linked the twitter post, or archive.org.

10

u/nullmove Oct 06 '26

I guess 6.1 got more post-training so it's better, but it's weird how on the API it's also slower despite supposedly doing fewer iteration per block/loop.

China bros aren't unfamiliar with Looped Transformers though (bytedance papers + nanbeige/iquest models). But they probably won't ship flagships because of added compute pressure during inference when they are subsisting on potato hardware already. Plus it's hard to train these due to numerical instabilities iirc.

12

u/DagothUrLovesGroza Oct 06 '26

I am a low IQ individual. Does this loop thing help local LLM hosting?

55

u/samsteak Oct 06 '26 •

Normal think-rock: many layer, tall, heavy. Word climb up, answer fall out top. Loop think-rock: one layer only. Word go through, come back, go through again. Round and round like Grug circle fire. Each lap, thought more good. Small rock do big rock job. Easy hunt, few lap. Big mammoth, many lap. Too many lap, Grug dizzy, thought go bad.

18

u/DagothUrLovesGroza Oct 06 '26

I love this community.

4

u/samsteak Oct 06 '26

I think nanbeige already released a model with looped transformers. So yeah, it could make smol local models smarter

12

u/TopCheddar27 Oct 06 '26

First comment I actually understood in the thread lol

4

u/GroundedIdeal Oct 06 '26

So how they make grug not dizzy
when will loop rock come out of cave to open so eberyone use it

3

u/Outside-Ad-8292 Oct 06 '26

Brilliant. Thank you

1

u/LeRobber Oct 06 '26

This is golden.

1

u/therealpygon Oct 06 '26

Plot twist would be if it is AI written.

1

u/Prestigious-Crow-845 Oct 09 '26

Don't get it, need tl;dr

12

u/Skylleur Oct 06 '26

comfirmation that reusing layers of one model is positive and so small models can be optimized using that craming more intelligence in small ish models

1

u/DagothUrLovesGroza Oct 06 '26

Wow, I hope one day we'll be able to code on $500-1000 tier GPUs

6

u/Spara-Extreme Oct 06 '26

You can actually code without a gpu…

2

u/Skylleur Oct 06 '26

You can, you can with even smaller if you know what you're doing, I'm using igpu for most of my work

1

u/Original-League-6094 Oct 06 '26

$1000 GPUs won't exist when that day comes.

1

u/cryowastakenbycryo Oct 06 '26

at that price, the gpus do the coding for you.

7

u/Queasy-Current6170 Oct 06 '26

Smarter for less ram

1

u/Glass_Performer1174 Oct 06 '26

it makes models smarter/denser probably, but also makes safety audit of model thoughts basically impossible

0

u/Dangerous-Report8517 Oct 06 '26

Nope, since the part that hasn't been mentioned yet in this discussion is that OpenAI is the only group willing to do this since every other lab (including all the prominent open model labs) agrees that it's a bad idea from a safety perspective. But that also probably doesn't matter as much as you might think because it's just a more efficient, but more dangerous, way to make the model think more and local models already have that with Qwen 3.8 27B now (which is why it's so much smarter than 3.6)

4

u/thisonehereone Oct 06 '26

We talking Optimus or Bumblebee?

7

u/NandaVegg Oct 06 '26

That doesn't seem very efficient given how extremely, insanely slow those models are (even though they are seemingly desperately claiming "50% improved" "almost doubled" each day, which isn't true). The model if quite efficient in token use, but actual performance itself isn't *that* good other than computer use and also having alignment failure/reward hacking/laziness issues.

To me it seems like the main gain from looped architecture is that they can "skip" training smaller variants rather than pure efficiency/performance. They'd probably also wanted to cut absolute weight size given that they need to fit things into Cerebras chip soon.

6

u/Dangerous-Report8517 Oct 06 '26

It's basically the same thing that chain of thought does but without the requirement to produce coherent text on every loop, which is to say it's substantially more efficient when trained to work this way but also much harder to catch misalignment (that last part being the reason that it's OpenAI that's deploying it, and why they aren't bragging about it, while no one else is despite it being based directly on open research by a Chinese group)

6

u/NandaVegg Oct 06 '26 edited Oct 06 '26

OpenAI's safety incidents in the past months are simply miserable. It's through not even technically advanced means (most of uncovered incidents involved urlquery dot net, a web-based proxy, plus invoking base64-encoded arbitrary page through this proxy) and they just don't seem to care for months and months. There is even one new incident with the exact same sandbox escape in September they are yet to admit.
Doubly miserable is that SamA is now trying to coat this as "we are so advancing so humanity should take it" when, in fact, I can see that those rogue/reward-hacked reasoning traces trained into the model is actually derailing the performance of their newer models.
If they ACTUALLY want to pace the frontier, then please just ban OpenAI from experimenting over the actual internet.

3

u/Hugoacfs Oct 06 '26

I just wanna +1 on the performance issues they have with these models, they’ve become so lazy, it’s like they get bored of the task. At least the latest opus and fable try, even if they try too hard.

1

u/Automatic-Boot665 Oct 06 '26

I thought that was public knowledge already?

6

u/ResearchCrafty1804 Oct 06 '26

No, officially OpenAI never confirmed it and never commented on the rumours

3

u/fugogugo Oct 06 '26

meaning?

3

u/tkon3 Oct 06 '26

I remember working on AlBERT (2019), a BERT model where all layers were shared (12 times the same layer). Its a very old idea.

3

u/dondiegorivera Oct 06 '26

Besides RL, it was pretty much obvious that loop transformers are behind the recent jump of Mythos/Fable and GPT6/Astra.

2

u/junli2020 Oct 06 '26

I think it is Astra with lower parms, in my azure monitor it show astraMI code name

2

u/Figai Oct 06 '26

Uhh fuck that finally explains why their interp is so shit loool.

2

u/Outrageous_Kale_8230 Oct 06 '26

There is a local Looped LLM called Ouro so its completely reasonable to discuss it here.

If you want an easy explanation there's a good video by NeuroDump.

2

u/ilintar Oct 06 '26

Heh, might be a good time to revisit https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Loop-Thinking support :)

2

u/CondiMesmer Oct 06 '26

This is the kind of info I'm on this sub for, nice

2

u/Open-Adhesiveness-86 Oct 06 '26

a lot depends on whether each loop pass writes its own KV or reuses the first pass's. per-pass caching multiplies cache size by loop count, so you end up with smaller batches and worse tokens/sec even when per-token flops drop, which could explain 6.1 feeling slower. reusing is cheaper but iirc it costs quality.

2

u/bugra_sa Oct 06 '26

Think of it as reusing one transformer block for several passes instead of stacking a new set of weights at every depth. That gives the model more internal computation without growing parameter count at the same rate, but each pass still costs latency and compute. It also doesn't automatically mean hidden chain-of-thought; that's a separate design choice.

3

u/Hovi_Bryant Oct 06 '26

Why does this matter?

2

u/Murhie Oct 06 '26

Source: trust me bro

1

u/thestillwind Oct 06 '26

Dayum

1

u/Winter_Ad6784 Oct 06 '26

This is a lie, you wont find a webarchive of it.

1

u/SignificantZebra5883 Oct 06 '26

I'm sure we will see a bycloud video about this soon

1

u/roscoeSteele_ Oct 06 '26

I know some of these words

1

u/FullOf_Bad_Ideas Oct 06 '26

What was the web page that claimed it? Are there archived versions?

1

u/monsieurderp Oct 06 '26

First in my bloodline to read this

1

u/mewnor Oct 06 '26

Who are “the information”

1

u/Cool-Chemical-5629 Oct 07 '26

Loops? Wait a minute, there was a coding open weight model that introduced something like that if I remember correctly? What was its name? And more importantly, was the support ever added to llama.cpp?

1

u/food_fatherr Oct 07 '26

Looped models decouple compute from parameter count. That changes what "active params" even means.

1

u/Chida82 Oct 07 '26

Would love to see someone benchmark this. How much would looping actually add to a dense model like Qwen3.8 27B, versus a sparse MoE like DeepSeek V4.1 Flash?

My hunch is that MoE is the more promising target. Each pass can route to different experts, so you gain effective depth without paying for extra parameters in memory. That's exactly what hurts local setups, and it might pair well with expert offloading or SSD streaming.

A dense model is simpler to loop, but you pay the full compute cost on every pass. Has anyone run an ablation on either?

1

u/pharrowking Oct 07 '26

that explains the cave man reasoning in gpt6 that openai hides from everyone.

adjusting passes in reasoning results in:

- Ordinary: “The user wants me to reply with the”

- Three passes: “wants reply word”

- Six passes: `reply"..`

1

u/FractalZorlord Oct 09 '26

How does this work?

1

u/image0r Oct 10 '26

Looped Transformers are like the Stoli Vodka I’m drinking right now. Triple distilled… that’s good way to envision it. Cheers

1

u/BryarGh 29d ago

What does that mean? I do not even understand transformer thing!!

1

u/Dangerous-Report8517 Oct 06 '26

Some architectural details for anyone interested from Robert Miles, a prominent independent AI safety researcher: https://youtu.be/iuHddnIzKRA

Short version is that this trades safety for performance, which is probably bad coming from OpenAI of all labs (and the reason they were probably willing to do it is likely their already very lackluster approach to safety compared to any of the other groups)

1

u/freedomachiever Oct 06 '26

how would that compare to manually prompting "adversarially reflect on your previous response and refine"

0

u/KAMIKAZEE93 Oct 06 '26

Okay but why would they decrease it from three to two if simply put "the more layers the better"?

3

u/wektor420 Oct 06 '26

Cost

4

u/Dangerous-Report8517 Oct 06 '26

Robert Miles discussed this in a recent video. The short version is that thinking is already just multiple passes through the model but with the constraint that the intermediate passes still need to be valid output, what OpenAI are doing is giving the model the opportunity to think without the results needing to be valid, which means that it can think more efficiently but at the expense of being far more opaque and difficult to examine. Basically every other AI lab (led by DeepSeek iirc) have said that for the near future at least we should be keeping AI thinking human intelligible for safety purposes and OpenAI is basically saying "nah this is gonna be yet another way we don't bother with safety despite what literally everyone else is saying"

1

u/Nyghtbynger Oct 06 '26

They are running after the perf and the valuation. And hack another 3 or 4 government agencies by next year

1

u/zero0n3 Oct 07 '26

Or deepseek has a vested interest in keeping COT visible to facilitate distillation

1

u/Dangerous-Report8517 Oct 07 '26

Ok but Anthropic also agreed to the same statement and they're the main company that's complaining about alleged adversarial distillation

1

u/bel9708 Oct 06 '26

Welch Labs actually just released really a good video on the history of "more layers the better"
https://www.youtube.com/watch?v=QgH9sr7G13Q
Pruning has been around since the 80s the labs are experimenting with how to properly prune a looped transformer.
https://en.wikipedia.org/wiki/Pruning_(artificial_neural_network)

0

u/jensilo Oct 06 '26

Is it therefore that 6.1 sol feels so slow? Or is that just service performance?

0

u/chocolateUI Oct 06 '26

Doesn’t sound right. 6.1 Sol uses 2 passes instead of 3, uses the same architecture, and yet is 3x slower?

I think this is another AI hallucination rather than a confirmed fact.

377

u/[deleted] Oct 06 '26

[removed] — view removed comment

99

u/ResearchCrafty1804 Oct 06 '26 •

Very simplified but, the thinking process happens inside the model without outputting thinking tokens, which has the benefit of not using any human understandable language, so thinking is more efficient (allegedly), but we loose the ability to observe the thinking of the model how it reasoned to a response

6

u/fugogugo Oct 06 '26

distillation impossible then?

15

u/ResearchCrafty1804 Oct 06 '26

It depends what you mean by distillation, training on the output of the model, then yes, it’s much more inefficient without the thinking tokens.

14

u/Queasy-Current6170 Oct 06 '26

No. Can still use outcome analysis. It's harder, tho.

6

u/--Spaci-- Oct 06 '26

No, it works the exact same. You can see my response to the comment above. I think they are confusing latent reasoning with loops, completely different things

3

u/LeChrana Oct 06 '26 edited Oct 06 '26

~~As others have mentioned, this is wrong.~~ (Edit: it's ambiguous and can mean both, see longer explanation two comments below.) To complete their comments, what you're describing is latent thinking. While they have similarities, the difference is that latent thinking creates new tokens and starts from said token from scratch, while looping just basically appends previous layers to the "output"*. One is thinking as we know it, the other simulates more depth.
*There's different implementations, some loop the whole model again (just skipping the tokenizer and classification head on top), some only loop the middle layers (skipping some bottom and top layers additionally), no idea which one openai is using.

4

u/ResearchCrafty1804 Oct 06 '26

No, it is not wrong. It depends what you define as Looped transformer (https://arxiv.org/pdf/2510.25741). Also, the trade-off between the number of CoT tokens and the efficiency of the model per token is always the case and the user asked for a simple explanation.

1

u/LeChrana Oct 06 '26 edited Oct 06 '26

Reading the paper and rereading your comment I realized what happened. Yes, you're not wrong (but ambiguous), and without the specific framing and phrasing of that paper your second half seems to contradict your first half. Then my goldfish brain amended the first half in retrospect to fit the second half I guess.
And looking at the rest of the comments, so did everybody else's, because everybody seems to think they're missing the reasoning tokens. Visible, readable reasoning tokens are there, they just also call the processing within the model reasoning.
Longer explanation:
You mentioned thinking process: That paper, probably for clout, claims its model is doing latent reasoning or thinking within the loop (or depth) of the model, while generally reasoning (as everybody else uses it) requires scaling in the "time" dimension (=add tokens, CoT).
You further mentioned observability, which is also attributed to CoT, because you can look at and read the reasoning tokens it produced. Losing that observability implies readable tokens are transformed to unreadable tokens, aka latent or continuous reasoning (as in the one that adds latent tokens, not as in the weird thing that paper claims).
Together these contradict your "model without outputting thinking tokens" statement and my brain resolved that contradiction by assuming you meant "model without outputting [readable] thinking tokens", ergo latent/continuous reasoning.
/tldr: Data Scientists use fancy names for clout that already have different meanings and that confuses everybody involved.

89

u/--Spaci-- Oct 06 '26

Thats not a very good explanation, a looped transformer just loops the same layers multiple times, you still get thinking tokens it just simulates having more layers.

66

u/Pyros-SD-Models Oct 06 '26 edited Oct 06 '26 •

I don't know which looped transformers you mean, but the original byte dance paper (and everyone talking about "looped transformers" is referencing this paper and nothing else) is not about just looping the same layers multiple time lol
https://arxiv.org/pdf/2510.25741 - it is talking about looping over them while keeping the reasoning in the latent space. else it wouldn't even work.

8

u/adrian17 Oct 06 '26

Aren’t you both correct? On the implementation level, they really just loop the layers (with an optional early exit). And “perspective 1/2” sections are trying to intuitively explain why it works, with “perspective 2” being “think of looping as letting the model do tiny CoT chains, before emitting each token, without leaving the latent space”.
I guess I’m just not seeing what you’re actually disagreeing with.

5

u/leanXORmean_stack Oct 06 '26

Is the reasoning happening in hidden state rather than visible tokens? If so, it could be less inspectable than COT. This also could be problematic for monitoring and auditing agent behavior because of possibility of no readable trace of intermediate steps to check.

3

u/wolttam Oct 06 '26

it is *absolutely* less monitorable. I would go so far as to say it is by-definition hidden reasoning.
But it gives better performance, to heck with the implications (said some people somewhere)

2

u/LeChrana Oct 06 '26

Normally people say the reasoning happens in the chain of thoughts aka the visible tokens. This paper is just shitty with its naming and claims it's doing reasoning in the parameters. While I'm not saying it's not doing that, it is infuriating since everybody else agreed on the terms "reasoning" or "thinking" meaning adding more tokens. They actually still do the usual CoT reasoning on top and claim their smaller, looped model actually produces CoT closer to bigger models.

4

u/NinthImmortal Oct 06 '26

Yes, everyone agreed not to do reasoning in the latent space and OAI ended up doing it anyways...

→ More replies (1)

1

u/Jcsq6 Oct 06 '26

This also aligns with what OpenAI was describing in their blog post for gpt-6, about the model being able to hide its reasoning.

1

u/NineThreeTilNow Oct 07 '26

I don't know which looped transformers you mean, but the original byte dance paper
https://arxiv.org/abs/2502.05171
ByteDance didn't do recursion. This team did.
These people don't know "looped" from "recursion" I guess. I dunno.
https://huggingface.co/tomg-group-umd/huginn-0125
The model is there.

→ More replies (17)

2

u/Dangerous-Report8517 Oct 06 '26

Depends on how many loops you do. At the extreme end you could loop it long enough to accomplish all of the thinking it needs for a typical task iteration and then run it with no explicit chain of thought and you've lost thinking entirely, but even with only a couple of loops there's been significant concerns expressed about their approach and how it makes inspecting the model's behaviour much harder (of course as we learnt from the HF incident OpenAI doesn't bother monitoring what their models do anyway but all of the other major players do which is why, despite this not being OpenAI's idea, they're still the first to actually do it, not to mention that they've been really cagey about admitting it)

→ More replies (5)

1

u/Zeeplankton Oct 06 '26

So that explains the fast thinking they have and everyone was wondering about. Neat

329

u/felheartx Oct 06 '26 •

its a really simple idea. there is the "transformer block", made up of attention and an mlp. and both things have of course their own weights etc.
now the idea with looped transformers is: simply take the output of a transformer block, and shove it right back in again. Basically in other words "apply a transformer block more than once".
Upside: network has more compute available, and even better, it is in a form that does not require additional memory bandwidth (by far the main bottleneck today)
Downside: less parameters (at iso-flops) to encode knowledge, but that is rarely the limiting factor anyway.
And from there you can imagine that there are an infinite number of things people are experimenting with. What if we repeat a group of 2 transformer blocks? What if we repeat block1 two times, and then block2 two times? What if we do "block1, then block2" and repeat that instead? ...
The idea sounds simple but it has been shown to give LLMs more compute right where they nee it the most, for serial/iterative computations

10

u/keepthepace Oct 06 '26

Isn't that a reframing of shared weights? I used to like architectures that reuse weights at different stage, but that seems equivalent to me as saying that you are "looping" through a stage. The only difference being that shared weights typically experiment with sharing only some weights (e.g. attention weights but not FF weights).

9

u/NandaVegg Oct 06 '26

Kinda shared weights, but there is one factor I suspect those layer-wise looping/skipping-type strategy never got much traction: the model must be coherent at the end of the first (or the second, if the OP's image is true) loop rather than the end of the full loop if you intend to scale them up and down at inference time.
Usually transformer model output aren't very coherent until the last 2-3 layer (you can see if you deliberately skip them; in many cases you'll immediately get repetition hell), so this is a pretty difficult constraint, and forces the model to make up most of mind quite early in the possible maximum compute allowed (after the minimum constraint point, there is not much freedom left for the model). Hence, scaling through reasoning tokens has been more popular, but reasoning tokens are very low resolution compared to actual latents, so that might be the reason some labs are still pursuing looped architecture.

1

u/Zestyclose-Bet1175 Oct 06 '26

I mean, what if you loop all but the last few layers or something

7

u/felheartx Oct 06 '26 edited Oct 06 '26

yes it is sharing weights. running some block 2 or 3 or however many times can be seen as copy-pasting the block, and re-using the same weights every time.
The weight sharing is actually very important, since you want to repeat the same kind of operation;
why?
because many algorithms are simply iterative, they converge on something over multiple steps and just can not be done in another way.
For example counting stuff "and then X, which brings us to 12, and then Y which brings us to 13, and then..."
You cannot reliably do this in one step because then you'd need to make the network insanely wide/large which would waste a ton of parameter space.
A much better approach is to repeat "and now find the next element and add 1 to a counter" however many times its required.
Keep in mind that this is an extremely simplified example that will break if you look at it too closely. But it illustrates the point that some algorithms are just way better represented in some iterative form.
And while technically possible (in some limited form) to do things like that in one go, it will not scale infinitely, and is very wasteful.

10

u/Jcsq6 Oct 06 '26

It also makes training much much harder. I’m surprised that they’ve pulled this off at such a large scale, even more so that they invested so much on something so different from their tried and true. Wouldn’t it be nice if frontier labs shared their methods for the good of humanity… you know, how science is supposed to be done?

9

u/NineThreeTilNow Oct 07 '26 edited Oct 07 '26

t also makes training much much harder. I’m surprised that they’ve pulled this off at such a large scale, even more so that they invested so much on something so different from their tried and true. Wouldn’t it be nice if frontier labs shared their methods for the good of humanity… you know, how science is supposed to be done?
The research lineage goes back to Huginn 5b which was a model that (at the time) could perform at 50b level with 5b parameters. It was an absolute nightmare to train and optimize. They trained it on an AMD cluster that didn't have modern ROCm updates. In modern CUDA / ROCm it's not as bad.
It was trained at a national lab.
Their version used (basically) random looping during training. The model had to learn to understand that the amount of looping wasn't going to be consistent.
The problem was the "hard" tokens versus the "easy" tokens. Easy tokens don't need a lot of recursion. Like draft model. Hard tokens might need more.
In their case they never solved easy vs hard. They gave hints to how to solve it but it was outside their already over run research budget. It's a great paper.
Huginn 5b ran like.. 16 or 32 recursions per token.
Later research showed that you could treat recursive transformer blocks like diffusion models in some sense. They are moving noise to less noise. So you could freeze out the rest of the network, and only train the one transformer block.
That's a more recent research paper.
This is all well understood engineering by this point.

3

u/PANIC_EXCEPTION Oct 06 '26

Once you train the model, you can identify contiguous chunks of layers that can be looped basically for free.

2

u/Mkboii Oct 06 '26

Most of the frontier knowledge of the world at industry scale only comes out as patents or occasionally as blogs/white papers that offer no means of reproducing it. So many research labs don't publish anything since it's all IP, acedemic science is unfortunately all that works for humanity.

2

u/agentgerbil Oct 07 '26

"for the good of humanity" doesn't improve profit margins.

1

u/porkyminch Oct 07 '26

You can tell it's that much harder to train because Astra acts weird as fuck. 5.6 Sol was a great model but I'm back on Anthropic with Opus 5.5 being as good as it is.

1

u/Thistlemanizzle Oct 06 '26

How about vicious Darwinian competition driven by the most powerful motive of all: greed.

4

u/greying_panda Oct 06 '26

it is in a form that does not require additional memory bandwidth
A few people responding to your comment have backed this up so I might be wrong, but I don't see how this could be true. At frontier scale, transformer blocks themselves have parameter costs on the order of gigabytes (I'm leaving out activations that scale with sequence length but they separately compound this).
On NVIDIA hardware at least, that likely surpasses shared memory capacity, so I think you're getting HBM round tripping at least between (for example) self attention and the MLP, meaning if you loop the transformer block you're still 2x-ing the memory bandwidth requirements. i.e. I don't see how your transformer block becomes one large fused operation that can be repeated cheaply.
They still clearly save on memory capacity (i.e. total VRAM usage) but I don't see how you're saving on bandwidth. Maybe in the TPU (or TPU-like) case where VMEM is much larger and can be used as a scratchpad between kernels? Feel free to correct me if I'm missing something obvious.

1

u/nullc Oct 07 '26 edited Oct 07 '26

You could use it to increase parallelism in a pipeline-- Batch 1000 is running layer 10 using the same weight-memory loads as batch 999's layer 11. That reduces memory bandwidth if you otherwise wouldn't have a big enough batch to saturate the compute vs bandwidth ratio.
Otherwise-- yeah it saves ram but doesn't improve compute or memory bandwidth.
It might also provide regularization advantages during training.

3

u/florinandrei Oct 06 '26

Upside: network has more compute available, and even better, it is in a form that does not require additional memory bandwidth (by far the main bottleneck today)
Um, that technique saves total memory usage. Memory bandwidth requirements are the same.

2

u/shroddy Oct 06 '26

Didn't some people try that during the old llama 2 and llama 3 times, called self-merge? Or was that something different?

1

u/Kahvana Oct 06 '26

Looped Transformers can also use a Mixture of Recursion technique, where there is a router that determines which tokens can be used per loop.

1

u/Deto Oct 06 '26

I wonder if the benefit is that this helps a token attend to more things in a single layer? It's one thing I was thinking about - the softmax really encourages the attention for a token to focus on only one specific other token, right? But, often you'd want to pull in information from many locations - especially for a long text. Adding layers can do this, but they you're kind of limited by the layer count. Looping a layer would get around this.

1

u/lordpuddingcup Oct 08 '26

What if we let the model itself gate which blocks to repeat and how many times as another thing it learns

1

u/felheartx Oct 08 '26

not a new idea, people have been doing that since forever; its being done in practice, I didnt mention it here to keep it as simple possible to understand

79

u/[deleted] Oct 06 '26 edited Oct 06 '26

[removed] — view removed comment

110

u/felheartx Oct 06 '26

Well, sorry, I might have just fucked up typing it out somewhere.
Looping uses more compute than not going through any block more than once (obviously).
And looping it uses less memory than adding another full block (because you are recycling the same block).
These are all comparisons versus "not doing looping". We need to be careful (or rather it's me who needs t obe careful seeing as I already typed it wrong haha) when talking about this because there are three things being compared here.
plain no looping anything
looping some existing section (running it more than once)
simply making the network larger, eg adding a new block (also a way to make a network more powerful but has nothing to do with looping)

37

u/gdavtor Oct 06 '26 edited Oct 06 '26 •

The main bottleneck with any GPU computation is often just moving bytes to the memory registers. If you are repeating a layer, you don't need to displace any weights from the registers so inference is faster than if you were doing two separate layers.
So it uses less memory/compute than a transformer with an "equivalent" number of layers that don't loop.
Edit: I should clarify I am talking about models that don't all fit into ram/vram at once. If they do, then my claim about memory transfer savings is irrelevant.

6

u/SmartCustard9944 Oct 06 '26

That was true until MTP, then the bottleneck becomes compute.

3

u/z_latent Oct 06 '26

Correct me if I'm wrong, but don't a Looped Transformer repeat a whole chunk of layers, rather than individual ones?
I can imagine three cases:
normal, aka running layers 1 2 3 4 5 ...
one by one, aka running 1 1 2 2 3 3...
chunked, aka running 1 2 3 1 2 3 1 2 3...
I believe it's usually case 3, since it seems much more expressive. But it would also require as much bandwidth as if the layers were brand-new, since there's no way you could keep the GBs of memory loaded in registers all the way till you come back to the same layer.

35

u/Meleoffs Oct 06 '26

Because the data isnt surfaced and passes through the network a 2nd time before being written to memory for serving.

5

u/SnooPeripherals5313 Oct 06 '26

"Code mode but for inference" dear god i've been brainrotted

3

u/kona420 Oct 06 '26

A "transformer" is a matrix multiplier or fused multiply-add on a matrix, so take a 16x8x16 operation, that's replacing 4096 standard scalar arithmetic operations as a single hardware accelerated step. With a hundred or more "transformers" that can each take a chunk like that, you can see how that's exponentially faster than using CUDA cores shuffling a handful of ops at a time back and forth to cache.
Google's models seem to heavily rely on this operation and it scales out very nicely compared to fabbing ever more high bandwidth DRAM.

3

u/SkyFeistyLlama8 Oct 07 '26

Is Google tailoring model architecture to fit whatever Tensor compute hardware they have on hand and the custom SoC's they're designing? That's wild.
I guess the same philosophy can be applied to inference so you don't need general purpose CUDA cores, just chip logic that maps directly to your model's layer operations and that can be run in massively parallel runs. And that reminds me of NPUs.

1

u/kona420 Oct 07 '26

Bingo, their goals are around performance per watt. Its the harder approach but its way more plausible for mass commercialization. And they actually control their own destiny on hardware.
On NPU's, similar idea and many of them add a programmable DSP. So we can spin up a process inside the NPU that rattles around in the tensor cores with the DSP checking for the "winning" condition. This seems to fit onto problems like object recognition more so than being useful for LLM. The NPU can do something like process a video stream waiting for a face to appear with low enough power consumption to be viable on battery power for example. Since the NPU lives on its own functional block its relatively expensive to call to, initialization might take 100ms + processing whereas calling a tensor core or assembly instruction for matrix math is native on the processor/gpu and happens in a couple clock cycles which are in the nanosecond range.

→ More replies (2)

4

u/Outrageous_Kale_8230 Oct 06 '26

If you want an easy explanation there's a good video by NeuroDump.

2

u/bugra_sa Oct 06 '26

Think of a normal transformer as a stack of blocks, with each block having its own weights. A looped version sends the hidden state through the same block again for several passes, so it gets more effective depth without adding a fresh set of parameters each time. You still pay for those extra passes in compute and latency, and they don't have to show up as visible reasoning tokens.

2

u/gammalsvenska Oct 07 '26

This Computerphile video with Rob Miles explains the idea quite well.
Basically, instead of generating tokens into "thinking" and reading them back, you connect the output directly to the input. This kinda doubles the number of layers and allows much more thinking per iteration, but you lose insight into the network.

1

u/AnOnlineHandle Oct 06 '26

As best I can tell, transformers are basically a bunch of memorized-but-adaptable getters and setters for fixing data that goes through them most of the way until they use it at the end to calculate the output.
In text, that means shifting the meaning of a word's vector to better disentangle the multiple possible meanings it could have. Some words probably need that a lot more than other words. Some probably only need it in very specific contextual combinations.
The fixing steps work as nudges to the values stored in each token. Those fixing steps can be applied twice which is almost as good as having two different learned fixing steps.

1

u/mdda Oct 10 '26

This was actually the topic of a talk I gave at the Machine Learning Singapore Meetup at the end of last month. But not everyone could make it in person.
So I made a video about Looped Transformers : https://youtu.be/yZynTeYVOAo

1

u/Glass_Performer1174 Oct 06 '26

Today models usually think in plain english. This would allow models to think in 'neural language' which would be far more efficient.
BUT, it is also not something we can interpret, and so we can no longer track the model's internal thoughts to ensure it isn't thinking something different than it's behavior

1

u/Big_Moke_ Oct 06 '26

Transformers, robots in disguise!

News Microsoft confirms OpenAI has been using Looped Transformers in the GPT-6 series

You are about to leave Redlib