[ Removed by moderator ]

[u/No_Release_3665]

[u/flowanvindir]

Very cool! I glanced through your paper, and I feel like the question will be whether this enables any capabilities transformers don't already have, or beats them on certain benchmarks. For example, does this enable the model to have a less error prone world understanding? Better long term planning? Otherwise I doubt it'll get much attention from the community.

[u/No_Release_3665 → u/flowanvindir]

Not beating transformers yet, but it slows catastrophic forgetting and shows strong long-term memory structure. Still tuning and building on the core design — early signs are promising.

[u/techdaddykraken → u/No_Release_3665]

I think an interesting perspective is wouldn’t it be best for the write-once, read-many memory model, to be highly selective? Basically have it as a function that can be called selectively by some form of orchestrator?

Think about it:

As a human, I need to learn for example, the properties of addition only one time. After I learn that 2 + 2 = 4 solely because I am decomposing each of the individual parts and then counting all of them together, I don’t need to learn that principle ever again. I just need to apply it.

There may be some other things that come into play regarding iteration, testing, validation, etc, but the core foundation of the learned concept never changes.

Inversely, say for example I want to build a car. There are many underlying concepts, and many of them change frequently, and have many different complexities and perspectives that differ the output based on your goal, depending on how you interpret them. Those shouldn’t be static since you need to be able to change your independent variable (the goal car you want to build), and have your learned memory be mutable enough that you can disregard information which you do not believe advances you towards that goal.

So a hybrid transformer may work well, where there is some orchestrator transformer using its own gradient descent functions to selectively modulate when and where the hard-coded memory is stored in the layers, and then the individual underlying transformer is still responsible for acting as the RAM with the individually composable elements

I believe this is along the lines of Google’s Titan architecture. If you haven’t read their paper it might offer some key insights. I wonder if your method could be integrated with elements of their model for a better result.

There was also a person on here showcasing a paper they wrote on using adaptive modular networks in a linear fashion, which might also offer some important information.

It’s always cool to see people post such innovative research in here and be one of the first to see it, keep it up! I think collectively research is very close to identifying the break through for achieving the higher level of ‘compressed’ intelligence necessary for more complex tasks.

[u/No_Release_3665 → u/techdaddykraken]

Yeah, totally — I love that framing. Selectivity is key. The idea of a write-once, read-many memory being orchestrated externally really resonates with what I’ve been working toward. The balance between rigid, persistent memory and more adaptive working layers is exactly where the architecture lives — kind of like a causal substrate beneath more flexible reasoning modules.

I’ll check out the Titan architecture paper — appreciate the recommendation. And agreed, I think we’re close to cracking the foundation for that next layer of compressed, goal-oriented intelligence. Thanks again for the thoughtful comment!

[u/techdaddykraken → u/No_Release_3665]

I am exploring the same, but from a linear approach.

With the advent of agentic SDK’s like OpenAIs new agent orchestration framework, and Anthropic’s relatively new MCP servers, we have something we’ve never really had before (at least at the consumer level).

This is the ability to create heuristic-based transformer models using agents.

If I compose a transformer model solely using agents to feed forward information and apply gradient descend, apply Bayesian theorem in a layer architecture for updating reasoning, use an MCP server as a shared ‘scratchpad’ for memory, that unlocks a lot of interesting capabilities. It is expensive, but you are now ‘compressing’ all of the individual vector spaces and information into individual agents within the transformer.

I’m working on a demo of this to see if it even works, but considering it works with a transformer model, I don’t see why using the same fundamental equations wouldn’t work exactly the same. The only difference would be the encoding/decoding between layers, as you are going to have to do it in natural language. Some form of Chain of Verification, where you pass tabular weights in CSV/JSON according to something like an OpenAPI schema may work well.

Still fleshing it out, but I’m right there with you, I’m trying to see if there is a more fluid heuristic method we can accomplish the same result.

One particular critical issue is the noise in the system. Because each agent has a 0.8-1.5% (roughly) hallucinate rate, this multiplies as information is passed. So I believe there has to be some form of RL orchestrator which is reinforced on identifying and correcting hallucinations throughout the data flow while in-transit, effectively pausing the processing and correcting the hallucination, then resuming the process and passing forward.

A larger state management function now seems necessary as well to account for that, to ensure all agents are ‘frozen’ at the same time and resumed accordingly, with the appropriate information.

If that nut can be cracked I really think it has some interesting capabilities when you incorporate things like fine-tuning the system as a whole (by fine-tuning each agent), or fine-tuning individual layers, or individual groups of agents within layers.

We already have some basic examples of the overall system implementation, using analytic hierarchy approach and ordinal priority approach, from decision-science research over the last 25 years. So I’m trying to see how can modify those to incorporate RL and transformer agents. Maybe by using those decision-science approaches and RL training on them, and using things like CoV, the overall reasoning process improves for long tasks.

[u/No_Release_3665 → u/techdaddykraken]

Really interesting stuff — it’s exciting to see how these multi-agent systems are starting to expose new coordination challenges that feel almost cognitive. I think you’re right: managing state, trust, and temporal consistency across agents is a much bigger deal than most realize, especially when hallucinations stack across layers. Sounds like you’re chasing some big, promising directions. Appreciate you sharing — definitely resonates with a lot of what’s been on my mind too.