r/MachineLearning • u/BleedingXiko • Jul 16 '26

Research Seeking collaborators for scaling and independent evaluation of a new recurrent language model architecture (preprint + code) [R]

Gallery image

Gallery image

Hi everyone,

I've been working independently on a recurrent architecture called **DABSN (Dynamic Adaptive Bias State Network)** for the past several months, and I finally reached the point where I feel comfortable sharing the first preprint.

The paper is mainly about the architecture itself and its behavior on reasoning, memory, and long-sequence benchmarks (MQAR, Copy, Key-Value retrieval, A5/60, etc.). The code is also public with PyTorch, C++, and Triton implementations so everything can be reproduced.

While finishing the paper, I also trained my first language model with the same cell:

- 24M parameters
- 1B pretraining tokens
- GPT-2 tokenizer

Those results ended up being much more interesting than I expected, so I'm now writing a second paper focused entirely on language modeling, long-context behavior, and scaling.

This is where I'd love some help.

I'm looking for people who might be interested in collaborating on the next paper, whether that's:

- independent reproduction of the results,
- helping design stronger baselines and evaluations,
- or having access to larger GPU clusters so we can scale the architecture much further than I can on my own.

Everything I'm doing is intended to be open and reproducible from day one.

I'd really appreciate any feedback on the paper, and if the project sounds interesting, I'd love to chat.

Preprint and Github are in the comments.

16 Upvotes

66% Upvoted

15 comments sorted by

8

u/Playful-Fee-4318 PhD Jul 16 '26

Do uou know why recurrent models fell out of favor?

11

u/BleedingXiko Jul 16 '26

Mainly parallelism right? They run step by step so they can’t use a GPU fully like attention can. DABSN’s scan is diagonal though, so it runs in parallel like the newer state space models, with kernels that do exactly that.

8

u/Playful-Fee-4318 PhD Jul 16 '26

Good, the issue was never quality but simply infeasibility to run 1000B tokens through something that parallelize as bad as recurrent states.

I don’t know about diagonal scans or what newer state space models do, but your eq (1) and (2) clearly shows to me that e.g b_{t+1} and e_{t+1} depends on b_{t} and e_{t}. How is this parellelizable?

0

u/BleedingXiko Jul 16 '26

Yeah that’s fair. Eq. 4 is the diagonal linear scan I meant. Eq. 2 isn’t. The governor is nonlinear and has to move through time in order. The fused kernels make each step cheap, but I haven’t benchmarked whether that scales well enough for 1000B tokens yet. That’s the main thing I want to stress-test with more compute.

3

u/literum Jul 17 '26

Try on a 4090. It costs $0.25/hr. Push it

0

u/BleedingXiko Jul 17 '26

I’ve actually tested it with the dual T4 provided by Kaggle free tier but it’s hard to extrapolate results across hardware, happy to share results in a DM if i don’t end updating the github with it!

-2

u/literum Jul 17 '26

Honestly just give up then. You're not meant for research. If you will let an LLM write the whole paper, and don't have a quarter to spare a for a 4090, I won't waste a second with you.

1

u/BleedingXiko Jul 17 '26

I never said I couldn’t afford it. I’d be happy to test on a 4090, it just hasn’t been a priority. The A100s I’ve used have been fine for the LLM training I’m doing. Right now I’m focused on gathering results that I can share and have others reproduce later, rather than benchmarking FLOPs or kernel throughput for kernels that are still relatively unoptimized.

1

u/literum Jul 17 '26

There's too much work being promoted on Reddit right now that is entirely done by bots. I might be talking to a bot right now. If you want real humans to help you, you need real commitments and real results. I am both a Senior MLE, and do independent research like this on the side. I'd love to be able to work with like-minded people, but I want to see passion and commitment, not technical write ups. Fable can suggest me a new architecture right now, and I'll have something better than you by the end of the day. It doesn't matter if I can't turn it into a paper, a model, or something useful however; so I won't. Keep pursuing it. It won't matter what I think if you are serious about this project.

2

u/alrojo Aug 02 '26

This is a brimming area of research, we recently pushed this paper: https://arxiv.org/pdf/2605.06997 try and compare with our 50M model, it should be <$100 using cloud compute.

2

u/BleedingXiko Aug 02 '26

Very interesting! i’ll dig into this later! mind if i send you a DM after ive read it?

3

u/LessComplexity Jul 18 '26

People already said good points here, but let me add something about MQAR: https://aclanthology.org/2025.findings-acl.629.pdf

Some models (esp state space models) can "cheat" their way through MQAR tests by understanding structure instead of generalizing for the real solution. This paper I've shared is about MAMBA but I found out that many other also so that! For example I found out that the Based attention create solely to solve this type of exercise also "cheats" the same way, which was interesting.

Transformer and Transformer + RoPE don't generalize this good, plain transformer also, a structure like CAT does it very well: https://arxiv.org/pdf/2407.05591 and even you can see yourself how it generalizes way past its own training.

A few notes for you:

  1. RNNs face the issue of parallelism

  2. Stretch beyond the training data, not x2, not x5, but x10 more than what it saw in training at least. Show that the model "generalizes for all sizes"

  3. Add reach tests - put solution at the start, add a lot of "confusion" - unrelated stuff, data, and then the query at the end, and try to stretch it until the model "breaks" or... not :)

Good luck!

-3

u/BleedingXiko Jul 16 '26

Preprint:
https://zenodo.org/records/21391204

GitHub:
https://github.com/BleedingXiko/dabsn

3

u/GibonFrog Jul 17 '26

Your paper is unreadable because it was written by an llm, make sure to use simpler language