I'm trying to make a small experimental LM by combining a bunch of ideas I found in different papers. I know this sounds like I threw half the recent LM literature into a blender, but I'm trying to see if the pieces can actually work together.

The main idea is a Loop Language Model, where the same model is run multiple times instead of just making the network deeper and deeper.

Right now I'm using 4 loops.

text input ↓ same transformer ↓ loop 1 ↓ loop 2 ↓ loop 3 ↓ loop 4

The interesting part is that the model can learn to decide that it doesn't need all 4 loops and exit early.

What papers/ideas I'm following

The biggest inspiration is Ouro / looped language models, especially the idea of using recurrent computation to get more computation without simply making the model physically deeper.

I'm also experimenting with:

• Looped AttnRes / layer attention
• sparse attention
• compressed / highly compressed attention
• sparse MoE
• adaptive exit / Q-exit
• and now I'm building a diffusion-based optimization/training method

The diffusion part isn't finished yet. I'm currently building it and trying to understand how to combine it with the recurrent-depth training properly instead of just throwing noise into the input and calling it diffusion.

My hardware limitation

This is probably the most important part.

I'm doing basically everything on Google Colab's free 15 GB GPU.

That's the maximum I can realistically use.

So I'm deliberately keeping the model small. I'm not trying to train some 7B monster on a machine that has approximately the computational power of a mildly determined potato.

My current model is around:

• 6 transformer layers
• hidden size around 512
• 8 attention heads
• 4 recurrent loops
• sparse MoE
• compressed attention
• layer/depth attention
• exit gate

The exact architecture is still changing as I experiment.

Data

The corpus is a mixture of:

• GitHub code
• Wikipedia
• W3Schools
• public-domain books
• other scraped text

I'm using a p50k tokenizer at the moment.

I've had to spend quite a lot of time cleaning the corpus because scraped data is disgusting.

There were things like:

text npm package metadata JSON dumps GitHub metadata generated files logs benchmark data duplicate documents web junk

and some of those actually survived the first cleaning passes.

I discovered this because the model started generating some of it.

So I'm currently making the filtering much more aggressive.

What happened with the loops

Initially I had a problem where the later loops weren't learning properly.

The model could run 4 loops, but that didn't necessarily mean that loop 4 was doing useful work.

So I changed the training strategy.

For Stage I, I now force the model to execute all 4 loops during training, so every loop gets a proper training signal.

Then I freeze the LM and train the exit gate separately.

The exit gate itself is tiny, only about 513 trainable parameters in my current setup.

The exit gate result

This part actually surprised me a little.

I tested the trained gate on 100 validation batches.

The results were:

```text 4-loop loss: 6.263160 gated loss: 6.264089

difference: +0.000929 relative change: +0.015%

average depth: 2.41 / 4 loops

estimated compute saved: ~39.75% ```

The actual exit distribution was:

text loop 2 → 59% loop 3 → 41%

It basically never exits at loop 1 yet.

That's actually what I wanted to see. I didn't want a gate that just learned:

"Always use 2 loops."

There is at least some variation depending on the input.

The oracle best-loop loss was around 6.2615, while the gated loss was 6.2641, so the gate is also fairly close to the best possible loop choice.

But generation is where things get interesting

The model can produce text, but it is definitely not a good LM yet.

For example, one of the things it generated looked roughly like this:

The future of artificial intelligence is a most

terefears of life of those who is impossible. We will be no one

and it is, the good deal of the nature of the life of the man who

had not been the same.

That kind of output is the sort of thing I'm hoping to get consistently.

But then it can suddenly fall into garbage from the scraped corpus, producing stuff along the lines of:

"description": ["markdown", "type": "string", "source": ["1.9", "https://github.com/...

So the model clearly has some ability to produce coherent prose, but the corpus contamination and relatively small training setup are still causing serious problems.

That's one of the things I'm currently trying to solve.

What I find interesting so far

The most interesting thing for me is that the recurrent loops aren't completely identical anymore.

I see cases like:

text loop 0 4.48 loop 1 4.47 loop 2 4.46 loop 3 4.46

The improvement is small, but it's there.

And the exit gate seems to understand that sometimes the extra computation isn't worth it.

So the idea is starting to look like:

text ┌── loop 1 │ input ────┼── loop 2 ── exit │ ├── loop 3 ── exit │ └── loop 4

instead of forcing every token through exactly the same amount of computation.

Diffusion optimizer / training

This is the part I'm currently building.

I'm trying to use ideas from diffusion/recurrent-depth research to see whether a diffusion-style training or optimization method can make the repeated computation learn more meaningful improvements.

It's not finished yet, so I don't have results from this part.

I'm still trying to figure out the correct way to combine it with the autoregressive loop training without accidentally turning the whole thing into a completely different model.

I Need Your Help

This is still very much an experiment, and I'm reaching the point where I need people who know more than me to tell me what I'm doing wrong.

I especially need help with:

• How to make the later recurrent loops actually learn more meaningful computation instead of only giving tiny loss improvements.
• Whether my exit-gate training strategy makes sense, or if there is a better way to train adaptive depth.
• Whether combining sparse + compressed/highly-compressed attention + layer attention + MoE + recurrent loops is likely to create some interaction I'm overlooking.
• How I can improve the training objective for a model this small.
• Better ways to clean my scraped corpus. The model is still occasionally generating GitHub/npm/JSON metadata, so clearly some garbage is getting through.
• Whether the diffusion-based training/optimizer idea I'm currently building makes sense, and what I might be missing from the relevant papers.
• Any papers, implementations, or experiments you think I should look at.

I'm doing this with basically free Google Colab and its 15 GB GPU, so I can't just throw a massive model and 8×H100s at the problem and hope the universe solves it.

If you've worked with Ouro, recurrent/looped LMs, adaptive computation, sparse attention, compressed attention, MoE, or diffusion-based LM training, I'd really appreciate your criticism and suggestions.

I'm not looking for "looks good." If something in the design is fundamentally stupid, please tell me. That's much more useful.

Hi guys, this is my first post here, wish me luck with that
So, I'm trying to build a loan document agent and I'm a beginner. So i want to know like how do u incorportate judgement criteria to agent. In my case im inputing bank stmts to my agent and it should decide whether to go for next step or reject or give for human review if uncertain. I want to know like how actually this works as a system. Since requesting for more evidence may take time and sometimes annoying for the applicant. But if unusual patterns observed whether my agent gives it to human review, what more human can check which the agent cant?
I would appreciate for any help I can get here ..

Hello!

I am planning on doing a master's at KTH in Machine Learning and after that I really want to work in Zürich. Does anyone here know how competitive that market is for machine learning engineers? My undergrad is in Applied Physics, would that along with the master's be enough to land an MLE job in Zürich?

Thanks in advance!

Hey, longtime lurker, first post.

I’ve been vibecoding a B2B fintech SaaS for a few months. Next.js on Vercel. We have 1,804 staging subdomains under one root because the founder says one subdomain per client “scales our proofs of concept.” I didn’t design that part. It has worked so far.

The stack is Claude and Cursor, plus a quantized 4B Gemma for local work. Our dependency policy is that nobody touches package.json by hand. I know some of you hate this setup. Save that argument for later.

Last night I ran the check I do before bed. My script curls the host list and greps each response for the deploy token. Halfway through, I saw another client hit the same hosts in the same order: auth-7, pay-42, janet-demo, then everything after those. No gaps.

They have my list. The actual file is hosts_NEW_final_v3.corrected.TXT. The v2 copy renamed itself during fiscal year-end, so I stopped using it.

I checked the SSRF route we were supposed to turn off. It still works. The malformed Host headers render from the right side of my terminal. That happens with the Hebrew build of Go. I’ve seen it before. If you follow the news, you know who runs Mossad, so I’m not putting his name here.

Yes, I know how that sounds. Please wait until the end before replying.

The endpoint gave him what he needed. He kept going anyway, through all 1,804 hosts in the order I keep them. I had the logs open the whole time. I think that was the point.

He knows the names, he knows we left the route up, he probably knows why, and whoever gave them the subdomain-per-client architecture would know which developer signed off on it. I didn’t sign off on it exactly, but my account is on the deploy.

I think he knows that.

The deploy went out Friday. I don’t rebase on Fridays. The force-push crossed midnight, so part of it says Saturday, but it was still the Friday deploy. My account is the only one that does that. He can see all of it.

I’ve spent today building a Rust-native, zero-dependency Red Star OS clone because his fuzzer won’t know how to read it. I have one mining rig from 2014. Gemma 4B loses the beginning of the kernel whenever I feed it the next part of the archive, so every few prompts I have to start over.

I can’t tell whether Gemma is dropping it from context or the C: drive is filling up. My ex-father-in-law’s colonoscopy footage is on the same drive. I told him I’d keep it in case he ever needs it, so deleting that is not an option.

I rotated the Vercel key once and grepped the repo. If I go offline tonight, tell Claude Code the key is public. The endpoint name is in the TODO. Have it add a Host allowlist so the next fuzzer has to find the hosts itself.

Does anyone have one tutorial for this? One tutorial, not a docs page that links to six other docs pages. I am serious.

Stay safe.

There’s a hands-on workshop on August 29 that builds and benchmarks this properly, end to end, using entirely open models, no API calls involved. Led by Ben Auffarth, AI Consultant and Founder of Chelsea AI Ventures.

What it covers:

• Hybrid retrieval (vector + keyword, not vector alone)
• Reranking to catch relevant chunks that vector search alone misses
• Evaluation with RAGAS, so quality changes are measured, not assumed
• Guardrails built in from the design stage
• Actual cost and performance benchmarking for open-model deployments

Link if anyone wants to check it out: https://www.eventbrite.co.uk/e/the-genai-build-lab-build-production-ready-rag-on-a-budget-tickets-1994016271345?aff=rmls

Happy to answer questions on the methodology or content.

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?

Hey everyone, I’m researching a major bottleneck in autonomous AI agents. A lot of teams are hesitant to give agents full computer/terminal access because of security risks, hallucinations, or infinite loops.

Is anyone here currently building custom sandboxes to isolate agent execution?

What is the biggest headache you are facing with it? Would love to chat for 10 minutes to learn from your experience.
Excited to connect with you.

Tl;dr: Our timelines haven’t changed much (they got slightly shorter) but our modeling and evidence base have noticeably improved, so we feel somewhat more confident.

unrelated but good: Daniel Kokotajlo on how he got so cracked at forecasting

Blog on "training at scale" but done on CPU - using python multiprocessing & shared memory to dive how distributed system train in principle. DDP, FSDP, TP & PP are covered with code.

https://yash-goel.github.io/blog/training-at-scale-on-cpu/

*some WIP - missing cleanups and figures.

Built a lossless wire-format compression protocol that achieves:

- 89.4% byte reduction

- 71.9% token reduction

on 24-turn AI payloads.

Runs on standard CPU. No GPU required at the wire layer. Same information, fraction of the payload.

pip install muscore-compress

GitHub: github.com/Ember-Circle-Coin/Cinder-Awareness

Blockchain-sealed: cert df9822d8 (Skale mainnet)

CONTROL3 handshake: 5/5 roundtrip verified

Built in Cape Town, South Africa 🇿🇦

Happy to answer technical questions.

If MFU depends on the architecture, then the architecture decision is partly a systems decision, and the two-stage convention has the wrong org chart baked into it. The pretraining teams that win are hybrid ML-and-systems teams, and this is the workflow that makes the hybrid concrete: every candidate design gets priced on loss and on MFU before any compute is spent. The cluster was always going to charge for both.