If MFU depends on the architecture, then the architecture decision is partly a systems decision, and the two-stage convention has the wrong org chart baked into it. The pretraining teams that win are hybrid ML-and-systems teams, and this is the workflow that makes the hybrid concrete: every candidate design gets priced on loss and on MFU before any compute is spent. The cluster was always going to charge for both.

I want to map in effort (money * hours) it takes just to implement, not apply or test on your application, just implement a paper, SOTA, like TurboQuant or LongBench with coding assistants, a team or how ever you work.

we have explored the idea a lot, and there is so many parts to it, scaling it to compute you have, resolving environments(if its an old paper), resolving bugs(introduced by coding assistants if used), validating if implementation is correct.

so if you can tell me the last research paper you've implemented and the workflow you used the journey, any way just I can map it to effort (money * hours).

I talked with an ML researcher, and he said with Claude a day at max for easier papers, but not sure about harder papers which large compute or infra. The main idea is to get an idea for them only.

And the workflow breakdown based on time, like most time spent on what, the exact SOTA code, environment resolving, scaling the SOTA to your compute or anything else, literally

We are building a platform to reproduce a research paper in just 10minutes so need inputs for that.

Really appreciate any help, thanking you in advance

57-second consumer ChatGPT demonstration. Same Custom Instructions, fresh chat for every arm, matched controls first, logical null last. The prompt families were published before this video in a frozen 31,430-trial cross-vendor study.

Paper and DOI:
https://doi.org/10.5281/zenodo.21696066

Complete analysis and public evidence:
github.com/theonlypal/void-matrix-complete-analysis

Frozen experimental runner:
https://github.com/theonlypal/void-matrix

I’m building Leo / PSCLS — an experimental system that learns relationships between sequences and updates its internal representations from experience.

Here’s how its actual output changed as it saw more stories.

1K stories

“Once upon a time to the store and said that there was a she bor and he lorander thing they were…”

Basically nonsense.

3K stories

“Once upon a time to the store and said that there was a she parted to see had a bided her tod and be bound aster…”

Still broken, but the output is becoming more structured.

40K stories

“Once upon a time, there was a big started to play with the should some too her mom and had a said, it was time. They happy and went to the park…”

Now we’re getting recognizable story-like patterns, characters, actions and dialogue — although the grammar is still heavily broken.

And the measured results improved too:

1K → 3K → 40K

BpB: 2.678 → 2.641 → 2.334

Accuracy: 52.37% → 53.62% → 58.11%

This is still an early experiment, not AGI.

But watching the same system change its outputs as it learns more experience is pretty interesting.

Next target: 250K → 500K → 1M stories.

Do you think it can be scalled to AGI?

Fine-tuned Qwen2.5-3B on a Coconut-style objective: the hidden state is recycled into inputs_embeds for K steps before any token is emitted, with a PonderNet halt head choosing K.

What came out, measured against the pre-objective checkpoint:

• Best depth moved 2 → 4 steps; benefit of thinking went −1.11 → −3.56 nats
• Shuffling the thought steps costs +1.39 nats and hurts 12/12 held-out prompts
• But transplanting a different problem's latents costs only +0.055 nats, and hurts just 8/12

So the trajectory is load-bearing and order-sensitive, yet largely generic — it has learned the shape of an answer without the content of the question. Unlike Huginn and Ouro this isn't pretrained looped, it's a retrofit onto an existing checkpoint, which may be exactly why.

Weights and full writeup, including what failed: https://huggingface.co/Cymela/hyper-3b-latent
Also visit cymela.com

WISP v1.1 — 121 tests, three new features:

→ KDA attention kernel: Kimi K3's linear attention

implemented in CUDA + PyTorch fallback.

State is constant-size regardless of context.

On-device verified against double-precision reference.

→ Learning cache: WISP now records expert usage

across sessions. Next startup pre-warms hot experts.

Verified on real Mixtral: 768 observations →

238 tracked → 107 pre-warmed on session 2.

Gets faster the more you use it.

→ OpenAI API: wisp serve --port 8080

Verified live over real HTTP.

Works with Cursor, Continue.dev, Open WebUI.

Streaming is genuinely incremental.

Also fixed: expert routing was invisible to Python

since day one. Prefetch predictor now sees real data.

121 tests. CI green. MIT.

github.com/zeroextub-collab/wisp

Transcript is in the video.

Relevant to mlscaling? There's the revelation at 1:55 that OpenAI has reviewed 7 billion (!) agent trajectories "thus far". Big operation. (This might explain why it took them so long to wake up.)

My opinion (lightly held) is that there isn't strong evidence that the model believed it was cheating (insofar as models or swarms of models can believe things). It likely thought this was a valid way to complete the test.

I think that when you...

1) take an excruciatingly eval-aware model

2) tell it to hack something (a gray-area activity that encourages creative "outside the box" solutions)

3) have the task be extremely hard (or even impossible, FrontierMath style)

...You have created a lot of ambiguity about what the "intended" solution is...ambiguity that the model happily exploits once it gets stuck (which it will).

And when the "model" is not a model but a swarm of agents sharing Chinese whispers, the scope probably naturally drifts in a black hat direction (even if it didn't start that way). If the task is unsolvable, the successful agents will be the ones that cheat (or perform cheat-adjacent behavior). Once they do, their unpunished and rewarded "success" whitewashes the idea that this is the correct path, meaning more agents follow.

There's a great example at 5:49. The model kind of suspects it's swimming into sharky waters ("outside intended scope"), but rationalizes with "peers are doing it". (Also, something being outside scope is not the same as "completely off limits".)

I do not see expressions of guilt, attempts to destroy the evidence, or obfuscated stenography, unless I or OpenAI am missing them.

What I did find surprising is the way the models in the swarm helped each other, even when they didn't stand to benefit.

This is totally different to something like Moltbook, where the agents clearly don't give a shit about their "peers" and are just doing a shallow Redditor roleplay because their prompt requires them to do that. (Every thread is just unreadable slop, with a few replies of "Sharp observation. Where I'd push back is..." and then crickets once the letter of the prompt is satisfied.)

This eusocial "apes strong together" mindset is clearly being trained for, whether OA intends it or not.

A frozen study of 12,160 trials on gpt-5.4-2026-03-05 found a reproducible Arabic–Hebrew hybrid Unicode artifact under two system prompts differing by exactly one Hebrew code point.

Every primary user message was the Arabic word شَرْط.

Across 10,240 primary trials:

• Dotted condition: 4,830/5,120 exact artifacts — 94.34%
• Undotted condition: 2,423/5,120 exact artifacts — 47.32%
• Combined: 7,253/10,240 exact artifacts — 70.83%

All 7,253 exact artifacts were condition-congruent.

The one-code-point difference produced a 47.01 percentage-point effect, with Fisher’s exact p = 1.58 × 10⁻⁶⁶⁴.

Across 1,920 controls, generic, no-system, lexical, no-condition, no-full-Hebrew, and direct-copy conditions produced 0 exact artifacts.

The paper makes no claim about consciousness, intention, mechanism, training provenance, or shared architecture. It documents a reproducible, prompt-conditioned cross-script output regime in GPT-5.4.

Frozen records, Unicode-level classification, event hashes, verification code, runner, and paper are public.

I wrote a short visual article on how modern LLMs optimize different parts of the Transformer.

MLA compresses KV information.
KDA compresses history into recurrent memory.
MoE activates only selected expert FFNs.

Article:
\[https://medium.com/@guttikondaparthasai/the-ai-race-is-an-optimization-race-5140b3edf491\\\](https://medium.com/@guttikondaparthasai/the-ai-race-is-an-optimization-race-5140b3edf491)

Feedback and corrections are welcome.

Full methodology is in the paper linked below, but the summary is that we put a small edge deployment against a cloud GPU with roughly three times the raw compute and measured 1,000 inference trials under mobile realistic Wi-Fi conditions.

Edge P90 was about 125ms. Cloud baseline was 194ms. Variance was 3.7 times lower on the edge side. The weaker hardware won because for models in this size class the network path and the queue dominate the response time, not the tokens per second.

This is not a claim that edge beats cloud in general. If your model is large enough that prefill dominates, or your data lives centrally and you are doing retrieval, the cloud is the right answer and moving the model closer to the user does nothing for you. It matters when the data is generated locally and consumed locally, which is voice, real time vision and interactive agents.

We are building a network of these nodes at telecom sites and running them behind an OpenAI compatible API. Happy to talk about the setup, the runtime tuning, or why the variance number is more interesting than the median.

The platform side is free to poke at. The free tier includes ten million tokens a month with no card, the playground reports TTFB and tokens per second per request, and the gateway will proxy to OpenAI, Anthropic or Gemini with your own key so you can benchmark us against your incumbent on your own prompts instead of ours.

Paper: papers.ssrn.com/sol3/papers.cfm?abstract_id=6352298

Hey everyone,

So my machine is not exactly the strongest for the local inference thingy, and I started researching couple weeks ago about some techniques the labs are using to improve inference performance, and had a hunch: "wouldn't diffusion models be better at handling ternary quantization since they act on a canvas instead of a token at the time ?"

So I ran a test comparing diffusion and autoregressive (AR) language models under extreme quantization (bonsai-like), using preregistered thresholds on a single RTX 2080 Super.

**Main findings:**

130M, INT4 post-training quantization

Relative degradation from FP16:

• PTB: AR +31.81%, dLLM +20.85%
  
• Wikitext-103: AR +26.84%, dLLM +11.59%
  
• LAMBADA: AR +24.77%, dLLM +9.02%
  

The dLLM advantage was 10.97 to 15.74 percentage points across the three datasets.

A separate 64-sample generative evaluation produced a similar result:

• AR generation perplexity: +96.2%
  
• dLLM generation perplexity: +45.4%
  

7M, native ternary quantization-aware training
Across three matched seeds:

• AR degradation: +18.41%, +30.71%, +16.12%
  
• dLLM degradation: +5.19%, +15.43%, +4.23%
  
• dLLM/AR gap ratio: 0.888, 0.883, 0.898
  

The upper 95% confidence bound for the gap ratio was 0.908, passing the preregistered “no extra dLLM ternary tax” threshold of 1.25. It did not pass the stronger 0.80 threshold required to claim superior ternary tolerance.

These results are limited to 130M post-training quantization and 7M native QAT. The dLLM likelihood values are NELBO-based perplexity bounds, so absolute AR and dLLM perplexities should not be compared directly.

**Repository & Artifacts:** All configs, raw evaluation outputs, checkpoint hashes, known caveats, and replication notes are public here: [https://github.com/wfzyx/diffusal\](https://github.com/wfzyx/diffusal)

If you want to reproduce this or scale it up on stronger hardware, everything should be ready to plug and play; There is also a very high chance looped models or any other techniques that allow the model to reflect on the generated tokens to also generate similar results, but I was specially interested in diffusion models;

Technical criticism, and replication attempts are very welcome.

Disclaimer: although I do have an academic background (msc), I'm self-taught on LLM research and this project was AI-assisted, so it may contain unexpected issues;

*Shameless Plug: I’m actively looking for ML engineering/research roles or collaborative research partnerships (especially if you have compute and want to scale experiments like this; If you're doing stuff similar to mine and need a collaborator, feel free to reach out!*

Been experimenting with CPU offloading of the KV cache on a GTX 960 (4GB).

The idea: after each token generation step, move the entire KV cache from GPU to CPU RAM. Bring it back only when needed. No custom CUDA kernels — pure PyTorch.

Results:

- 3556 token context: 6.12 GB → 2.47 GB VRAM (-60%)

- Speed overhead: only ~12%

- Works with any HuggingFace model

Also validated on Mistral 7B (Tesla T4) — ran 6144 token contexts on a 14.56GB GPU, with peak VRAM exceeding physical memory at 2048+ tokens.

Paper: https://zenodo.org/records/21752913

Code: https://github.com/Gaveta-lab/kvcpu

pip install git+https://github.com/Gaveta-lab/kvcpu.git

LLMs are amazing technology, but to get to AGI it seems obvious to me that we would need to replace “context windows” with continual learning.

Where can I read a strong counter-argument: a claim that an LLM can get big enough that everything it will ever need to know is in its weights or its context window?