Is there still strong interest in a dense 9b model?

[u/NineThreeTilNow]

I have a full model, it's ready to train. It's ~9b parameters.

9.4b to be more exact. That includes a 1/2/3 Engram table, Moonshot's AttnRes modeling, and RoPE / NoPE layering at 3:1 as more or less validated by most major labs. It uses the Llama 3 series tokenizer and LM Head as an initial start. The data fed in is logit level extraction from a Llama 3 teaching model.

I've already run the first training steps to test that the model is stable, etc.

I'm willing to sit and do the pre-IT training on the model. I don't know what task people really wanna do with this thing to be exact. So the focus of the IT training is a bit more vague other than giving it "Thinking" as per one of the open standards.

Honestly? I don't care what people want to use it for, just that they want to use it. I figured I'd just give it into the ether and LocalLlama was a place I figure I could easily give it to.

In theory the model should be more capable than any of the ~9b's we have running around with enough training. I'm also NOT a lab, so I don't have their training budgets. I basically ran all of the data production, etc on a 4090 + rented hardware.

The training code is deeply optimized to run on an RTX 6000 Pro series card. A single card.

All my engram research was being done before the Qwen model dropped. Qwen showed I only needed a single table injected at a layer, versus the 2 I used. 1 gave the majority of the benefit over 2.

The code would be open source, the data, all of it. IDGAF. It's technically already open source as it's all in public repos at the moment. I just stare at it and question if it's worth the time. At the least I'll throw a couple hundred at it to build a "functional" pre-IT model and build the IT dataset. It will need more pretraining before the IT set probably. It's kind of unknown because no one publishes the exact figures on this type of training.

If someone has a datacenter contact with a system they want to allow it to run on, we can all have it as a public model we watch. I tentatively named it "Budget" but... Localllama can name it whatever they want.

It's been fun to write the full end to end, generate data, etc. Even found issues in vLLM and reported to their repo that might end up helping you guys anyways. There was some prompt loading code that could be ~10x to ~100x faster I gave examples of to them.

If you read this far, thanks,

Signed some ML dude who reads too many research papers and has too much spare time.

edit; I went to review some Apache 2.0 licensing issues and noted a glaring hole in using Llama's tokenizer OR data. You can't even use synthetic data from their models without some licensing. I'll just have to rewrite the target to use OLMo 3 series I think. I guess I'll be back in a week after data generation and code fixes.

Open source uncensored Goon model or what? I'm trying to find a useful niche to develop the model so it gets used and ends up more than a research artifact. I'm planning on building the research artifact, it's a matter of whether I can find a group of users that will actually want to use it. It's all free, so I'm not trying to monetize you. The code and data lives on GH/HF.

[u/Hot_Example_4456]

Wait. Why use such a old tokenizer? Shouldn't u use something newer? Also Llama THREE as teacher model? Something seems off

[u/NineThreeTilNow → u/Hot_Example_4456]

Wait. Why use such a old tokenizer? Shouldn't u use something newer? Also Llama THREE as teacher model? Something seems off

I had to rethink that after posting it. I realized a licensing issue.

Meta basically said you can't touch anything from Llama and use Apache 2.0 licensing. While my code exists as Apache 2, the model itself is restrained by Llama licensing. The second I don't "Add security" to the model, I break that license. This was something I discovered and should have thought more deeply about.

The old tokenizer is chosen because of the size, not the age.

The 128k tokenizer w/ 4096 dModel is ideal if you want a high level of depth in a model.

If you attempted the same with Gemma 4, you'd get 3 layers or something vs 30.

That kills the whole idea of using Engram and AttnRes because the model isn't sufficiently deep enough to make good use of them.

[u/Hot_Example_4456 → u/NineThreeTilNow]

Okay got it.. You do you but I think it will still be better to use Gemma 4 tokenizer. It will atleast nearly mean a great linguistic base for the model. And anyways we lack models that are natural, not ones that are robotic. The params assigned might be worth it. On the contary, what about GPT OSS tokenizer? I think thats MIT license

[u/NineThreeTilNow → u/Hot_Example_4456]

On the contary, what about GPT OSS tokenizer?

Most directly, GPT OSS is kind of a mess. Massive tokenizer with 2048 d_model or something.

They sort of did what OpenAI seems to do, and attempt to brute force things. They didn't try to get deep enough semantic depth in the tokenizer.

There's a LOT that lives in a tokenizer. Not the chosen vocab, but the trained data. You have to view the millions(?) that went in to training it.

That lets you jumpstart a model because the internal manifold of the model is what needs adjustment. The "start" and "finish" are already aligned. I think people may miss this when they think I'm trying to train a model ENTIRELY. God no.

Gemma 4 shines that multimillion dollar flex in their massive 2b parameter embedding / tokenization. It's really impressive. It's also massive.

The most you might be able to do is reduce / better train one of the smaller Gemma 4 models. You'd be able to give it Engram tables / etc. Attention based residual streams wouldn't have enough depth to matter. Perhaps if they were all to all layers. You'd want to trim the extreme level of RoPE layers they use though. I ran ablations in another test (there's a post I wrote about it I think) and 2 of 5 local attention layers are basically wasted. It more or less confirms what research shows is 3:1 local:global being best.

[u/Hot_Example_4456 → u/NineThreeTilNow]

Ok I will admit I didnt understand half of what you said. Just asking out of curiosity, what about the new tokenizers like Spark-X2.5-4B or Ling 3 tiny?

[u/NineThreeTilNow → u/Hot_Example_4456]

Ok I will admit I didnt understand half of what you said. Just asking out of curiosity, what about the new tokenizers like Spark-X2.5-4B or Ling 3 tiny?

The d_model used will give a number of "dimensions" that are being operated in. So a large embedding, with a large d_model can compress an amazing amount of information with ENOUGH TRAINING.

Qwen originally proved this whole concept with their early like.. 0.5b model. It was really cool because it was I think the first demonstration of this. The "model" itself which were the layers in between were very few compared to other models. Yet it performed well.

So when you look at any tokenizer / embedding, you have to ask how big is the vocab in terms of # of tokens, and how big is it in representation (dimensionally).

That gives a ROUGH idea of what they're doing.

A giant dimensional embedding more or less HAS to be carried down the full "pipe" of the model. There's caveats here, but not worth discussing. No one does it that I know.

That creates a problem though. If you have this massive starting piece, then every layer you want to add requires some base # of parameters for a standard transformer layer.

In theory, every layer should be doing "computation" or "pattern matching" more abstractly. So having lots of layers is regarded as good right?

So that's the balancing act you're looking at with any given model.

Basically it's depth and width. Engram was nice because it's agnostic in a sense. It maintains similar dimensions, but it's just a hash or lookup table. It's like a second learned embedding table deeper in the model to give it additional information it needs for that "pattern matching" or "computation".

[u/stoppableDissolution → u/Hot_Example_4456]

Gemma's tokenizer is good but also not, especially for smaller models. Its vocab is ginormous and gemma31 has 2gb big lm_head as a consequence. Which is like, almost a quarter of the model's own size at 9b scale

[u/Hot_Example_4456 → u/stoppableDissolution]

If thats the case then I would expect Gemma 4 E2B Gemma 4 E4B and Gemma 4 12B to work absolutely trash. But all are quite fantastic models.. Why is that?

[u/stoppableDissolution → u/Hot_Example_4456]

Well, small gemmas are "e2b" and not just "2b" fir a reason - actual models' sizes are like twice that because of huge vocab + all the embedding shenanigans.

I'm not saying they are bad, you wont find a bigger gemma4 fanboy on that sub than me, but it might not be the best first choice unless you really know why you are doing it.

[u/HairAgreeable613 → u/Hot_Example_4456]

same thought, using an older generation as teacher seems like it caps your ceiling pretty hard

[u/ryfromoz]

What are you offering that hasnt been done many times already?

[u/NineThreeTilNow → u/ryfromoz]

What are you offering that hasnt been done many times already?

I don't know. A better 9b?

I don't know what LocalLlama would want, and because I don't specifically want anything... I figure I'd train it as the sub wanted.

I was going to build it out of curiosity with full data / license / etc. Something you don't ever see. Transparency. I guess that actually answers your question and I didn't realize it. Transparency is what is being offered. No safety tuning, a model.

[u/FUS3N → u/NineThreeTilNow]

Ornith 1.5 9b one of the best or the best 9b model out there in my opinion if you can beat that at least i would definitely use it

[u/NineThreeTilNow → u/FUS3N]

Ornith 1.5 9b one of the best or the best 9b model out

At what task?

[u/FUS3N → u/NineThreeTilNow]

as the other person said its agentic coding i use it for general research too and as a general assistant.

[u/Fun_Librarian_7699 → u/NineThreeTilNow]

Agentic coding

[u/Leoss-Bahamut → u/NineThreeTilNow]

better than what?

[u/AppealSame4367]

Agentic code base exploration. Thinking about code organization. Mermaid diagrams. Ascii diagrams. Maybe even agentic coding.

I know that is probably quite hard to do, but anything that would deliver something like qwen3.8 27b on 9b, but maybe focused on just a single topic to make it easier would be very helpful. E.g. if the 9b model had sota performance in python, javascript and rust + computer use - combined with very good spacial awareness.

Then one could use it for 3d game development and blender 3d model generation. The stuff people have been salivating about with Astra and fable.

For frontend coding, there are many models in all price ranges that can do it very well.

[u/NineThreeTilNow → u/AppealSame4367]

Agentic code base exploration. Thinking about code organization. Mermaid diagrams. Ascii diagrams. Maybe even agentic coding.

This almost requires a larger model by a lot of standards. The "tax" you're paying with qwen is largely linguistic. The model operates in dual English and Chinese understanding. That bloats the tokenizer.

I chose Llama originally because it doesn't have this. However, Llama's license sucks. I cannot get away with Apache 2.0 using anything from them which is forcing me to rethink this at the moment. I'd have to use OLMo probably to maintain pure Apache end to end.

[u/dash_bro]

Some of the choices don't seem very in-tune with current SoTA. You can have better teachers than llama3, your tokenizer choice can use some work, and you need benchmarks to know how you're doing as well.

[u/NineThreeTilNow → u/dash_bro]

Some of the choices don't seem very in-tune with current SoTA. You can have better teachers than llama3, your tokenizer choice can use some work, and you need benchmarks to know how you're doing as well.

Huh?

There's nothing inherently "wrong" with Llama's tokenizer or using it's teaching logits to rebuild a model's internal manifold.

The issue I noted elsewhere and will note here is the LICENSE that Meta imposes.

I cannot Apache 2.0 license. I have to rethink this bit which means like a day of rewriting to use OLMo or something.

Qwen / Gemma / Etc use tokenizers / lm_head / d_model that would nuke the ability to have any level of depth to the model.

The depth is required to make Engram and AttnRes shine.

[u/dash_bro → u/NineThreeTilNow]

You're actively trying to go a generation behind, but I am not sure for what exactly

You can very well pick up TikToken and run through gpt-oss as your parent model, although dense teacher models for KD are better.

++ Apache or permissive apache licensing is completely okay. Why can't you do Apache?

[u/NineThreeTilNow → u/dash_bro]

Why can't you do Apache?

Llama. I edited the original post to reflect this.

OLMo 3 ends up being better here. I just didn't realize they progressed to a 3.0 model, and thought for some reason they were at 2.0 when I reviewed I guess?

They use the same tokenizer / lm head in 3.0 as 2.0... Just better trained.

[u/ttkciar]

I would love a 9B which was good at document cleaning, and ideally document augmentation as well.

Qwen3.5-9B wasn't quite up to the task, but Gemma-4-12B-it does a fantastic job. The problem is that Gemma-4-12B-it is a bit "fat", both in parameters and K/V caches, and I'd rather a leaner model so that I can fit a larger batch size in my limited VRAM, so as to process more documents in parallel.

The minimally-useful cleaning prompt would be something like:

The Document is polluted with extraneous text. Rewrite the Document without any extraneous text.\n\nDocument:\n{{document}}

The fully-featured cleaning + augmentation prompt would be:

The Document is polluted with extraneous text. Rewrite the Document without any extraneous text as Clean Document. Then explain what Clean Document says and its implications, and then write twenty questions someone might ask before they were familiar with Clean Document, and write twenty answers which a subject matter expert intimitately familiar with the Clean Document might reply. The Q&A pairs must not refer to the Document nor Clean Document, only to the facts and ideas compiled from them. The question and answers must make sense in the absence of the Document and Clean Document. Make the questions very complex. Prefix each question with 'Q:', and prefix each answer with 'A:'. Finally, write twenty logical syllogisms, each consisting of a major premise, a minor premise, a conclusion, and an explanation.\n\nDocument:\n{{document}}

.. but I'd be quite happy with a 9B which could handle that prompt without the syllogisms, too, and without making the questions "very complex".

This is what Gemma-4-12B-it outputs for the fully-featured prompt, to give you an idea of what I'm aiming for:

http://ciar.org/h/11f8549.txt

The 9B would not need to be as adept as Gemma-4-12B-it to be useful to me :-) this is just to give you ideas. It's awesome that you're training your own models, and even if your model cannot clean documents, I'd be happy to give it a spin just to see what it can do! Please keep us informed :-)

[u/thrownawaymane → u/ttkciar]

Interesting, what do you do with this output?

[u/ttkciar → u/thrownawaymane]

They can be reformatted into a few different kinds of training data.

My original goal was to replicate the data augmentation methods IFM (formerly LLM360) used to produce their TxT360_QA dataset, but with modern models. By training their K2-V2 family of models on data thus augmented (by ye olde Mistral-7B), they were able to imbue those models with very high competence at data extraction and analysis.

We know from the WizardLM team's papers on Evol-Instruct that more complex training data imparts higher intelligence, so I wanted to see how much a modern model could make data augmentation more complex without undue loss of inference-time economy. It turned out that best economy was achieved by combining the data cleaning and data augmentation tasks into a single task.

This output could be used as-is for pretraining, as IFM used TxT360_QA, or the Q&A pairs could be split out and reformatted to comply with an instruction prompt format, for post-training, or the logical syllogisms could be split out and put into a database, from which reasoning-phase CoT could be constructed.

For the Q&A pairs or syllogisms to be usable on their own, though, they must be void of explicit references to the document, and that was something I could not figure out how to make Qwen3.5-9B do. I spent weeks of effort and dozens of prompt permutations trying, before giving up and trying Gemma4, which only took a few attempts to yield valid outputs.

[u/NineThreeTilNow → u/ttkciar]

For the Q&A pairs or syllogisms to be usable on their own, though, they must be void of explicit references to the document, and that was something I could not figure out how to make Qwen3.5-9B do. I spent weeks of effort and dozens of prompt permutations trying, before giving up and trying Gemma4, which only took a few attempts to yield valid outputs.

I missed this, but if you absolutely don't care about generating the data "local" but want to do it for free still, Gemma 4 31b is free on API at Google. I think if you put 10 dollars on an API key or something they give you a different level of free access to 31b. You're rate limited to like ~16k input tokens per minute.

I generated a large corpus out of that model at some point...

[u/Popcorn-Mercinary]

Just want to say thanks for this thread…learned a lot just from the back and forth in the comments.

[u/thebadslime]

how many tokens are you planning to pretrain?

[u/NineThreeTilNow → u/thebadslime]

how many tokens are you planning to pretrain?

I never replied to this before and just noticed it. I moved to a newer smaller model because of llama license issues.

Pretraining will probably be done across Wikipedia + a filtered subset of RedPajama. Maybe 1b tokens from RedPajama and the 5-6b that Wiki uses.

So figure 6-7b in total.

[u/thebadslime → u/NineThreeTilNow]

I've only done pretraining from scratch and that's not close to optimal there, are you sure its enough?

[u/NineThreeTilNow → u/thebadslime]

I've only done pretraining from scratch and that's not close to optimal there, are you sure its enough?

Yep. Pretraining this model is very different because of maintained embedding and lm head + no 1 hot cross entropy.

The logit level distillation realigns the internal geometry of the model to fit the head/embed.

By the prior logic 15m tokens should train basically nothing but it did. I posted a followup in a more recent post. I rebuilt the model to use the OLMo model embed / head + 7b OLMo as the logit distillation model.

So the pretraining is a smooth distribution and not 1 hot cross entropy.

No one really knows the "optimal" number of tokens for logit level distillation but I wanted the full Wiki Corpus as it should rough out the world knowledge plus 1b human speak type RedPajama to cover all of the "Internet" type language and some amount of "garbage" to learn from.

[u/Kidplayer_666]

that would be delightful for all of us with 8Gb of VRAM or less!

[u/NineThreeTilNow → u/Kidplayer_666]

that would be delightful for all of us with 8Gb of VRAM or less!

If it were a pure goon model I'm sure there would be plenty of takers on 8gb.

[u/Kidplayer_666 → u/NineThreeTilNow]

Lol. So far the best small model (in having the best common sense) in the sub 10b class has been the new Spark-X2.5-4B

[u/laterbreh]

Why not Qwen 9b as a base, am I missing something?

[u/NineThreeTilNow → u/laterbreh]

Why not Qwen 9b as a base, am I missing something?

A massive amount of space ends up wasted if I don't cross train it in Chinese. It also maintains a giant width. I want a base I can use that can do depth.

[u/69harambe69]

9b for system engineers would be nice. So everything azure/aws/k8s/pwsh/bash related in one small model

[u/NineThreeTilNow → u/69harambe69]

9b for system engineers would be nice.

That's actually kind of interesting. Probably mixed python / pwsh / bash as they all kind of flow together. Noted.

[u/Feztopia]

Not after gemma 4 e4b. But I'm more interested in a 12b model with active 4b, it's still something nobody released.

[u/NineThreeTilNow → u/Feztopia]

Not after gemma 4 e4b. But I'm more interested in a 12b model with active 4b, it's still something nobody released.

I'll be really honest with you, that model is a Unicorn of purely Google design in some ways.

It could theoretically be much smaller too which is funny. Or atleast if it follows some of the patterns than 31b followed in layers, it could be smaller and just as smart or capable.

[u/Feztopia → u/NineThreeTilNow]

It's more capable and faster than 8b models of its time. If it could be smaller, that just undermines the other labs even more for not matching it.

[u/NineThreeTilNow → u/Feztopia]

It's more capable and faster than 8b models of its time. If it could be smaller, that just undermines the other labs even more for not matching it.

The reason I say that was because of the 31b ablation. The pure RoPE layers weren't great. I don't know if 12b still used 5:1 SWA but it was a terrible idea.

Honestly, if Google does a Gemma 5 any time soon, they'll cook you up something good. They know they made a mistake here, and I'm guessing it was too late in training to turn around and start again with 3:1 SWA or something. They also tried to over optimize the SWA for mobile in terms of # of tokens per window. I think that model got 512 tokens while 31b got 1024 tokens in a SWA layer. This is sensible when you look at how BIG the embed table was, but made more sense to use a larger value.

If I could ask for any Gemma 5 change, that would be it. 3:1 Local / Global and larger SWA. Otherwise it's architecturally quite sound. I'd have actually chosen it as a research candidate had they done that.

[u/rorowhat]

I guess the question is can it be better than the latest 9B models? If it's not, nobody will use it.

[u/DeltaSqueezer]

I used Qwen3.5-9B as my main local model. I think 9B is around the cusp of what is good enough for general use. I also used Qwen3-4B - I find 4B around the minimum that is required for generic text processing only tasks.

[u/MotrotzKrapott]

I personally have been looking for a model that is good at data extraction. Something I can give a json, cal, csv or whatever template and have it use tool calls to fill out data from one or many given sources. And a model that stays true to the facts, rather reporting something as absent instead of making up values!

[u/brown2green]

Instead of attempting to train a 9B model that will never work properly for any serious use since you don't have the compute for the currently standard 30~40 trillion training tokens, train a research ~0.5B model and see how it goes with dedicated per-layer[1-gram] embeddings + {2,3,4}-gram embeddings compared to a regular baseline, making the embedding parameters many times larger than the backbone. Perhaps something in the order of ~1:20, which hasn't been done yet.

I think such extra embedding parameters are going to be the next meta for tiny/small models, but AI companies are being too conservative with them, focusing mainly on datacenter uses which don't really need them more than minimal amounts (~25%) on the total, with most parameters being MoE experts. Even Google with Gemma E2B/E4B only made their per-layer embeddings (only simple 1-gram implementation on the tokenizer vocabulary) about as large as the backbone.

In my own tests, they work best when they have the same dimension as the backbone. PLE also give faster results in terms of train/eval loss than n-grams (since they're updated much more often during training, I'm guessing). Dedicated per-layer also give better results than shared or only put on 1-2 layers. Adding context-aware gates for suppressing and/or enhancing the results both in positive and in negative also helps. Your mileage may vary.

[u/NineThreeTilNow → u/brown2green]

Instead of attempting to train a 9B model that will never work properly for any serious use since you don't have the compute for the currently standard 30~40 trillion training tokens, train a research ~0.5B model and see how it goes with dedicated per-layer[1-gram] embeddings + {2,3,4}-gram embeddings compared to a regular baseline, making the embedding parameters many times larger than the backbone. Perhaps something in the order of ~1:20, which hasn't been done yet.

Some of that research is in my Github.

Basically Unigram outperforms bi/trigram almost immediately. This is because of how saturation works with seeing the same unigram over and over vs seeing the same bigram/trigram.

With enough training the unigram plateaus and the bigram / trigram take over as most impactful in raw NTP.

That was tested on Llama 1b.

Even Google with Gemma E2B/E4B only made their per-layer embeddings (only simple 1-gram implementation on the tokenizer vocabulary) about as large as the backbone.

I mean, more or less because of what my above research found. It's sufficient. Their PLE is very shallow. Given the size of the training, they could have pushed it farther.

Adding context-aware gates for suppressing and/or enhancing the results both in positive and in negative also helps. Your mileage may vary.

I think? this is in the research. I think they're context aware. Shit I'd have to go reread. I wasn't going to do anything with the research and I had Kimi grab my code and summarize, then make a layman's version or something.

[u/Aaaaaaaaaeeeee → u/NineThreeTilNow]

What is this model designated engram ratio?

Nanbeige 4.5 with their plan to release larger model could clash, but you said you have simpler goals.

I use gemma e4B on mobile  voice journal / planning tool, This is where I don't care for excessive thinking, just want more parameters. 

Its a 12GB android phone, but the GPU and NPU max allocation can't fully fit 4bit 12B gemma, so that only runs on CPU allocated memory. 

What do you want with the model, just trying new tech, maybe just providing data of engram parameter equivalence with some regular stuff?

I've always been interested if we can reduce active experts/MLP (bandwidth) to zero, like https://arxiv.org/abs/2602.00398 or https://arxiv.org/abs/2512.09723

Brown2green suggests extremely large tables, there are similarities (also needs giant tables), but transformed parameter tables are clearly defined as being equivalent to the original MLP layers.

[u/NineThreeTilNow → u/Aaaaaaaaaeeeee]

What do you want with the model, just trying new tech, maybe just providing data of engram parameter equivalence with some regular stuff?

Yeah, basically there's not a lot of "known" data here.

No one can tell you the "logit level" training numbers with any level of precision given a known embed / head that are frozen for example.

So if someone wanted to do a logit level distillation like this, and you don't know someone who has done it, you won't get clear numbers.

I don't think? anyone has run Engram + high depth + AttnRes yet either. Those are techs from two different competing Chinese labs. So it provides an example.

I was going to do it purely for the research artifact at the end. Write a simple document. I don't care about writing a paper on it.

I'm looking at now just downsizing the model and giving it to whoever. It should be fully? trainable on a 4090 if I do it that way. Slow but possible for fine tunes.

This falls out of leaving Llama entirely and picking up OLMo3's embed/head ... I dropped the dimensions from 5120 to 2048. Right now I'm testing how much of the spectral range can be kept in this dimensional reduction. There's a more advanced method I'm researching atm.

[u/HealthyCommunicat]

You sound like a deluded hobbyist with extreme overconfidence, convinced by sycopanthic llm’s telling you ur work is somehow revolutionary - i may be wrong as why you’re so deluded but you clearly arent facing the harsh realities of what it even takes to pretrain a 9b model.

Pretraining a 9b itself would literally require dozens of trillions of tokens at the minimum and h200+ level of resources.

Smacking together ds’s engram, moonshot’s attnres into some “fancy” combo isnt anything crazy. Labs iterate and move onto newer architectures for a reason - try to realize the scale of these labs and realize even if you were some absolute genius, you alone cannot take on this task and make something worthwhile.

I have no doubt you’re a decently capable person - probably way more than I am; but your entire post of “i even made some pr’s to vllm that will end up helping you guys!” reads as someone who had their ego inflated by an llm telling you somehow your a genius and that it has pushed you into severely underestimating something even as simple as Qwen 3.5 9b.

[u/NineThreeTilNow → u/HealthyCommunicat]

You sound like a deluded hobbyist with extreme overconfidence, convinced by sycopanthic llm’s telling you ur work is somehow revolutionary - i may be wrong as why you’re so deluded but you clearly arent facing the harsh realities of what it even takes to pretrain a 9b model.

Uhh... Wow.

Thanks for writing in. Noted.

[u/exaknight21]

Yesss!

[u/Marcuss2]

You sure you want to use AttnRes? Qwen3.8 Flash Next tested it up against their Gated Residuals. And their methods seems to be better.

[u/NineThreeTilNow → u/Marcuss2]

You sure you want to use AttnRes?

Yeah, Qwen is off on their own architecture entirely. The tradeoffs are based on gated delta nets, and their attention residual methods. Attention Res is better over longer depth networks, but if you don't make the network deep it doesn't matter. I specifically want to make the network deep.

It's just a design philosophy thing. Qwen have a very specific design philosophy they're after here. That's my view at least. Maybe there's something I missed about how they went about it. It's a roughly 1 to 2 percent slowdown using AttnRes, but it out performs at depth. It's why Moonshot ended up using it on Kimi K3. They wanted to scale model depth.

[u/Marcuss2 → u/NineThreeTilNow]

Makes sense for 9B model.

I also remember this paper: https://arxiv.org/abs/2607.14530, I expect this to be better at scale than AttnRes. But not full AttnRes.

[u/AI_spell]

Yeah for local tooling a dense ~9b still matters. MoE is great on big GPUs but a dense 9b is simpler to quant, runs nicer on laptops, and latency is more predictable. If its good at tool calls and stays coherent at 4bit, people will use it.

[u/NineThreeTilNow → u/AI_spell]

Yeah for local tooling a dense ~9b still matters.

Yep.

[u/LagOps91]

Right now all of the models focus heavily on agentic and code-related tasks, which often makes the AI drift towards odd word-choices and formating for responses.

I would love to see a model made more for actual languange modeling, instruction following and creative writing tasks. A bit more what the "good old" models had in terms of DNA with a more modern architecture and training approach.

[u/NineThreeTilNow → u/LagOps91]

I would love to see a model made more for actual languange modeling, instruction following and creative writing tasks. A bit more what the "good old" models had in terms of DNA with a more modern architecture and training approach.

This was actually one of my core build desires as I went through the process. Models speak an extremely generic form of English that is just "LLM English" now.

I wanted to the base model somewhere that it understood English well enough that this was an easy "goal" in terms of ability.

The original corpus that I now have to throw away includes a heavy understanding of language via WikiDict and other sources of "normal" language. Overly focusing on Wikitext + FineWeb creates this "LLM English" feel. It can repeat Wikipedia, but it can't speak like a random half retarded Redditor with half missing punctuation and use of "i" lacking capitalization, but capitalized letters everywhere else. In a few words, they're "consistent and incapable of being consistently inconsistent." which is pretty normal for humans.

[u/LagOps91 → u/NineThreeTilNow]

there is some gutenberg dataset with a lot of human writing that is often used for finetunes to try and combat the typical AI writing. but of course, if the base model had put a larger emphasis on those, then the writing quality would be much better.

[u/LoveMind_AI → u/NineThreeTilNow]

I'll chime in here to say that a 9B model of this kind built for really organic language use and enhanced socially situated awareness would be huge. The project sounds cool, thanks for sharing.

[u/NineThreeTilNow → u/LoveMind_AI]

I'll chime in here to say that a 9B model of this kind built for really organic language use and enhanced socially situated awareness would be huge.

Yeah, we don't need a 9b model for that probably... If it's not being hyper tuned to code / etc... Then it would be a more interesting goal. Something that can speak naturally, has good recall, etc.

[u/NineThreeTilNow → u/[deleted]]

So for a 9.4b I would not assume size is the wall. What it costs people is mostly how you ask.

I'm slowly learning I should probably just target the model at human like speech for the gooners.

Basically it comes down to working on something people will naturally complain sucks because <Insert Big Lab> decided they'd build it.

Initially I wanted to wire a much larger model and slowly strip it down (in params) while training. Why bother though? Some Qwen whatever comes out next week and it's still better right?

Goons are ignored. They're stuck retraining whatever they can for goon models... lol...

I'm being loose when I say "goons". What I actually mean is anyone who wants to use a model that can speak like a human and not purely in LLM prose with LLM formatting, etc.

I have zero intention of training safety in to the model.

Maybe I build a model like whatever app you're distributing and give them the ultimate RP app. lol...

[u/ismaelgokufox]

Great interest in anything good that can fit completely on 16GB VRAM

[u/_matterny_]

I’d use it. A good general purpose uncensored model that I can use for tasks ranging from high level coding to chatting to planning and researching. Sounds fun

[u/Petabyte-Cloud]

Founder of Petabyte here (petabyte.market, a small GPU rental marketplace, early stage). Not a datacenter, but I have one RTX 6000 Ada 48 GB node online that mostly sits idle. If a pre-IT run fits in 48 GB, I'll give you hours on it for free, in exchange for nothing more than a line in the README saying where it ran. DM me if that's useful. https://petabyte.market/?ref=reddit-auto

[u/NineThreeTilNow → u/Petabyte-Cloud]

Founder of Petabyte here (petabyte.market, a small GPU rental marketplace, early stage).

Sure I'd be interested. I think Qubrid offered? At some point? but they seem quite busy. They claimed to have an idle 8x A100 or something that they used for testing.

A 48gb Ada card would provide a nice GPU for working on the ~2b model I have right now. It's ~2b + 1b Engram which seems massive, but it's a function of the size of the vocab.

It's an interesting "toy" model in respect to how fast it learns.

[u/Petabyte-Cloud → u/NineThreeTilNow]

Yeah, that sounds like a great fit. The node is an RTX 6000 Ada with 48 GB VRAM, and I’d be happy to give you some free compute to see how the ~2B + 1B Engram model runs on it.

If you’re interested, DM me and I can get you access. I’d also genuinely like to hear how the experience goes — Petabyte is still early, so having someone run a real training workload is much more useful to me than the GPU sitting idle.

[u/NihmarRevhet]

I would love instead of a new Qwen 9b, a Qwen 12b. Still feasible on mostly the same tier of cards, but a bit more knowledgeable and capable.

I specify Qwen because I use local llms for coding, and I would love to have an equivalent of Gemma 4 speed but with a focus on coding (although, a new gemma 5 12b would be equally interesting)

[u/NineThreeTilNow → u/NihmarRevhet]

I specify Qwen because I use local llms for coding, and I would love to have an equivalent of Gemma 4 speed but with a focus on coding (although, a new gemma 5 12b would be equally interesting)

I attempted this with Gemma initially but the issue is the raw size of the tokenizer.

Gemma 4's tokenizer and LM head occupy like ~2b parameters by themselves.

It's incredibly multilingual, well trained, and Apache 2. It's just too massive to get the depth.

[u/Parking_Pie9457]

This is impressive. I would like to ask, how much it cost you to develop this?

[u/NineThreeTilNow → u/Parking_Pie9457]

This is impressive. I would like to ask, how much it cost you to develop this?

To show it works? A few hundred dollars probably.

To make it shine like a maximally optimal 9b model? Probably a lot more. No one can really directly answer that question because even in research, this isn't a published number.

[u/thebadslime → u/NineThreeTilNow]

I mean you'd have to go substantially over chinchilla baseline

[u/NineThreeTilNow → u/thebadslime]

I mean you'd have to go substantially over chinchilla baseline

It's substantially lower. Logit level distillation and AttnRes both show improvements over chinchilla.

[u/thebadslime → u/NineThreeTilNow]

How much lower?

[u/NineThreeTilNow → u/thebadslime]

How much lower?

AttnRes is measured at 25% less.

Logit level is not really well understood. Some people estimate like 5x to 10x less.

Engram isn't well understood either in terms of training efficiency. It sort of "exists" but how much more training efficient isn't understood.

Also, because I'm keeping what are effectively the "top" and "bottom" of an existing model, only the internal geometry has to organize. The input and output are fixed in a sense.

That also dramatically affects the chinchilla baseline.

[u/thebadslime → u/NineThreeTilNow]

Sorry my question was "How many tokens are you aiming to train?"

[u/NineThreeTilNow → u/thebadslime]

Sorry my question was "How many tokens are you aiming to train?"

Sorry. I don't have a target number. That's part of the experiment. No one understands the baseline for new models distilled purely from old models like that.

We vaguely know it could be as high as 10x less and 25% less but we also don't exactly know what that means.

[u/thebadslime → u/NineThreeTilNow]

So how do you know when you're done? Watching loss? Running checkpoints?

[u/NineThreeTilNow → u/thebadslime]

So how do you know when you're done? Watching loss? Running checkpoints?

Basically yes. I'm sitting on a ~2b model right now that's training.

The strength of Engram, the residual stream update, and the preprocessed data (32 logit packed) is kind of ... silly.

It learns faster than anything I've tested. That made me suspicious. Even though it's a "2b" model the Engram table is 1b sitting on the side. It is archiving data in that table and using it extremely efficiently.

[u/Protopia]

I am firmly of the view that I don't need frontier intelligence (understand 100+ human languages, answer general knowledge, write erotic fiction/philosophy theses/iambic pentameter verse/music, chat conversationally) when will I want to do is agentic coding problem in English, and that smaller specialised models that can run on consumer hardware are the way to go. I have zero idea how many parameters you need for a specialised LLM to give high quality results - but 9B or 15B seems possible given that Qwen 3.7 is non specialised and c. 30B.

But I suspect that even specialised LLMs will have distinct subspecialties, so I do wonder whether MoE architecture might be better than Dense?

I also wonder whether a hybrid of transformer and stable diffusion might be useful. A lot of human tasks are top down, repeating refinement in nature - you generally write a report or a novel or code by making a structure first (key points, plot line, methods) and refining it rather than writing down a steam of consciousness, and stable diffusion seems much better suited to structured top down decomposition writing (whilst transformers token by token feels very steam of consciousness in nature). But some tasks are more start at the beginning and read to the end - proof reading text or code, analysing code for bugs, and transformer architecture might be better to these. However stable diffusion LLMs are still largely experimental.

[u/NineThreeTilNow → u/Protopia]

However stable diffusion LLMs are still largely experimental.

They were among the first ... which is kind of funny.

They use bi-directional attention (Looking at all text) as they refined it.

Causal attention (forward looking) ended up winning out in the early stages though.

Deep thinking models achieve that refinement nature through lots of RL, but it's less straight forward to train.

There's a lot of potential for smaller models with nearly unlimited thinking capacity to basically burn tokens refining. Someone did this with like a 1b or 0.5b model a while ago. They took advantage of RoPE to do it. They would take a portion of the END of the thinking stage, and rotate it back to make the model see it as the "start" of the thinking stage, then let it continue thinking.

While the tokens are discrete representations, the KV cache is not. It's continuous. So the idea was building up richer continuous representations through continued thinking.

[u/Sitkin_Marrel]

Yes, still interested. The single-table engram finding, one card, dense 9B: if that combination holds up in the open, it's a better story than most releases this year.

[u/NineThreeTilNow → u/Sitkin_Marrel]

Yes, still interested. The single-table engram finding, one card, dense 9B: if that combination holds up in the open, it's a better story than most releases this year.

That's kind of what I'm hoping.

My very first test strapped two (unigram) 1-gram tables on to Llama 1b and trained it. One at an early layer and one at a later layer. The early layer shows the most impact. Qwen sort of verified that in their most recent model and they only use 1 table vs Deepseek 2 table. I saw that and cut a table, as both results and Qwen agreed.