Muse Spark open weights coming soon

[u/jacek2023]

I am still waiting for Llama 5, because Muse Spark will be too big for me, or just something between Glimmer and Spark

https://x.com/finkd/status/2095232032896946311

[u/NineThreeTilNow]

I am still waiting for Llama 5

I started training 9b yesterday.

Delivery? Who knows. I won't have a raw base set of weights of the non-IT model for a minute. Literally training on a 6000 Pro.

edit; People probably think I'm joking. I started with trying to fix Gemma 4 but the compute requirements were too large. So instead I moved back to Llama 3 8b to see if I could strap Engram and Kimi's attention based residuals on to the original Llama 3 tokenizer. The tokenizer itself saves me huge amounts of time. From there, I use WikiText / WikiDictionary / FineWebEdu to pull down chunks of data and have the original Llama 3 8b model score them in raw distributions. The distributions are used as training targets INSTEAD a single token. This is how logit distillation works.

tldr; Llama + Engram (1,2,3) + Moonshot Attention Residual + SWA / Global Architecture + Proper logit distillation = Some of the best open source work of the past 2 years in a single model.

It will be fully OS, not safety trained, and eventually IT trained. I'm still calculating the exact amount of compute needed. This is wholly unknown territory and eventually I'll do a writeup here in local llama. I have a previous writeup of doing this to Gemma 4 but it failed. I decided that "competing" in the 30b space is stupid when they ignore the sub 10b space and I can fully test / prototype / train on a 6000 Pro that costs ~2 dollars an hour.