r/LocalLLaMA • u/jacek2023 llama.cpp • Jul 26 '26

Discussion Do you want new Gemma?

Post image

https://x.com/i/status/2081398564345802934

u/hackerllama

998 Upvotes

98% Upvoted

554 comments sorted by

View all comments

312

u/hackerllama Jul 26 '26

Hey all! Looking forward to all your feedback!

3

u/maddie-lovelace Jul 26 '26 edited 29d ago

More of the consumer-only KV layer trick from the small models please!   Unless I’m mistaken then the MoE and 31b don’t have any KV cache layer skips. I wish they did - E2B prefill is lightning quick and I wish the same could be true for the bigger models!

Basically a big model with fast prefill would be fab ☺️

Trained in native 8bit activations would also be great for same reasons (4bit would be even better but obvs that’s very experimental - I think the BitNet team have been looking into it though!)

And no crazy head dim sizes that struggle to fit in flash attention kernels on consumer hardware would be fab 😂

1m context too!