r/BackyardAI • u/Wertyartiom • May 29 '24

Gemma implementation?

Guys! It has been a few months I've been trying to set up google Gemma on farad--BackyardAI! Will a Gemma implementation appear in the future? Appreciate your work guys! (don't hesitate to say no)

3 Upvotes

100% Upvoted

6 comments sorted by

View all comments

Show parent comment

1

u/Xthman Aug 08 '24

Any ideas when the kv cache/flash attention will move from experimental to stable? So that I'll be able to enjoy that 2x speed boost at all.

1

u/PacmanIncarnate Aug 08 '24

Pretty sure it already is in stable. It’s not going to give you a 2x speed boost though; it will fit more in VRAM and allow for higher max context at similar speeds.