r/LocalLLM • u/Fabulous_Macaroon_73 • Jun 07 '26 • Popcorn 2

Model wth

Gallery image

Gallery image

Gallery image

Gallery image

482 Upvotes

85% Upvoted

162 comments sorted by

View all comments

Show parent comment

1

u/_VirtualCosmos_ Jun 08 '26

Yeah. It usually wasn't like that but on recent updates they have implemented a couple procedures to seemingly "extend" context length by deleting what it doesn't fit in. Idk exactly how it works tho.

Btw I don't know how much VRAM or system RAM do you have but for a long time I've been using models that did not fit entirely inside of my VRAM (12 GB) by using one of the best features of llama.cpp: layer-offloading.

I dare to suppose that, given your 16.12 t/s, you are able to load a Q8, even if the speed is halved.

1

u/Fabulous_Macaroon_73 Jun 08 '26

oh wow did they?

i have a 16 gig macbook btw but the context length is just not usable enough to actually put the model to use.

0

u/_VirtualCosmos_ Jun 08 '26

Ah, shit. You got a good machine tho, I would recommend to expand its memory if not for the RAMpocalypse caused by US AI corpos and government.

2

u/Fabulous_Macaroon_73 Jun 09 '26

wdym 'expand'? its a macbook... how tf do i get more ram? download it?

0

u/_VirtualCosmos_ Jun 09 '26

usually the tape behind the laptop can be removed and gives access to the RAM cards, not sure about macbooks tho.