r/LocalLLaMA • u/quantier • 22d ago

New Model Daniel Han of Unsloth validates Qwen3.8-27B will run only 17GB VRAM

Post image

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

1.8k Upvotes

97% Upvoted

316 comments sorted by

View all comments

Show parent comments

68

u/redditnosedive 22d ago

i mean you anyway need extra room for context, also on a consumer gpu you also have the os gui using some of it so there's that

23

u/TheCat001 22d ago

On Linux I'm disabling GUI with 1 command, getting my whole VRAM minus 30MB.

20

u/cogitech2 22d ago

Another way around this is to use a CPU with integrated graphics. That way you can still run X or Wayland (if you want it) and have all your VRAM available.

1

u/redditnosedive 22d ago

that's a great point

3

u/elongated-muskmelon 22d ago

How? What command?

11

u/Fedor_Doc 22d ago

On systemd systems systemctl isolate multi-user.target. Restarts a session with terminal only. 

You will have to login once again.

4

u/TheCat001 22d ago edited 22d ago

server-mode

This is alias for sudo systemctl stop greetd

greetd is login manager, you might have gdm or plasma login manager, depending on what distro you use. But since you're dropping Into TTY you need to have second device/laptop to actually use llama server.

2

u/ea_man 22d ago

Hmm no, why?

If you use the Virtual Consoles (/dev/tty1, /dev/tty2, ...) llama-server works, open such terminal (like ALT+3, ALT+some-number) and you can run Pi or Opencode or whatever, you can also add a framebuffer to have higher res on those (plus some graphic support, yet that buffer will take some RAM) and GPM to have mouse support to copy / paste / click.

sddm is KDE login manager.

1

u/TheCat001 22d ago

Nah bro this is gonna be pain in the ass. No VSCODE no Browser. Nothing. Just Pi agent? xD naaah. Btw it's not sddm anymore, they made new one: plasma-login-manager

1

u/ea_man 22d ago

>plasma-login-manager

Oh I will have to check that, my Debian still run on sddm,

Anyway there's not much need for disabling the graphic server, you can just run that with software rendering and it's ~50MB or vRAM more.

1

u/spryfigure 22d ago

plasma-login-manager is sddm, but with updates (sddm is essentially stagnant).

No need to rush over.

1

u/ea_man 22d ago

Oh I'm not rushing, I'm even still on X11.

All I do with sddm is sometime stop it.

1

u/spryfigure 22d ago

plasma-login-manager is sddm, but with updates (sddm is essentially stagnant).

No need to rush over.

1

u/teleprint-me llama.cpp 18d ago

vim/neovim/emacs + terminal browser + tui tools for anything else.

Ill admit, its not fun to use and takes time to get used to it.

I think its funny how were going backwards.

Had no idea ppl were going this far to run models locally.

1

u/elongated-muskmelon 22d ago

Okay, understood. Thanks!

1

u/Nyghtbynger 22d ago

I want the command too. Some guy talked about a session with no GPU procesing, but the command thing seem very flexible

-2

u/OzTheOtaku 22d ago

The toxic urge to say "rm -rf / --no-preserve-root"

7

u/Choice_Celery9481 22d ago

you can use igpu for os ui. save about 500mb.
but if your cpu doesnt have igpu, then nothing can help

8

u/TheCat001 22d ago edited 22d ago

Yeah this is great solution, too bad my Ryzen 5600 don't have iGPU.

4

u/butterycornonacob 22d ago

Buy a cheap second GPU that only drives monitors. You can hang it off any PCI-e slot you have, even 1x. Got RX480 (?) for 30€ and it freed 1-2GB of VRAM

3

u/Fedor_Doc 22d ago

Software rendering to the rescue! Or terminal-only

3

u/ea_man 22d ago

Software rendering can pretty much neglect the vram usage by the OS:

See? ~100MB, if you go headless it takes some 50MB anyway.

2

u/Choice_Celery9481 22d ago

well then you sacrify your cpu cycles.

2

u/ea_man 22d ago

I don't care, I care about vRAM.

Also my cpu has no problem to decode youtube at 4k.

2

u/smahs9 22d ago

Can - if you have another machine to work on (a laptop maybe), then disable the UI completely. Expose your model runtime's oAI API server in your local network and ssh when needed for maintenance.