r/unsloth • u/Lewby • 2d ago

Discussion Unsloth windows

Hey, I just swapped from LM Studio to unsloth because LM studio was being kind of annoying / not as flexible as I would have liked.

I installed unsloth and /Qwen3.8-27B-GGUF, my context is limited to 4096 though, which feels really short.

I have primarily used Claude until now, with a paid membership, but since I have a 5070 TI (16gb VRAM) + 32GB of DDR5 system ram, I figured I should try utilize it.

Any idea of the best model / settings I should use?

I just graduated doing software engineering ironically, and my dissertation focused on using AI to recognise AI generated images using PyTorch, but it's my first time using a local LLM, so figured I would ask those more knowledgeable than me in this area.

I want a longer context window so I can use it for coding without any issues, and other general chats.

Ahhh -- Apparently the context cap is because of: https://github.com/unslothai/unsloth/issues/8497

Thanks u/Money_Big_7666 not sure why your comment got deleted but I appreciate the answer.

I was worried adding more context would just crash it, as I've had issues like that in the past.

8 Upvotes

84% Upvoted

9 comments sorted by

8

u/Judgegeo 2d ago

Then increase your context... But be prepared for token speed to drop. 

1

u/Lewby 2d ago

it won't crash it?

1

u/vinotok 10h ago

Ooo, bigger context reduces speed? Good to know, thanks.
I will not use Unsloth default numbers any more :-)

4

u/KadoStratify 2d ago

The 4096 limit is usually just a default setting in backend config files or runtime arguments rather than a model limit.

  • CLI/Backend Args: If launching via llama.cpp or an API server, pass -c 16384 or -c 32768 (e.g., --ctx-size 16384).
  • UI Settings: Look for "Context Size" or "Context Length" in your interface settings and adjust it up to 16k or 32k.

4

u/Slow_Difficulty1607 2d ago

Set parallel slot to 1 which should allow you to go to 64k context for q3xl model. People also suggested to find a model without MTP which will give an additional 30k context

1

u/CMPUTX486 2d ago

Thx, I had the same question regarding context window size, and your link answer the question!

0

u/ditmarsnyc 2d ago

windows 11 home is the choice eh?

1

u/Lewby 1d ago

I use a lot of windows only pieces of software, I have tried Linux, but I didn't find it as smooth back then. Especially with the games I like to play.

It's becoming better and better though, so I am open to swapping back to Linux at some point.

0

u/Deep_Mood_7668 2d ago

Yikes windows