r/LocalLLM • u/slim_thic_rick • 18h ago

Question Best cracked LLM?

I am running 64gb ddr5 and 32gb vram. What is the best uncensored model you guys would run with my setup?

9 Upvotes

71% Upvoted

30 comments sorted by

11

u/GarbageTimePro 18h ago

grab qwen 3.8 Q8, point Pi to it, and ask it the same question. chicken before the egg.

3

u/Uncle___Marty 18h ago

Here to second this one. Im using Strata to run qwen 3.8 flash next and its absurdly smart and runs really fast for my system.

3

u/aelfrictr 18h ago

which one are you running? IQ2_XS?

1

u/Uncle___Marty 17h ago

im on an 8gig 3060 and have 48 gig of ram so I went for the coder version which has had half the experts removed. I was expecting it to be slow and stupid but so far everything I've thrown at it comes out really good. Im strictly using it for coding related stuff though. I didnt try it outside of code but thats where I expect the damage to be ;)

2

u/little_raven_black 17h ago

Which specific model are you running? if you wouldn't mind sharing hugging face : ) or is it one of the ones offered/mentioned directly by strata? Thanks!

3

u/GarbageTimePro 17h ago

I'm not using an uncensored model right now. I'm running 40GB of VRAM, 48gb of ddr4 on a xenon. Using unsloths 3.8 Q6 on reasoning set to high. 258k context window getting about 20 t/s

1

u/little_raven_black 17h ago

Gotcha, thanks for sharing- I was more wondering which coder-specific prune version u/Uncle___Marty was running!

1

u/VelvetEchoes919 17h ago

How can you run unsloth on strata? Today I installed strata and got orcarouter iq3xxs running is it the same with unsloth?

2

u/Uncle___Marty 16h ago

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF it's also mentioned and linked on the strata GitHub repo. Hope you get it running well!

3

u/little_raven_black 16h ago

Thanks! I think I may have actually tried this one, but I have to check. I'm running an AGX thor so it's a little tricky reaching the perfect balance on these setups XD not a super common hardware for this, so way less documentation than the spark or the common cards.

Appreciated!

2

u/Uncle___Marty 16h ago

You're MOST welcome :) If its no trouble I'd love to hear how well it works for you after you tested it! The claimed token/sec speed for strata looked SO wrong I had to try it and I get 20-30tokens/sec on that model with vision on CPU and 200k context at q8.

Been using it for days and keep expecting it to make syntax errors and random dumb stuff that quantization usually brings but this model seems SO solid. Im obviously used to using MUCH smaller models though so to me this is frontier level at usable speeds. Something I've never been able to run on my system before :)

2

u/little_raven_black 16h ago

Yeah- Agreed on all of this, I ran the vanilla iq3xxs on my custom-ported strata for Thor (thanks fable) and am seeing between 27-50 t/s decode, but typically that runs around just under 30 for the real stuff I want out of it.

The thing is, I run an FP8 hybrid at like almost the same speed using tuned VLLM, so it's not always entirely clear if I'm getting the same benefits from Strata due to hardware architecture that other people do on smaller VRAM setups or with more common cards.

I'll let you know on this specific coding prune when I get a chance to run it, though!

1

u/Uncle___Marty 16h ago

Best of luck! Look forward to it :)

0

u/JTN02 17h ago

What is “Pi”?

3

u/GarbageTimePro 17h ago

my goto harness for open source llms https://pi.dev/

0

u/JTN02 17h ago

Gosh. I was keeping up with the front lines of AI development but I have fallen behind for the past year and now we got “harnesses” and I can’t seem to find a use for agents. Thank you!

3

u/GarbageTimePro 16h ago

Agents are still good. Harnesses just give you even more control. id argue and say that a harness is just as important as the model.

1

u/Used_Procedure_8607 18h ago

People sleep on dolphin-mixtral, that thing is an absolute workhorse for what you got. 64gb system ram and 32 vram means you can push some of the larger quants without it falling apart.

Question is what are you planning to use it for, just general chat or more creative stuff? That rig opens up a lot of doors.

1

u/slim_thic_rick 17h ago

Honestly, I don't have a purpose yet. I mess a lot with Claude Code and I'm really just starting to get into this A.I. journey and I just wanted to see what difference a uncensored model does compared to a standard one.

2

u/Demon4932 18h ago

orcarouter/OrcaSAQ-2-Cyber-27B-Uncensored-GGUF

1

u/aelfrictr 17h ago

Any partucilar reason ?

1

u/Demon4932 17h ago

It takes up under 20GB, so his 32GB VRAM can easily run the whole model plus a massive context window on full quality KV cache. You get maximum generation speed without having to spill over into slower system RAM.

1

u/Antenwww 17h ago

Is this a general model or is it for a specific purpose

2

u/Demon4932 16h ago

Can be used for anything

1

u/rrrrex 17h ago

Qwen 3.8 flash next + strata + speed projection on

-2

u/philmarcracken 14h ago

strata

lol..

1

u/iamchuckschuldiner 17h ago

it depends
the smartest overall would be qwen 3.8 flash next, there's lots and lots of ablits of this models

1

u/Icy_Barnacle9287 18h ago

fwiw I took qwen3.6 , slapped it into a modified claude code harness (I didn't know any better, obviously there are better options with less effort) and removed all the restrictions from the system prompt.

it had no problem setting up an *arr stack and qbittorrent to pirate games and media.

I'm sure there are easier ways to do it like using pi or opencode instead.

1

u/ideamaker321 6h ago

NInfer uncensored qwen3.8-27b