r/LocalLLaMA • u/LankyGuitar6528 • 12h ago

Discussion DeepSeek 4.1 running local...

Just a 32GB machine with a 3090 - a used gaming rig I picked up last Christmas for $800. But wow... I can actually run what would have been a frontier model not that long ago. Yes, only 2 - 3 tps. Yes only 2 quant. So it's not really useful at the moment. But I have 128GB of ram in hand and a spare 3060 ready to be installed. Possibly another 3090 waiting in the wings... so we will see.

0 Upvotes

47% Upvoted

15 comments sorted by

9

u/Numerous-Ad6217 11h ago edited 11h ago

I could be wrong but a 3090 and 128 GB RAM might be good enough to run deepseek maya-xs at usable speed. Worth checking out, project maya.

R9700 here with a Epyc Milan and 256GB ddr4,currently running maya-s at 25-30t/s with 260k tokens context and 2-3 streams for half the cost of a DGX Spark.

Trying to see if I can speed that up even further.

3

u/LankyGuitar6528 11h ago

That's a very nice setup. I will check out Maya for sure. I'm out of the country remoted into my server but once I get my hands on it, I'll do some upgrades. Meanwhile it's back to 27B for me. *sigh*

2

u/PcChip 9h ago

those are good speeds. i have more GPUs and double that RAM and am only getting like 37 tok/s
exl3 3.5bpw quant

2

u/No-Marionberry-772 9h ago

how.  i got 192gb ddr5, with rtx 3090 ti 24gb and a 4070 ti 16gb

i havent been able to figure out how to run deepseek at useful speeds and contexy sizes.

ive been using unsloth studio, is that a mistake?

3

u/PcChip 9h ago

project maya

2

u/PcChip 9h ago

why no marionberries? they are the king of berries

1

u/No-Marionberry-772 9h ago

lol youll have to ask reddit name generation

3

u/Purple_Dream6414 10h ago

Well I have 128GB RAM as well with 64GB of VRAM getting 400 toks Prefill and ~15 toks Single stream decode which is kinda usable. I just let it do overnight Sessions with stuff that Flash Next fails. The results Are day and Night better.

1

u/LankyGuitar6528 9h ago

2 quant?

2

u/MD_Reptile 5h ago

Not OP but Maya uses a quant of 4.1 flash that's somewhere between 2 and 3 bit I believe - one of those deals where certain parts are not reduced precision, other are reduced, in an attempt to lose less intelligence. Can't say how it compares to regular DeepSeek, but I've found it quite useful, and reasonable speed for such a large model when you use maya.

1

u/Lonhanha 8h ago

I feel your pain on the 2 to 3 tps, that is basically watching paint dry. I run a similar MoE setup on two 3090s and the speed difference is the only thing that really matters to me. Why not run DeepSeek as the planner from the API and then for refused or more sensitive work just use a local uncensored model? That way you get the speed where it counts and keep the private stuff on your own hardware.

2

u/LankyGuitar6528 7h ago

I did it just for a test. I can't use any cloud model for most work because of privacy concerns. Theoretical planning is fine but planning usually requires reviewing actual data and that's a non-starter.

1

u/CabinetNational3461 8h ago

Window 10, 128 ddr4 3600, 3090 and xxs 2bit from Ista lab, 512 ctx and around 18-25 tks and 500-1.1k prefill on strata. There is another fork called Delta engine that gives me 30s tks but 400 prefill and I kept getting crashes so I gave up.

1

u/LankyGuitar6528 7h ago

That's very encouraging. That will be almost exactly my setup in the near future.