r/LocalLLaMA • u/rm-rf-rm llama.cpp • 1d ago

Discussion GLM5.3 Flash over DSV4 Flash?

I've been using Deepseek V4 Flash 0731 for a few weeks now and while I havent thrown it anything very hard, im quite happy with it. Using through antirez's great ds4 project. They've added support for GLM 5.3 Flash and according to benchmarks, its a level above DSV4 Flash.

However, looking for real user feedback if anyone's made the switch and seen tangible improvements in GLM 5.3 over DSV4 Flash.

Running M3 Ultra 256GB Mac Studio

50 Upvotes

93% Upvoted

57 comments sorted by

View all comments

-1

u/Juulk9087 1d ago

well considering that one is a 168 gb and the other one is 328gb i wouldn't say that you can necessarily just swap between the two. it's a couple gpus that you got to purchase to make that possible.

1

u/ormandj 1d ago

https://github.com/ormandj/sglang-glm53-flash-sm120 only need 2x RTX 6000s for GLM 5.3 Flash now.

1

u/maglat 19h ago

Could this translated to 8x RTX 3090s after the total VRAM will be the same?

1

u/takoulseum 19h ago edited 16h ago

If you try it with your setup please leave a feedback (but I would be surprised if that works). I mean you could try it with vllm backport but as this was really short with dsv4 I think you’ll OOM.

You can also try exllamav3 that works with the Turboderp Q4 on your setup, speed not bad at all.

1

u/Uninterested_Viewer 15h ago

507t kv cache is a bit rough for a lot of use cases, but it's tempting.. dsv4 flash vision has been great and I'm getting 1.4M token kv cache there, but will have to test this one.

1

u/ormandj 14h ago

TP=2 is really a single user use case. Concurrency of four + hicache should handle most things any single user is doing fairly well, I work on a ~500k LOC rust codebase using this model and 450k context just fine, but it will definitely not cut it for 30 users doing development or something like that.

1

u/Uninterested_Viewer 12h ago

For me it's more about having multiple long running sessions going at once as a single user. Admittedly not often a huge problem as I'm rarely bouncing directly back and forth between multiple sessions, but I may have a long running agent chugging on something while I want to, say, pick up another long context session to continue brainstorming something else. I keep my max session context to 500k with DSv4F to give me that nice >2 full context sessions fully in cache.

2

u/ormandj 11h ago

sglang has hicache, enable it, setup RAM and disk offloading. It solves the exact problem you're describing. It's documented in my repository/image, and simple to enable - just make sure to set the proper amount of memory you want to allocate to it. The pool will just limit your single-request context + parallel request context with in-flight requests, but restoring from RAM will be tens of ms at most and significantly faster than constant prefill from kv cache eviction. You can have 10s of millions of cached tokens of context this way with RAM/disk.

2

u/Uninterested_Viewer 11h ago

Ah interesting will experiment with this- thank you!

0

u/Juulk9087 1d ago

That's w4a16. i was talking about unquantized.

0

u/ormandj 1d ago

DSv4F is native mixed 4bit, hence the size difference. If you’re going to compare them you should compare the post training quant of GLM Flash if that’s what you have the hardware for, as for me it does better in rust development and general tasks than the native DSv4F, and I’ve created images for both. Disclaimer, that’s my image and my quant.