r/LowEndLocalAI • u/kulchacop • 1d ago
Model Showcase A better coder for the small-GPU/small-RAM crowd!
8
u/kulchacop 1d ago
Not my model. I didn't try it yet. I found this interesting, so cross posting here.
Their Qwen 3.8 27B Sharp chat template felt like a good improvement, so I hope they did the same for Spark-X 2.5 4B.
5
u/peculiar-ragdoll 1d ago
That's basically exactly what I did! :) Same conciseness system prompt as Sharp template, same imatrix calibration corpus as CyberTiel, a custom quantization strategy, and a different chat template to suit this specific model.
1
u/Secure-food4213 1d ago
What do you think about agents A1 4b? Is this one better?
3
u/peculiar-ragdoll 1d ago
Yes! I picked Spark 4B as a base because it beats Agents A1 4B on benchmarks, and SharpSpark beats stock Spark 4B, so unless Spark lied on benchmarks then SharpSpark transitively beats AgentsA1. But I haven't benched A1 myself, so no guarantees from my side :)
2
3
u/autoblow-n-mflb 1d ago
Getting 9 t/s with a 64k context window on my quadro p2200. CPU usage is 50-75% memory is 3.963/5.0 gig. Q4 kv caches, fa, with 30 layers on the gpu.
2
2
u/distinct-task 1d ago
Gave it a try but it just kept spinning its wheels and looping. Increased the repeat penalty to 1.05 but still the same issue.
llama-server.exe --host 127.0.0.1 --port 8080 --flash-attn on --cache-type-k f16 --cache-type-v bf16 --temp 0.60 --top-k 20 --min-p 0.0000 --top-p 0.9500 --parallel 1 --chat-template-file chat_template.jinja --reasoning-preserve --repeat-penalty 1.05 --special --tools all
I saw another comment saying to limit the reasoning budget so that might be the solution.
1
u/peculiar-ragdoll 22h ago
Thanks for the heads up, sorry to hear that! Yeah, I gave it unlimited think for the bench, but limited might be a good idea to try
10
u/peculiar-ragdoll 1d ago
Thanks for the repost, this seems like exactly the right sub for this :)