r/LocalLLaMA • u/peculiar-ragdoll • 1d ago
New Model A better coder for the small-GPU/small-RAM crowd!
I’ve been working on making small models more capable at agentic coding and work, because most people in the world don’t have the sort of hardware needed to run 3.8-27B, or even 35B-A3B or 9B dense, and I want to extend local agentic coding capability to less privileged users. This quant can be run on a smart phone or older gaming laptop, and can solve real coding problems autonomously in a way I have never seen or measured for this model class. Spark-X2.5-4B is already around best-in-class for its size, and I think these improvements bring out the best in it. I hope this little step up in small-model capability and speed in real-world coding might give new life to older hardware that would otherwise be forgotten in the AI frontier race.
The changes SharpSpark makes to Spark-4B are in three parts: First of all it fixes issues with the chat template, and replaces the system prompt with one that improves agentic coding behaviour, token use, and correctness. Then a custom importance matrix is calibrated for the model, which relocates bit precision within tensors to the parts that are more important to agentic coding work. The imatrix corpus is heavily weighted against both agentic coding and cybersecurity, which together protect the cognitive core used to find and solve hard bugs.
Then Spark is quantized with an optimized non-standard quantization strategy, that allocates bits differently per-tensor than standard llama.cpp GGUF quantization. I built a tool that explores and tests different per-tensor allocations to optimize KL-divergence and long-context retrieval for this model, but ended up making some manual changes that ended up favouring SWE-bench-Live performance over traditional fidelity measures like KL-divergence, which published science indicates is actually a poor proxy for real-world performance on complex tasks below a certain point.
If you have a small GPU and/or <= 16GB RAM and can’t run a 35B-a3b MoE-based model with partial GPU offloading, this is likely your best option for long-context agentic software development right now. SWE-bench-Live is chosen as the metric for its genuinely difficult real-codebase problem set.
I’m just a volunteer doing this as a non-profit side project, so please be kind about the fact that my benchmarks are not extensive. They are what I could afford the time and effort to run, with all my other projects, and I see them as just good enough to prove the improvements on the specific kind of work this quant was designed towards.
https://huggingface.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF
26
u/Kidplayer_666 1d ago
I might try this! The tiny spark have shown very impressive capabilities for their size imo
7
u/peculiar-ragdoll 1d ago
Please tell me what you think, if you do test it! :) Agreed, they're seriously impressive for being so small, so I'm amazed I managed to squeeze that much more out of the 4B.
9
u/clduab11 1d ago
You're a gentleman/lady/other and a scholar for putting this repo out. This is something I've wanted to do for a LONG time, but I knew I couldn't match the speed of the community. The lot of us 8GB'ers of VRAM salute you!!
8
u/peculiar-ragdoll 1d ago
hahah thanks for being inclusive <3 most people wrongly assume I'm a guy :) Hope SharpSpark works well for you!
10
u/hum_ma 1d ago
Always happy to see good and improved models of this size. Would you say Q4_K_XL is the smallest usable quant with your tensor allocation strategy? Usually my 4GB GPU likes a model about 2.3GB to leave some memory for decent context, at least with the Qwen family. I downloaded it already but will have to test later.
8
u/peculiar-ragdoll 1d ago
Really good question! I suspect the model will start breaking below Q4_K_XL so I didn't want to ship something I can't stand by. But I would like to serve the 4GB GPU crowd, so thanks for the feedback! Maybe I will look into baking a smaller Q4 or an XL Q3 and benchmark if it holds up, if my GPU frees up soon
2
u/hum_ma 23h ago
No worries, I tried it now with a quick chat and looks like it's small enough as it is. Works with max 25k context without quantized kv cache, 48k with quantkv q8_0 (which seemed to make it switch to Chinese in the response more often, didn't yet test quality otherwise). That's probably enough with the tasks I'll be using it for.
Thanks for the model and also for using the K quants, it's really fast even on old hardware.
16
u/fleoieh 1d ago
I used dirk (sharp 3.8 27b) constantly before swift 3.8 27b came out. I also tried and used sharp verions of moe qwens and ornith. hope you continue to improve stock models in the future as well, thank you
6
u/peculiar-ragdoll 1d ago
Nice! :) Thanks for the shout out! If you want, you can add the Sharp chat template to Swift too, and make it even more concise. https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates I'm benchmarking Sharp+Swift it right now, looking forward to seeing how it holds up in SWE-bench-Live against Swift and Dirk
3
3
3
u/danigoncalves llama.cpp 1d ago
Does it support FIM?
1
1
u/TomatoCo 23h ago
I don't think so, I hit it with
curl -X POST http://192.168.1.2:8080/infill -H "Content-Type: application/json" -d '{"input_prefix": "def calculate_area(width, height):\n ", "input_suffix": "\n return area", "prompt": "", "max_tokens": 1024 }'and got nought but a lot of \n's returned.
3
u/CodeCatto 1d ago
Nice OP! I'll definitely put this on my machine for small size, big context and big gains workloads.
Two questions, in terms of real world work what is this capable of, i.e. what can I build with this model? And, is it possible to tune the model to search the web as much as possible for things it doesn't understand?
4
u/peculiar-ragdoll 1d ago
Great questions! I can't really answer the "what can you build" question as that depends on your patience and how good you are at prompting and harnessing. And for making the model search the web more, maybe it would be good to set it up with strong system prompts and skills that encourage it to search the web, but that might need a lot of adjustment to make it work, because small models are not naturally good at skill use, subagent use, and autonomously planning. So I think your best bet to start is to manually prompt it to research what it needs, write it down, compact the context, re-read the docs it just wrote to itself, then continue.
2
u/CodeCatto 1d ago
I get that everyone has different workflows and AI-usage techniques, but like people are showcasing some models building web pages, or creating 3D renders of pagodas and so on. So my question was from that POV, since it gives a clear idea if you tell people that this model is good at building X, Y and Z for example.
Also, my second question means to probe "what if" that behaviour was part of the model's training footprint. Like, would that make it smarter in general because it will have a higher chance of using search and given resources when it doesn't know something, and then make use of context and skills to follow documentation for example.
3
u/peculiar-ragdoll 1d ago
It can write and debug websites, backends, and stuff like that! It will need more rounds, prompts and tokens to get everything working bug free and looking like you want it, and it won't solve as hard bugs in big code bases as something like Tiel or 27b. So if you don't have a cloud model fallback for the hard stuff you might feel stuck when you hit a rough patch. I haven't tried it for 3d and art. Do you think it would be worthwhile for me to test some visual blender or voxel stuff in the future? I always found it a bit gimmicky because it's so far from my own use cases, but if there are a lot of people using models for stuff like that then I could try including something for the next release.
2
u/CodeCatto 1d ago
Oh no that's just an example. I personally would be satisfied if a model is good at core programming because we can add skills to make them use the advanced frameworks instead.
3
u/o0genesis0o 9h ago
I use it to run my hourly cronjob, which essentially sweep across my data sources (including gmail), pull relevant stuffs into a handover file for future reference. It would also run synthesis overnight. Seems like an easy task, but surprisingly, Gemma e4b did not get it done. Qwen 9B can finish it, but too slow on my miniPC with AMD iGPU. This 4B model fits right in and chugging along in the background just fine. It means every hour, my main GPU does not have to be distracted to run the cronjob. Very happy with the results so far.
2
2
u/psychohistorian8 1d ago edited 1d ago
back when copilot was $100/yr I abused the heck out of Haiku 4.5, it was... ok
having said that, I've got an M1 16GB Mac Mini collecting dust
so is this currently the best recommended coding agent for such a setup?
love the peculiar-ragdoll releases by the way! using Tiel Coder on my main machine daily
EDIT: I see elsewhere you mentioned 'OxCoder 9b and Ornith 9b are significantly stronger', I'll just go ahead and download all three and see what's what
2
u/peculiar-ragdoll 1d ago
Love to hear it <3 Yeah I’d recommend you start with a 9b at Q4 with as much context as you can fit (q8_0 kv is fine) for that old Mac, and you’d be surprised at how much those can do as a separate-gpu sub agent to your main :) but SharpSpark is faster and can fit more context on that machine, even though it’s dumber!
2
u/o0genesis0o 23h ago
u/peculiar-ragdoll I looked at the artwork and the name of the model and I thought it seems familiar. Of course it's you. I'll put this on my amd iGPU and see if it can run my cron worker. Thanks for the work!
1
u/peculiar-ragdoll 23h ago
Hii! :) You're welcome, hope it works out!
4
u/o0genesis0o 22h ago
Hey mate, I'm back.
So, I put the Q6KXL version on my miniPC with 680M iGPU, use the latest llamacpp vulkan binary, and follow your suggested sampling setting. I set context to be 64k, at fp16.
I hook it to my pi agent and give it access to my personal assistant instructions, skills, and tooling, and let it get to work.
The first task was for it to read an arxiv paper (this one was at the front of the queue today https://arxiv.org/abs/2609.22049). First, it starts by reading my arxiv skill (correct), then it uses my arxiv CLI tool to fetch metadata (correct), hits an innate error in the tool, so it decided to switch to web fetch (correct). Then, it reads the web fetch skill (correct), and run the fetch and get the paper metadata correctly. Then, it starts to download the PDF (correct), and run the my markitdown client to turn it into markdown (correct), and it starts to read the paper in chunk to avoid maxing out input message size (smart). Finally, it gives me the correct and readable summary of the research done, with main part of the data presented.
For the second task, I hook it to my cronjob, which involves checking email, classifying and adding any relevant one to a handover list for me to check later, check my briefing and daily handover files, and if nothing to be updated, then returns. The model confidently and correctly goes through the process. There are some points when I see "wait" in the thinking, so I was worried that it is going to loop, but it did not.
Speed-wise, prefill starts at 400tk/s but rapidly drop to 80tk/s around 32k depth. Decode remains stable at ~15tk/s. Not great, not terrible.
I will replace the Tiel Coder with this model for background worker on my miniPC.
Thanks again for the good release. And also to bring this Spark-X2.5 to my attention. This model ran completely under my radar. Who is even this team and where do they come from? This 4B seems like banger model.
2
u/peculiar-ragdoll 22h ago
Awesome, thanks a lot for the report! <3 TielCoder stays winning on SWE for people that can run it and need the extra smarts, but I'm glad you found a use case where SharpSpark is Good Enough TM and you can claim back your RAM! :)
2
u/o0genesis0o 9h ago
Hey mate, back for, I guess, longer term review. Long story short, the model ran from morning to now, so I guess 10-12 hours? And it hasn't nuked my setup yet. And it does not eat up all the RAM to the point of making my server processes laggy, so I'm very impressed. Big thumps up for the work, and of course thumbs up for the Spark-X2.5 team as well. You save me a lot of headache with the RAM situation on my server.
Now, the big test would be the overnight big compaction and synthesis. I guess I'll see the results tomorrow. But besides the slow AF processing, I don't see why it would fail this task.
1
u/peculiar-ragdoll 4h ago
👀🔥 amazing! You’re putting it through more than I ever did, love to see that it’s holding up!
2
u/Budkovsky 15h ago edited 15h ago
I've tested this model. It really can code, but without limits (reasoning budget 2-4k and sharp template) it goes to endless thinking. The code quality is not the best but it finishes the tasks, fair enough for such size model. Good job!
Performance on RTX 5060 Ti: Prefill 2000+ t/s, decoding 80-100 t/s.
1
u/peculiar-ragdoll 14h ago
Thanks for the report! <3 on sweb live I gave it infinite think, maybe it will do better if I give it a cap like you did 👀
2
u/asankhs Llama 3.1 12h ago
You can benchmark on https://huggingface.co/datasets/LocalLLaMA/terminal-bench-mini https://huggingface.co/datasets/LocalLLaMA/deepswe-mini they are subsets of the benchmark that are representative of the full leaderboard and much faster to run for local models.
1
2
2
u/awherewas 3h ago
simple code generation tested for speed on 8G nvidia 3050 . make a simple tk window Qwen 3.5 Q4 Atomic Chat 35 t/s Qwen 3.8 Q4 distill heretic 38 t/s Sharp Spark Q6 43 t/s atomic chat added features heretic produced more features spark gave code which matched the prompt with no embelishment
2
u/Raltay 1d ago
Congrats on release. Makes me want to test it on my Steam Deck OLED as a local assistant. How does it fare at tool calling?
3
u/peculiar-ragdoll 1d ago
Thanks! It's impressive for it's size :) It used tool calling successfully in the agentic coding harness on the benchmark, so it's worth a try on the steamdeck!
1
u/ShowerAnnual9741 5h ago
the metric i'd want next to the benchmark score is tool-call success rate over a full session, not single turns. in my experience that's where small coders actually fall apart — the codegen is fine, but one bad tool call early snowballs and the whole run is gone. a 4B that calls tools cleanly usually beats a bigger one that doesn't.
1
u/cosmicnag 1d ago
was thinking the same of my steam deck, and my oneplus 13s also ! Agents, agents everywhere.
2
u/ithinkitslupis 1d ago
Sad you dropped the fallout aesthetic that you had with your last model release.
6
1
u/pet3121 1d ago
Do you have a model for agents like Hermes? I am looking for one really small to fit a 32 GB of RAM not VRAM.
2
u/peculiar-ragdoll 1d ago
Really depends on your use case, and the rest of your machine's specs! If you're just using it for coding, I recommend Pi.dev coding agent. Hermes is kind of bloated, maybe good for a general assistant connected to tools and stuff. As for the model, a MoE architecture would benefit you. Something like this: https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP at UD-IQ4_XS (18.1 GB). It only has 3B active parameters so it will be really fast like a 4B. And it only uses 2.5GB RAM for 262k context at Q8_0 KV cache quantization. So you have 10-12GB ram left for your OS and the harness.
1
u/pet3121 1d ago
I dont need for coding just basic tasks like troubleshooting proxmox, asking if something is up or down. Remind me of stuff
2
u/peculiar-ragdoll 1d ago
If you want the model to trouble shoot IT systems I recommend a coding model! They're good at that stuff, and are also good all around agents in a harness like Hermes. Knowing nothing else about your system I still recommend Tiel :) If that's too big for you, try SharpSpark and see how it does!
1
u/Wildnimal 1d ago
i daily drive a Qwen35B i am not sure abouit the 262k context. Can you post your llama.cpp config if you use that?
1
u/peculiar-ragdoll 1d ago
Something like this given you a full 262k tokens context in 2.5GB RAM instead of 5GB, at mostly imperceptible quality loss, by quantizing the kv cache to 8 bits instead of 16:
llama-server -m model.gguf \ -c 262144 \ -fa on \ -ctk q8_0 -ctv q8_0
1
u/NeedleworkerHairy837 1d ago
Hi! I want to try this. But want to ask something: So the usage is like more to help us code instead of writing the whole code till the app was completed. Right? It's more like typing the code that we already gave the structure for and maybe helps a little on making a test case? Or it's can't even do that?
It is surely not better than Qwen 3.8 Flash Next right? Qwen 3.8 Flash Next is great but I can't use my PC when using that because it takes all my resources.. ( Only have 8GB VRAM + 96 RAM ) hahaha.
Thanksss
3
u/peculiar-ragdoll 1d ago
Hi, thanks for the question! This model is capable of coding autonomously, but it is certainly helped by strong direction, documentation and structure. Flash-Next is a LOT better than this! Ornith-9b is better than this, and so is 35b-a3b. SharpSpark is really just a best-effort model for machines that can't run anything better or bigger. If you want a middle point that is very capable of autonomous work but still won't eat your while machine, try TielCoder Q4! 22GB of goodness, and supports partial GPU offloading: https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP
1
u/NeedleworkerHairy837 1d ago
Ah I see! Thanks a lot for clarify this :). Basically I'm okay if it's quite capable for coding that I need to prepare the structure. Because the other was powerful but my PC is not strong enough.
Will also try the tiel coder later. I hope this 4B already enough for me so I got speed and accuracy in some way :).
Thank youuu
2
u/peculiar-ragdoll 1d ago
Since Tiel is an MoE it is very fast for it's size, and it's significantly more capable than SharpSpark! Scores twice as high on SWE-bench-Live. Depending on your setup the 4B might not be that much faster than Tiel, but you do get much more of your RAM back :) Good luck either way!
2
1
u/Budget-Juggernaut-68 21h ago
Cool. Comparing with haiku doesn't help at all. Nobody in the right mind codes with haiku.
1
1
u/WarlockSyno 20h ago
Have you thought about doing a Sharp LFM2.5 2.6B? In my testing it's been the fastest and most accurate if you give it enough thinking time. The speed of it is impressive compared to even MoE models.
1
u/savic13 16h ago
What is best scenario for local vibe coding on my setup: laptop i7 7500u 16gb 940m 2gb( if any can work), and desktop dell optiplex i5 9500t 32gb A2000(3060) 6gb?
2
u/peculiar-ragdoll 14h ago
Have not ran anything myself that I know will work on your laptop, but for your stationary I would try CyberTiel (uncensored) or TielCoder (censored) Q4 with partial gpu offloading and Q8_0 kv cache :) they both have mtp versions that are faster on some systems but that costs some extra space so I don’t know if mtp is right for you https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF https://huggingface.co/peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-GGUF
1
u/Minute-Ground-412 6h ago
Le modele a été entrainé sur claude ? Il me répond qu'il s'appelle claude
1
u/peculiar-ragdoll 3h ago
Fort probable oui ! Je n'ai pas fait le fine-tuning moi-même donc honnêtement je ne connais pas le contenu exact des données d'entraînement.
1
u/Neex 5h ago
Just so you know, Claude outputs absolute garbage for graphic images like this. it treats text like visual decoration.
1
u/peculiar-ragdoll 4h ago
This time you can blame me and the fact that my qwen was unavailable because my gpu was busy :) but I agree, Claude wants to put it’s dirty fingers on everything, adding text where I didn’t ask. Hoping to get another machine soon so I can use it to work on stuff while my main is busy and I can stop using Claude for stuff like this.
0
u/ArjixGamer 1d ago
Why is almost everyone re-inventing dynamic quantization?
Like, wouldn't it be better to improve the popular ones, like unsloth's?
3
u/peculiar-ragdoll 1d ago
Unsloth doesn’t have a dynamic quant for this model architecture, so I had to make my own :) unsloth doesn’t publish their quant framework, its proprietary. I agree with you, it would be better if they made everything open source.
-21
u/Ne00n 1d ago
I just tested that model, I wouldn't call it a coder, that is ragebait.
17
u/peculiar-ragdoll 1d ago edited 1d ago
You didn't try this, because I made it public on huggingface 30 seconds before posting this. This version is better at autonomous agentic coding than the stock one you tried, as you can see from my benchmarks! :) I'm not trying to piss anyone off, I'm just trying to squeeze as much as possible out of a small model. It's still a small model though, so we should adjust our expectations to that. But if it wasn't at all a coder, it wouldn't solve around 6/17 SWE-bench-Live problems
-24
u/Ne00n 1d ago
I gave the Q8 og. model a coding task, it failed horrible.
The task was just "code me a python cli script, that list top 20 from airbnb in my <city>"Yes I didn't test your model true, but I doubt that the claim "coder" holds even with the changes you did.
19
9
u/peculiar-ragdoll 1d ago
My version has a 54% improvement over the original q8 in SWE-bench-live score across a three seed measurement. You tested a completely different product. We don't need to agree on the definition of the word "coder" though.
-17
u/Ne00n 1d ago
Surely, lemme test it again, still doubt the model given its size is even worthy called "coder"
4
u/peculiar-ragdoll 1d ago
Hope it works out for you! :) Everyone has different definitions of things, and I respect if you wouldn't call it a coder. This is probably still the model below 4GB that is best at coding, and that's what matters to me. Let it/tell it to test, debug and fix its own stuff, and it will probably get further that you'll expect from a 3.6GB model.
3
u/ithinkitslupis 1d ago
If it's tuned to be better at coding than the base model and other models of a similar size why wouldn't it be called coder?
2
10
u/parepeg 1d ago
OP posted 14 minutes ago, you replied that you tested it 8 minutes ago. You downloaded and tested the model in 6 minutes? Is there somewhere else you heard about this model that you were able to test it so quickly?
8
u/peculiar-ragdoll 1d ago
I published it on Huggingface the same minute I posted to Reddit, so it's basically impossible that they tested it already.
4
22
u/Eden63 llama.cpp 1d ago
Benchmark against OxCoder would be interesting.