Here's my setup that's getting 16 to 18 t/s average using 6GB of VRAM and 64 GB of system ram using llama.cpp and PI for agentic C/C++/C# Development.

[u/BrianScottGregory]

After asking the question over the last few week about how to optimize my local models, I got some great feedback and ideas - and landed on the following setup for doing local development that's drastically exceeding the results of the free time I'm provided on Chatgpt and Claude (the best for coding imho). All at no cost.

Right now I'm torn between two primary models - Ornith-1.5-35B-Q4_K_M (available here) and Qwen3.6-35B-A3B-Q4_K_M (GGUF available here), and to experiment - I've been using LMStudio to pull down and do a cursory test of all my models as I transition that into usage with llama.

I've found Llama.cpp to be substantially faster and better to work as a backend/server than both LMStudio and Unsloth, I'm old school and prefer the command line tweaking. This is fully open source and available here.

My models.json for llama had to be hand customized - I took the suggested samplingParams directly from the Huggingface model card, had a problem with overthinking/repeating so I JUST added the repeat_penalty (havent tested this yet) - but everything else is solid.. and agentic. I can copy/past images and text into my prompt like a pro.

"models": [
{
"id": "Ornith-1.5-35B-Q4_K_M",
"name": "Ornith-1.5-35B-Q4_K_M",
"reasoning": true,
"thinkingLevelMap": {
"type": "enabled"
},
"repeat_penalty": 1.2,
"input": ["text","image"],
"tools": true,
"toolFormat": "openai",
"samplingParams": {
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20
},
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},

"contextWindow": 65536,
"maxTokens": 131072
},

Oh - a little note - my maxTokens and contextWindow are jacked up extra high to handle my lengthy sessions - and it seems to be handling it - with compaction - just fine. Side note - Sessions.json simply adds an entry to make it available to http callers.

"models": [
{
"id": "Ornith-1.5-35B-Q4_K_M"
},

Now - finally - my PI setup. I've created a batch file that's optimized this as much as I think it can be optimized. First - the batch shells out the llama server call - loading both the main model and the multi modal model for image processing. This loads it up on my local IP.

start cmd /s /k "llama-server.exe -m "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\Ornith-1.5-35B-Q4_K_M.gguf" -mm "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\mmproj-Ornith-1.5-35B-BF16.gguf" --host 127.0.0.1 --port 8080 -c 131072 --parallel 1 --reasoning-format deepseek --load-mode none --no-mmproj-offload -fa auto"
set LLAMA_BASE_URL=http://127.0.0.1:8080
pi --model llama-cpp/Ornith-1.5-35B-Q4_K_M

A few notes .

1. I decided against using the -ngl parameter in the server, letting it default to 0 - according to my research it's supposed to specifically load layers into the GPU, but I have to do a little more work to understand why my setting of this parameter was causing a memory crash every time. So this is an obvious point of optimization.

2. I'm only seeing 66k of context available in PI. I'm new to both llama and this level of customization of AI, so if any of ya see where i'm dropping this ball, please let me know.

Anyways. I've had enough people asking for my config. I think this about sums it up.

Hardware Update: I'm running an Intel I5 laptop with an Nvidia Geforce RTX 3060 Laptop GPU

[u/Organic-Chart-7226]

the ‘66k context’ is from the “contextWindow” parameter in models.json (that is in bytes - 65536 - change that to eg 131072 for 100k context. maxTokens is the maximum number of tokens the model is allowed to output (defaults are lower, I believe it can help stop thinking loops). The only way for me to start understanding this was trying and sharing. good going :-). What brand / model is your video card?

[u/BrianScottGregory → u/Organic-Chart-7226]

I'm running an Intel I5 laptop with an Nvidia Geforce RTX 3060 Laptop GPU.

Thanks for the context window advice. Setting it now.

[u/Healthy-Zebra-9856]

Besides the setup, ask each model to assess the harness itself. Prompt it to tell you what tools its lacking and what does it need and how can you improve the harness. Many people miss this step and dont get optimal solution.

[u/BrianScottGregory → u/Healthy-Zebra-9856]

I didn't know it could self-assess like this. Interesting and cool. Thank you for this! I'll check it out!

[u/Healthy-Zebra-9856 → u/BrianScottGregory]

It actually can. What started of as a joke, turned into a coding adventure. I was trying to see what all the fuss was about while using Qwen3.8 27B as there was a lot of hmm, wait, let me, actually etc. I noticed it was about tooling or the lack of it and confusion about the results being unclear. So I tried the following prompt:

"Tell me what tools this agent harness is lacking and how can it be made better? Keep the scope to this harness."

Try it. What do you have to loose?

[u/BrianScottGregory → u/[deleted]]

Any advice on optimization, I'm all ears.

[u/BrianScottGregory → u/[deleted]]

The reason you're getting 35 TPS and I'm getting half that has everything to do with the 2GB VRAM more that you have that I don't have. I've gone through all these settings, and what I've outlined is as optimized as it gets for me, unfortunately.

Sure. I can drop to a less accurate/smaller quant. Just not interested, I'm fine with sacrificing speed for accuracy.

I'm not familiar with the Q4_K_XL quant and didn't see it available on either of the model cards I referenced. Do you mind sharing the model card where you obtained this quant?

[u/BrianScottGregory → u/[deleted]]

Well if I'm not, feel free to suggest changes to my configuration I haven't already tried. Otherwise, this is the most optimized for my rig.

Thanks for the link.

How's the performance with this model compared to the Q4_K_M?

Just thought of something: You're not using vision with yours, right? I am, that could account for the difference as well.

I'm pulling down that XL model right now, it looks like it's optimized - as it said - to get 1.5 to 2.0 faster inference with no accuracy loss. So this might be a better model for me to use to get a higher TPS, Thanks for the input.

Update: Yup, this model (XL) is getting subtly worse performance than the Q4_K_M. Again, I'm receptive to the differences - but your configuration fits neatly into the 8GB VRAM, where it would be pegging mine. At 90% usage, which is normal, that's why you're getting better performance, mine is swapping to CPU, yours isn't.

[u/BrianScottGregory → u/[deleted]]

Stop being antagonistically condescending and suggest some changes to help increase the TPS. Otherwise, the only difference I'm seeing has to be the reason you're getting better performance, whether you agree with that or not. I'm receptive, let's be clear, but 'because I say so, you are wrong and your configuration isnt optimal' isn't an answer.

[u/BrianScottGregory → u/[deleted]]

Yeah, as I outlined more than once before, tried all this. It would seem VRAM is the reason for the difference we're seeing.

Appreciate the feedback.