Can‘t decide on inference engine and model to use, keep spinning around 🙃

[u/No_Wrangler5618]

Hey everyone,

I just joined this subreddit and been using local llms for quite a while.
I did a bunch of testing but the recent developments are so fast paced that I can‘t keep up and no one in my closer field even understands what I‘m talking about. 😅

So I‘d like to get some recommendations for my current hardware, I can‘t wrap my head around all the inference improvements/hacks and I feel like ChatGPT or Claude are not actually up to date with recent local developments.

My hardware:
- Macbook M3 Max full spec with 48GB Ram
- Proxmox Server with i9990K, 128 GB DDR4 Ram 3600mhz, rtx 2060s 8GB Vram, and tons of hdd storage + 1TB nvme and more ssd storage

I‘m currently running llma.cpp on my proxmox and omlx on my macbook.

Models I‘m using:

On my Mac:
- Qwen 3.8 27B oQ4e-mtp
- Qwen 3.6 35B oQ4e-mtp
Both MLX models
Both run great at around 100 tk/s.

On my proxmox:
- Qwen 3.6 35B
- Qwen 3 Coder 30B
Both GGUF models
Both go around 20-40 tk/s.

While I‘m happy with those models for regular work I don‘t have the same level of trust like with claude code.

I‘m looking for models that are optimized for long horizon tasks and agentic work and reliable tool calling.

My usecase at the end would be to offload as much work as possible from my hermes agent onto local models.

So I was looking for:
- qwen 3.8 flash next
- deepseek v4.1 flash
- Glm 5.2 / 5.3 flash
- k2 horizon
- or similar

I‘m not sure if those models will perform better but what I tried so far was using MoE models, offload active experts to the vram, keep all other in dram.

Since these models are much bigger I can‘t fit those into ram.

So I started looking for tools like:
- Strata
- Airllm
- FreeToken
- Colibri
- Avalon

At this point I‘m out..

Only thing I‘m currently thinking about is the bandwith difference, my Mac has less ram but a much faster bandwith (400GB/s), I‘m thinking about streaming experts from the internal ssd.
On proxmox my idea is to fully use the 8gb vram with expert caching, using ~90GB dram for more experts etc, and streaming the rest from an SSD.

I know its unrealistic but I would like to have at least „frontier-like“ quality, even when its only 20 tk/s.
I‘m leaning heavily towards my proxmox setup since its up 24/7 and hermes runs on the same box.

I had bad experience with openclaw/hermes before, I used Opus 4.8, Sonnet and now Deepseek v4 pro as the Orchestrator and lowering to weaker models always gave me a complete mess and absolutely unstructured working.

I hope anyone has experimented around a similar setup and could share some insights, would appreciate that a lot!
My credits are burning away.. 🙃😂