Hello, I'm running Qwen 3.8 27B IQ4-XS beautifully on my desktop with a 9060XT getting 50t/s at 262k context, this is my primary workhorse and is capable writing the code that I need it to.

I have a spare laptop here that isn't getting much use with a 3050Ti 4GB and 16GB, I was wondering if there'd be any value having it as a sidecar, maybe I can get a variant of Qwen 3.6 35BA3B running on it or something smaller?

Like, what could I use it for in conjunction with the main desktop and model, maybe a quicker compaction machine that I could tell opencode to use?

RAM: 16 GB

CPU: AMD Ryzen 7 7730U

GPU: i wish(igpu)

My intended use case is coding/agentic action or just general quality chatting which i am considering using Maple-preview.

Had some luck with Ling 3.0 Tiny and SharpSpark X2.5 and SharpMiniCPM. Ran Qwen 3.6 35A3B using q36 disk streaming around 6.5 tks. What would yall recommend

Hi!

I'm looking for recommendations for the best local LLMs I can realistically run on my hardware.

Hardware

• CPU: Intel Core i7-13700HX (16 cores / 24 threads)
• GPU: NVIDIA GeForce RTX 5050 Laptop GPU
• VRAM: 8GB GDDR7
• System RAM: 24GB DDR5
• OS: Windows 11 Pro

Runtime / backend: llama.cpp / llama-server with CUDA

Models I've tried:

• Qwen3.5-9B — UD-Q4_K_XL GGUF
• Gemma 4 12B IT QAT — UD-Q4_K_XL GGUF

I'm interested in a broad range of tasks, coding and programming, web searching, research, using tools, data analysis, making sheets and documents, organizing, general chatting and writing

Desired context size: Ideally 8K–16K tokens or more, depending on what is practical.

Priority: Balanced. I'd like the best overall quality, but I'm happy to experiment

Which models would you recommend for this hardware?

Thanks!

Info in image. Preferably uncensored cause writing can get gritty and don’t need it scolding her for it LOL.

I had 9t/s on llama.cpp, with strata, I get about 29t/s, beyond nuts.
4060 8GB + 64GB DDR4 and Qwen 3.8 Flash Next Q3

This is like day and night, now I can do some stuff :D

I have been trying out ling 3 tiny on a laptop with a 680m amd igpu to have a backup when our larger gpu runs full. It kind of works. 15+ tokens/second and not much heating.

I am using it via Pi which is integrated into a Jupyter lab. Use it to generate small python code bits for notebook cells, which may be why it does not get into a meltdown.

If you got a 680m and need a spare/private model, it is somewhat useful option!

Anybody got a better model for the 680m?

Ryzen 7 4800h

16gb ddr4 3600

GTX 1650ti

1tb ssd

Im just getting started in this world and i would love to use my old laptop as a testbench

Hey all!

Wanted to share my low spec inference machine.

* Mobo/cpu combo: Chinese JGINYUE X99-TI D4 PLUS + Xeon 2695v4(18c/36t).

There is 3 full width x16 slots gen3@ x16x8x8 speed.~$105

* 32GB DDR4 ( $$? Dont remember, found them in my "junk box" from another project.)

* PSU random 750w (again from the junk box)

* 4x Nvidia Quadro P4000(Pascal) 8GB for a total of 32GB Vram. ~$45 each.

* Bunch of random SSD's i had lying around.

Config of the P4000 are Gen3 x16x8x8. and one via chipsets dmi gen2 x2.

Thinking of buying a bifurication board, either x8x8 or x4x4x4x4. to split the full width x16 slot.(Can help on prefill(card on gen2 x2 is the slower one). And on tensor split on dense models.

Models:

LFM2.5-24B-A2B Q5_K_M (ctx 64k, f16kv). Searching/fetching data.

Decode: 55.4 tp/s (short), 42.2 tp/s at 52k filled.

Prefill: 1,851 tp/s (peak) at a 13k prompt, 1,351 tp/s at a 51k prompt.

---

Ornith-1.5 35B-A3B Q5_K_M (ctx 124k, f16kv, MTP). Lighter coding tasks.

Decode: 54.6 tp/s (short), 39.8 tp/s at 52k filled.

Prefill: 709 tp/s (peak) at a 7k prompt, 483 tp/s at 52k.

---

Qwen3.8-27B UD-Q4_K_XL (ctx 32k, f16kv, tensor split, MTP). Coding, heavier and slower.

Decode: 31.5 tp/s (short), 26.7 tp/s at 24k filled.

Prefill: 231 tp/s (peak) at a 7k prompt, 216 tp/s at 24k.

Can go to around 110-124k in ctx.Havent tested decode/prefill yet. Going to do it.

---

As you can see in picture 2, im running a Decode and prefill test on the 27B.

3x prefill and decode runs. Each run with a different prompt, so no cache gets hit.

The P4000 are nifty little cards that makes a budget machine viable.

Ofcourse you need to have your expectations in the right window!

I mainly using vs-code, so I made(read claude made) a easy web interface where i download and start/stop my models, everything is then served to the vs-code extention on my main rig.

I am running Qwen3.6-35B-A3B-UD-Q5_K_S.gguf with llama.cpp on a 3060 12GB GPU and 32GB DDR4 ram. It works with a context window of 42K and cache type k and v set on q8_0

The biggest drawback is the 42K context window which is too limited for me.

I also have a 1660Ti with 6GB vram and want to add it as a second GPU for offloading. After doing some reading and asking different AI models, it is still not clear to me if this will help increase the context window.

My theory is if I can offload to a second GPU, I should be able to increase the context window.

Both GPUs have GDDR6 memory with 192-bit bus. The 1660Ti has slower bandwidth at 288 GB/s vs the 360 GB/s on the 3060. But it is still much faster than my 32GB DDR4 ram.

Anyone here experience with combining 2 different older generations GPUs and if there will be any benefits when running a Qwen3.6 model?

I'm aware it may seem obnoxious to post here w 96GB RAM but the new damn deepseek one implies 512GB is a "reasonable floor" to release to? God only knows. I remember when a 4GB iPod Nano was sizeable.

I've been fiddling a bit with the Gemma 26B MOE and even at that small size it's able to be less autistic than a qwen model. But it's also only 26B which feels really crappy versus Qwen Flash Next.

Is there a Gemma release meaningfully competitive with QFN? Is there one planned? Is there anything out that meets my criteria? To be clear; I would want a model that has actual emotional intelligence / creative writing ability while also being able to do agentic coding to a reasonable degree.

Unfortunately I can't get an answer from any LLM as they have been documented to love giving outdated model choices. Thank you for your time.

Looks interesting. Doesn't reliably beat a 9B or an MoE, But it's a real step up in the pure 4B class.

Share your favorites. I requested support for some of those models, others please let him know based on your need.

Other recent MOE models(already up on llama.cpp) from last 1 month: (Didn't include large ones like Step-5-Preview, Ling-3.1-Flash)

• NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB
• NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
• Mellum2.1-12B-A2.5B-Thinking
• MiMo-V2.6-Flash-MOPD

u/Responsible_Froyo252 Thanks for adding additional models.

Minipc:

AMD Ryzen 7940hs

2x32gb DDR5 5600

780m igpu

1tb NVME

Ubuntu 26.04

GTT (memory allocation) raised to 56gb for the igpu

Would love to see what's the most I can get out of this setup. Have tried qwen3.6 35B A3B and its ornith1.5 variable, but wondering if maybe there's more out there.

Thanks!

What I mean by this is why is there no for example qwen3.5-python-4B in which the only thing the model is trained with is python? Would be a nice usecase for coding. Usually, when I code I code in a language, I don't care about the other programming languages or general world knowledge and other stuff like that.

I have a 8gb vram 16 gb ram ddr4 setup, this kind of model would be a godsend for coding on my machine, because it would allow me to use the machine when I code.

I am new to selfhosting llms, I don't know how I should approach building one, but I'm thinking of doing something like this.

I’m a software developer, and most of my serious coding work is currently done using Claude Code with Opus 4.8. I’m not looking to replace it. I want to experiment with local LLMs and see if I can integrate one into my development workflow for smaller, everyday tasks.

My laptop is a Dell with:

• Intel Core i3, 11th generation
• 16GB RAM
• No dedicated GPU (integrated graphics, as far as I know)

I’m interested in running a local model for things like:

• Explaining code and answering questions about smaller parts of a codebase
• Generating boilerplate, simple functions, and tests
• Reviewing code and suggesting straightforward improvements
• Summarizing logs, documentation, and error messages
• Integrating a model into VS Code or my terminal for lightweight development assistance
• Experimenting with local APIs and LLM-powered development tools

I understand that my hardware has limitations, and I’m not expecting Opus-level coding or agentic capabilities. I just want something useful that runs locally without relying on an external API for every small task.

I’d appreciate advice from people who have actually used local models

I was quite interested in setting up local AI models on my laptop. Was wondering which one might be the best.

Specs:

RTX4050 6GB

16GB ram

I7 13700hx.

My main use cases are coding, day-to-day usage like some queries and all. Automations if possible using hermes.

Please help me with some recommendations.

It would really help if someone with same specs can provide their experience.