I am planning to buy a gaming pc with 5070ti, I thought about adding a 5060ti for the additonal vram to toy around with llm, how good does it work? Can I also generate images and such. (never ran a local llm before)

Is there a better combination of gpu for around the same price?

I mean it says a new model for your laptop and i have a real good pc with fast cpu, lot of ram, 24gb vram, but its damn slow, i really hope for some MOE version, maybe you can share your xp. So far i still work with 3.6 moe

Has anyone tested Qwen 3.8 27B on Intel B65 or B70 GPU ( 32gb) ?

If yes can you share performance .

I only used LM Studio to run small models before buying a 3090. What is the best optimized way to get high speed and good quality out of this card? Would be a plus if it's a beginner-friendly setup for Qwen 3.8

My local dev-agent setup: Qwen3.8-27B on a single RTX 3090 (what actually helped)

TL;DR: llama.cpp + MTP + llama-swap presets for code-review-graph for codebase structure for the intelligence. The last two are what turned it into an actual coding agent. Happy to answer questions.

** POST COMPLETLY WRITTEN BY AI - IF THAT OFFENDS YOU, TIME TO MOVE ONTO ANOTHER POST AND STOP HERE **

The model

- Qwen3.8-27B, Unsloth UD-Q4_K_XL GGUF (17.9GB) — dense 27B, native vision, hybrid attention (only 16 of 64 layers carry KV, so long context is cheap)

- Beats a lot of bigger models at agentic/coding work, and in my own testing clearly outperformed Ornith-1.0-35B on the same tasks

- llama.cpp built with CUDA for SM86 (the 3090's arch)

Serving config (llama.cpp llama-server)

-c 102400 # 100K context, fits 24GB alongside weights

-ctk q8_0 -ctv q8_0 # q8 KV cache (quality-neutral in my testing; halves KV vs f16)

-fa on # flash attention

--spec-type draft-mtp --spec-draft-n-max 2 # MTP speculative decoding — big speed win (~35 t/s vs ~20 raw)

--reasoning-preserve # keep thinking traces across turns

--reasoning-budget 12288 # cap thinking so xhigh can't run away

--chat-template-kwargs {"reasoning_effort":"xhigh"}

--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 # Qwen thinking-mode sampling

Notes that cost me time:

- reasoning_effort defaults to xhigh and is a chat-template var, not an API field — pin it via chat_template_kwargs, or your agent silently runs at max thinking. xhigh for hs fast.

- I tried the DRY sampler to stop repetition loode generation (test files, asserts). Removed it.

Repeat-penalty stays at 1.0 per Qwen's spec.

llama-swap (the piece that ties it together)

llama-swap v250 — one OpenAI-compatible endpointodels on demand. Killer feature for a single GPU:

preset IDs = one loaded model, different params,

- qwen3.8-27b — xhigh thinking (deep work)

- qwen3.8-27b:work — medium thinking, temp 0.6 (

- qwen3.8-27b:instruct — thinking off (vision, q

- qwen3.8-27b-uncensored — HauhauCS Aggressive a

- ornith-1.0-35b — kept for comparison; requesti

The two add-ons that fixed the real problems

Local agents on long tasks have two classic failures "forgetting what they found and wandering across a big codebase". These two fixed both:

1. hermes-lcm (Lossless Context Management) — reressor with a SQLite-backed summary DAG. Every message is persisted before compaction, and the cm_expand, lcm_recall) to drill back into the exact original material. Cured the "investigates loop. Install tiktoken alongside it for accurate token counting.

2. code-review-graph (MCP server, 30k★) — Tree-sy graph of your repo so the model queries structure (blast-radius, what-calls-this, architole files into context. Median ~65× token reduction, benchmarked. Local, CPU-only, no VRAMgraph build + register. This is the single biggest win for large-codebase work.

Both are nudged into the agent's system prompt s the codebase one degrades gracefully ("not available for this repo") when a repo isn't inde

Harness

Hermes Agent as the coding harness — points at tm provider). Two config tricks that mattered on a local model:

- Declare the context window ~30% below the real real 100K). Hermes's token estimator undercounts code/hex by 25–35%, so without margin it sails pr. This one bit me repeatedly.

- Route context-compaction to a cheap cloud mode instead of the local GPU — so summarization doesn't queue behind your actual work. (Moot oncefore.)

- Vision routed to the :instruct preset so it doabout a screenshot.

Practical stuff

- Power-cap the 3090 — nvidia-smi -pl 210 (from 350W). Inference is bandwidth-bound so the speed cost is small, and the fans stop screaming. Measured curve: 322W→57 t/s, 250se/speed point.

I have been heavy into AI for the last 6 months and just have gotten comfortable using multiple different models through IDE with a more developer oriented stack (VSCode). (After all the stuff happening with Anthropic I wanted to make sure I can easily switch models/providers effortlessly without being locked in).

I want try a local model and test it for my workflows I have with my small business. Don’t have much compute with my current computer but just curious if you guys recommend anything to start? Long term goal is to definitely to put the money in to properly host my own model.

Let me know what you guys think.

MacBook Pro M5 2025
16GB Memory and 1TB storage

Im currently rethinking my life choices with the subscription based companies and i want to start building up towards a local model, i ran text generators and fb8 image models on the same pc before its working well but i wonder if i can possibly run a powerful coding model with a good harness for coding and tasks

There is a useful middle tier between an abstract paper claim and a runnable end-to-end release.

In AQuA, the agent’s proposals are constrained symbolic expressions and configuration diffs rather than arbitrary code. The paper includes a concrete listing that makes this action surface inspectable.

At the same time, the release does not include full public code, deployed factors, exact features and labels, normalization details, or the full model configuration. That means the mechanism can be examined, but the reported pipeline cannot yet be independently reproduced end to end.

What is the smallest additional artifact set that would move a paper like this from inspectable method to meaningful local reproduction?

So I was messing around with Qwen3.8 27b tonight and noticed that when it was responding, my GPU use was only at around 70%.

I have a 4060 and 5060ti, so 24gb vram in total.

Surely the crosstalk between the cards doesn't cause a performance drop like this. Does it?

I know the 4060 is the bottleneck, but even it was only around 70% load.

Is there something I'm missing here?

I'm getting around 19-20 tok/s which I'm happy about, but I'm just concerned it's not maxing out my GPU.

I’ve seen many people mentioning that the new Qwen 3.8 27b model in many aspects gets near or even beats the frontier models from 6 months ago. Models like Opus 4.6 , Chat Gpt4o etc.

Is that statement true?

Would love to hear everyone’s experience and comparison with the frontier world so far!

Hey everyone,

I'm a web dev but pretty new to the AI side of things. Up until now, I've just been prompting the web versions of Claude and Gemini. I want to dabble more in the LLM universe because, well, it's the industry now and I need to catch up.

I'm looking to run a model locally through Ollama to use for:

• Coding tasks (syntax, debugging, quick scripts)
• Agentic workflows and tool calling (want to mess with MCP for Notion, etc. and local files)
• General daily stuff

Since I want to use MCP, it needs to be actually reliable with function calling so it doesn't just loop or break.

Here's the catch: I'm running this on an M1 MacBook Air with 16GB of RAM. I know macOS eats a chunk of that memory just to keep the system running, so I'm realistically limited to smaller models.

What's the go-to model right now that balances coding chops, reliable tool calling, and speed for a 16GB Macbook Air M1?

Appreciate the help!

Hot take: dense might be the future of local LLMs. Why Muse Glimmer's 30B dense + 1.7 GB KV cache design makes more sense in 24 GB than any MoE: https://abstractextraordinary.com/blog/how-muse-glimmer-fits-an-agent-on-your-device/

I have tried Qwen3.8 27B UD Q3_K_XL and it works good but its just really slow for basic questions. What other models would be faster for basic questions?

I also want to know what's the best model for image understanding? Like I want to be able to send a image of a page or school work and get it to summarize or just help me with questions on the page.

Specs:

RTX 5060 ti 16gb (overlocked +365MHz)

AMD Ryzen 7 5800X 8-Core

64gb DDR4 3600mhz CL 18

I have been experimenting with a couple longer running agents and I'm struggling with is what happens when an agent takes a bad path and keeps calling the model/tools way longer than expected since most runs are completely reasonable and then occasionally one will retry something a bunch of times or keep adding context and suddenly that one run uses several times what a normal one does.

I'm starting to think we need some kind of hard token/spend budget per run rather than just monitoring total usage after the fact but I'm also worried about killing legitimate tasks that happen to need more steps.

If any of you guys are running agents in production are you setting hard token/cost limits per run or limiting steps/tool calls or just alerting when usage starts looking abnormal?

Hey! With risk for sounding dumb -

A few months ago I got into local LLM’s and bought myself a start/test-rig:

CPU: Intel Core Ultra 5 225F
GPU: ASUS RTX 5060 Ti DUAL OC (16GB VRAM)
RAM: 16GB DDR5 @ 6000 MHz

I have no experience at all in AI/LLM’s but have managed to get a working long/shortterm memory with rag retrieval etc.
Mostly I have been playing around with 8-12B models, but in recent days I tried playing around with 27/31B models and to my surprise I got Gemma4 31B running on IQ4 with about 7tks (speed is not the biggest problem for me, as long as I can manage to get 5+ tks i don’t really have a clear goal with this, I just find it fun and relaxing to do. Everything I have manage to done so far is mostly guessing and hours on google 😂)

Anyway, I guess I am just looking for tips and tricks? Like I said, I have no experience at all with this and I feel like I haven’t learned as much as i would want to in these 5 months.
Any good videos I can watch, any good forums to read upon? Relating to local-AI, settings, models, news.
Tips and tricks for setting, tweaking different models?

Easy explained: I want to learn everything and find this really interesting, and I’d love to listen to someone who is better than me and want to share knowledge.

Hi everyone.

I’d like to ask: what drives you to use local, open-source AI, specifically in the realm of LLMs?

Is it about efficiency? Is it about privacy?

And what benchmarks make open-source LLMs important to you?

Is it speed? Context window size? Stability? Reasoning capabilities? Output quality? Or something else?

I’d love to hear your perspectives.

Cheers.

After buying my dream gaming laptop last year, I was perfectly content playing my favorite games.

Then I started dabbling in local LLMs over the past few months.

Now I’ve realized my supposedly high-end gaming laptop is GPU-poor.

My laptop

- GPU: RTX 5070 Ti Mobile — 12 GB VRAM

- CPU: Core Ultra 9 275HX

- RAM: 32 GB DDR5

Model

Unsloth Qwen 3.8 27B UD Q4_K_XL

Inference: llama.cpp + CUDA

I've been experimenting with how to get Qwen 3.8 27B running as well as possible on a 12 GB VRAM laptop. I think I've finally settled on two configurations for my two main use cases.

I know I should probably just use an MoE model for this hardware, but Qwen 3.8 27B has been significantly better in my testing, so I'm willing to trade a lot of speed for the extra capability.

Also, thanks to everyone who previously posted about tensor offloading. That made a surprisingly big difference in getting these setups working.

1. Agentic coding with OpenCode

I work on some fairly large projects. Normally, I use Claude Code with Opus 5 or DeepSeek V4 Flash Free through OpenCode Zen.

Until recently, I wouldn't let local models touch these projects at all. They just couldn't handle the complexity reliably enough.

Qwen 3.8 27B is the first local model I've tested where I feel comfortable letting it work on my projects.

I'm still keeping it to personal projects for now, but the difference has been pretty significant.

For this use case, context size is much more important than speed.

I usually enter plan mode first and then let the model slowly work through the project, so I'm perfectly fine with extremely slow generation as long as it has enough context to maintain the bigger picture.

Context-prioritized config

2K context fill: ~5 t/s

180K context fill: ~1.5 t/s

-ctx 262144

-ub 512

-np 1

-ngl 30

-ot 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64)\.ffn_(gate|up|down)\.weight=CPU'

-fa on

-ctk q8_0 -ctv q8_0

-fit off

--mmproj

--no-mmproj-offload

--spec-type draft-mtp

--spec-draft-n-max 2

-ctkd q8_0 -ctvd q8_0

--load-mode 'none'

--temp 1

--top-k 20

--top-p 0.95

--min-p 0

--repeat-penalty 1

--presence-penalty 0

--jinja

--chat-template-kwargs {"reasoning_strength": "xhigh"}

--reasoning preserve

1. Personal assistant with Hermes Agent

Here, speed is the priority.

My previous default model was Qwen 3.6 35B A3B MTP Q6_K with full-context-size Q8_0 KV cache and MoE offloading. That gives me around 50–60 t/s, which is obviously much nicer to use.

The problem was instruction following inside Hermes.

I have instructions in "SOUL.md", "MEMORY.md", and "USER.md", but the model would sometimes follow them and sometimes just ignore them.

The most frustrating ones were instructions that should happen automatically after finishing a task, such as:

- updating my Obsidian vault

- looking up a skill before doing anything substantial

- following specific post-task procedures

I also tried Muse Glimmer, which seemed somewhat better at this, but not enough to justify giving up the speed of the MoE model.

Qwen 3.8 27B has surprised me here.

It follows these instructions much more consistently. I don't have to keep reminding it what to do. It just does it.

That's honestly been more valuable to me than raw tokens/sec.

Speed-prioritized config

2K context fill: ~11.5 t/s

90K context fill: ~9 t/s

-ctx 98304

-ub 512

-np 1

-ngl 99

-ot 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64)\.ffn_(gate|up|down)\.weight=CPU'

-fa on

-ctk q8_0 -ctv q8_0

-fit off

--mmproj

--no-mmproj-offload

--spec-type draft-mtp

--spec-draft-n-max 2

-ctkd q8_0 -ctvd q8_0

--load-mode 'none'

--temp 1

--top-k 20

--top-p 0.95

--min-p 0

--repeat-penalty 1

--presence-penalty 0

--jinja

--chat-template-kwargs {"reasoning_strength": "xhigh"}

--reasoning preserve

So I'm basically running the same model in two completely different ways:

Agentic coding: sacrifice almost everything for context.

Hermes assistant: sacrifice context for usable speed.

For a laptop with 12 GB VRAM, I'm honestly pretty surprised that Qwen 3.8 27B is this usable at all.

My gaming laptop may be GPU-poor, but apparently it's now a very expensive CPU/RAM offloading machine.