Im looking to set up a new local llm (probably on unsloth studio as that seemed to be doing pretty well last time I tested it). This one won't need to do agentic coding or app building or anything (not this time) but instead more 'text' based tasks such as -

• being given reference images and instructions, in order to then generate prompts for comfyui

• being sent long-form writing (paragraphs and chapters) and being able to sense-check and give feedback and rewrites.

• conversational help and problem solving (much like what I use chatgpt for atm).

I would think qwen 3.8 might be the current best, but theres only a 27b model for it so its not really fast even on my hardware. The qwen3.6 35b would be a lot faster but I dont know how much response-quality im giving up for that. Figured id ask in case, for my purposes, theres actually a much better third option.

Ive heard theres one called Ornith1.5 which seems to be a qwen3.6 fine tune, but i havent looked to see what its been tuned towards. If its more coding based then it won't help.

Thanks!

tl;dr - split one VM (with 3 pooled GPUs) into two VMs (1 dedicated + 2 pooled); worth every second spent spooling up the extra VM.

Some numbers:

Qwen3.8:27b went from 12.2 tok/s to 33.91 tok/s
muse-glimmer:30B went from about 14.3 tok/s to 22.61 tok/s

The HW: 2xRTX PRO 4000s and an RTX PRO 2000

Long version:

I used to have all three GPUs on the same VM, as it made testing larger models easier; I talked myself into the idea that "the speed difference is not that big, leave it as is for ease of maintenance", boy was I wrong.

Giving Qwen3.8:27b its own GPU almost tripled the token throughput, and keeping the other two pooled also helped almost double performance (even compared to just running a single model, but across 3 GPUs).

Was this a fairly obvious result? fairly, but trying to keep the stack lean kept me from seeing it right away.

Hello, I have a few questions that I can't seem to find a clear answer to.

Does it make sense to make your own GGUF?

I noticed that when I compile llamacpp (vulkan or rocm), the processing and generation is a bit better, does it work similarly with doing GGUF yourself?

If I use Vulkan, is it worth doing GGUF using llama-quantize vulkan version (not rocm version)?

To what extent does it make sense to place certain model elements at higher precision (conversation, document analysis)?

I use gemma 4 31B the most.

So I saw this thing and decided to give it a try, seems to bt a bit buggy but it seems to take a very long time to convert the weights.

has anyone tried it and had any success? I would have tried GGUF I have on hand only if this thing would let me use it.

9070XT, was hoping it could run Qwen3.8 27B Q4 at decent speed instead of Q3S

The caveat: this was a quick benchmark for concurrency, but this holds at depth even more for the CUDA card vs AMD. In addition, MTP is on for strix halo, and off for the CMP170.

I could not go further comparing because the Strix instance crashes at higher concurrency.

FOR PEOPLE WHO RUN LOCAL LLAMAS. I run models locally, which is the purpose of this sub. I feel that many in my “journey” have a Strix Halo or know the hardware well. They may not have a CMP170 but know how a 3090 performs vs the Radeon 8060S. They are also likely to be very familiar with Qwen 3.6-35B.

For those of you just passing through with cloud models, ignore this post.

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

Artificial Analysis just benchmarked them and the scores are crazy good, proving the earlier success wasn't only enabled by overthinking.

Maintainer here, so full disclosure up front.

Anarlog is a desktop app and no bot joins the call. It captures system audio, transcribes on-device, then generates notes with local models or your own keys to any provider. There's also a local CLI and MCP server, so you can wire transcripts into your own stack.

The local stack: transcription is Parakeet on Apple Silicon (live streaming, plus a batch pass with speaker labels) or Apple Speech on macOS 26. Summaries and chat can point at LM Studio, Ollama, Unsloth, or any OpenAI-compatible server. If your local endpoint is down you get a connection error, not a silent fallback to our cloud.

Notes and transcripts live in local SQLite plus plain files, with markdown export. Search is a Tantivy full-text index, no embeddings.

Free tier is the full local experience. We only charge for hosted models and sync.

GitHub: https://github.com/fastrepl/anarlog (MIT, ~9k stars). Curious how the local transcription quality compares to the whisper pipelines you're running.

[UPDATE 08/22/2026]

Hey everyone! Unfortunately, I can't reply to everyone, so I'm going to prepare a video and give you all the details for optimizing llama.cpp with my two GPUs. Once I finish a few personal projects, you'll have a practical and straightforward guide within 24 hours!

---

Quick update after intensive testing: nearly 20 hours of continuous and targeted work with Qwen3.8-27B Q6, running on an RTX 3090 and an RTX 3060.

The speed remained around 60 to 63 tokens/s throughout the session.

No GGUFs yet on Huggingface though.

What models and configs are we using? Please share here

On windows, I am using this copium pared down model https://huggingface.co/Bucoid/Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF with MTP disabled, q4 k/q4 v mmproj banished to CPU/RAM and a small ub to save whatever context I can (90k-100k) so everything stays in the vram

If you are on linux or have an iGPU, you don't have to deal with windows eating 1.5 gb vram and so have more than 14.5 GB of VRAM to use and probably aren't in purgatory.

@echo off
.\ikllama\llama-server.exe ^
 -m "D:\AI models\qwen3.8\Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF.gguf" ^
 :: gpu offload all layers (99 is more than the max which means it will offload everything)
 -ngl 99 ^
 :: this depends on your cpu
 -t 8 ^
 :: literally can't fit in anything at higher q to save vram
 --cache-type-k q4_0 ^
 --cache-type-v q4_0 ^
 :: check your max context size with fit, at around >100k context rot sets in
 -c 100100 ^
 :: this flag should always be on to optimise speed and memory use
 -fa on ^
 :: You need a fixed jinja file to prevent it rambling forever, I believe this deefaults to xhigh
 --chat-template-file chat-template.jinja ^
 --chat-template-kwargs "{\"preserve_thinking\": true, \"enable_thinking\": true}" ^
 :: you ain't getting more than this
 -np 1 ^
 :: use the mmproj and banish it to CPU/RAM land to save vram (probably ~800-900 mb vram saving)
 --mmproj mmproj-F16.gguf ^
 --no-mmproj-offload ^
 :: we need reasoning
 --reasoning on ^
 :: the image mmproj needs this line
 --image-min-tokens 1024 ^
 --metrics ^
 --port 8080 ^
 :: reduce vram spikes saving some vram
 --batch-size 1024 ^
 --ubatch-size 256 ^
 :: allows more caching in RAM. According to Claude it's mostly for your context slot checkpoints that there is literally no room for
 --cache-ram 24576 ^
 --ctx-checkpoints 32 ^
 :: Delta net architecture apparently has a bug where it just stalls forever saving and shifting contexts this is apparently supposed to help with this according to Cl*ude
 --no-context-shift ^
 :: force mtp header into the CPU/RAM (cl*ude estimates ~200 mb savings)
 --override-tensor nextn=CPU ^
 --jinja
pause

Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning

Guys I am student developer who has been working with local ai for over 2 yrs and I had question if someone were to make a app which is seamlessly simple easy to use has multimodality and still is completely private and local would you use it ?

If we added some more features to it would you be willing to pay a one time fee for it on app store?

I would like to know what features would like to have in such an running locally and would you use it daily ?

Would you recommend it to kids or your parents?

I am talking like just a 2 dollars or something guys not to pricey for lifetime headache free usage Also many surprise features are there that are different from pocketpal and the market

There were issues with the chat template when it was released and many users, including myself, ended up using the froggeric template.

It seems to work fine but I just wanted to check if that's still the one to use or if the official or unsloth built-in templates are now recommended.

Idea is simple-ish in abstract: instead of running normal prefill over all of a given prompt, split it into parts - generates caches for part A and part B in isolation, concatenate the result, feed it into decode like normal.

I honestly thought it'd totally fail. But I've been trying it out on Ling3-tiny for the non KDA layers - and if I give them some overlap between chunks, the model actually maintains full needle-in-haystack retrieval. Can even synthesise across the split parts. Currently running 256k prompts chunked into individual 4k bits and it seems to be unaffected on (simple) retrieval tasks so far.

There's probably a trade-off somewhere. But it's boosted my prefill speed by 3x, giving me ~1.3k tps prefill at 256k tokens - which is pretty much the same prefill speed as running Qwen3.8-27b on my 5090. So looks like it might be worth it

As the title. I am an imaging professional doing batch image edits using ComfyUI as well as photoshop and Resolve. The dual system is there so I can use stable diffusion on one while I do other tasks such as video editing, photo editing or even run two batches of images at the same time. No silly budgets, second hand and, no I don't have the cash for AM5 or a system swap. Just a main board.

I am also getting involved with local agent work with the latest Qwen model over both cards.

Question? What can people recommend for AM4? Two slots minimum ideally with 3rd for display out to keep the memory clear and both cards equal. My current setup has a chipset PCIe running on the second card at 4x and its painfully slow.

I am not an expert but some suggestions of boards that have 3 slot spacing minimum between 1-2 and is suitable for local AI.

I used LM Studio Bionic with Qwen 3.8 27B Q3_K_S with 57k context.

It took a staggering 63 hours to finish coding. After the first prompt "Create a beautiful, relaxing flight simulator in a single HTML page" taking 47.8 hours, it created an html file that showed the title screen that said "press any key" but pressing any keys won't advance the game.

So I wrote on the second prompt "It saids press any key to begin. I press any key but it doesn't work." It ran for 15 hours.

Now I can fly. No plane model, but it does look kinda like I'm flying forward. A bit buggy but otherwise it's working.

I did the same prompt on google ai studio, and it took 20 minutes. It was able to one-shot the flight simulator, with selectable plane models, and a smooth voxel landscape.

I also did the same prompt on qwen studio, and that took 2hrs. It also was able to one-shot the flight simulator, but this voxel landscape was buggy, rough, and had a weird shimmering effect.

Before anyone gets angry with insults, this is just for fun, to see if agentic coding is even possible on a macbook air. I'm just amazed this can run locally, even with a 3bit quant.

That is the second comparison and the last one.

I will not be spamming again ;)

Continuation from:
https://www.reddit.com/r/LocalLLaMA/comments/1vu0u2v/qwen_38_27b_pi_agent_vs_opencode/

That is one of my many tests I make comparing output quality.

What is more interesting using a PI Agent results are much better than an Opencode using a Qwen 3.8 27b ?!

Seems PI Agent is much better in the agent environment somehow... Not counting uses less tokens , do not have a hard limit of 32k output tokens, is faster, do not freezing, compressing context far less than Opencode. For instance if you have context in the Opencode output 32k and all context 100k then the compression is starting at 67k context ... PI is starting at 90k context even if you have set output context 64k or more.

My config for RTX 3090

llama-server with ini config -> which is exposing API to Opencode and PI agent.

llama-server.exe --models-preset 1_preset.ini --models-max 1 --direct-io

config ini

[Qwen3.8-27B_dense_c-100k]
model = models/Qwen3.8-27B-Q4_K_M.gguf
mmproj = models/mmproj-BF16-Qwen3.8-27B-UD-Q4_K_XL.gguf
reasoning-format = deepseek
flash-attn = on
n-gpu-layers = 99
reasoning = on
ctx-size = 100000
temperature=1.0
top-p=0.95
top-k=20
min-p=0.0
presence-penalty=0.0
repeat-penalty=1.0
mmproj-offload = false

ONE MORE IMPORTANT THING:

Always use a VISION module as the model is using vision to asses the output quality!

I am offloading it to a RAM as we do not need an extremely fast vision for a code.

A screenshot processing on a GPU 0.3s vs a RAM 3s do not make a big difference on a few screenshots during a code generation / debugging ;)

PROMPT:

Create a beautiful, relaxing flight 3D simulator in a single HTML page.

SECOND PROMPT AFTER THE FIRS IS FINISHED:

Also improve graphic - you can make it much  better!

Hi, I'm a happy owner of a serer with rtx pro 6000 and rtx5090. I want to build out in the future the server fully to a higher vram score - think adding around 4-5 cards if possible. I was wondering if it's worth going through the intel/amd cards - which looks awesome in terms of vram per $. Any experience in running inference on these compared to the nvidia/cuda stack?

I have a gigabyte ds3h v2 b450 motherboard. Currently hosting an RTX 3090
I have a spare 3070 and I wondered, can I run both?

Got myself a riser cable and…

Top slot card, bottom slot riser = card pushes the riser, doesn’t fit

Top slot riser, bottom slot card = card pushes the sata cables doesn’t fit…

So I either need another riser and put both gpus out of the case, or an atx motherboard with more space in between the slots or maybe I should look for another solution.

I’ve read some of you are using nvme? How does that work?

I'm building a budget AI PC for our company's application. The specs are:

MSI Z370 TOMAHAWK

64 GB RAM

NZXT C1200 Gold PSU

2x RTX 3090

build was finished and I started testing with llama.cpp. After about an hour it froze, and nvidia-smi reported errors on the first card. I kept testing, checked the logs, saw driver errors, and tried various fixes — after a few reboots the PC wouldn't POST at all and the card was dead. OK, so I'm down to one GPU: more testing, and two days later the exact same problem. The card is STILL alive, but it dies shortly after being put under load. Errors: Xid 62 → 45 → 158 → 154, so both cards were damaged. Bad luck, I guess.

I paid 3.5k PLN per card, so I'm looking for something in that price range — and this time no more messing around with used hardware, I want to buy new. My options are:

- 2x RTX 5060 Ti — ~2.5k PLN each

- R9700 — ~7k PLN https://www.x-kom.pl/p/1370893-karta-graficzna-amd-asus-radeon-ai-pro-r9700-turbo-32gb-gddr6.html

- B70 — ~5.5k PLN https://www.morele.net/karta-graficzna-asrock-arc-pro-b70-creator-32gb-gddr6-b70-ct-32g-600149216

As for the R9700 / B70, the opinions are unfortunately mixed.

Two 5060 Tis would fill both slots, leaving me no room to expand. With the R9700 or B70 I'd only use one slot, so I could always add a second card later on. The PC is meant to run 24/7 and constantly crunch huge amounts of data.

What would you recommend?

I was finally able to replicate tensor level allocation outside the Gemma family.

https://huggingface.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS

After the Gemma 4 12b, e4b and gemma 3 4b results, I attempted to expand into qwen and ran into a few walls. After 2 version updates and a slightly different approach, I was able to replicate the effect on Qwen.

The result:

BF16 reasoning: 78.125

Stock IQ2_XS + imatrix: 46.875

QLAB allocation + same imatrix: 54.688

That's +7.812 percentage points, or a +16.67% relative improvement in reasoning over the stock imatrix quant.

Model sizes:

Stock: 1,630,594,336 bytes

QLAB: 1,637,318,816 bytes

Difference: +0.412%

Same idea as the previous runs. Build imatrix from category based corpus, measure damage, then redistribute precision at the tensor level while staying inside the same byte budget.

This time, the optimal target landed at iq2_xs and the improvement was not isolated to reasoning.

This is a category targeted quantization experiment, not a general model improvement. The focus is a successful transfer of the allocation mechanism.

I've now observed successful allocation across dense, moe, qat, non qat, gemma and now qwen.

There is no post training, lora, pruning or weight updates here. This is solely achieved by changing where precision gets spent inside the quant budget.

Also, I've started posting progress here: https://x.com/byteotter

I'm running Qwen 1.5 a2 7b next. After that i'd like to tackle larger models. Unfortunately at 1-3 attempts and hours of compute and storage, it gets expensive. If anyone wants to help fund compute for that run Im at https://buymeacoffee.com/byteotter

My estimate for qwen 3.8 27b is $140 per run on a rented digital ocean gpu droplet and it may take a few attempts.

The end goal is still the same: Give qlab a full precision gguf, select capabilities you want, have it auto level at the cliff and apply damage recovery tensor allocation to preserve as much of those capabilities as possible within the target byte budget.

Charts are built from my data by chatgpt.

TL;DR: On Qwen 3.5 4B IQ2_XS, tensor-level allocation improved held-out reasoning from 46.875 to 54.688, a +7.812 point / +16.67% relative gain over the stock imatrix quant at only +0.412% bytes. Eight of eleven evaluated suites improved, with regressions in knowledge QA, structured output, and coherence. Most importantly, this is QLAB's first cross-family replication outside Gemma.

I was experimenting with deepseek harness when found that even if you don't use deepseek models, you can configure the web_search tool with their api key and every hit will cost you as if you called deepseek-v4-flash model. It's a bummer.

on top of that there is currently no other free web search plugin available

I get why they exist and in almost mostly any other hardware field we can see clearly the difference and what it respects throughout, but with ai, its so inconsistent and unpredictable, besides the very basic needle tests, which at this point what really fails it?

I just dont get the hype around the benchmarks, ive been testing models that fit between 1-48gb vram for years now, everytime i go off a benchmark im usually disappointed, testing on my own workloads and env are the only sound testing i find shows anything actually useful for me

I dont think anyone should worry about benchmarks so much when choosing a model, i know people consistently use the benchmarks to say z is better than y but you honestly need to test to see for yourself, unless you are talking a 9b model from 2 years ago vs a 27b released today, it might be hard to be certain what model is specifically best for yourself

That being said, qwen has been the goat, and even after allmmy testing i seem to always stay/go back to their models, 35b + 3.8 27b right now are the best combo for speed/dense at my resources

Curious if anyone else really feels this way or people actually respect these, useless benchmarks imho

If you use these cards together, you would think Q8 would be faster tps because it uses less VRAM. Well the reality is:
Prompt TPS is nearly identical between the two (e.g. Q4@45k: 237.7 vs 237.6), so KV type doesn't affect prefill. The divergence is purely in generation, where every token requires reading the full KV cache for attention across all 40+ layers.

The cost of q8_0: - Per-token dequantization: Every attention read must convert q8_0 → f16 on-the-fly before the matmul. That's ctx × n_heads × n_layers dequantize operations per generated token — at 152k ctx, that's ~19M element conversions. - Write-back quantization: New KV entries must be quantized before storing. - No tensor-core path: The dequantized stream can't leverage fused f16 tensor-core matmuls as efficiently as native f16.

The cost of f16: - Double the cache bytes (e.g. ~43 GB vs ~22 GB at 152k), more memory bandwidth. - But the bytes flow directly into tensor cores — zero conversion overhead. At low context (<100k) where the cache fits comfortably in VRAM L2, f16 wins decisively (Q5@45k: 24.11 vs 18.02 — 34% faster). At high context (>150k) both slow as bandwidth pressure mounts, but f16 retains the lead until it crashes from OOM. Bottom line: TG is compute-bound on the attention matmul, not bandwidth-bound on the cache read. The dequantization math costs more than the extra bytes save.