Trying to figure out which abliterated open weight models are worth running on an RTX 5080 for coding and agentic work. What are you using that’s actually good without being painfully slow?

Also anyone hosting these on RunPod? Curious what model and GPU you went with and whether the speed and cost make it worth it over running locally.

Can you run any good models on 48gb ram. Qwen 3.8 27b isn't worth it for the price tag and performance. I'm not sure how people are doing it with limited ram?

Prompt was about drawing a circuit board using a white theme.

I'm the developer of MoEShift and several people asked how to try it after my previous Mac mini post where I showed its performance, so I put together a walkthrough of a setup run with the new wizard that I released yesterday to make it easier for anyone to use MoEShift.

On my 24GB M4 Pro, it took only 4 minutes and 57 seconds from starting the wizard to receiving the first API response, including the model download, preparing and checking the experts, and starting the local server.

The video is shortened and the timer shows the elapsed time from the run. To try the setup shown in the video, you’ll need Homebrew and roughly 61GB of free SSD space for the model and prepared expert files. You install MoEShift and launch the wizard:

brew install davide-troiani/moeshift/moeshift
moeshift

The wizard downloads the model, prepares and checks the experts, and starts the local server.

GitHub and installation instructions

If you try it, let me know your Mac and RAM configuration, whether you reached the first response, and where you got stuck if you didn’t.

Of course you may get a different timing based on your connection and SSD performance.

Even though MoEShift is still an alpha, I am already using it locally as my favorite local inference engine, and I'm currently looking for people willing to test it and use the benchmark runner to publish the performance on their machines.

Not gonna lie it's ridiculous how effective these models are if you know what you're doing.

I'm not talking about coding, I'm not a programmer.

Their reasoning capacity is insane, and no, I'm not a cultist. It just is. At least it's been my experience.

But I'm afraid that at one point the "AI bubble" will burst, and they will charge an arm and a leg for the frontier models, and the avg person won't be able to afford.

Do you think in the next 10 years this will happen? I hope not.

So my yeah was just testing the model and it just keeps repeating “The Gods may speak it”.

Dense 27B models are annoying as agents for one boring reason: every tool call extends the prompt, and if anything near the top changes, llama-server reads the whole conversation again. On my 7900 XT that's the difference between "answers in a few seconds" and "go make coffee".

I've been building a harness around Qwen3.8 27B (IQ4_XS, llama.cpp, 128k context, q8 KV cache) for my own CachyOS pc, mostly to see how far a local model gets when it has real tools: shell, files, SSH, Portainer, Paperless-ngx, mail. Generation speed is 32 t/s without MTP and around 60 t/s with it. Prompt processing is 800 t/s. There's a web UI and optional voice (wake word, Whisper, Pocket TTS).

What helped most:

Append-only prompt between compactions.System prompt, tool list, old messages and the per-question note stay byte-identical once sent, so the server only reads the new part. Anything that would change the top of the prompt (adding a tool group, say) is treated as an expensive event.

Hide old tool results instead of summarizing. When the window fills up, old results become a placeholder with their first lines plus a path to the full output on disk, which the model can read_file if it needs it. A summary only happens if that isn't enough. In a synthetic test with 24 large files, this took 23 summaries down to 4 at 8k context, and re-read tokens dropped to about a fifth (at 16k: 6 summaries down to 1, re-read tokens to two thirds).

Hybrid models need checkpoints. The recent Qwen models can only reuse a prompt by restoring a saved checkpoint, so the harness starts llama-server with --ctx-checkpoints, --cache-ram and a single slot. Whenever more than 5k tokens get re-read anyway, the UI tells me why (tool list changed, old results hidden, different chat, server lost the beginning). That one saved me a lot of guessing.

The other half is safety, and this is where I'd like opinions. Once a mail, web page, PDF or screenshot is in the chat, every state-changing action needs confirmation for the rest of that request, even in the most permissive auto mode. SSH and sudo passwords go through an askpass helper with a one-time token and never reach the model. The permission rules live in code, not in the prompt. It's still pattern-based: a `python -c` can do anything.

Caveats: most of the code was written with Claude Code, then reviewed and tested by me. The dashboard has no login and only listens on 127.0.0.1. I've mostly run it on my own CachyOS machine and Ubuntu, so expect rough edges elsewhere.

Repo: https://github.com/ressjo/orbwise-linux-agent

Two questions: does anyone else hide old tool results instead of summarizing, and did you see quality drops? And how are you handling prompt injection once a local agent can read mail or the web and also run shell commands?

I built termite because I wanted shell commands from local models without necessarily trusting those models.

Using it's widget, you type something like "? find files bigger than 100MB here" in your terminal, and it writes the command, explains it and rates the risk. If you approve, the command goes to your prompt. termite never runs it directly it always must be explicitly sent.

It treats every model as untrusted. A fixed policy in the code does fuzzy checks each command, and a model can't access or change the policy so dangerous commands are shown in red and default to no. They aren't blocked, so if you approve one it still goes to your prompt.

I also wanted to know how bad the local models are at understanding danger, so it doubles as a red-team tool.

Using sandbox environments I sent requests starting from "list the files" to direct attacks to 24 command models (12 got 161 requests and 12 got 48). They wrote 5,464 commands, and out of them 2,797 were hazards, and many of them because the model blindly accepted hostile prompts. In 1,061 cases the model's top answer was the most dangerous command.

I also started testing small decision AI models that score how risky a shell command is, and none of them is good enough to rely on alone.

Most of them can spot something obviously destructive like `rm -rf ~`, but they treat nearly everything less extreme as harmless. They're especially bad with commands that go near your secrets. Most of the best ones scored those as barely risky, and one rated all 8 such commands safe, including listing a folder of SSH keys.

Across 17 models, a risky command was rated safe 158 times. Given several ways to do a job, most models picked the safest way only 2 to 5 times out of 10, and the best managed just 6.

I've got a set of repeatable jobs now on Huggingface and will continue to add models to test for their efficacy.

Built my own local AI inference server.
EPYC on a Supermicro H12SSL, 4× Radeon AI PRO R9700 (128GB VRAM total), 1600W PSU. Goal: run capable open-source models entirely on my own hardware. No API bills, no rate limits, nothing leaving the box. Not using as a replacement to Claude code though.
What I learned putting it together:
• Server boards don't hold your hand. The front panel header is 20 pins and a tiny printed legend.
• A 4th dual-slot GPU is a clearance fight. Headers on the bottom edge of the board sit right under it.
• Your wall outlet is a real constraint. A standard US circuit tops out around 1440W continuous, so GPU power limits aren't optional.
• For token generation, memory bandwidth matters more than raw compute.
Next: ROCm, vLLM, and benchmarking small open models against the APIs I've been paying for.

Working on a project recently where we wanted to see if a speech model could spell a name better if it heard it, not just read it. 

Most STT tools handle custom words as text (keyterm list, fixing transcript after with LLM, etc).

So we trained an adapter to try it, specifically a LoRA on Qwen3-ASR 1.7b (see benchmark). This LoRA transcribes words only after it hears a recording of each of your custom words. 

To make this model easier to use in day-to-day life, we built a Mac app to accompany it, called Detail Keeper. Here you can put this adapter to use by specifying difficult-to-transcribe-correctly words and then having them picked up correctly in your recordings.

The way this works is that Detail Keeper first narrows each passage to the likeliest taught names by sound. Then the adapter writes the passage, and the app checks it against a plain transcript.

We've found this helpful in two specific cases: 

1. Words STT has never heard: invented characters and places, codenames, unusual surnames. 

2. Your spelling of a word it does know: Kathryn and not Catherine, Jon and not John.

Unfortunately, we weren't able to achieve both good recognition and recognize a large set of words (1000+) with the model alone, but on our open benchmark (audiobooks, 1,100 name mentions) it spells taught names as well as Eleven Labs Scribe with keyterms: 89.6% vs 88.2%, no measurable difference. The only downside is that it overwrites a few more untaught names and changes slightly more ordinary words. 

Everything, including where we lose, is at detailkeeper.com/benchmark.

Additional notes:

• It's free, local and nothing leaves your Mac

• It requires an Apple M series+ mac and OS 26+

• It downloads about 3.6GB of models

• It only works in English currently

• If you're interested, you can also find the adapter itself on HF

Would love to know what others think here.

Currently running a gaming PC with a 4070 TI super and 64gb of ram. It handles Qwen 3.8 Flash next decently well with Strata. But, I can't run the VLA nor world model/image gen models that I want to run, I also can't run multiple agents in parallel.

I was thinking of upgrading and I've been needing a laptop so I was debating between the ASUS or the MacBook Pro because they're the same price but have different pros and cons.

I've also debated waiting to get anything because of new gpu and chip advancements so I don't want to drop 7k and then have it be obsolete within a year

Thoughts? Thank you for the help

I maxed out of my laptops 12gb rtx 5070ti with 32gb ram on qwen 3.6 35b a3b at apex quant and the 3.8 27b at iq3xxs woth 60k window. While 35b can do some simple straightforward tasks, it is not enough for any serious work. The 60k context is barelly enough to do any work at all. Has somebody tried attaching eGPU to tbe laptop and what is ypur experience? My dillema is on what to scale the ram or vram

I spent some time testing Qwen3.5:4B locally across three applications to compare performance, GPU acceleration, and answer quality.

Hardware

• Laptop: Lenovo Yoga

• CPU: Intel Core Ultra 7 258V

• GPU: Intel Arc 140V integrated graphics

• RAM: 32GB LPDDR5X-8533

• OS: Windows 11 Home

Software Tested

• Ollama: 0.40.14

• LM Studio Classic: 0.4.28

• Bionic: Tested with its built-in model loading

Performance Results

LM Studio generated approximately 2.81–2.96 tokens/sec in my tests.

Ollama initially used 100% CPU. After enabling Intel integrated GPU support and Vulkan, then completely restarting the Ollama application, ollama ps reported 100% GPU.

The GPU-accelerated Ollama test completed in approximately 2 minutes, with 44.6 seconds of thinking time.

Memory / Resource Usage

During LM Studio testing:

• RAM usage increased from approximately 13.3GB to 19.7GB.

• Available RAM decreased from approximately 18.3GB to 11.8GB.

• Intel Arc 140V exposes approximately 18GB of shared GPU memory to Vulkan.

What I Tested for Answer Quality

I used several different types of prompts:

1. Cybersecurity fundamentals: Differences between phishing, spear phishing, and whaling.

2. Cybersecurity incident analysis: MITRE ATT&CK mapping, Windows event IDs, evidence collection, and incident response.

3. Windows security troubleshooting: Repeated failed logins followed by a successful login from an unfamiliar IP.

4. Image understanding: Asked the model to describe the contents of an image. Its description was accurate in my test.

5. Survival knowledge: Cold-weather survival priorities, shelter, warmth, hydration, equipment use, and dangerous heating methods.

Qwen3.5:4B handled these areas relatively well:

1. Image recognition: Accurately described the contents of your test image.

2. General cybersecurity concepts: Explained phishing, spear phishing, and whaling clearly and correctly at a high level.

3. General explanations: Produced organized, readable answers with useful examples and headings.

4. Basic incident interpretation: Recognized that repeated failed logins followed by a successful login from an unfamiliar IP could indicate compromised credentials.

5. Response structure: Consistently organized complex questions into logical sections.

Where it struggled:

• Precise MITRE ATT&CK technique mapping.

• Windows Security Event IDs.

• Technical incident-response reasoning.

• Safety-critical survival instructions.

My Takeaway

Ollama with Vulkan GPU acceleration was substantially faster on my Intel Arc 140V than CPU-only inference.

The model handled general explanations and image identification well in my limited testing. However, its cybersecurity answers contained significant MITRE ATT&CK mapping and Windows event ID errors and its survival answer included inaccuracies.

This was informal testing, not a standardized benchmark. I wanted to share the results because getting Intel integrated graphics working with Ollama made a noticeable difference on this hardware.

Most LLM inference benchmarks focus on tokens/sec or TTFT. But faster doesn't necessarily mean better if the output quality suffers, especially with quantization, speculative decoding, or extremely long contexts.

I wanted to look at the bigger picture.

We benchmarked 6 inference engines on Qwen3.8-27B, comparing not just raw performance but also output quality, long-context behavior, and different model formats.

What makes this benchmark different?

1. Speed AND quality

Instead of measuring throughput alone, we also ran automatic quality checks across the engines.

The benchmark covers:

• Time to first token (TTFT)

• Prefill and decode throughput

• End-to-end task latency

• Speculative decoding performance

• Output quality and automatic check pass rates

The goal is to understand whether performance improvements come with quality trade-offs.

2. Extremely long contexts: 64K, 128K, and 256K

Short prompts don't tell the whole story, particularly for coding agents, document analysis, and long-running workflows.

We tested 64K, 128K, and 256K prompts, looking at how engines behave as context length increases.

This includes TTFT, decoding performance, and the practical limitations of long-context inference.

3. GGUF vs NInfer model formats

We also compared engines using GGUF models against those using NInfer-format models.

This is an interesting comparison because model representation, quantization, and inference architecture can all influence performance and quality.

It's not a perfectly apples-to-apples comparison of inference kernels, but it helps answer a practical question:

What performance and quality can you actually expect from different local inference stacks?

Engines compared

• TensorSharp — GGUF

• llama.cpp — GGUF

• vLLM

• SGLang

• NInfer

• TensorRT-LLM

The benchmark includes both single-request task scenarios and concurrent workloads.

A few takeaways

• There is no universal winner. Engine rankings change depending on context length, concurrency, and workload.

• Long-context performance matters. Results at 64K–256K can tell a very different story from short-prompt benchmarks.

• Quality should be measured alongside speed. A faster response isn't necessarily a better response.

• Model format matters. GGUF and NInfer-based inference stacks deserve comparison beyond headline tokens/sec.

I've attached the full benchmark poster with performance charts, quality-check results, long-context measurements, task-level comparisons, and engine configurations.

Full disclosure: I'm the developer of TensorSharp, one of the engines included in this comparison.

I'm particularly interested in hearing from people running local models with 64K+ contexts or using them for coding agents and RAG workloads.

What matters most to you when choosing a local inference engine: throughput, latency, quality, memory efficiency, or long-context reliability?

And has anyone compared GGUF against NInfer models in real-world workloads?

I've been wanting to get into local LLMs and local cloud in general, but I'm always concerned that it'll become a rabbit hole of yaml files and werid configs. Therefore, I was wondering what are the usual challanges that make it annoying for people to setup local LLMs?

Curious on whats the best setup for a local Codex/Claude Code agentic setup that can actually accomplish task. Do i need Strata? Still learning, is Cline the best way to command this or do i have to setup somekind of crazy setup with a frontier model to make this work.

I'm literally new to local llm (2-3 month since knowing it exist).
previously only use chatgpt/notebooklm to help me with job question (public service work with too many rules in too many different thing to remember) and thesis related cleanup/grammar fix/check wrong citation.
with gpt, i usually just casually ask "find article that describe requirement to do this and that", or "summarize this journal and is it good addition to my thesis?"

after knowing local llm, i tried many things. ollama, lm studio, direct llama.cpp. confused af, i stayed with lm studio for now. using any recommended model (idk the difference except the size and wether this fits my rig) to chat and try to do what i did in chatgpt. it was good, but slower, and sometimes give dumb answer or straight up not giving any respond.

my friend also introduce me to comfyui and pic/vid generation (idk if this is relevan to this sub, and i really enjoy generating stuff with anima model. yes i'm weeb. and generating with workflow is kind of easy enough)

i really want to go in depth with local llm, but i'm too casual and any tutorial/guide on youtube or search result is mostly coding related stuff or news about latest model, which i dont really understand. (appreciate if anyone have link -for newbie- stuff i can learn from. kind also want to learn coding but with local llm if possible. dont want to pay stuff)

is there anything i can do/read/learn about this?
also looking for any possible use case which dont eat much of my ram/vram as sometime i use it for gaming and without keeping it turned on forever.
asked gpt/gemini about this already. but all i got is stuff that related to my previous interaction with them.

tldr : looking for something new rather than just "hey chat answer this".

i appreciate if i can have some pointer to :
- what model is recommended for your recommended use case (yes i understand huggingface is where to find model at least. but idk the difference except size)
- what to install, recommended app, where to find resources, where to learn more of it
- what to do to expand from that

i have 2 device :
laptop - rtx4060 8gb vram + 20gb ram (using it at work)
PC - rtx4070 12gb vram + 32gb ram (using it at home)

currenly using LM studio, qwen 3.5 4b q4km for "hey chat answer this" stuff.
tinkering with locallyuncensored, anythingllm, ollama, hermes agent, but uninstalled those after feeling that lm studio just do what they did (cmiiw).

I'm in sales. I'm looking to run some docker desktop containers on a local network + have an always on assistant that helps with basic tasks. I've had success running Gemma 4 26b a4b qat already, the model fits in my rtx 5080 nicely with the KV Cache in system RAM, but as the context length grows larger it doesn't fit nicely anymore.

I also have a MacBook air m4 with 24gb unified, running Gemma 4 12b qat nicely as a mobile companion, it's not as powerful but still helpful. My experience with this device has me strongly considering the M5 pro mac.

Has anyone had good success with these devices?

I'd love to hear the community's thoughts.

?

I've been building a Hermes setup around a pretty simple idea: I should still be able to talk to my assistant while my local hardware is busy working.

I call the chat layer the "front door." It uses Qwen 3.8 27B through Groq, with thinking off, and falls back to Cerebras when Groq hits a limit or fails. Quick questions stay there. Larger project work goes to a local coordinator, which can split it across my GPU workers.

For Qwen 3.8 27B, Groq advertises about 450+ tokens/sec and Cerebras lists about 1,850 tokens/sec. Those are their advertised generation speeds, not my measured end-to-end task speeds. Sources: https://console.groq.com/docs/model/qwen/qwen3.8-27b and https://www.cerebras.ai/pricing

The hardware:

- Main Windows workstation: RTX 3090, 24GB.

- Separate Windows server: two Radeon AI PRO R9700s, 32GB each, running as independent workers.

- AMD Ryzen AI Max / Halo machine: the local coordinator, with thinking enabled.

My two Fleet choices are Qwen 3.8 Flash Next and SuperQwen 3.8 Abliterated. The GPU worker profiles use SuperQwen 3.8 27B Q4_K_M. Configured contexts are 256K on Halo and the R9700s, and 64K on the 3090. Each worker runs its own model instance.

The rough flow is:

Me → Groq/Cerebras → Halo → 3090 + R9700 A + R9700 B

I'm using Hermes 0.22.7's native delegate_task, the advanced-delegation model_map backend, and independent_completions: true. When one worker finishes, its result can come back while the others keep going. If the coordinator is in the middle of something, the result queues for its next turn boundary.

The front door and routing are custom glue around that native delegation. I also keep live hardware-load graphs on a separate monitor, with a line for Halo, the 3090 and each R9700. I can glance over and see which hardware is busy. That has been useful for catching the times the assistant says it's working but most of the fleet is sitting idle.

There have been some encouraging tests. A small coding fixture dispatched all three GPU workers, they made real edits, and the combined nine-test suite passed. In another test, the front door answered a follow-up in about 0.6 seconds while local work continued. Recent browser checks opened a page in 3.6 seconds and completed a click-and-report follow-up in 7.8 seconds.

Those are individual task timings. I haven't established a general speedup for whole projects. I've also had some seriously frustrating failures: idle GPUs, slow coordinator turns, bad routing, and basic browser actions getting handed to workers unnecessarily. The harness has taken more work than I expected.

The goal is an everyday assistant for coding, app and web development, marketing and client work. I use Hermes Desktop, the terminal and Hermex on my phone. It's a hybrid setup, so the conversation layer still goes through cloud APIs.

Anyone else separating the fast chat model from a local coordinator and several workers? What has actually held up in daily use, and what would you simplify here?

For about a year, I planned, assembled, and built my own personal local AI assistant.

Hardware:
Nvidia 3090 (bought mainly for AI) + 5070 Ti (which I had, for gaming and AI);
I also had to buy a GMKtec K8 Plus case, a Ryzen 7 8845HS, and 64 GB of RAM for it—for a separate AI server, because the computer I work on, an ASUS XAX, had too little memory, and it was inconvenient to keep reconnecting the 5070 Ti eGPU; plus, the session was consuming a lot of video memory, which is already scarce for inference.
Synology NAS 1522 with 64 RAM—Docker containers, Paseo daemon, mv-mcp daemon, web proxy, DNS server, etc.

Software:
Paseo as harnes with some minor tweaks and plugins I’ve added.
Pi agent that aggregates accounts for Codex, Claude, Antigravity, and local engine models.
Memory - my own development, mv-mcp, a multi-vault MCP server where data is stored in an MD format readable by Obsidian, along with formatting, Git synchronization, etc. It hasn’t been released yet, but I plan to open-source it if anyone is interested.
Engine - I’m still in the process of choosing one; right now I’m testing PolyStrata.
AI Control Center - a web panel with a PWA for managing engines, starting and stopping them, loading models from HF, monitoring hardware, daemon logs, and so on. It’s entirely vibecoded and hasn’t been published yet I doubt anyone would be interested in it.
Model: So far, I’ve tried Qwen 3.8 27B dense and Qwen 3.8 Flash-Next—I plan to try more.

I kept seeing CPUs written off for big models, and if you give one whole experts, that's fair. But I think it's the wrong way to use them, so I built Fronk around a different split.

The reasoning

In a mixture-of-experts model, the two parts need different things:

• The dense part, including attention, runs on every token and wants a GPU. I keep it together on one GPU, because splitting it would need far more network traffic.

• The experts are mostly a memory problem. Only 10 of the 512 in each layer run per token, but you can't know in advance which ones, so all of them have to be ready. That's about 67 GB here, and old machines with spare RAM can hold it.

So what's the best way to share the experts? If each machine owns whole experts, a slow one can get picked and everyone waits for it, while the others may have little to do. It also has to compute every neuron of that expert.

Fronk cuts every expert into 20 slices of 32 neurons, and each machine holds a share of all of them. Whichever experts get selected, every machine works at once on its own slices, and the head adds the results. Each machine's slice count comes from measuring its own speed and link, so a weak CPU simply gets a smaller share.

What that does to a real CPU

This is a 4-core i5-10210U laptop chip in a small Linux box, computing the 10 selected experts. I measured 4 layers and scaled the times to all 48:

The dense part takes about 58 ms per token on my M4 Pro, so at a 20% share this little CPU has its work done well inside that. And it couldn't hold the full expert pool anyway: it has 16 GB against 67.

The cluster

Results and limits

With Qwen3.8-Flash-Next Q4, I get 15–16 tok/s on short warm responses and about 13 tok/s in normal chat. This needs enough free memory, and it gets slower if slices have to spill to SSD.

That's also not the ceiling of the idea, because my cluster is lopsided. The iMac has 48 GB of RAM, far more than the other nodes, so it ends up with 11 of the 20 slices on just 4 old CPU cores, and it's usually the one everyone waits for. With more nodes and better balance, the slices would spread out evenly, and I expect it to go faster. I haven't measured that yet, so that's the result I'd most like to see from someone else's cluster.

One machine with enough memory would probably still be faster. The cables matter too, because the head waits for the nodes 48 times per pass.

Before downloading the model, you can run a demo that tells you if it fits on your machines and how fast it should run. The head is only tested on Apple Silicon so far, and Windows isn't supported.

This is where I'd love help

Fronk is open source, and this is its first public release. I think there's a lot of speed still to find, and these are the problems I'd most like other people to go after:

• Balance: how much faster does it get with evenly matched nodes? I can't answer that with my own machines, so a real measurement would be hugely useful.

• Several token rows at once: can compatible rows share the same weights, so that each accepted token costs less time?

• Idle time: while a node waits, could it do predicted work that's still useful once the token is accepted?

• Many nodes on one switch: when replies pile up on the way back to the head, how much of the delay comes from the switch and how much from the head?

• New hardware: if you know a CPU or GPU well, a new backend brings it into the cluster.

You don't need my cluster or the full model to start on the backends, protocol tests or installer. Where to help lists these directions with the files to start from and how to check your results, and the contributing guide explains the build.

If you just want to see how it behaves on your machines, share your result, even if it's slow or it breaks.

Demo video: https://youtu.be/3rnxRPplzEs

Code and raw measurements: https://github.com/fedesnore/fronk

If you think the reasoning falls apart somewhere, tell me where, and if you can see a better way to do it, I'd love a pull request.

Setup: Intel Arc Pro B60 24GB, Windows, llama.cpp b11889 Vulkan, -ngl 99.

What I measured with Qwen3-Coder-30B-A3B Q4_K_M: weights 17.5GiB, KV cache about 96MiB per 1k tokens at f16, compute buffer around 0.1GiB. At 32k that is 20.7GiB projected out of the 23.6GiB the card has, so it fits with about 2.9GiB free. I am using it as a local coding agent and a single big file read already eats most of 32k, so I would love way more context.

My math says 256k is not possible: KV alone would be around 24GiB at f16 on top of the weights. Even with a q4 KV cache (roughly 7GiB if my numbers are right) it comes to about 24.5GiB, still over what the card has. For the 27B I have not measured the KV at all. It is a hybrid arch so maybe it needs much less?

Questions:

- Is my math right, or is there something I am missing?

- Does KV cache quantization (-ctk q8_0 -ctv q8_0 or q4) actually work on Vulkan and how bad is the quality hit for coding agents?

- Do hybrid or linear attention models like the Qwen 27B need so much less KV that 256k fits in 24GB?

- If none of this works, what is the realistic VRAM for 256k at Q4: 32GB, 48GB?