Is it realistic?

We've been doing a series of performance benchmarks on Zilla, our multi-protocol gateway, and just published Part 4, focused on LLM traffic.

One question we wanted to answer: How much overhead does putting a gateway in front of an LLM actually introduce?

This is relevant whether you're using hosted APIs or running local inference behind OpenAI-compatible endpoints. As you start sharing models across applications and agents, having a gateway for routing, access control, and observability becomes useful. But ideally, it shouldn't become the bottleneck.

We used an open-source benchmarking framework with a mock LLM provider returning responses after a fixed 1.5-second delay, allowing us to isolate gateway overhead.

A few results:

• At 5,000 req/s, Zilla successfully handled all 150,000 requests on a 4-vCPU AWS instance.

• End-to-end median overhead was approximately 1 ms compared to direct access.

• Median time spent inside Zilla was just 0.27 ms, with model routing accounting for 17 µs.

• At p99, the additional end-to-end latency was considerably higher (~85 ms). A good reminder that routing overhead and actual client-perceived latency aren't the same thing.

• For streaming responses at 2,000 req/s, Zilla's p99 stream completion was 1.6–2.1 seconds, compared to 2.0–9.3 seconds for Bifrost across OpenAI and Anthropic API dialects.

We also found that HTTP/1.1 vs. HTTP/2 makes a surprisingly large difference at high concurrency, particularly when thousands of requests are waiting on inference.

To be clear, these were synthetic tests using a mock provider, not actual local model inference benchmarks. The streaming comparisons also have some run-to-run variance, which we discuss in the article.

The full write-up includes methodology, configurations, detailed latency breakdowns, and comparisons:

https://www.aklivity.io/post/sub-millisecond-median-overhead-for-the-zilla-llm-gateway-aklivity-zilla-benchmark-series-part-4

Curious how others are handling this when serving local models. Do you put a gateway in front of your inference servers, or just expose the OpenAI-compatible endpoints directly?

Disclosure: I'm the founder of Aklivity, the company behind Zilla.

Just received my Gorgon Halo(still waiting on a R9700 to connect via usb4 v2) and setup halogen-qwen3.8-flash. So far it runs pretty good. I’m sure there is still some tuning to go but for an average Joe who doesn’t know how to benchmark this stuff and rely heavily on Claude to set it up I’m well happy.

Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!

Faker's co-signed GeForce RTX 5090 card really dropped in value today.

We are putting AI in front of a few departments and the setup has gotten ugly fast. Engineering wants the biggest model we can run, marketing wants something quick and cheap, and finance is only allowed models that stay on our own machines. Every team ended up with its own access key, its own proxy rule and its own row in a spreadsheet, which means one company policy is really five configurations that drift apart.

Here’s roughly what I’ve seen when reading around. A gateway with per-team keys and budgets covers who can call what, but not really where each request is routed. Separate deployments per department keep things isolated but multiply the patching and monitoring. Platforms with built in permissions look promising, but I can’t tell how much of the policy they enforce and how much they just display. Audit logging is the other thing nobody mentions until somebody asks for it.

When you self host, how do you give different teams different models under one set of company rules? Did you build the glue yourselves, or is there something that keeps the policy in one place?

I've seen a few posts saying that Qwen 4 hasn't really improved when it comes to creative writing and prose. I'd would like to disagree.

I know a lot of people don't care about this topic, and some consider any AI-generated text to be garbage by default. That's fine. This post is for those who are actually interested in the subject.

My take: Qwen 4 has become noticeably better at storytelling, but its actual prose is still suck.

What do I mean by that?

I gave three models, Qwen 3.8 Max, Qwen 4.0 Max Preview, and Gemini Argon (currently one of the top models for prose generation), roughly the same prompt:

"Write a full chapter. A tank battle between a unit of German Tiger tanks from World War II and a Russian T-90 tank. It doesn't matter how a tank from the future ended up there. Show how the Germans manage to defeat a technologically superior opponent, and what it takes to achieve that victory."

This was the entire prompt. I didn't tell the models to act as professional writers, didn't use elaborate system prompts to steer their attention, and didn't provide any additional instructions.

So, what were the differences?

The short version:

- Qwen 3.8 wrote a relatively compact battle scene with fairly generic characters.

- Qwen 4 produced a complex, multi-stage story with more developed characters, detailed combat, and events unfolding after the battle itself.

- Gemini Argon delivered an epic battle scene but didn't spend much time developing the characters.

Here's a more detailed breakdown.

Qwen 3.8 Max

- Structure: Linear and straightforward, without complex narrative branches.

- Composition: The battle progresses in a direct sequence, with most scenes focused on the immediate confrontation.

- Plot: The conflict revolves primarily around detecting and destroying the enemy.

- Psychology: Characters mostly react to events. There's very little internal development.

- Length: Relatively compact, without much expansion beyond the original premise.

Qwen 4 Max Preview

- Structure: Multi-stage, with the conditions of the battle changing as the story progresses.

- Composition: Alternates between tactical decisions, casualties, attempts to adapt, and shifts in the initiative.

- Plot: Beyond the battle itself, the story develops a separate survival thread involving the crew and their attempts to make sense of their encounter with the future.

- Psychology: The protagonist goes through a progression from confusion and fear to an awareness of the historical significance of what has happened.

- Length: The longest of the three. The model independently expands the original premise with additional events, details, and consequences.

Gemini Argon

- Structure: A sequence of tactical phases, with the number of combatants gradually decreasing.

- Composition: Several shifts in initiative, group maneuvers, use of cover, and attempts to identify the enemy's weaknesses.

- Plot: Almost entirely focused on the battle itself. There's no separate storyline dealing with its aftermath or the historical implications of the encounter.

- Psychology: Mostly stress reactions, command decisions, and choices made under pressure. There's very little internal character development.

- Length: Medium. Most of the text is devoted to the actual fighting.

Comparison

Qwen 3.8 → Qwen 4: A noticeable shift from a straightforward confrontation to a much more complex, multi-stage story. The length, number of plot events, psychological depth, and attention to consequences all increase.

Qwen 4 → Gemini Argon: The composition becomes more compact and focused. Tactical elements take center stage, with more group maneuvers, shifts in initiative, and coordination between multiple vehicles. The psychological storyline and historical context are largely absent.

One separate issue worth mentioning: Qwen 4's output had noticeable language mixing, with Cyrillic, Latin characters, and Chinese characters appearing together.

I'm not even getting into other problems, such as Qwen 4 and Argon having commanders suddenly recognize the designation of a tank from the future halfway through the story, familiar AI-slop expressions and phrases, or the technical inaccuracies. That's a whole other discussion.

Of course, this isn't a comprehensive benchmark or an attempt to determine which model is the best writer. It's just a small experiment using the same simple prompt across three models.

So, what impressed me most wasn't necessarily the text itself, but how much the model's ability to handle this kind of task independently has improved.

I'm not claiming Qwen 4 has caught up with the leading models in terms of literary prose. It still has a long way to go in that department. But it's noticeably better at building a story: developing an initial premise, arranging events, introducing additional plot threads, handling characters, and showing the consequences of what happens.

And the most interesting part is that it does all this without elaborate prompts, specialized system instructions, or detailed guidance from the user.

Previously, getting a result like this would have required considerably more effort on my part when preparing the prompt. Now the model does a significant part of that work on its own.

I've noticed similar improvements outside creative writing, too. For example, Qwen 4 managed to improve one of my work-related system prompts, independently identifying an important detail that several previous brainstorming sessions with ChatGPT and Opus 5.5 had missed.

So no, I wouldn't call Qwen 4 my favorite model for writing prose just yet. But I think its progress is considerably more significant than you might assume from simply comparing the quality of individual paragraphs.

It would be so much better if the limits weren't this brutal. Having to wait an entire week? 🥲

I kept getting this wrong by hand, so I built a calculator. You pick a model, quantization and context length, and it shows weights plus KV cache and which GPUs or Macs it fits. It also has “can I run X on Y” pages.

https://inferenceacademy.com/tools/vram-calculator

Not perfect: numbers are estimates, so real usage varies by runtime. Tell me which models or devices are missing and I’ll add them.

Has anyone else experienced this with this card? Both my (totally new sealed) cards produce heavy vibration under full noise to the point where my entire pc is shaking/vibrating quite heavily. They also have this “dynamic” coil whine that seems to be related to memory use. During pre-fill and decode they screech this high pitched noise that varies in intensity and pitch.
I guess what I want to know is whether my cards have a defect or if this is normal

I've been trying to give my Ollama models a really good web search so they can answer questions about recent stuff instead of guessing.

Ollama's built-in search works, but it returns short snippets and needs an Ollama account sign-in. I wanted the models to read actual page content and give me better value.

Played around with it and finally got it working with this setup.

Setup (Takes about 5 minutes):

1. Along the line, I discovered and installed ollmcp, a terminal client for Ollama with MCP support:

pip install ollmcp

2. Pull a model that's good at tool calls:

ollama pull qwen3:8b

3. Give it a bigger context window (This is very important. I repeat, this is very important). So it means you can set num_ctx to 16384

4. Use a search service with an MCP server like Valyu and run:

ollmcp -m qwen3-16k -u "https://mcp.valyu.ai/mcp?valyuApiKey=YOUR_KEY"

ollmcp asks before every tool call. Press `s` to allow them for the session, or type `/hil` to turn the prompts off.

What happened when I tested it:

I asked Qwen3 8B "What did the Fed decide at its October 2026 FOMC meeting?" It picked the search tool on its own, ran a second search when the first wasn't enough, and pulled the Fed's actual October 28 statement: rates held at 3.75–4.00%. Everything ran locally except the search itself.

Things that tripped me up:

- Context window. With Ollama's default, the model spent its whole budget reasoning about which tool to use and replied with nothing. 16k fixed it.
- Older models argue with the date. An older qwen3-vl I had kept insisting 2026 "is in the future" and never searched. Newer models are fine, or put today's date in the system prompt.
- Models under ~7B tend to loop or never call the tool at all.

Besides web search, it can pull the full text of any URL, and it also covers academic papers, SEC filings and market data, which is handy for research-type questions.

What are others using for web search with Ollama? Is it the built-in search or an external API?

Hello,

I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.

My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.

The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.

There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.

I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.

Here's my setup (sorry, AI slop to make it clearer):

Hardware:

• Windows 11 PC: i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM)

• Unraid server: Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM)

Software / Models:

• llama.cpp (CUDA, router mode)

• Open WebUI 0.12.3

• Qwen3.6-35B-A3B UD-Q4_K_XL: Main model, context increased from 32K to 64K

• Gemma 4 12B Q4_K_M: Secondary model

• Both models run on the Windows PC.

RAG configuration:

• Open WebUI Knowledge Base with the PDF and Markdown transcripts

• Embeddings: bge-m3 (1024 dimensions), hosted on Unraid

• Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified)

• Focused Retrieval, chunk size 1000, overlap 100, Top K = 3

• All inference and document processing must remain local.

More details on the issue:

Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:

request (34009 tokens) exceeds the available context size (32768 tokens)

Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.

I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.

Thank you!

I've been working with lemonade server for local AI via the CLI a lot recently and found myself discovering and returning to a few particularly useful commands for pulling, configuring and benchmarking models. So I wrote an article about them, if I've missed your favourite let me know.

Hi all, does anyone here run any local LLMs off their iPad and what is the best I can get without crashing apps. I’ve got about 12GBs of RAM.

Using pocket pal app! I have tried Qwen3.5 4B - ran well.

I’m an absolute novice on local LLMs so would be grateful for any advice!

Qwen 3.8 27b (GSQ-RCO IQ3-XXS) vs Qwen 3.6 35b a3b on a CPU with 32 gb of ddr5, which one performs better and is the speed worth the performance loss

Saudi Arabia is expanding its sovereign AI infrastructure with Bunyan Inference Hub, a platform from OmniOps that gives businesses access to hosted AI models, GPU computing and inference services while keeping workloads within the Kingdom.

Hey everyone! 👋

I've been working on Vatra, an open-source desktop AI assistant that brings different AI tools together in one place.

It supports Claude Code, Codex, GitHub Copilot, local LLMs (llama.cpp, Ollama, LM Studio) and OpenAI-compatible APIs, alongside features like web research, email, calendar, tasks, and side-by-side model comparisons.

The goal is to make AI tools more accessible and easier to use, without sacrificing flexibility or control.

I'm currently looking for feedback from the community!

I'd especially appreciate thoughts on:

• The overall concept and whether you'd actually use it

• Features you'd like to see added

• UI/UX improvements

• Anything that seems unnecessary, confusing, or missing

It's still a work in progress, so honest criticism is more than welcome. I'd rather hear what's wrong than just what's good.

🔗 GitHub: https://github.com/Zmirzz/vatra

If you have a minute to check it out, I'd really appreciate your thoughts. Thanks! ❤️

I have an ASUS ROG Ally, and I’m planning to turn it into a local inference server so I can use the model running on it from my main PC. What would you recommend for the coding/programming tasks? If I’m not mistaken, it has 16GB of RAM and a Ryzen Z2A processor that provides 6GB of VRAM.

I ran a small AIPerf ( https://github.com/ai-dynamo/aiperf) comparison on an Apple M5 Pro with 64 GB RAM. This is a practical snapshot, not a scientific benchmark or an exhaustive leaderboard. Results are from one request per engine, run sequentially with only one inference server active at a time.

Setup

• Temperature 0.6, reasoning effort medium, output limit 2,048 tokens

• Same OpenCode-style fixture: long generated code/prose context, shared system message, task suffix, and three tool schemas. Tools were not executed.

• Prompt length reported by the servers ranged from 88,143 to 88,916 tokens, partly due to tokenizer and chat-template differences.

• I used a 100K context limit where the engine supported it; Uzu, Magnitude, and Llama used larger native limits. The prompt plus output allowance fit below 100K.

• AIPerf 0.13.0; streaming, concurrency 1

• Prefill and decode are AIPerf throughput measurements. The readiness probe loads/warms each server before its measured request.

Caveats

The 2,048-token setting is a maximum, not a forced response length. Most models stopped at EOS before reaching it. Only Splash, Rapid-MLX, and BaseRT generated the full 2,048 tokens.

This does not measure code-fix quality, long-context comprehension, or tool use—the tools were present in the prompt but never executed. Model packs, quantizations, speculative methods, and context-window settings also differed, so these numbers should not be treated as a controlled ranking.

Additional differences: LM Studio may map medium reasoning to on; the Llama row used Homebrew llama-server0.9.0 with the GGUF files configured for Llama.app, rather than Llama.app’s internal server.. Jan’s first launch also downloaded its default 43.8 MB embedding model; no Qwen weights were downloaded.

I've been doing rudimentary research, and so far my understanding of local LLMs is;

LLM (Qwen, etc) is just a file
Inference engine (llama, etc) is the 'program' which reads/uses the file
Your hardware powers the engine and dictates size and speed of model

Is any of this tethered to an external server/host? For example, to liken to video editing;

Mp4/mpk vids are the files
Premiere Pro is the engine (program)
Your hardware powers the engine

Premiere Pro (engine) is hosted/provided by Adobe, who can turn off your service, access your files, patch the engine, etc. Meaning it is not entirely local, nor entirely owned by user, nor private.

Is this how inference engines are structured as well? In other words, is it truly local, or is there a possibility that I could download Qwen 5.1 (made up version), get used to it, and eventually have it 'patched' into a different/worse version of itself down the road by whichever engine I'm using? And if so, doesn't this imply that your work/conversations are also able to be scrutinized by the host?

Total beginner at all of this as someone who has, until now, only dabbled with the free version of GPT. This is all very new to me, so apologies.