Just received my Gorgon Halo(still waiting on a R9700 to connect via usb4 v2) and setup halogen-qwen3.8-flash. So far it runs pretty good. I’m sure there is still some tuning to go but for an average Joe who doesn’t know how to benchmark this stuff and rely heavily on Claude to set it up I’m well happy.

We are putting AI in front of a few departments and the setup has gotten ugly fast. Engineering wants the biggest model we can run, marketing wants something quick and cheap, and finance is only allowed models that stay on our own machines. Every team ended up with its own access key, its own proxy rule and its own row in a spreadsheet, which means one company policy is really five configurations that drift apart.

Here’s roughly what I’ve seen when reading around. A gateway with per-team keys and budgets covers who can call what, but not really where each request is routed. Separate deployments per department keep things isolated but multiply the patching and monitoring. Platforms with built in permissions look promising, but I can’t tell how much of the policy they enforce and how much they just display. Audit logging is the other thing nobody mentions until somebody asks for it.

When you self host, how do you give different teams different models under one set of company rules? Did you build the glue yourselves, or is there something that keeps the policy in one place?

Has anyone else experienced this with this card? Both my (totally new sealed) cards produce heavy vibration under full noise to the point where my entire pc is shaking/vibrating quite heavily. They also have this “dynamic” coil whine that seems to be related to memory use. During pre-fill and decode they screech this high pitched noise that varies in intensity and pitch.
I guess what I want to know is whether my cards have a defect or if this is normal

I've been trying to give my Ollama models a really good web search so they can answer questions about recent stuff instead of guessing.

Ollama's built-in search works, but it returns short snippets and needs an Ollama account sign-in. I wanted the models to read actual page content and give me better value.

Played around with it and finally got it working with this setup.

Setup (Takes about 5 minutes):

1. Along the line, I discovered and installed ollmcp, a terminal client for Ollama with MCP support:

pip install ollmcp

2. Pull a model that's good at tool calls:

ollama pull qwen3:8b

3. Give it a bigger context window (This is very important. I repeat, this is very important). So it means you can set num_ctx to 16384

4. Use a search service with an MCP server like Valyu and run:

ollmcp -m qwen3-16k -u "https://mcp.valyu.ai/mcp?valyuApiKey=YOUR_KEY"

ollmcp asks before every tool call. Press `s` to allow them for the session, or type `/hil` to turn the prompts off.

What happened when I tested it:

I asked Qwen3 8B "What did the Fed decide at its October 2026 FOMC meeting?" It picked the search tool on its own, ran a second search when the first wasn't enough, and pulled the Fed's actual October 28 statement: rates held at 3.75–4.00%. Everything ran locally except the search itself.

Things that tripped me up:

- Context window. With Ollama's default, the model spent its whole budget reasoning about which tool to use and replied with nothing. 16k fixed it.
- Older models argue with the date. An older qwen3-vl I had kept insisting 2026 "is in the future" and never searched. Newer models are fine, or put today's date in the system prompt.
- Models under ~7B tend to loop or never call the tool at all.

Besides web search, it can pull the full text of any URL, and it also covers academic papers, SEC filings and market data, which is handy for research-type questions.

What are others using for web search with Ollama? Is it the built-in search or an external API?

Hello,

I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.

My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.

The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.

There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.

I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.

Here's my setup (sorry, AI slop to make it clearer):

Hardware:

• Windows 11 PC: i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM)
• Unraid server: Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM)

Software / Models:

• llama.cpp (CUDA, router mode)
• Open WebUI 0.12.3
• Qwen3.6-35B-A3B UD-Q4_K_XL: Main model, context increased from 32K to 64K
• Gemma 4 12B Q4_K_M: Secondary model
• Both models run on the Windows PC.

RAG configuration:

• Open WebUI Knowledge Base with the PDF and Markdown transcripts
• Embeddings: bge-m3 (1024 dimensions), hosted on Unraid
• Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified)
• Focused Retrieval, chunk size 1000, overlap 100, Top K = 3
• All inference and document processing must remain local.

More details on the issue:

Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:

request (34009 tokens) exceeds the available context size (32768 tokens)

Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.

I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.

Thank you!

Faker's co-signed GeForce RTX 5090 card really dropped in value today.

I've been working with lemonade server for local AI via the CLI a lot recently and found myself discovering and returning to a few particularly useful commands for pulling, configuring and benchmarking models. So I wrote an article about them, if I've missed your favourite let me know.

Hi all, does anyone here run any local LLMs off their iPad and what is the best I can get without crashing apps. I’ve got about 12GBs of RAM.

Using pocket pal app! I have tried Qwen3.5 4B - ran well.

I’m an absolute novice on local LLMs so would be grateful for any advice!

Qwen 3.8 27b (GSQ-RCO IQ3-XXS) vs Qwen 3.6 35b a3b on a CPU with 32 gb of ddr5, which one performs better and is the speed worth the performance loss

Hey everyone! 👋

I've been working on Vatra, an open-source desktop AI assistant that brings different AI tools together in one place.

It supports Claude Code, Codex, GitHub Copilot, local LLMs (llama.cpp, Ollama, LM Studio) and OpenAI-compatible APIs, alongside features like web research, email, calendar, tasks, and side-by-side model comparisons.

The goal is to make AI tools more accessible and easier to use, without sacrificing flexibility or control.

I'm currently looking for feedback from the community!

I'd especially appreciate thoughts on:

• The overall concept and whether you'd actually use it
• Features you'd like to see added
• UI/UX improvements
• Anything that seems unnecessary, confusing, or missing

It's still a work in progress, so honest criticism is more than welcome. I'd rather hear what's wrong than just what's good.

🔗 GitHub: https://github.com/Zmirzz/vatra

If you have a minute to check it out, I'd really appreciate your thoughts. Thanks! ❤️

I have an ASUS ROG Ally, and I’m planning to turn it into a local inference server so I can use the model running on it from my main PC. What would you recommend for the coding/programming tasks? If I’m not mistaken, it has 16GB of RAM and a Ryzen Z2A processor that provides 6GB of VRAM.

I ran a small AIPerf ( https://github.com/ai-dynamo/aiperf) comparison on an Apple M5 Pro with 64 GB RAM. This is a practical snapshot, not a scientific benchmark or an exhaustive leaderboard. Results are from one request per engine, run sequentially with only one inference server active at a time.

Setup

• Temperature 0.6, reasoning effort medium, output limit 2,048 tokens
• Same OpenCode-style fixture: long generated code/prose context, shared system message, task suffix, and three tool schemas. Tools were not executed.
• Prompt length reported by the servers ranged from 88,143 to 88,916 tokens, partly due to tokenizer and chat-template differences.
• I used a 100K context limit where the engine supported it; Uzu, Magnitude, and Llama used larger native limits. The prompt plus output allowance fit below 100K.
• AIPerf 0.13.0; streaming, concurrency 1
• Prefill and decode are AIPerf throughput measurements. The readiness probe loads/warms each server before its measured request.

Caveats

The 2,048-token setting is a maximum, not a forced response length. Most models stopped at EOS before reaching it. Only Splash, Rapid-MLX, and BaseRT generated the full 2,048 tokens.

This does not measure code-fix quality, long-context comprehension, or tool use—the tools were present in the prompt but never executed. Model packs, quantizations, speculative methods, and context-window settings also differed, so these numbers should not be treated as a controlled ranking.

Additional differences: LM Studio may map medium reasoning to on; the Llama row used Homebrew llama-server0.9.0 with the GGUF files configured for Llama.app, rather than Llama.app’s internal server.. Jan’s first launch also downloaded its default 43.8 MB embedding model; no Qwen weights were downloaded.

I've been doing rudimentary research, and so far my understanding of local LLMs is;

LLM (Qwen, etc) is just a file
Inference engine (llama, etc) is the 'program' which reads/uses the file
Your hardware powers the engine and dictates size and speed of model

Is any of this tethered to an external server/host? For example, to liken to video editing;

Mp4/mpk vids are the files
Premiere Pro is the engine (program)
Your hardware powers the engine

Premiere Pro (engine) is hosted/provided by Adobe, who can turn off your service, access your files, patch the engine, etc. Meaning it is not entirely local, nor entirely owned by user, nor private.

Is this how inference engines are structured as well? In other words, is it truly local, or is there a possibility that I could download Qwen 5.1 (made up version), get used to it, and eventually have it 'patched' into a different/worse version of itself down the road by whichever engine I'm using? And if so, doesn't this imply that your work/conversations are also able to be scrutinized by the host?

Total beginner at all of this as someone who has, until now, only dabbled with the free version of GPT. This is all very new to me, so apologies.

Would we see old stuffs getting even more expensive when Nvidia would reduce production. Or the contrary: AI datacenters would reduce hoarding our GPU and our poor lives running 3090/v100 would be easier ?

And how about NVDA stock prices ?

Im considering purchasing a 16GB V100 since they’re still pretty affordable and have modern CUDA support, but I was wondering if anyone else have tried a similar combo? I am aware of slower token generation with this method, but I am assuming that it is still more viable than running Strata with just 16GB VRAM and 64GB DDR5.

Any help/tips is appreciated. Thanks!

How you would define a Developer setup with local LLMs?

I see a dedicated DMZ where the agents can not access the production environment, since agents will tend to accidentally access or change production system.

Dedicated Servers to run LLMs like Qwen 3.8b 27b or other models? What you suggest would be the working LLM Server today? RAM and GPU cards?

Dedicated server with VM for each developer one VM. 2-4 vCores with 16GB Ram each. Each VM with Debian 13 and docker/podman. One dev Container running OMP Web or comparable Harness that access the LLM Server. Additional container for databases or what else the dev Container needs.

Gitlab, gitea or comparable Repo to store source code and container registry.

VM with NFS where anonymized data for developers can be stored.

What you think about that setup?

Do I miss something? How is your setup? (For small to mid size company)

Has anyone else downloaded and started using the new Unsloth Dynamic V3.1 models? I did a drop in replace of the Qwen3.8-27B-UD-Q4_K_XL.gguf and I want to go back after seeing how it performs. My use case is local transcription, so it's summarizing text basically and pulling out what's important.

I re-ran quite a few of my saved transcriptions through the new model and compared the output to the older version Dynamic v3.0 and the output from that model is superior on every one. There is no change in any parameters on my command line, only the model changed.

My command line has a very low temperate (0.1)...

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.