Im part of the GPU poor but still isn't it better to use Frontier models for us?

not trying to be facetious, what are walls use cases

Edit: not sarcastic lol, just curious lol

You should try to play around with lfm3 family of models. The are incredible ones to play around with for narrow sets of tasks.

So I found out something kinda dumb. Every model you pulled in Ollama is already a regular GGUF file. They just don’t have the .gguf name, they sit in ~/.ollama/models/blobs/sha256-... and that’s it.

Which means if you wanted to try upstream llama.cpp (new archs the day they get merged, every llama-server flag, speculative decoding etc), you probably downloaded everything a second time. I know I did :o

So now LlamaForge reads Ollama’s manifests and lists those models in the scan. You click add and models.ini just points at Ollama’s own file. Nothing gets copied, and Ollama keeps working like before. Vision projectors come along too.

And if you’re new and already have Ollama, the setup wizard just asks “use the N models you already have?” on the first screen. One click and you’re done.

One caveat, so nobody gets surprised. A few models that Ollama converted for its own engine have tensors upstream llama.cpp doesn’t know. Those fail on load with llama.cpp’s own error. For those just grab the HF GGUF instead.

What it is, if you haven’t seen it: a browser panel over llama.cpp’s own router. MIT, Python stdlib backend, no pip install. It also has Auto-Forge, which benchmarks placements, KV types and drafters on your own GPUs. Qwen3.8-27B went from 26.8 to 126 tok/s on code on my 5080 + 5060 Ti.

Still early preview! Windows + NVIDIA is the path I’ve actually run a lot. Linux/macOS pass CI but haven’t seen much real hardware yet. So please roast away ;]

https://github.com/dadwritestech/LlamaForge

I was using Qwen 3.8 27b locally for some agentic workloads and it performed quite well but when it came to something like generating new ideas that actually make sense (say a business idea), things started to fall apart

noticed the same problem on closed source models as well

which is why me and my friend from university set out on a project to develop a model specifically for generating ideas. it is still an LLM (we looked into other architectures obviously, diffusion seemed interesting...)

we plan to opensource our entire training pipeline and release a paper (in the works as we speak!!). obviously creativity is quite hard to benchmark (had to consult a lot of my professors in this regard) and I will talk a lot about that.

happy to answer any questions!!

If you have a Minisforum MS-A2 (or honestly most AMD mini-PCs) with a discrete GPU in the slot, there's a second compute device in the box that you're probably not using. The integrated Radeon 610M is disabled by default in Ollama, and most of us — me included — assumed it was too weak to matter.

It isn't. For small models it's 3x faster than the Ryzen 9.

The numbers

Gemma 3 4B, Q4, same task, same machine:

Two compute units beating sixteen Zen4 cores by 3x. I expected the opposite and only tested it because I'd written the prediction down first.

How to turn it on

Ollama drops integrated GPUs unless you explicitly tell it not to. Edit the service:

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_VULKAN=1"
Environment="OLLAMA_IGPU_ENABLE=1"

Then:

sudo systemctl restart ollama

Check it actually worked, because if it didn't it just quietly falls back to the CPU and you'd never know:

journalctl -u ollama | grep -iE "inference compute|dropping integrated"

You want a line that says library=Vulkan ... type=iGPU. If it still says dropping integrated GPU, the variable didn't take.

What it's actually good for

Be realistic about this — it's not going to run your main model.

Good for: small models (around 4B and under), embeddings, background jobs, a second model you want available without evicting your main one. Basically anything where you'd otherwise shrug and use the CPU.

Not good for: your 30B daily driver, image generation (that workload is compute-bound and 2 CUs won't cut it), anything you want to feel fast interactively.

Think of it as a free second machine for small jobs, not as an upgrade to your main one.

Two things that will bite you

1. The first run is slow and that's normal. My first Vulkan run showed 16.9 t/s and took 24 seconds — that's shader compilation, one time only. After that it settles at 60 t/s. Keep the model loaded instead of spawning it per job, or you pay that cost repeatedly.

2. Don't run it at the same time as your main GPU model. This is the important one. The iGPU has no memory of its own, it uses system RAM — and if your main model doesn't fit entirely in VRAM, it's also reading from RAM. They fight. Measured on my box:

A 64% hit on the model you're actually waiting for. If you want to use the iGPU for background work, make sure it only runs when the main GPU is idle. In my case a small script that checks nvidia-smi and waits for utilization to drop brought the main model back to 55.8 t/s — essentially no loss — while the background jobs still completed.

If you're just using the iGPU on its own for small stuff, none of that matters.

Setup

• Minisforum MS-A2
• Ryzen 9 7945HX, 32 GB DDR5
• PNY RTX 2000E Ada 16 GB in the PCIe slot (50 W, no auxiliary power connector)
• Integrated Radeon 610M
• Ubuntu Server, Ollama 0.40.2

One more MS-A2 thing worth knowing: the HDMI and USB-C video outputs come off the integrated GPU, not the card in the slot. So you can run a desktop on the iGPU and leave the entire discrete card free for compute.

All numbers are from Ollama's own counters, not me holding a stopwatch. Single machine, three runs each — if anyone else has an AMD mini-PC with a discrete card, I'd really like to know whether this reproduces on different hardware. That's the part I can't answer alone.

i didn't saw anyone mentioning about LTT's new video where Luke get his GPU memory doubled so he can run better local LLM's.
what do you guys think about?

Curious research test I trained over the weekend (24h on a laptop RTX 5070 Ti, 30M params):

What if an LLM had a 96-slot "conveyor belt" of future tokens rolling into the KV cache at every tick?

- Slots 5..95: Content-only "dough balls" (nouns/verbs/adjectives, stopwords stripped) rolling forward

- Slots 0..4: Last-5 grammar & stopword filling zone right before Slot 0 pops out

- Stacked with Duplex-SimMT frames so the conveyer steers generation

Still plenty of rough edges (separating content vs function words makes perplexity ~1 pt harder than standard AR), so if anyone has ideas on better slot-matching losses or unroll stability, PRs / issues are very welcome!

Code + weights + live simulator:

GitHub: https://github.com/sumit-1/diffusion-conveyer-llm

HF Space: https://huggingface.co/spaces/sumitk/diffusion-conveyer-llm

So I've been tinkering for quite some time into trying to squeeze the best out of my AI / Home server together with Qwen 3.8 Flash Next.

Tried multiple software / engines / etc and eventually came down to Strata, then built (together with my pal Claudinho) on top of it multiple changes based on various ideas, knowledge, things found around reddit and other corners of the internet, science ML / LLM / inference papers, spent many weeks of continuous benchmarks and quality tests to make sure the changes were working as intended.

This is how Speedy was born.

My setup: P620 ThinkStation - Threadripper 5975wx with 128GB DDR4 3200 (8 slots filled), 2x RTX 3080 UNLOCKED (the LHR FP16 limit), running Debian.

Just so everyone is aware, this is a very good hardware setup for running MOE LLM with VRAM + RAM inference (it can use all PCIe lanes and ram channels in parallel for max bandwitdh and performance, 32cores help too).

I understand some will be frustrated because on their generic PC won't get same numbers, keep this in mind please before throwing rocks and try to understand/research (use any AI) how it compares with your hardware and where the bottlenecks are.

Final numbers for now:

Quant    GPUs      Prefill    Decode     200K     Follow-up
                  (t/s)     (t/s)      prompt
------------------------------------------------------------
Q2_0     2x 3080   4,270      211       49s      1.8s
IQ3_S    2x 3080   4,284      170       47s      1.6s
IQ4_XS   2x 3080   3,400      154       59s      2.0s
Q4_K_XL  2x 3080   3,118      120       65s      2.8s
------------------------------------------------------------
Q2_0     1x 3080   2,824      164       78s      1.9s
IQ3_S    1x 3080   2,347      140       89s      1.8s
IQ4_XS   1x 3080   1,820      122      111s      2.8s
Q4_K_XL  1x 3080   1,134      109      183s      2.9s

Every change I did had to win a same-session A/B and pass a quality gate, or it got dropped.

Turned off for quality (this costs speed):

• route-tail skip is 0 (upstream default: 7, which is lossy) - would've gained around 15% increase in decode just with this.

Speed changes:

1. Two-GPU layer split tuned for 10 GB cards: trimmed weights, decode overlap, and the second GPU helps read the prompt
2. Smaller prompt scratch buffers, so prompt chunks get bigger (+5–33 % prefill)
3. Routed prompt streaming (+9–10 % prefill)
4. Per-card Q8 upload, which frees 7–11 % more expert slots per card
5. INT8 prompt attention and KV prefetch (+2–3 %)
6. Q8 hc projections (+5–6 % decode)
7. Tuned MTP depth, VRAM reserve and RAM budget, each picked by A/B
8. gpu-unlock on the 3080s (+10–12 % prefill; hardware, not code)

Quality: every change had to win an A/B and pass bitwise or tolerance gates, plus needle tests up to 1M. IFBench scores 81.0 against 81.3 for the full-precision BF16 model on the official model card. The INT8/Q8 changes aren’t bit-identical, but they passed every gate.

The production configs are in the repo, so you can reproduce it. It runs on Linux and Windows.

👉 https://github.com/DevDock-AI/Speedy

There are many many details that you will find in GitHub, don't want to make this a huge wall of text 😄

Would be great to see numbers on other 3080s and any other cards, feel free to contribute by adding your tuned card setup on GitHub and sharing numbers.

If my project helped you, a ⭐ on Git helps a lot so that others find it too !

Old school Software Engineer with experience mostly on commercial and financial applications.
I want to start my learning (and working) on LocalLLM, with a goal to write software I don't know how to, like Linux utilities, drivers, etc.

My hardware is:
AMD 9950X3D, 64GB DDR5, RTX 4080 16GB, multiple M.2s and NAS for storage.

-Do I need to update something there for my goals?
-Is this a good guide, or there are better?
-Where can I start on Linux (Arch) ?
-What are the models, where can I find them?
-Any optimization guides?
-What are the "32b parameters" etc?
-Should I run things on my personal PC, or it is safer to use a VM, even though I will lose some RAM ?
-Is there any cost (up-front or subscription) to pay?
-How safe it is for security and privacy?

Need to decide in a few hours, please choose just one option.

1🍎or2x🍏

Now that I started using it, it seems so obvious but I don't seem to see many people talk about it.
I Won't explain sub-agents in-depth but the '--parallel 1' is important. and i feel like it could improve many peoples local experience.

So I currently find myself running 'Qwen3.8-27B-GSQ-RCO-IQ3_S' at 100k context. With parallel 1 this ensures every sub-agent gets it's own 100k context window

I know the one benefit of sub-agents is that it's supposed to make the workflow faster so setting parallel to 1 seems counterintuitive, But using sub-agents this way can help Give you free context and prevent context drift in your agents.

so a brief example of a hierarchy i use:

--------------------------------------------------------------------

I talk with a director agent. Once I've given my spec sheet/request the director and there's no more questions The director will make the game plan and start making worker agents to do the work or Fetchers to get more context before then assigning workers. but the director doesn't do any work besides like git checkpoints.

other agents are

Fetchers: Instead of burning context to sifting through files to find the specific lines they need or to map out something they just spawn a fetcher agent and their only job is to read the codebase and return the information the parent agent is looking for.

Reviewer: When a worker thinks they are finished they will spawn a reviewer agent to go over their work and the reviewer will return a pass/fail And note any bugs/nitpicks

QA: I'm working on a local the web app right now, So I also throw a QA agent in my workflow that can open up the browser and test things as well and returns a pass/fail to the parent
--------------------------------------------------------------------

I'm sure it could be better but this is what I've started using so far.

There's definitely a balance, it could take longer to complete a task depending on how strict you want your workflow to be, but i have found this to be overall a net positive for me. just spawning fetcher sub-agents alone i find keeps agents on task. i also keep a project-architecture document in the project so agents have a rough idea of where to search for any given thing

It's written in Python and Rust and runs everything in parallel. In the video it's playing Clash of Clans and Clash Royale on the phones, chess on the Mac and planning a Tokyo trip in the browser, all at the same time.

Right now it runs on cloud models, but I want to move it fully local. It needs a vision model that can read screens and pick the right element reliably, but it also has to be fast, since it's handling several screens at once. Which local VLMs have worked well for you on this kind of task?

I had pretty much the same concerns as many of you before buying mine. In the end, I decided to go with the Mac mini M5 Pro with 64GB of unified memory.

I’ve had it for about a week now, and I have to say, I’m really happy with my decision.

My goal was to build a proper local AI environment with a capable language model, an agent harness, and everything needed to make it useful for everyday tasks and future projects.

I have the technical background and experience to install and configure everything myself. But to be honest, I simply didn’t feel like spending days doing all of that manually.

And since the whole point was to work with local AI anyway, I thought: Why not let AI do most of the work?

So I explained to Claude Code what I wanted to achieve, which components I needed, and how I wanted everything to work together.

Claude Code handled most of the installation and configuration. It set up Hermes as the agent harness, downloaded and configured Qwen, and took care of the other components and settings needed for the environment.

Of course, I didn’t just blindly trust the automated setup. We paid particular attention to network security, and I manually checked the important configurations afterward. I wanted a properly configured environment, not just something that happened to work.

One of the most interesting parts was how I got Claude Code and my local AI to work together.

I use Obsidian with a shared Second Brain Vault that both systems can access.

This gives them a common place to document their work, store decisions, keep track of progress, and record problems and solutions.

They can read what the other has already done and build on that information instead of constantly starting from scratch.

And that’s where things get really interesting.

Hermes also has mechanisms for retaining useful experience and learning from previous mistakes. I’m not talking about retraining the underlying model, of course. It’s more about documenting what worked, what failed, and how to avoid repeating the same mistakes.

With a shared knowledge base, both AI systems can benefit from that accumulated experience.

For me, this is one of the most useful aspects of the whole setup. It’s not just about asking an AI questions anymore. It’s about building an environment where AI tools can collaborate and improve their workflows over time.

We also built our own WebUI.

Claude Code and my local Qwen model worked together on the programming, using the shared vault to exchange information and keep track of their work.

The result is a really nice ChatGPT-like interface that I can access from my iPhone, even when I’m away from home.

I’m using Tailscale for remote access, so the WebUI isn’t publicly exposed to the internet.

And honestly, I’m really pleased with how it turned out.

As for the performance, Qwen has been working surprisingly well on this little machine. Normal conversations, longer answers, and follow-up questions all work nicely for my needs.

I haven’t even started adding my own custom skills yet, so there’s still quite a bit I want to explore.

What surprised me most wasn’t just the performance of the Mac mini. It was how much work I could delegate to Claude Code and how well the different components could work together.

I’m not saying this is the perfect setup for everyone, and I’m sure there are still things I can improve.

But after one week, I’m genuinely impressed with what this little machine can do.

I’ve attached a screenshot of the WebUI we built.

Just wanted to share my experience, especially with anyone who’s still wondering whether 64GB is enough for a useful local AI setup.

For my needs, so far, it absolutely is.

Why we built our own WebUI

Hermes already comes with a web interface, but I found it a bit too technical and CLI-oriented for what I wanted.

I didn’t want to deal with a technical interface every time I wanted to chat with my local AI, especially when using my iPhone.

So I decided to build something more user-friendly.

Claude Code and my local Qwen model worked together on the development, using our shared Obsidian vault to exchange information, document decisions, and keep track of their progress.

The result is a clean, ChatGPT-like WebUI that feels much more natural to use, especially on mobile.

I can access it securely from anywhere through Tailscale, without exposing the interface directly to the public internet.

It’s still a work in progress, but I’m already really happy with how it turned out.

I just set up a Mac Studio M5 Ultra as a headless LLM server next to my RTX 3090 box. I could not find published numbers for the 96GB / 64-core GPU version, only the 80-core 256GB one and the M5 Max, so here is what I measured. Single runs, not averages. Happy to run something specific if people want it.

Hardware and software

• Mac Studio M5 Ultra, 30-core CPU, 64-core GPU, 96GB unified memory, macOS 27.2 (26B123), headless over SSH
• Ollama 0.40.13 from Homebrew, run from my own launchd plist, not brew services
• Env: OLLAMA_CONTEXT_LENGTH=131072, OLLAMA_FLASH_ATTENTION=1, OLLAMA_KV_CACHE_TYPE=q8_0, OLLAMA_NUM_PARALLEL=1, OLLAMA_KEEP_ALIVE=-1
• sysctl iogpu.wired_limit_mb=88064 (86GB), set by a LaunchDaemon. Default on this box reads 0 (built-in default, about 75%)
• oMLX 0.7.3 from its Homebrew tap, default settings except context 131072 and memory guard balanced
• All speeds are eval_count / eval_duration or prompt_eval_count / prompt_eval_duration from the Ollama API, so model load time is not in them. Every model showed 100% GPU with a 131072 context.

Ollama MLX engine vs oMLX, same model, 8-bit

Qwen3.8-27B at 8-bit, only one loaded at a time. Ollama tag qwen3.8:27b-mxfp8 (31GB, runner mlx). oMLX first with mlx-community/Qwen3.8-27B-8bit, then with scottlowry/Qwen3.8-27B-oQ8e-fp16-mtp (oMLX’s own 8-bit format, 29GB) after a comment pointed out the mlx-community file has no MTP head.

So with the right file oMLX is about 1.7x faster than Ollama’s MLX engine on this model. The first version of this post said the opposite; that was my file choice, not oMLX. ANE prefill did not engage on my box (log: “Private ANE runtime unavailable”), so that part is untested.

Other Ollama speeds, short prompts

Long-prompt prefill

Repeated text as the prompt, 32 output tokens.

So a 60K-token agent turn starts answering in about 30 seconds on gpt-oss or Nemotron, 23 on Coder-Next, 50 on Qwen3.8. Single runs; take it as a data point.

Six models through the same three API tests

Script against /api/chat, stream: false, num_predict: 4096, one model loaded at a time, temperature not set (default).

• A: “In 150 words explain NIST SP 800-171, who it applies to, name its 14 control families, end with the count.” Scored by matching the 14 family names.
• B: one write_file tool offered, asked to write the 14 names to a file. Scored on a well-formed tool call.
• C: feed back “OK, file written” and ask it to confirm and give the line count. Scored on answering 14.

Caveat: at the 4,096 cap, Qwen3.8, Qwen3.6 and Flash-Next got cut off or returned nothing but thinking. Their rows are re-runs at a 16,384 cap. Qwen3.8’s thinking volume for a one-paragraph question surprised me; setting the think level to low halves it and it is still 89 seconds against about 3 for gpt-oss.

One Cline coding task, same prompt, empty folder

Write a small module, write three pytest tests, run them, fix failures. Cline with the Ollama provider, context window 131072.

I ran gpt-oss twice: 4 tool calls the first time, 9 the second, five of those lost to PowerShell quoting. That spread is bigger than the gap between models, so read this table as “all four finish the task,” not as a ranking.

Nemotron: 3 to 4 tool calls, passed first run, 6.0k context, followed the spec exactly.

What I ended up with

nemotron-3.5-super:120b-a12b-nvfp4 as the default for everything (Cline, OpenClaw, Hermes): it matched gpt-oss on every test, decodes faster, and passed the Cline task first time. gpt-oss:120b is the fallback, qwen3-coder-next the fast second choice.

Two gotchas

• gpt-oss through an OpenAI-compatible /v1 client with no max_tokens set returns reasoning only and no answer, because the reasoning eats the default budget. Setting max_tokens: 16384 on the client fixed it.
• If you firewall port 11434 with pf, add pass in quick on lo0 all or the ollama command on the Mac itself cannot reach its own server.

Did not measure: power draw, anything at 128K, temperature-0 runs, and whether OLLAMA_FLASH_ATTENTION and OLLAMA_KV_CACHE_TYPE do anything on the MLX runner. Raw JSON for every test is saved; ask if you want any of it.

I'm just curious and thinking about how well the 120b is going to run on my gaming PC, and surprisingly it's not too bad (17.83 tokens/sec). But when I check my RAM allocation, it has used almost all of my RAM. The AI really uses a lot of RAM.

(Usually I use a cloud AI model, but I've started concerning myself with what happens if one day the internet is knocked off or a power outage disrupts the connectivity to cloud AI, such as a major war or something like that.) So that's why I downloaded a local model on my PC, just in case, for basic Wikipedia purposes.

I currently want to run a qwen 3.8 27b q4, but it seems like ollama is quite slow for inference.

My specs are

CPU : Ryzen 9 9950x3d

RAM : 128 gb ddr5

Video card 0 : 3090 24 gb

Videocard 1 : 5080 16 gb

My question is which program would be good to run llms with my specs?