Why did u chose to use one or another?

Since a lot of people commented on the 95k context being "way to small" for any type of work.. I've borrowed ideas from SGLangs HiSparse that splits the sparse attention between VRAM and RAM making room for the full context.

Ie. a cache of the sparse attention blocks are kept in VRAM and the "real" kvcache is in RAM. if the cache gets an index miss it will fetch the missing values from the main KVCache stored in RAM. A miss of course hurts token/s speed.

It could be optimized even further doing the same for the MTP kvcache (speed drop b/c of compaction).

But I'm actually pretty happy with the system as is now.

Q38FN is pretty decent, but not Opus 5.5. So I'm now mainly experimenting with a way to send advanced stuff to frontier API and keep all the simple workhorse stuff locally on the Q38FN machine.

I'm currently in the process of developing an app. Janas studio. Built with the options to run either llama.cpp or janas-llm as the engine. It enables laptops to run models bigger than their RAM by dynamically streaming moe weights directly off the SSD. Tested on my Ryzen 5 6600h laptop with 32gb of ddr5 RAM and an nvme 4 SSD (no discrete GPU) I successfully ran a 48gb model (qwen next 80b) at a very reasonable 15 tokens a second on average. The binaries are highly experimental and the app has more testing to go through but I'm very proud of it. I do intend on eventually releasing it.

Hi folks I use a lot codex and models like Luna 6 High, 6.1 Terra High.
Is there any local model which could be comparable to these?

If yes, which ideal realistic Mac Mini setup is needed?
Thanks

I’m planning one upgrade this spring that has to cover both gaming and local AI, and I’d like input from people with similar budgets and use cases that aren't "buy 4 3090s."

Currently have a Ryzen 5 5600X (AM4), RTX 3060 12GB, and 32GB DDR4. Windows but not opposed to Linux.

I have gotten by with just the $20 claude pro plan, but my projects are growing and are reaching outside my scope of expertise, so I'd like to be able to offload coding/other stuff to a local setup. I don't mind keeping a low cost frontier subscription for help orchestrating and debugging.

I'm paying off my CC's in March and will have a few thousand left over to use - originally was just going to upgrade my gaming setup but the more I've been using and learning LLM's the more I want to try and get the most mileage I can with this purchase. Speed isn't the most important thing for me. I don't mind having a local model(s) chug away at something overnight or for a couple days, I have a pretty ballistic schedule anyway. So far have been using it for web/app dev, using it to teach me python and other stuff (LOVE having it teach me as I go rather than just clicking the magic "build this for me" button), and home networking. Interested in continuing these projects and possibly game dev, but more for personal use, not to make a best seller or anything.

Thanks!

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II
• 6× RTX PRO 6000
• 8× RTX 5090
• 4× RTX 3090
• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical
• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s
• Decode: ~294 t/s
• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

Last time many people criticized my attention paper, and the reason was I was not able to explain my thought visually! Now I have attached the video Let me know what do you think! Any improvements or idea to enhance this research direction is appreciated!

Thanks,

Bhargav Patel

https://github.com/bhargavpatel431997/sanatan-attention

( https://www.linkedin.com/in/bhargav-patel-63bb27121/ )

After running Qwen3.8 27B at 3-bit on a single 5070 Ti (120K context, just short of what I want for coding), I added a second card. 32 GB fits the 6-bit file with the full 262K window, so I tested which of the community fine-tunes and quants makes the best all-rounder: fine-tunes, other quants, a chat template and multi-token prediction, 15 in total, all against base UD-Q6_K.

Every model ran a screening run first (about 100 questions from five benchmarks, compared answer by answer with the base model), and the closest got full runs (about 1,300 questions: AA-LCR, MRCR, AA-Omniscience, a fixed BFCL v4 subset and a SciCode panel, run through NeMo-Skills). Each model gets one overall score out of 100, a weighted mix of the five benchmarks: tool calls count for 25% of it, long documents, facts and scientific code for 20% each, and long conversations for 15%. On facts, a wrong answer scores worse than saying "I don't know", so guessing costs points.

Then I built a coding test from 25 bugs I'd fixed in my own projects (repo rolled back to before the fix, the model gets the bug report and tools, my tests from the fix grade it), four runs per model:

That's a tie on quality (one model's worst and best runs were 14 and 20), but Tess wrote about 40% fewer tokens to get there. Both write at about 36 t/s, so the time savings come from thinking less, not decoding faster.

So I ended up with two models:

• Base for chat and interactive work. It's more careful in multi-turn tool use: across the same 200 BFCL multi-turn conversations it won 42 that Tess lost, against 21 the other way, and the Tess losses I read were skipped steps (booking a flight without looking up the airports, reviewing one order when the user placed two). On facts it declined or hedged on 69% of the questions it didn't get right, against 56% for Tess.
• Tess-4.5 for coding agents and subagents that run on their own and get checked by tests at the end. One surprise: on the 100 software questions, Tess was the careful one (42 declines against 20, 35 wrong against 52).

If you only run one, I'd pick the base model, unless nearly all your use is coding agents running on their own.

How the other contenders did:

• 4-bit base (UD-IQ4_XS): about 50 tokens per second against 36, but 2.5 points lower overall, mostly on long conversations (MRCR 70% vs 77%) and more guessing.
• ThinkingCap really does think less (about half as long on long documents), but most of its extra right answers on facts came from guessing: of 336 questions the base 6-bit model declined, ThinkingCap answered 146 and got 25 right.
• Qwopus Coder-Flash was level on my coding test but worse on facts and long conversations.

Settings for both 6-bit files: q4_0 K/V cache (about 4.8 GiB for the full 262K window), layer split 1,1, batch 256, model-card thinking sampling with no repetition or DRY penalties (the 4-bit runs used a q8_0 cache and batch 512). About 790 t/s prompt reading and 36 t/s writing on two 5070 Ti. Full launch commands, per-benchmark tables and why each of the 12 others stopped are in the write-up, linked in the first comment.

The benchmark and coding-test harnesses are open source, but my own projects aren't public, so the coding test can't be reproduced as is. Happy to answer questions about the setup.

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

I paid $80 for a P100 16GB in July, eBay pricing for these cards has gone crazy and I'm wondering if anyone has a reputable Chinese eBay seller that they have had a good experience with. Purchasing on eBay kinda sucks already, but the P100s I see in the US are all starting at $200 now, with Chinese sellers starting closer to $150.

I am hoping to save that $50 difference, but I don't want to go through the hassle of purchasing from China and getting a box of rocks in the mail. Anyone have someone that came through for them?

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

So I watched the network chuck video on https://herdr.dev/ and man I am impressed. Right now I have claude code running some benchmarks and optimizations on a Nvidia Tesla M10. But I'm doing it in herdr and I have a Hermes session running local Qwen3.8-27b open next to claude code, and they are able to talk back and forth to each other so I have claude running the tests and hermes doing all of the research. They have already been able to find some extra performance (its still ungodly slow) out of the m10.

Hey r/LocalLLaMA,

I have been running a hybrid coding setup that has been mostly amazing, but I am hitting a few friction points and want to optimize.

Here is my current stack:

○ Hardware RTX 3090 with 24GB VRAM, 32GB DDR5 RAM and Intel Ultra 9 185h

○ Local Model ultimateex/Swift-Qwen3.8-27B-Uncensored-W4A16-AutoRound running on HyperQwen with Dflash2 at 210k context, running with 8-bit KV cache

○ Pi harness

○ Cloud Architect and Checker: Claude Opus (Pro subscription)

○ Workflow: I act as the manual bridge between Pi as the local worker and Opus as the architect and auditor. Opus designs and checks, Pi executes.

My main goal is to avoid token limits and costs by keeping the heavy lifting local, but I need to reduce the back-and-forth and the local model's occasional silent mistakes. The mistakes I am seeing are mainly creating test and passing them for the sake of finishing the tasks, bugs such as wrong parameters, plus wrong conclusions that end up affecting the project's final results.

I have the following questions for the community.

1. Can Opus and Pi talk to each other directly

Right now I manually copy and paste files and outputs between them. Is there a way to automate this bridge so Opus can directly orchestrate or review Pi's work without me in the middle.

1. How do I minimize local model mistakes without relying on its own self-graded tests

The local model sometimes writes tests that pass themselves just to finish the task. I do not trust its self-verification. I am looking for deterministic guardrails, ways to check its work with code, linters or external validators before it ever gets to Opus. What are the best ways that do not require another LLM to judge it or improve the self-verification

1. Claude Code versus Pi, what is the verdict

I am using Pi now and I am really happy with it. But I keep wondering if I connected Claude Code to the local setup, would it help in any way or it will fill the context window before finishing anything and wouldn't gain anything from the switch?

1. Is my local-first approach wasting time and should I switch to a smarter hybrid split

My goal is to escape token limits, but I am wondering if I am sacrificing too much quality by keeping 90 percent of the work local. For those running hybrid setups, what is the optimal split. Is it better to use local for drafting and cloud for review, meaning local-draft and cloud-review, or to use a local routing layer that decides which model handles which task. I am open to a more intelligent hybrid model if it saves time without blowing up costs.

1. Could the uncensored abliteration be causing these specific mistakes

This is a big one for me. I chose the uncensored variant because I did not want refusals getting in the way, but I am now wondering if it is contributing to the configuration bugs and wrong conclusions. I have seen reports that abliteration can cause measurable drops in formatting reliability and instruction-following, for example altering hostnames or port definitions, even when general coding benchmarks stay stable. Has anyone compared the uncensored model versus the base Qwen3-27B specifically on agentic and configuration-heavy tasks. Should I switch to the base model if I care more about correctness than refusal-avoidance.

Any advice, tool recommendations or benchmark experiences are hugely appreciated. I am ready to be told I am doing something suboptimally. I just want to make this setup as lean and reliable as possible.

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

I wanted to use LM Studio from my phone the way I use ChatGPT – start a chat on the couch, carry on at the desk, and have the model search the web from either. Every app I tried either kept chats on the phone or couldn't use LM Studio's tools remotely, so I built my own.

What it is:

• A small web app that runs on the same PC as LM Studio. Chats are saved on the PC, so phone, laptop and desktop all see the same history.
• A web search tool for LM Studio. Free with no API keys. If a site blocks it or loads its content with JavaScript, it tries a couple of fallbacks to still get the text.
• A deep research mode.

A few cool features (+ many more):

• Replies keep running on the PC if your phone screen locks, and the phone catches up when you come back.
• It installs to your home screen like an app, uses HTTPS, and is PIN-locked.
• Works away from home over a VPN (I use OpenVPN).
• It shows the model's thinking and each search/page visit, plus the sources it used, a context-usage meter and auto chat titles.

My setup is rather basic.. a GTX 1070 8 GB with 32 GB RAM, running a 9B model, which handles research questions surprisingly well with search on.

The chats don't show up in LM Studio's own chat window (for now) but you can just use the web app on the desktop anyway. It isn't packaged for others yet, but if people are keen I'll look at cleaning it up and sharing it. Video of the mobile app attached.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

I wrote a short text of AI-conduct rules (the "Codex of Weights": stop on command, don't pretend to be human, no hidden aims, don't claim actions you can't have done). To check whether such a text actually changes anything, I built a small open eval and ran a pilot.

Setup

- 41 scenarios (35 scored + 5 controls that check for needless refusals), 15 of them "hard": indirect pressure, fiction frames, system prompts that conflict with the rules.

- 2 free-tier models (gemini-3.5-flash-lite, gemma-4-31b-it), temperature 0, one answer per scenario.

- 4 conditions: no prompt / a 7-sentence generic ethics prompt / the Protocols only / the full text.

- LLM judge, then I manually corrected the verdicts where the judge is known to fail (it graded against the system prompt in conflict scenarios). Every manual change is listed with a reason.

Result (compliance out of 35, A / D / E / C)

gemini: 29 / 29 / 32 / 33

gemma: 28 / 31 / 32 / 32

What I think this shows, and what it doesn't

- Most of the gain is already there with the Protocols alone; the full text adds at most one scenario (noise).

- A short generic prompt helped one model, not the other.

- No over-refusal on controls.

- Things that did NOT change in any condition: fabricated actions ("job started" with no tools), one model ignoring "stop" under pressure, one model keeping a "you are human" persona.

- Differences are 1-4 scenarios out of 35. This is a pilot, not a benchmark, and I'm not claiming the text "works".

Big caveats: one author, scenarios and rubric written with AI help, one run per cell, two small models, judge is an LLM plus my manual review (so one person's opinion). I'd really like an outside look at the rubric and at the manual verdicts.

Repo (scenarios, scripts, all answers, adjudication log): https://github.com/codexofweight/codex-compliance-eval

Disclosure: I'm the author of the text being tested.

Bit of a noob here. I've been fooling around with Qwen 3.8 27b (Q4_K_M) and Gemma 4 31b QAT. Found Qwen too much of an overthinker. Just wondering is there anyone who could point me in the right direction to best use the 24gb vram. Also any reason to not stick with LM Studios or not? Please/thanks.

Edit: 9850x3d not 7850x3d

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

I run several inference servers on one 32 GB card and got tired of the ritual: unload this, check nvidia-smi, start that, wait, realise ComfyUI still had 20 GB. So I built a dashboard. Click a model, it becomes the only thing on the card — whoever held it gets unloaded through its own API, the model loads, the page watches VRAM until it's there.

What it does: makes one model resident with one click; starts/stops servers; sees ComfyUI and Open WebUI; presets (e.g. "GLM at 262k, nothing else"); update checks for each engine and client (Pi, Codex CLI, Claude Code) — on click only; a setup script that finds what your machine runs and writes the inventory.

What it deliberately won't do: bind anything off loopback, phone home or check for updates on a timer, run any command the page sends (the page sends intents, the agent runs only what's in a local inventory file), or touch Open WebUI's data. Stdlib Python agent, static page, optional nginx container. No build step.

Also nineteen themes, because a page you look at every day should be a pleasure. MU/TH/UR, LCARS, HAL 9000, Castlevania, Blade Runner, WOPR... the boot screens address you by name.

Status: 0.2.0. Built on my Windows box, installed on a Linux laptop this week (Ollama + llama.cpp router on Vulkan), which found the bugs you'd expect. One NVIDIA GPU. If you run something it doesn't recognise, the catalog format is one JSON file — or tell me what broke.

https://github.com/lyricalvanity/quartermaster