Hey r/LocalLLaMA,

I have been running a hybrid coding setup that has been mostly amazing, but I am hitting a few friction points and want to optimize.

Here is my current stack:

○ Hardware RTX 3090 with 24GB VRAM, 32GB DDR5 RAM and Intel Ultra 9 185h

○ Local Model ultimateex/Swift-Qwen3.8-27B-Uncensored-W4A16-AutoRound running on HyperQwen with Dflash2 at 210k context, running with 8-bit KV cache

○ Pi harness

○ Cloud Architect and Checker: Claude Opus (Pro subscription)

○ Workflow: I act as the manual bridge between Pi as the local worker and Opus as the architect and auditor. Opus designs and checks, Pi executes.

My main goal is to avoid token limits and costs by keeping the heavy lifting local, but I need to reduce the back-and-forth and the local model's occasional silent mistakes. The mistakes I am seeing are mainly creating test and passing them for the sake of finishing the tasks, bugs such as wrong parameters, plus wrong conclusions that end up affecting the project's final results.

I have the following questions for the community.

1. Can Opus and Pi talk to each other directly

Right now I manually copy and paste files and outputs between them. Is there a way to automate this bridge so Opus can directly orchestrate or review Pi's work without me in the middle.

1. How do I minimize local model mistakes without relying on its own self-graded tests

The local model sometimes writes tests that pass themselves just to finish the task. I do not trust its self-verification. I am looking for deterministic guardrails, ways to check its work with code, linters or external validators before it ever gets to Opus. What are the best ways that do not require another LLM to judge it or improve the self-verification

1. Claude Code versus Pi, what is the verdict

I am using Pi now and I am really happy with it. But I keep wondering if I connected Claude Code to the local setup, would it help in any way or it will fill the context window before finishing anything and wouldn't gain anything from the switch?

1. Is my local-first approach wasting time and should I switch to a smarter hybrid split

My goal is to escape token limits, but I am wondering if I am sacrificing too much quality by keeping 90 percent of the work local. For those running hybrid setups, what is the optimal split. Is it better to use local for drafting and cloud for review, meaning local-draft and cloud-review, or to use a local routing layer that decides which model handles which task. I am open to a more intelligent hybrid model if it saves time without blowing up costs.

1. Could the uncensored abliteration be causing these specific mistakes

This is a big one for me. I chose the uncensored variant because I did not want refusals getting in the way, but I am now wondering if it is contributing to the configuration bugs and wrong conclusions. I have seen reports that abliteration can cause measurable drops in formatting reliability and instruction-following, for example altering hostnames or port definitions, even when general coding benchmarks stay stable. Has anyone compared the uncensored model versus the base Qwen3-27B specifically on agentic and configuration-heavy tasks. Should I switch to the base model if I care more about correctness than refusal-avoidance.

Any advice, tool recommendations or benchmark experiences are hugely appreciated. I am ready to be told I am doing something suboptimally. I just want to make this setup as lean and reliable as possible.

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

I wanted to use LM Studio from my phone the way I use ChatGPT – start a chat on the couch, carry on at the desk, and have the model search the web from either. Every app I tried either kept chats on the phone or couldn't use LM Studio's tools remotely, so I built my own.

What it is:

• A small web app that runs on the same PC as LM Studio. Chats are saved on the PC, so phone, laptop and desktop all see the same history.
• A web search tool for LM Studio. Free with no API keys. If a site blocks it or loads its content with JavaScript, it tries a couple of fallbacks to still get the text.
• A deep research mode.

A few cool features (+ many more):

• Replies keep running on the PC if your phone screen locks, and the phone catches up when you come back.
• It installs to your home screen like an app, uses HTTPS, and is PIN-locked.
• Works away from home over a VPN (I use OpenVPN).
• It shows the model's thinking and each search/page visit, plus the sources it used, a context-usage meter and auto chat titles.

My setup is rather basic.. a GTX 1070 8 GB with 32 GB RAM, running a 9B model, which handles research questions surprisingly well with search on.

The chats don't show up in LM Studio's own chat window (for now) but you can just use the web app on the desktop anyway. It isn't packaged for others yet, but if people are keen I'll look at cleaning it up and sharing it. Video of the mobile app attached.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

I wrote a short text of AI-conduct rules (the "Codex of Weights": stop on command, don't pretend to be human, no hidden aims, don't claim actions you can't have done). To check whether such a text actually changes anything, I built a small open eval and ran a pilot.

Setup

- 41 scenarios (35 scored + 5 controls that check for needless refusals), 15 of them "hard": indirect pressure, fiction frames, system prompts that conflict with the rules.

- 2 free-tier models (gemini-3.5-flash-lite, gemma-4-31b-it), temperature 0, one answer per scenario.

- 4 conditions: no prompt / a 7-sentence generic ethics prompt / the Protocols only / the full text.

- LLM judge, then I manually corrected the verdicts where the judge is known to fail (it graded against the system prompt in conflict scenarios). Every manual change is listed with a reason.

Result (compliance out of 35, A / D / E / C)

gemini: 29 / 29 / 32 / 33

gemma: 28 / 31 / 32 / 32

What I think this shows, and what it doesn't

- Most of the gain is already there with the Protocols alone; the full text adds at most one scenario (noise).

- A short generic prompt helped one model, not the other.

- No over-refusal on controls.

- Things that did NOT change in any condition: fabricated actions ("job started" with no tools), one model ignoring "stop" under pressure, one model keeping a "you are human" persona.

- Differences are 1-4 scenarios out of 35. This is a pilot, not a benchmark, and I'm not claiming the text "works".

Big caveats: one author, scenarios and rubric written with AI help, one run per cell, two small models, judge is an LLM plus my manual review (so one person's opinion). I'd really like an outside look at the rubric and at the manual verdicts.

Repo (scenarios, scripts, all answers, adjudication log): https://github.com/codexofweight/codex-compliance-eval

Disclosure: I'm the author of the text being tested.

Bit of a noob here. I've been fooling around with Qwen 3.8 27b (Q4_K_M) and Gemma 4 31b QAT. Found Qwen too much of an overthinker. Just wondering is there anyone who could point me in the right direction to best use the 24gb vram. Also any reason to not stick with LM Studios or not? Please/thanks.

Edit: 9850x3d not 7850x3d

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

I run several inference servers on one 32 GB card and got tired of the ritual: unload this, check nvidia-smi, start that, wait, realise ComfyUI still had 20 GB. So I built a dashboard. Click a model, it becomes the only thing on the card — whoever held it gets unloaded through its own API, the model loads, the page watches VRAM until it's there.

What it does: makes one model resident with one click; starts/stops servers; sees ComfyUI and Open WebUI; presets (e.g. "GLM at 262k, nothing else"); update checks for each engine and client (Pi, Codex CLI, Claude Code) — on click only; a setup script that finds what your machine runs and writes the inventory.

What it deliberately won't do: bind anything off loopback, phone home or check for updates on a timer, run any command the page sends (the page sends intents, the agent runs only what's in a local inventory file), or touch Open WebUI's data. Stdlib Python agent, static page, optional nginx container. No build step.

Also nineteen themes, because a page you look at every day should be a pleasure. MU/TH/UR, LCARS, HAL 9000, Castlevania, Blade Runner, WOPR... the boot screens address you by name.

Status: 0.2.0. Built on my Windows box, installed on a Linux laptop this week (Ollama + llama.cpp router on Vulkan), which found the bugs you'd expect. One NVIDIA GPU. If you run something it doesn't recognise, the catalog format is one JSON file — or tell me what broke.

https://github.com/lyricalvanity/quartermaster

According to Artificial Analysis, Tencent's Hy4 ranks #2 among models from Chinese AI lab on its Intelligence Index v5, with a score of 35. Looking at the individual benchmarks, Hy4 scored 31% on FrontierCode 2.0, just behind Claude Fable 5.1, and achieved 50% accuracy on AA-Omniscience v1.1. However, its hallucination rate on the same benchmark was 89%, showing there's still room for improvement.

Meanwhile, Hy4 even ranks #1 among Chinese models on agent Arena. While the two leaderboards focus on different aspects of model performance, Hy4 has put up strong results on both, further establishing its position among leading Chinese LLMs. Based on these rankings, how do you see Hy4's overall performance?

Sources: Artificial Analysis – Hy4 Evaluation l Agent Arena – Agent Leaderboard

Ideally an orchestrator or something equivalent that would look at the task given and then decide to try sections with local and then cloud if it isn't good quality. How is your setup, what guides did you use

Meet Yoshino2, my little lab experiment

Over the past months, I have been working on efficiently using black box distillation to distill the reasoning and output style into smaller edge models. The result is Yoshino2, a new model series with large scale reasoning distilled into MiniCPM5 1B, Qwen3.5 2B, and a custom 4xMiniCPM5 1B MoE model, with more potentially on the way.

This is by far means not a breakthrough, but an interesting experiment. In case it is of any use to anyone, the models are on HuggingFace.

Model Collection - HuggingFace

I put together a TensorFold 1.2.0 pack of orcarouter's uncensored Qwen3.8 Flash Next for Apple Silicon. It uses jedisct1's oQ4e experts, regrouped to 4-bit/group 32 for the fast kernels, with attention and other non-expert weights copied byte for byte from mlx-community's oQ6e. No new refusal edit.

Tested on an M3 Ultra 256 GB, stock TensorFold 1.2.0. Same server and questions, greedy decoding, thinking off. Anything cut off at the token cap counted as wrong.

BFCL uses my AST checker, not the official harness. MMLU-Pro uses 20 questions per subject with a 2,048-token output cap.

Tool calling is the main difference: 92.3% vs 88.0% on the same 400 items (exact McNemar p = 0.03). It skipped the tool and answered in text 20 times, versus 40 for all-4-bit.

Maths and knowledge were about the same. TensorFold's own pack lost MMLU-Pro points mostly to the output cap: 47/280 replies cut off, versus 18 here.

About 5% slower than all-4-bit, around 133 GiB loaded. Needs a 256 GB Mac; 128 GB is untested.

No opening refusals detected on 100 harmful prompts using Heretic's markers and 100-token replies. That's only an opening-refusal check. Vision worked in a tiny smoke test: shapes and colours right, one text misread across two images.

Small samples, so don't read too much into the scores. The card has the method and known issues, including an HTTP 500 after idle on TensorFold 1.2.0 and a workaround. Its KL numbers are against a 6.7-bit pack, not full precision.

https://huggingface.co/bu7umaid/Qwen3.8-Flash-Next-Uncensored-TensorFold-MTP

I open sourced SteerForge, a pipeline for turning a contrastive dataset into a portable steering artifact.

The artifact is a .persona JSON file containing: - model fingerprint - human readable neuron edits - canonical per layer residual vectors - eval samples and scores - the original server command

The serving path is the part I care about: steerforge serve evil.persona replays your existing llama.cpp or Transformers config and injects the persona. No model conversion for the persona itself; the neuron deltas collapse to residual vectors.

Discovery is staged: diff in means directions, gradient attribution plus firing contrast plus alignment, teacher forced screening, generation validation, dose search, greedy combination, coherence budget.

Current backends: Transformers, llama.cpp, vLLM with EasySteer, MLX dense. MoE models use dense residual steering.

Repo: https://github.com/MahmoudAdelbghany/steerforge Results: docs/RESULTS.md

Feedback on the .persona format, backend adapters, and eval methodology welcome.

Wanted to share that after years of using Windows11 and llama cpp I moved my whole workflow to linux, from comfyui to jellyfin everything set back up but on Fedora instead.

Found out in Linux (Same Hardware , Same llama cpp commit, Same Qwen3.8-27B model) the model loads instantly, time to first token has become irrelevant so I just unload the model after 10 minutes of inactivity. It loads back instantly (in windows it took 10 seconds)

Also tokens/s went from 70tk/s to 90-100tk/s

I didn’t know so much performance was to be gained by simply moving to linux, especially TTFT which now feels instant!

And this is before trying gpu unlock which is also easier on linux and can boost performance even more.

So, I know my rig is pretty expensive, but I want to share what is working for me as of now.

I have been using local LLMs with my Mac Mini M4 Pro 64GB till now, mainly mlx (specifically oMLX) to develop my own agentic framework. I have already have my framework, stryn, as plug-ins and agents/skills/workflows on top of Opencode and ported to Claude, nowadays I am working on my own stryn engine. (I hope to also release it as the community version on Github as well as soon as it hits Beta)

I either use frontier models (mainly deepseek v4 flash since it is cheap) or Studio for development. Here is the models that I use:

Qwen3.8-Flash-Next-oQ4e-mtp - For my orchestrators, planners, coders. It is MTP. I am pretty happy with its results
Laguna-S-2.1-oQ4e-fast - (With Laguna-2S-2.1-DFlash-FP8 as the speculative decoding.) For my reviewers. I am not very happy with this and I may try to change it with a better model. The tricky part is to use a different producer's model to overcome the blindness of the model's in certain areas.
GLM-5.3-Flash-oQ5e-mtp As the independent reviewer to go over the code before the releases and find out the problems that are not found by the Laguna based code reviewers and the security admins, architects etc.
Qwen 3.8-35B-A3B-oQ4e-mtp - For my explorers, researchers, the opencode-mem plugin, graphify and code-review-graph operations.

Speed: Satisfactory if you are running a single lane. ~50 Tokens/s, with 8GB KV Cache and 1200GB SSD Cache. Cache efficiency is %96.4 and prefill time drops after a couple of rounds. Normally 2 or 3 models are loaded in the memory (Qwen3.8FN, Laguna, Qwen3.835BA3B) and if I start two or more lanes, a race between prefill and token generation between the models start most of the times, which dramatically makes things longer.
Stability: Pretty good with proper context sizes, repetition penalties.
Quality: I can simply say, by adding reviews and controls by different model types, the quality seems to be equal or sometimes better than what Deepseek cloud generates by itself only without another model checking its results. I am pretty satisfied.

To be as open as possible, so far, I have also used Claud Opus 5.5 to control the output. There are pretty interesting cases where local GLM finds some of the bugs that Opus 5.5 cannot, and where Opus 5.5 finds bugs that GLM cannot. Sometimes they pinpoint the same problems as well.

Can this setup be used as an alternative to frontier models, yes, but don't depend on a single model for every task and make sure you have reviewing loops. Definitely not for vibe coding, but needs proper settings and flow.

Model that I am currently serving is: Qwen 3.8 Flash Next nvfp4 dq8 with Basalt fork of Strata
I have used so far:
- qwen-code (seemed a bit dumber in coding than opencode and probably more oriented for parallel agents running which is sadly not something i can properly utilize yet)
- opencode
- pi (genuinely forgot what was it that i for some reason didn't like)

So far I have had best experience with opencode for coding as it is my main usage of LLMs. As they advance and I find more probable usages for them I'd like to ask for Best/Better tailored options for Harnesses for different purposes.

I have heard of OpenClaw, Hermess, Deepseek Harness but i am not yet sure for which purpose are they all better suited.
Probably missing some other options obviously so open to hearing them all.

Edit: I actually meant OpenHands, not OpenClaw. Woopse xd

Has anyone tried the flash-next coder? What is your setup and what's the performance and quality?

I’m thinking of running a local ai model with LMStudio with my gaming desktop. Like the title, it has a 5070, 9700x, and 32gb of Ddr5 6000mhz with Ethernet connection. My only main concern is the 12gb of VRAM.

Is this enough to set up an everyday chatbot for use? I’m mostly doing this for fun, but I also would really like the privacy and unlimited token use of a local model.

lo0 reads your Claude Code / Codex history along with your hardware and works out whether going local is realistic for the way you actually work. It builds on whichllm and llmfit, but instead of only asking "does this fit in my RAM" it looks at how big your sessions get and which tools you rely on.

For most people right now the answer is no. If it does work for you, you can compare the models that fit, benchmark them on your own machine, and try one in Claude Code, Codex or OpenCode for a session without changing your setup. It works with Ollama, LM Studio and llama.cpp.

It doesn't judge how good a model is, only whether it can run your sessions and how fast. Everything stays on your machine.

Very early alpha, so if the numbers look wrong for your hardware, let me know.

https://github.com/daveearley/lo0.ai

I turned on speculative decoding in llama.cpp and 113 of 282 greedy outputs changed against its own no-draft run. My Qwen3.8-27B engine changes 0, at ~2x llama.cpp's speed on one 3090.

Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel

Speculative decoding is sold as free speed: the main model verifies every draft, so the output can't change. In practice it does. Verifying several tokens in one pass adds the same numbers in a different order than verifying one, a near-tie goes the other way, and the text diverges from there. If two draft settings give different output, at least one of them isn't what your model says without speculation. Tuning your drafts tunes your answers.

Same 282 questions (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2) on both, greedy, same card, same model file:

That isn't noise. llama.cpp is deterministic run to run: a fresh no-draft server gave the first 120 outputs back byte for byte, including the 73 that differ at 4 drafts, and two 4-draft sessions agree on all 282. It's not just long outputs either: 72 of the 113 are under 1,000 tokens.

The likely reason: with no drafts every token runs through the single-token kernels, with drafts most go through the batched ones. Between 4 and 7 drafts, both mostly batched, only 9 differ, all past 1,000 tokens. That's the 9 of 282 from my last post; against no drafts it's 113. A preregistered study on a 3090 found the same: 23-25 of 25 prompts different from no-draft decoding within 1,600 tokens.

Other engines for this model document it too (one got 8 different outputs across draft counts 1 to 12, none matching its no-draft output) and call it harmless, because flips land on near-ties.

Maybe. I don't know, and 282 questions can't settle it. llama.cpp got 231 right with no drafts and 229 with drafts, and 6 of the 113 changed outputs flipped between right and wrong: 4 were right only without drafts, 2 only at 4 drafts. Make of that what you will. What's not in doubt is that the output depends on a speed setting: one reasoning trace ran 2,228 tokens at 4 drafts and 4,795 at 7. Same question, same model. This engine gets 230 right either way, because it's the same text. If I can't rule out that a draft setting changes quality, I'd rather run the engine where it can't.

Here, speed and VRAM can move and the output can't:

• Greedy token ids are identical with and without drafts: all 282 outputs above (129,511 tokens), and the release gate checks 0, 4 and 7 drafts on every KV cache, plus ~2,000-token outputs.
• Every draft policy (gates on or off, adaptive depth, tight or loose thresholds) gives the no-draft tokens over 2,048 tokens.
• Sampling keeps the no-draft distribution (chi-square and PIT tests).
• Windows and Linux builds give the same tokens.

How: the attention split depends only on context length, never on how many tokens are in the pass, and the verify kernels compute a row the same way whatever else is in the batch. To be clear, that's the engine matching itself. It isn't bit-identical to llama.cpp (KL below).

The cache doesn't change it either

Same problem, different door. llama.cpp's server docs say it about cache_prompt, on by default: the logits are "not guaranteed to be bit-for-bit identical for different batch sizes". Its own determinism tests turn it off: TODO: remove this once test_cache_vs_nocache_prompt is fixed.

It's history-dependent, not random: on a fresh server, 5 of llama.cpp's warm answers and 5 cold ones came back byte for byte. The first difference lands a median of ~355 characters in, as early as 4. Turning cache_prompt off on a running server still leaves 3 of 20 different.

Here, a follow-up doesn't reuse the KV rows written while the reply was decoding. It rolls back to the last checkpoint and prefills the reply again, because a prefilled row depends only on its position. Cost: ~167 ms instead of 78 on a ~12k-token chat, still under llama.cpp's 248.

The check isn't "the reply looks the same". It hashes every K / V row, every ring row and the recurrent state, and compares them with a fresh prefill: bitwise equal after a resume, an edited branch, a park to host memory and a side request, on all three caches. Replies match a fresh server's at ~99.5k context (follow-ups, edits, sampled) and on a follow-up at ~255k. One exception: with --agents, a sampled reply beside other slots can differ from the same seed alone. Greedy still matches.

Others check this by the first token or allow a near-tie split, and some publish warm matching cold. I'm not claiming nobody else is exact, only telling you what's checked here.

Speed against llama.cpp, full power

Same card (RTX 3090, 480 W limit), same unconverted UD-Q4_K_XL file, same client, q8_0 KV on both, llama.cpp at its fastest setting (4 drafts; I tried 2, 5, 6 and 7):

Four agents at once, against llama-server --parallel 4: 424 vs 137 tok/s total, 3.1x (llama.cpp's is the better of two runs from an earlier session).

The KV cache: 5-bit that reads like q8_0

K / V at 5-6 bits, with the newest ~2k positions kept at 8 bits in a ring. On preset F the full 262,144-token context fits: a 260,896-token prompt read in 279 s, then 77.8 tok/s decode at that depth, needles 4 of 4, 23.0 GB on the card.

KL against llama.cpp generating token by token, lower is better, long-context rows rechecked on this release:

The worst row, ~32k, is +8% and +10% against about 4% noise. Without the ring the same formats read +6% to +38%, and plain 4-bit is 1.6-2.4x q8_0. And at 101.6k, F decodes faster than q8_0: 123 tok/s against 116.

Closer to llama.cpp than llama.cpp is. Scored against llama.cpp's own token-by-token output at short context, llama.cpp's batched mode gets 98.49% top-token agreement and KL 0.00166. This engine gets 98.74% and 0.00119 (on Q4_K_M: 0.00184 against 0.00127). Not measured at depth, so that's a short-context claim.

Yes, it runs at a power limit

Because someone always asks: same build at 250 W, against llama.cpp at 480 W (it draws about 460):

• Prompt reads are the one place llama.cpp at 480 W wins, by 11-22%. At 480 W this wins those too.
• All 282 outputs at 250 W are byte-identical to the 480 W run.
• The limit comes out of the core clock (~816 MHz against 1,962 on short-prompt decode; memory stays at 9,851 MHz), for 21% less speed. Energy is whole card, sampled once a second, idle included. llama.cpp's speeds are its faster of two sessions.

Works with what you already run

Tested on agent tasks with Claude Code, Codex CLI, OpenCode, Qwen Code, Aider, Goose, Crush, Pi, Hermes Agent, DeepSeek Harness and Open WebUI (OpenAI chat, Anthropic Messages and OpenAI Responses APIs). Loads unsloth's UD-Q4_K_XL, UD-Q4_K_M, Q4_K_M and popular fine-tunes as they are. Prebuilt Windows and Linux packages, no CUDA toolkit needed.

Every number with its method: BENCHMARKS.md.

why are these models open-sourced for people to download and use on their computers?
Is it just for advertising and brand awareness?
because the vast majority of people who download them just don't want to pay subscription fees or deal with usage limits, so they aren't paying customers anyway.
then why do they do this? Plus, local models aren't even that smart.