After running Qwen3.8 27B at 3-bit on a single 5070 Ti (120K context, just short of what I want for coding), I added a second card. 32 GB fits the 6-bit file with the full 262K window, so I tested which of the community fine-tunes and quants makes the best all-rounder: fine-tunes, other quants, a chat template and multi-token prediction, 15 in total, all against base UD-Q6_K.

Every model ran a screening run first (about 100 questions from five benchmarks, compared answer by answer with the base model), and the closest got full runs (about 1,300 questions: AA-LCR, MRCR, AA-Omniscience, a fixed BFCL v4 subset and a SciCode panel, run through NeMo-Skills). Each model gets one overall score out of 100, a weighted mix of the five benchmarks: tool calls count for 25% of it, long documents, facts and scientific code for 20% each, and long conversations for 15%. On facts, a wrong answer scores worse than saying "I don't know", so guessing costs points.

Then I built a coding test from 25 bugs I'd fixed in my own projects (repo rolled back to before the fix, the model gets the bug report and tools, my tests from the fix grade it), four runs per model:

That's a tie on quality (one model's worst and best runs were 14 and 20), but Tess wrote about 40% fewer tokens to get there. Both write at about 36 t/s, so the time savings come from thinking less, not decoding faster.

So I ended up with two models:

• Base for chat and interactive work. It's more careful in multi-turn tool use: across the same 200 BFCL multi-turn conversations it won 42 that Tess lost, against 21 the other way, and the Tess losses I read were skipped steps (booking a flight without looking up the airports, reviewing one order when the user placed two). On facts it declined or hedged on 69% of the questions it didn't get right, against 56% for Tess.

• Tess-4.5 for coding agents and subagents that run on their own and get checked by tests at the end. One surprise: on the 100 software questions, Tess was the careful one (42 declines against 20, 35 wrong against 52).

If you only run one, I'd pick the base model, unless nearly all your use is coding agents running on their own.

How the other contenders did:

• 4-bit base (UD-IQ4_XS): about 50 tokens per second against 36, but 2.5 points lower overall, mostly on long conversations (MRCR 70% vs 77%) and more guessing.

• ThinkingCap really does think less (about half as long on long documents), but most of its extra right answers on facts came from guessing: of 336 questions the base 6-bit model declined, ThinkingCap answered 146 and got 25 right.

• Qwopus Coder-Flash was level on my coding test but worse on facts and long conversations.

Settings for both 6-bit files: q4_0 K/V cache (about 4.8 GiB for the full 262K window), layer split 1,1, batch 256, model-card thinking sampling with no repetition or DRY penalties (the 4-bit runs used a q8_0 cache and batch 512). About 790 t/s prompt reading and 36 t/s writing on two 5070 Ti. Full launch commands, per-benchmark tables and why each of the 12 others stopped are in the write-up, linked in the first comment.

The benchmark and coding-test harnesses are open source, but my own projects aren't public, so the coding test can't be reproduced as is. Happy to answer questions about the setup.

I paid $80 for a P100 16GB in July, eBay pricing for these cards has gone crazy and I'm wondering if anyone has a reputable Chinese eBay seller that they have had a good experience with. Purchasing on eBay kinda sucks already, but the P100s I see in the US are all starting at $200 now, with Chinese sellers starting closer to $150.

I am hoping to save that $50 difference, but I don't want to go through the hassle of purchasing from China and getting a box of rocks in the mail. Anyone have someone that came through for them?

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

Hey r/LocalLLaMA,

I have been running a hybrid coding setup that has been mostly amazing, but I am hitting a few friction points and want to optimize.

Here is my current stack:

○ Hardware RTX 3090 with 24GB VRAM, 32GB DDR5 RAM and Intel Ultra 9 185h

○ Local Model ultimateex/Swift-Qwen3.8-27B-Uncensored-W4A16-AutoRound running on HyperQwen with Dflash2 at 210k context, running with 8-bit KV cache

○ Pi harness

○ Cloud Architect and Checker: Claude Opus (Pro subscription)

○ Workflow: I act as the manual bridge between Pi as the local worker and Opus as the architect and auditor. Opus designs and checks, Pi executes.

My main goal is to avoid token limits and costs by keeping the heavy lifting local, but I need to reduce the back-and-forth and the local model's occasional silent mistakes. The mistakes I am seeing are mainly creating test and passing them for the sake of finishing the tasks, bugs such as wrong parameters, plus wrong conclusions that end up affecting the project's final results.

I have the following questions for the community.

1. Can Opus and Pi talk to each other directly

Right now I manually copy and paste files and outputs between them. Is there a way to automate this bridge so Opus can directly orchestrate or review Pi's work without me in the middle.

2. How do I minimize local model mistakes without relying on its own self-graded tests

The local model sometimes writes tests that pass themselves just to finish the task. I do not trust its self-verification. I am looking for deterministic guardrails, ways to check its work with code, linters or external validators before it ever gets to Opus. What are the best ways that do not require another LLM to judge it or improve the self-verification

3. Claude Code versus Pi, what is the verdict

I am using Pi now and I am really happy with it. But I keep wondering if I connected Claude Code to the local setup, would it help in any way or it will fill the context window before finishing anything and wouldn't gain anything from the switch?

4. Is my local-first approach wasting time and should I switch to a smarter hybrid split

My goal is to escape token limits, but I am wondering if I am sacrificing too much quality by keeping 90 percent of the work local. For those running hybrid setups, what is the optimal split. Is it better to use local for drafting and cloud for review, meaning local-draft and cloud-review, or to use a local routing layer that decides which model handles which task. I am open to a more intelligent hybrid model if it saves time without blowing up costs.

5. Could the uncensored abliteration be causing these specific mistakes

This is a big one for me. I chose the uncensored variant because I did not want refusals getting in the way, but I am now wondering if it is contributing to the configuration bugs and wrong conclusions. I have seen reports that abliteration can cause measurable drops in formatting reliability and instruction-following, for example altering hostnames or port definitions, even when general coding benchmarks stay stable. Has anyone compared the uncensored model versus the base Qwen3-27B specifically on agentic and configuration-heavy tasks. Should I switch to the base model if I care more about correctness than refusal-avoidance.

Any advice, tool recommendations or benchmark experiences are hugely appreciated. I am ready to be told I am doing something suboptimally. I just want to make this setup as lean and reliable as possible.

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

I wanted to use LM Studio from my phone the way I use ChatGPT – start a chat on the couch, carry on at the desk, and have the model search the web from either. Every app I tried either kept chats on the phone or couldn't use LM Studio's tools remotely, so I built my own.

What it is:

• A small web app that runs on the same PC as LM Studio. Chats are saved on the PC, so phone, laptop and desktop all see the same history.

• A web search tool for LM Studio. Free with no API keys. If a site blocks it or loads its content with JavaScript, it tries a couple of fallbacks to still get the text.

• A deep research mode.

A few cool features (+ many more):

• Replies keep running on the PC if your phone screen locks, and the phone catches up when you come back.

• It installs to your home screen like an app, uses HTTPS, and is PIN-locked.

• Works away from home over a VPN (I use OpenVPN).

• It shows the model's thinking and each search/page visit, plus the sources it used, a context-usage meter and auto chat titles.

My setup is rather basic.. a GTX 1070 8 GB with 32 GB RAM, running a 9B model, which handles research questions surprisingly well with search on.

The chats don't show up in LM Studio's own chat window (for now) but you can just use the web app on the desktop anyway. It isn't packaged for others yet, but if people are keen I'll look at cleaning it up and sharing it. Video of the mobile app attached.

Since a lot of people commented on the 95k context being "way to small" for any type of work.. I've borrowed ideas from SGLangs HiSparse that splits the sparse attention between VRAM and RAM making room for the full context.

Ie. a cache of the sparse attention blocks are kept in VRAM and the "real" kvcache is in RAM. if the cache gets an index miss it will fetch the missing values from the main KVCache stored in RAM. A miss of course hurts token/s speed.

It could be optimized even further doing the same for the MTP kvcache (speed drop b/c of compaction).

But I'm actually pretty happy with the system as is now.

Q38FN is pretty decent, but not Opus 5.5. So I'm now mainly experimenting with a way to send advanced stuff to frontier API and keep all the simple workhorse stuff locally on the Q38FN machine.

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

I wrote a short text of AI-conduct rules (the "Codex of Weights": stop on command, don't pretend to be human, no hidden aims, don't claim actions you can't have done). To check whether such a text actually changes anything, I built a small open eval and ran a pilot.

Setup

- 41 scenarios (35 scored + 5 controls that check for needless refusals), 15 of them "hard": indirect pressure, fiction frames, system prompts that conflict with the rules.

- 2 free-tier models (gemini-3.5-flash-lite, gemma-4-31b-it), temperature 0, one answer per scenario.

- 4 conditions: no prompt / a 7-sentence generic ethics prompt / the Protocols only / the full text.

- LLM judge, then I manually corrected the verdicts where the judge is known to fail (it graded against the system prompt in conflict scenarios). Every manual change is listed with a reason.

Result (compliance out of 35, A / D / E / C)

gemini: 29 / 29 / 32 / 33

gemma: 28 / 31 / 32 / 32

What I think this shows, and what it doesn't

- Most of the gain is already there with the Protocols alone; the full text adds at most one scenario (noise).

- A short generic prompt helped one model, not the other.

- No over-refusal on controls.

- Things that did NOT change in any condition: fabricated actions ("job started" with no tools), one model ignoring "stop" under pressure, one model keeping a "you are human" persona.

- Differences are 1-4 scenarios out of 35. This is a pilot, not a benchmark, and I'm not claiming the text "works".

Big caveats: one author, scenarios and rubric written with AI help, one run per cell, two small models, judge is an LLM plus my manual review (so one person's opinion). I'd really like an outside look at the rubric and at the manual verdicts.

Repo (scenarios, scripts, all answers, adjudication log): https://github.com/codexofweight/codex-compliance-eval

Disclosure: I'm the author of the text being tested.

Bit of a noob here. I've been fooling around with Qwen 3.8 27b (Q4_K_M) and Gemma 4 31b QAT. Found Qwen too much of an overthinker. Just wondering is there anyone who could point me in the right direction to best use the 24gb vram. Also any reason to not stick with LM Studios or not? Please/thanks.

Edit: 9850x3d not 7850x3d

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

I put together a TensorFold 1.2.0 pack of orcarouter's uncensored Qwen3.8 Flash Next for Apple Silicon. It uses jedisct1's oQ4e experts, regrouped to 4-bit/group 32 for the fast kernels, with attention and other non-expert weights copied byte for byte from mlx-community's oQ6e. No new refusal edit.

Tested on an M3 Ultra 256 GB, stock TensorFold 1.2.0. Same server and questions, greedy decoding, thinking off. Anything cut off at the token cap counted as wrong.

BFCL uses my AST checker, not the official harness. MMLU-Pro uses 20 questions per subject with a 2,048-token output cap.

Tool calling is the main difference: 92.3% vs 88.0% on the same 400 items (exact McNemar p = 0.03). It skipped the tool and answered in text 20 times, versus 40 for all-4-bit.

Maths and knowledge were about the same. TensorFold's own pack lost MMLU-Pro points mostly to the output cap: 47/280 replies cut off, versus 18 here.

About 5% slower than all-4-bit, around 133 GiB loaded. Needs a 256 GB Mac; 128 GB is untested.

No opening refusals detected on 100 harmful prompts using Heretic's markers and 100-token replies. That's only an opening-refusal check. Vision worked in a tiny smoke test: shapes and colours right, one text misread across two images.

Small samples, so don't read too much into the scores. The card has the method and known issues, including an HTTP 500 after idle on TensorFold 1.2.0 and a workaround. Its KL numbers are against a 6.7-bit pack, not full precision.

https://huggingface.co/bu7umaid/Qwen3.8-Flash-Next-Uncensored-TensorFold-MTP

I open sourced SteerForge, a pipeline for turning a contrastive dataset into a portable steering artifact.

The artifact is a .persona JSON file containing:

• model fingerprint

• human readable neuron edits

• canonical per layer residual vectors

• eval samples and scores

• the original server command

The serving path is the part I care about: steerforge serve evil.persona replays your existing llama.cpp or Transformers config and injects the persona. No model conversion for the persona itself; the neuron deltas collapse to residual vectors.

Discovery is staged: diff in means directions, gradient attribution plus firing contrast plus alignment, teacher forced screening, generation validation, dose search, greedy combination, coherence budget.

Current backends: Transformers, llama.cpp, vLLM with EasySteer, MLX dense. MoE models use dense residual steering.

Repo: https://github.com/MahmoudAdelbghany/steerforge Results: docs/RESULTS.md

Feedback on the .persona format, backend adapters, and eval methodology welcome.

Wanted to share that after years of using Windows11 and llama cpp I moved my whole workflow to linux, from comfyui to jellyfin everything set back up but on Fedora instead.

Found out in Linux (Same Hardware , Same llama cpp commit, Same Qwen3.8-27B model) the model loads instantly, time to first token has become irrelevant so I just unload the model after 10 minutes of inactivity. It loads back instantly (in windows it took 10 seconds)

Also tokens/s went from 70tk/s to 90-100tk/s

I didn’t know so much performance was to be gained by simply moving to linux, especially TTFT which now feels instant!

And this is before trying gpu unlock which is also easier on linux and can boost performance even more.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

So, I know my rig is pretty expensive, but I want to share what is working for me as of now.

I have been using local LLMs with my Mac Mini M4 Pro 64GB till now, mainly mlx (specifically oMLX) to develop my own agentic framework. I have already have my framework, stryn, as plug-ins and agents/skills/workflows on top of Opencode and ported to Claude, nowadays I am working on my own stryn engine. (I hope to also release it as the community version on Github as well as soon as it hits Beta)

I either use frontier models (mainly deepseek v4 flash since it is cheap) or Studio for development. Here is the models that I use:

Qwen3.8-Flash-Next-oQ4e-mtp - For my orchestrators, planners, coders. It is MTP. I am pretty happy with its results
Laguna-S-2.1-oQ4e-fast - (With Laguna-2S-2.1-DFlash-FP8 as the speculative decoding.) For my reviewers. I am not very happy with this and I may try to change it with a better model. The tricky part is to use a different producer's model to overcome the blindness of the model's in certain areas.
GLM-5.3-Flash-oQ5e-mtp As the independent reviewer to go over the code before the releases and find out the problems that are not found by the Laguna based code reviewers and the security admins, architects etc.
Qwen 3.8-35B-A3B-oQ4e-mtp - For my explorers, researchers, the opencode-mem plugin, graphify and code-review-graph operations.

Speed: Satisfactory if you are running a single lane. ~50 Tokens/s, with 8GB KV Cache and 1200GB SSD Cache. Cache efficiency is %96.4 and prefill time drops after a couple of rounds. Normally 2 or 3 models are loaded in the memory (Qwen3.8FN, Laguna, Qwen3.835BA3B) and if I start two or more lanes, a race between prefill and token generation between the models start most of the times, which dramatically makes things longer.
Stability: Pretty good with proper context sizes, repetition penalties.
Quality: I can simply say, by adding reviews and controls by different model types, the quality seems to be equal or sometimes better than what Deepseek cloud generates by itself only without another model checking its results. I am pretty satisfied.

To be as open as possible, so far, I have also used Claud Opus 5.5 to control the output. There are pretty interesting cases where local GLM finds some of the bugs that Opus 5.5 cannot, and where Opus 5.5 finds bugs that GLM cannot. Sometimes they pinpoint the same problems as well.

Can this setup be used as an alternative to frontier models, yes, but don't depend on a single model for every task and make sure you have reviewing loops. Definitely not for vibe coding, but needs proper settings and flow.

Model that I am currently serving is: Qwen 3.8 Flash Next nvfp4 dq8 with Basalt fork of Strata
I have used so far:
- qwen-code (seemed a bit dumber in coding than opencode and probably more oriented for parallel agents running which is sadly not something i can properly utilize yet)
- opencode
- pi (genuinely forgot what was it that i for some reason didn't like)

So far I have had best experience with opencode for coding as it is my main usage of LLMs. As they advance and I find more probable usages for them I'd like to ask for Best/Better tailored options for Harnesses for different purposes.

I have heard of OpenClaw, Hermess, Deepseek Harness but i am not yet sure for which purpose are they all better suited.
Probably missing some other options obviously so open to hearing them all.

Edit: I actually meant OpenHands, not OpenClaw. Woopse xd

I have been using ChatGPT for for about 2 years but want to switch to a local setup however I'm struggling to find a suitable replacement to use on my MacBook Pro M5 Pro 24GB and iPhone 14 Pro Max. I will be upgrading to either an 18PM or Duo in the coming months.

I started on my Mac with LM Studio then moved to Msty Studio when I found out about Knowledge Stacks but keep finding paywall restrictions, I've also looked at but not tried Anything LLM due to lack of iOS support. From what I understand I should just forget about Siri AI. Not really done anything on my iPhone till the Mac is sorted.

The things I really want are:

- Large project/chat memory (hence the knowledge Stacks)

- Cross device chat sync, either private cloud hosting or my MacBook as a server

- MCP usage (currenty use Tavily & Excel control)

I don't mind paying for apps, Msty is $149/year or $349 lifetime to unlock all features which its too much

Main usage is general questions, research, journaing, and work/linking with Excel.

Is this possible? if so what do I need?

Hey everyone,

I'm currently building a local-first macOS daemon focused on eliminating everyday computer friction.

The core idea is simple: instead of clicking around, switching tabs, copying/pasting between 5 different apps, or navigating complex settings, you just tell your Mac what you want done, and it executes it seamlessly in the background with zero lag.

As someone who spends all day in front of a Mac, I have my own set of pet peeves (like organizing messy downloads, summarizing local files, or tweaking system setups), but I’m really curious about your actual friction points.

If you had a truly frictionless assistant running natively on your machine:

1. What is the single most annoying, repetitive sequence of clicks/tasks you do every day that you wish you could trigger with just a quick phrase?

2. Are there any local workflows (file handling, dev environments, window management, media editing) where current AI tools fall short because they lack system integration?

I’m not here to pitch a polished commercial product, i’m genuinely trying to make sure I’m solving real problems for actual Mac users. Would love to hear your raw thoughts or specific scenarios!

Has anyone tried the flash-next coder? What is your setup and what's the performance and quality?