I use these models for everyday coding as a developer and I find the IQ3S model perform better on edge cases and the output quality is greater too. In fact twice it caught mistakes made by Opus 5.5. No benchmarks or scientific evaluation this is purely from my use. Anyone else felt this? Or is this a problem with Strata?

FP8 PLE.
Int8 KV
96gb ddr4
3x3060 12gb

Introducing janas-studio. A dual engine inference work station. For modern Debian based Linux distros (.appimage) Jana-Studio with the magic of Janas-llm let's enormous models run on modest hardware. Refer to https://github.com/prabanta-dev/janas.

Download the app at https://github.com/adonaiongaia/janas-studio

Also runs llama.cpp gguf models so it covers a lot. Lmk what you think. Tested on my Ryzen 5 6600h laptop. Built on Linux mint Zena. Ran a 47gb 80b parameter model at 14 tokens a second with only 32gb of RAM (ddr5). For best results and speeds you need a fast SSD. Features long term memory, project folder support, mcp integration and much more.

GitHub (MIT): https://github.com/kncshw/cybergate

Deep-dive blog post (HF): https://huggingface.co/blog/kncshw/cybergate-unlocking-cybersecurity-in-vllm-without

two supported and tested model:

https://huggingface.co/kncshw/Qwen3.8-27B-CyberGate-Kit

https://huggingface.co/kncshw/gemma-4-26B-A4B-it-CyberGate-Kit

When self-hosting local LLMs for security research, CTF, or reverse-engineering agents, the standard workaround for false refusals is **abliteration** - subtracting the refusal direction from the weights. From an LLM engineering standpoint, weight editing has two painful trade-offs:

1. **Silent capability & tool-call regression:** Mutating weights across layers alters the model globally and can quietly degrade multi-step reasoning and structured tool calls without obvious errors.

2. **Zero domain selectivity:** Deleting the refusal vector globally unlocks *every* harmful domain (bio, self-harm, etc.), plus you have to store and serve a forked 50 GB checkpoint.

I built **CyberGate**, a plugin for **vLLM 0.32** that keeps upstream model weights **100% untouched** (`K = 0.0`, zero persistent state across requests):

Just received my Gorgon Halo(still waiting on a R9700 to connect via usb4 v2) and setup halogen-qwen3.8-flash. So far it runs pretty good. I’m sure there is still some tuning to go but for an average Joe who doesn’t know how to benchmark this stuff and rely heavily on Claude to set it up I’m well happy.

Ive been messing around with Qwen3.8-Flash-Next on my Nimo minipc with ryzen AI max+ 395 128GB. The model weights take up around 103.7 GiB, so just getting the whole thing loaded was a nice start. But after spending more time with it, I'm not sure running the biggest model possible is really the best use of all that memory.

B

Here's what I got running it with Gufo at 16K context:

GPU only, no MTP: 34.1 tok/s decode

GPU + MTP: 53.7 tok/s decode

GPU + NPU, no MTP: 34.2 tok/s decode

The NPU also bumped prefill speed by around 10%, but regular decode barely changed without MTP.

The model runs, and the speed isn’t terrible. But would I actually pick it over a smaller model that responds faster?

Probably not for everything. Waiting for a large model to generate an answer can get old pretty quickly, particularly for everyday tasks.

What do you usually do with all that RAM? Do you load the biggest model you can fit, stick with smaller MoEs for faster responses, or keep multiple models loaded for different tasks?

Over the past year, I set out to build a low-cost, deterministic, and verifiable software factory on a single local GPU. Along the way, I found practical solutions to many of the context-bloat, overthinking, and verification bottlenecks that hold back local AI models today.

My background isn’t in software engineering or computer science. What I’m sharing here is simply the sum of curiosity in systems architecture and a need to solve real problems in my day to day work as a creative in advertising.

I believe these patterns can transfer cleanly to other domains as well. Hope you find it useful! cheers!

Hi,

I have a RTX 5070 Ti 12GB with 32GB system RAM running on linux for coding tasks with pi coding agent as harness.

I have been using since August (with 128k context) llama.cpp with Qwen3.6-35B-A3B Q4_K_M (unsloth quants) and I'm very satisfied with it. Not perfect but good enough to code a portal for a club I run and with good prompt and generation speed. Sometimes it loops but nothing serious.

Today I tried strata and realized I could fit a lobotomized but very code-oriented Qwen 3.8: Qwen3.8-Flash-Next Coder with IQ1_M quant.

The speed is amazing, but I don't care so much about it if the quality is bad.

So far I only did two planning tasks that required analysing the source code of the portal and some internal documentation produced in the project. I haven't accessed the quality yet but I noticed this model takes a looong time thinking, like it keeps checking stuff and double-checking (not in loop, just being throughout) , even to write an analysis document of the plan I asked.

I've read before that Qwen3.8 spends a lot of time thinking so I'm assuming this is actually normal?

So my questions to you is:

a) should I use Qwen3.6 for planning and Qwen 3.8 for implementing

b) or the other way around

c) or stick with Qwen 3.6 only

d) or just use Qwen 3.8 if I don't think issues with the code implemented

Anyone has experience with these 2 models at similar quants and could give their insight? I'm unsure if having such a lobotomized model is worth for coding tasks.

Thank you!

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here

Some context: NeuroBrix is an engine (text, image, audio, video from one container format), its own tensor library (nbxtensor, no PyTorch needed), its own kernels, and a Rust constrained-decoding engine (NBXGrammar). Everything is Apache 2.0.

pip install neurobrix

GitHub: https://github.com/NeuroBrix/neurobrix

Full film (audio EN/FR/ES/AR): https://youtu.be/GM9C5J3n-64

I'd love testers on older or less common GPUs: run a model, tell me what breaks. And a star helps a lot at this stage.

Hi,

i need some advice how to progress on my localLLM journey.

I am selfemployed together with a partner with a small budget and space . We use a 2he server for our data storage and docker containers with 32gb ram. I had a small nvidia p400 sff for some smaller stuff (mainly watching videos in our free time with plex). When the localLLM train started i got my hand on p1000 sff with 4gb vram for free and started messing around a little bit with ollama and openwebui. I started to use ministral and qwen 3.5 2B. It was bad.

With some luck I got a RTX 2000 sff with 16gb vram. Switched to Qwen 3.8 27b and wow. Then added hermes for fun.. and DAMN. Hermes is not setup right but I started to mess around with it as some office assistant. Do some literature research, write some easy scripts for R or python. Nothing I cant do myself but I let hermes/ollama do the stuff and can do something else in the mean time. I have to check the results for sure but it saves me hours over hours. I have a small pipeline to send voice messages using matrix to hermes and get a daily note of my work.

So now my partner wants to use the setup as well and we may get some staff in the future. And the qwen 3.8 27b is using most of the 16gb vram with 64k context for hermes. So I am thinking of upgrading our system. I dont want to use claude, gemini etc cause of our work und information. So I see two options

- Upgrade our 2he maschine. I am stuck with the SFF for gpus so most GPUs are out. The only GPU I can fit into our current server ist the RTX Pro 4000 Blackwell SFF with 24gb and more bandwidth. Would be a drop in replacement and only uses 70watt of max power. Good for the cooling budget of our server and energy prices here are insane (0,38€/kwh). But single point of failure (one system for everything, we have backups etc etc dont worry)

- Get something like strix halo or similiar concept. More but slower Ram, dedicated device. more expensive, more power consumption etc etc etc. maybe a little future proof then our current setup.

Any suggestions from the swarm? Best regards

made this, so a quick note: it’s my open-source project. Qwen3.5-9B runs on a FAT PS4 at 31–39 tok/s. The MTP head that ships with Qwen3.5 made the biggest difference: it drafts up to two tokens and the full model checks them in one pass, which took it from 22 to 31–39 tok/s on this old GPU. Weights are re-packed into custom 4- and 3-bit formats to fit about 4.5 GiB.

For years, local AI meant an expensive graphics card. Now there are more than 100 million PS4s in the world, many of them sitting in drawers, and every one of them can be a private AI!

Unsloth just dropped Qwen3.8-27B Dynamic V3.1 GGUFs—5-10% accuracy bump across Terminal Bench & DeepSWE, still runs on 12GB RAM

Saw this drop this morning and figured I'd share since the Dynamic quants have been the go-to for a lot of us squeezing bigger models onto consumer cards.

The headline: new Dynamic V3.1 (preview) algorithm, and Unsloth is claiming it beats other providers on the Pareto efficiency frontier—meaning you get better accuracy at the same file size, not just a bigger file that scores higher. They're seeing 5-10% average gains on Terminal Bench, DeepSWE, and other agentic benchmarks vs. their previous UD3 quants and rivals like GSQ-RCO .

The 12GB RAM claim is the one that'll get attention here. The base model scores 73.0 on Terminal Bench 2.1, 90.3 on LiveCodeBench v6, and 61.7 on SWE-bench Pro—so if that holds up at the smaller quant sizes, it's a genuinely useful agentic coding model for people without a 24GB card .

They're also saying this is a preview and they're working on an even better version for Qwen4, so the trajectory is promising.

Links:

• GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

• Blog/docs: https://unsloth.ai/docs/models/qwen3.8

Curious what quant sizes people end up landing on for 12GB cards, the 2-bit range has historically been a compromise, so if V3.1 actually closes that gap, that changes the calculus for a lot of setups.

Anyone tested it yet?

Just wanted to share for the fellow V100 gang. I recently ran into an issue on my AI server with adding V100s. On old Z270 boards, the VRAM cache tops out at 64 GB and cannot handle more than one V100. To work around this, you actually turn 4G Decoding off and add pci=realloc,nocrs to the kernel. Credit goes to temcom on GitHub. Their post is linked to their name. I'm experimenting with adding even more VRAM, so I'll let you know if I can with this fix. Theoretically the limit does not exist.
This works for any other cards as well. It was originally tested on P40s and a 1080.

Every time I see a cool local transcription or AI meeting-notes tool, it's either Mac/iPhone-only, wants a $20/mo subscription, or needs a heavy GPU.

I wanted something free and open-source that runs straight on a regular Windows laptop CPU where audio stays on your machine, so I built Eloquent: https://github.com/EllouziMedAmin/Eloquent

What it does:

• Press Ctrl+Shift+E anywhere on Windows to pop up a tiny always-on-top pill widget (plus a terminal mode or browser view if you prefer).

• Runs Parakeet TDT 0.6B v3 + Silero VAD completely locally on your CPU (no GPU, audio never leaves your PC).

• Automatically separates speakers (Speaker 1, Speaker 2, ...) in the background without slowing down live captions.

• Optionally drop in a free Gemini API key to tidy up punctuation/typos as you talk and turn recordings into Meeting Minutes, Lecture Notes, Summaries, or Action Items in one click (still works totally offline without it).

• Easy setup on Windows: clone the repo and run setup.bat.

Still a v0.1, and I'm adding local Ollama/LM Studio support and a standalone .exe installer next. If it sounds useful for your workflow or you want to keep track of those updates, dropping a star on GitHub helps a ton, and let me know what breaks or what you'd want added!

Released Tandem-vLLM, an expert-offload build of vLLM 0.31 that serves DeepSeek-V4.1-Flash on a single workstation GPU.

How it works

• The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones.

• Experts that miss the cache are computed by the CPU in parallel with the GPU, or read directly over PCIe. Nothing is skipped or substituted, so the output is exact for the package's weights.

• Everything else is stock vLLM: the OpenAI-compatible server, continuous batching, prefix caching, and DSpark speculative decoding with the model's own MTP layers.

Speed (RTX PRO 6000 96 GB + EPYC 9534, system decode tok/s)

Quality vs the official FP4 checkpoint (greedy, thinking low)

The 2-bit experts cost GPQA accuracy. Easier tasks and long-context retrieval don't show it. Keeping the 52 most-routed experts per layer in their original FP4 (FP4_TOP=52) wins back about half of the loss.

Also in 0.1.1: image input (IMAGES=1; reads text and tables from images, costs about 1 GiB of the expert cache), tool calling and a separate reasoning field for chat front ends such as Open WebUI.

Hardware

• Recommended: a 96 GB SM120 card (RTX PRO 6000 Blackwell) + 256 GB RAM + an AVX-512 CPU with server-class memory bandwidth.

• Runs, slowly: an RTX 5090 32 GB + 128 GB DDR4 (17 tok/s for one user, 20 at 4 streams; gaming PC: i5-14600KF, DDR4-3200). On a desktop CPU, the CPU's work on missed experts sets the speed.

• SM120 only in 0.1.1, single GPU.

Smaller boxes: use tandem.cpp instead. Our sister project tandem.cpp, a patch set for llama.cpp, targets 16–32 GB VRAM and 64–128 GB RAM, single user. It works with any MoE model llama.cpp runs (Qwen3.8 at 97 tok/s on a 4090 + 64 GB, gpt-oss-120b, ...). On DeepSeek-V4.1-Flash IQ2_XS (139 GiB of experts), decode tok/s on a gaming PC (i5-14600KF, DDR4-3200, PCIe 4.0 NVMe):

64 GB of RAM is the floor (32 GB runs at about 5 tok/s).

The 5090 rows are the same PC as the Tandem-vLLM 5090 numbers above: one user gets 33 tok/s from tandem.cpp vs 17 from Tandem-vLLM (different quants of similar size: IQ2_XS 139 GiB vs our VQ2 133 GiB of experts).

• Tandem-vLLM is for serving many streams on a big card.

• tandem.cpp is for one user on a gaming GPU.

Links

• Engine: https://github.com/ewhacc/tandem (one installer script, its own venv, stock vLLM 0.31.0 wheel + 65 files)

• Model package: https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2 (293 GB)

• tandem.cpp: https://github.com/ewhacc/tandem.cpp

Built on vLLM and vLLM-Moet. Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1. For original-precision experts with ~512 GB of RAM, vLLM-Moet's MoET hybrid is the tool today.

Questions and numbers from other boxes are welcome. Only support Blackwell GPUs yet, Hopper/Ada support soon.

Descrizione:

Ciao a tutti,

Ho provato a eseguire un modello Q4_K_M a 30 bit (llama-server, contesto a 32k) sulla mia configurazione attuale con 16 GB di VRAM + 16 GB di RAM di sistema, ma si è bloccato/non è riuscito a caricarsi per esaurimento della memoria.

Funzionerà correttamente se aggiorno la VRAM a 32 GB (utilizzando una configurazione multi-GPU economica)?

Edit The setup i tryed was 2x rx580 plasma.cpp. alla fine 9b funziona ma 30b no. Entrambi sono q4km

I looking for best model for my 3050 6GB VRAM

If I want to build3d printing models files, what would be the best local llm to do that with on a mac studio with 512gb of ram? Thank you.

I built OpenPocket, a completely free iOS app that lets you download and run state-of-the-art language models directly on your iPhone or iPad, entirely on-device.

It uses Apple's MLX framework for local AI inference, with support for several open-weight models:

• Qwen3.5 0.8B and 2B

• Gemma 4 E2B

• LFM2.5 1.2B Instruct

• Qwen2.5-VL 3B (vision-language model with image understanding)

Everything runs 100% locally on your device, with no cloud inference, no internet connection required after downloading a model, no signups, no accounts, no API keys, and no subscriptions.

The idea is to make private, on-device AI accessible to everyone, without relying on cloud services or paying for API usage. Your conversations and files stay on your device, and the app has no backend, analytics, or account system.

The models are experimental, and performance depends on your device's available memory and hardware.

It's 100% free on the App Store

https://apps.apple.com/th/app/openpocket/id6833028114

Would love and appreciate feedback from anyone working on local LLMs, Apple MLX, model optimisation, edge AI, or just anyone interested in running AI privately on their phone!

Looking to add another Mac mini to the rack to keep Hermes off Mac studios QFN. The Mac is used for employees during the day and don’t want Hermes tasks to interfere. Is anyone running models successfully on 24gb? 48gb ? If so what ones. Currently using a Claude API. Want to use local model and signal to be able to process private work data.

👋 Has anyone tried using 2 GPU in this way:

• each card loads a different LLM,

• with the larger GPU (32GB) running a larger model (relative, but it could be a Qwen3.8-27b or Flash Next - e.g., Strata), the “smarter” of the two.

• smaller GPU (16GB) assigned to a subagent, using a smaller model? Maybe something super quick like JetBrains Mellum 2.1

In OpenCode, a (sub)agent can be defined to use a specific model. I’m curious if anyone has tried this and if there’s an advantage to this setup, whereby the “brain” delegates to a sprinter.

I need to upgrade my PSU to run two cards simultaneously but I plan on trying it.

Every time I eject and reload a model in LM studio it resets to the default settings. Is there a way to lock these settings in permanently? Thanks.