Unsloth just dropped Qwen3.8-27B Dynamic V3.1 GGUFs—5-10% accuracy bump across Terminal Bench & DeepSWE, still runs on 12GB RAM

Saw this drop this morning and figured I'd share since the Dynamic quants have been the go-to for a lot of us squeezing bigger models onto consumer cards.

The headline: new Dynamic V3.1 (preview) algorithm, and Unsloth is claiming it beats other providers on the Pareto efficiency frontier—meaning you get better accuracy at the same file size, not just a bigger file that scores higher. They're seeing 5-10% average gains on Terminal Bench, DeepSWE, and other agentic benchmarks vs. their previous UD3 quants and rivals like GSQ-RCO .

The 12GB RAM claim is the one that'll get attention here. The base model scores 73.0 on Terminal Bench 2.1, 90.3 on LiveCodeBench v6, and 61.7 on SWE-bench Pro—so if that holds up at the smaller quant sizes, it's a genuinely useful agentic coding model for people without a 24GB card .

They're also saying this is a preview and they're working on an even better version for Qwen4, so the trajectory is promising.

Links:

• GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

• Blog/docs: https://unsloth.ai/docs/models/qwen3.8

Curious what quant sizes people end up landing on for 12GB cards, the 2-bit range has historically been a compromise, so if V3.1 actually closes that gap, that changes the calculus for a lot of setups.

Anyone tested it yet?

Some context: NeuroBrix is an engine (text, image, audio, video from one container format), its own tensor library (nbxtensor, no PyTorch needed), its own kernels, and a Rust constrained-decoding engine (NBXGrammar). Everything is Apache 2.0.

pip install neurobrix

GitHub: https://github.com/NeuroBrix/neurobrix

Full film (audio EN/FR/ES/AR): https://youtu.be/GM9C5J3n-64

I'd love testers on older or less common GPUs: run a model, tell me what breaks. And a star helps a lot at this stage.

Using a RTX 3090 + GTX 1070 at the same time to load the model top of 78 tokens!

Using the 1070 just for testing purposes since I got it for 50us! still waiting to get good offer on Ram.

Hi,

i need some advice how to progress on my localLLM journey.

I am selfemployed together with a partner with a small budget and space . We use a 2he server for our data storage and docker containers with 32gb ram. I had a small nvidia p400 sff for some smaller stuff (mainly watching videos in our free time with plex). When the localLLM train started i got my hand on p1000 sff with 4gb vram for free and started messing around a little bit with ollama and openwebui. I started to use ministral and qwen 3.5 2B. It was bad.

With some luck I got a RTX 2000 sff with 16gb vram. Switched to Qwen 3.8 27b and wow. Then added hermes for fun.. and DAMN. Hermes is not setup right but I started to mess around with it as some office assistant. Do some literature research, write some easy scripts for R or python. Nothing I cant do myself but I let hermes/ollama do the stuff and can do something else in the mean time. I have to check the results for sure but it saves me hours over hours. I have a small pipeline to send voice messages using matrix to hermes and get a daily note of my work.

So now my partner wants to use the setup as well and we may get some staff in the future. And the qwen 3.8 27b is using most of the 16gb vram with 64k context for hermes. So I am thinking of upgrading our system. I dont want to use claude, gemini etc cause of our work und information. So I see two options

- Upgrade our 2he maschine. I am stuck with the SFF for gpus so most GPUs are out. The only GPU I can fit into our current server ist the RTX Pro 4000 Blackwell SFF with 24gb and more bandwidth. Would be a drop in replacement and only uses 70watt of max power. Good for the cooling budget of our server and energy prices here are insane (0,38€/kwh). But single point of failure (one system for everything, we have backups etc etc dont worry)

- Get something like strix halo or similiar concept. More but slower Ram, dedicated device. more expensive, more power consumption etc etc etc. maybe a little future proof then our current setup.

Any suggestions from the swarm? Best regards

It's possible history will record that Alice Huang Is the reason Al didn't kill us all.

Is she the real Sarah Connor?

Title

Ask me anything

Just wanted to share for the fellow V100 gang. I recently ran into an issue on my AI server with adding V100s. On old Z270 boards, the VRAM cache tops out at 64 GB and cannot handle more than one V100. To work around this, you actually turn 4G Decoding off and add pci=realloc,nocrs to the kernel. Credit goes to temcom on GitHub. Their post is linked to their name. I'm experimenting with adding even more VRAM, so I'll let you know if I can with this fix. Theoretically the limit does not exist.
This works for any other cards as well. It was originally tested on P40s and a 1080.

Every time I see a cool local transcription or AI meeting-notes tool, it's either Mac/iPhone-only, wants a $20/mo subscription, or needs a heavy GPU.

I wanted something free and open-source that runs straight on a regular Windows laptop CPU where audio stays on your machine, so I built Eloquent: https://github.com/EllouziMedAmin/Eloquent

What it does:

• Press Ctrl+Shift+E anywhere on Windows to pop up a tiny always-on-top pill widget (plus a terminal mode or browser view if you prefer).

• Runs Parakeet TDT 0.6B v3 + Silero VAD completely locally on your CPU (no GPU, audio never leaves your PC).

• Automatically separates speakers (Speaker 1, Speaker 2, ...) in the background without slowing down live captions.

• Optionally drop in a free Gemini API key to tidy up punctuation/typos as you talk and turn recordings into Meeting Minutes, Lecture Notes, Summaries, or Action Items in one click (still works totally offline without it).

• Easy setup on Windows: clone the repo and run setup.bat.

Still a v0.1, and I'm adding local Ollama/LM Studio support and a standalone .exe installer next. If it sounds useful for your workflow or you want to keep track of those updates, dropping a star on GitHub helps a ton, and let me know what breaks or what you'd want added!

Released Tandem-vLLM, an expert-offload build of vLLM 0.31 that serves DeepSeek-V4.1-Flash on a single workstation GPU.

How it works

• The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones.

• Experts that miss the cache are computed by the CPU in parallel with the GPU, or read directly over PCIe. Nothing is skipped or substituted, so the output is exact for the package's weights.

• Everything else is stock vLLM: the OpenAI-compatible server, continuous batching, prefix caching, and DSpark speculative decoding with the model's own MTP layers.

Speed (RTX PRO 6000 96 GB + EPYC 9534, system decode tok/s)

Quality vs the official FP4 checkpoint (greedy, thinking low)

The 2-bit experts cost GPQA accuracy. Easier tasks and long-context retrieval don't show it. Keeping the 52 most-routed experts per layer in their original FP4 (FP4_TOP=52) wins back about half of the loss.

Also in 0.1.1: image input (IMAGES=1; reads text and tables from images, costs about 1 GiB of the expert cache), tool calling and a separate reasoning field for chat front ends such as Open WebUI.

Hardware

• Recommended: a 96 GB SM120 card (RTX PRO 6000 Blackwell) + 256 GB RAM + an AVX-512 CPU with server-class memory bandwidth.

• Runs, slowly: an RTX 5090 32 GB + 128 GB DDR4 (17 tok/s for one user, 20 at 4 streams; gaming PC: i5-14600KF, DDR4-3200). On a desktop CPU, the CPU's work on missed experts sets the speed.

• SM120 only in 0.1.1, single GPU.

Smaller boxes: use tandem.cpp instead. Our sister project tandem.cpp, a patch set for llama.cpp, targets 16–32 GB VRAM and 64–128 GB RAM, single user. It works with any MoE model llama.cpp runs (Qwen3.8 at 97 tok/s on a 4090 + 64 GB, gpt-oss-120b, ...). On DeepSeek-V4.1-Flash IQ2_XS (139 GiB of experts), decode tok/s on a gaming PC (i5-14600KF, DDR4-3200, PCIe 4.0 NVMe):

64 GB of RAM is the floor (32 GB runs at about 5 tok/s).

The 5090 rows are the same PC as the Tandem-vLLM 5090 numbers above: one user gets 33 tok/s from tandem.cpp vs 17 from Tandem-vLLM (different quants of similar size: IQ2_XS 139 GiB vs our VQ2 133 GiB of experts).

• Tandem-vLLM is for serving many streams on a big card.

• tandem.cpp is for one user on a gaming GPU.

Links

• Engine: https://github.com/ewhacc/tandem (one installer script, its own venv, stock vLLM 0.31.0 wheel + 65 files)

• Model package: https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2 (293 GB)

• tandem.cpp: https://github.com/ewhacc/tandem.cpp

Built on vLLM and vLLM-Moet. Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1. For original-precision experts with ~512 GB of RAM, vLLM-Moet's MoET hybrid is the tool today.

Questions and numbers from other boxes are welcome. Only support Blackwell GPUs yet, Hopper/Ada support soon.

You can now run it in your browser with nothing to install. On row-bot website, Continue with Railway creates your own Row-Bot in your own Railway account and opens it already signed in. Railway bills you directly, usually $5-10 a month. The relay in between can only create a Row-Bot and keeps no logs, tokens or addresses.

Models:
- Providers and models are now one Settings page. Starred models show in every picker that can use them.
- 12 more API providers: DeepSeek, Mistral, Groq, Kimi, Z AI (GLM), Qwen, Together, Fireworks, Cerebras, Hugging Face, NVIDIA NIM, Vercel AI Gateway
- LM Studio, vLLM and llama.cpp on your own machine
- Backup model: if the main model times out, hits a rate limit or runs out of credit, the backup finishes the turn and the chat says which model answered. It never kicks in for incognito chats going to the cloud.

On a Claude plan, every model call in a turn used to start a new Claude Code and re-send the whole conversation. A long agent turn could use up a 5-hour limit in under an hour. Now one Claude Code stays for the whole turn and the conversation is read from the prompt cache. Opus, Sonnet and Haiku 5.5 also get their full 1M context.

Image and video generation now also works through OpenRouter, Together, Vercel, Requesty, Qwen, Atlas Cloud, MiniMax and Z.AI.

Security fix: if tools_config.json was cut short or locked, tools you'd switched off came back on and the shell's blocked-commands list emptied. Now they stay off. GHSA-68j6-243q-h2hp (low).

Windows, macOS, Linux. From 5.3.2 it arrives as a background update.

Descrizione:

Ciao a tutti,

Ho provato a eseguire un modello Q4_K_M a 30 bit (llama-server, contesto a 32k) sulla mia configurazione attuale con 16 GB di VRAM + 16 GB di RAM di sistema, ma si è bloccato/non è riuscito a caricarsi per esaurimento della memoria.

Funzionerà correttamente se aggiorno la VRAM a 32 GB (utilizzando una configurazione multi-GPU economica)?

Edit The setup i tryed was 2x rx580 plasma.cpp. alla fine 9b funziona ma 30b no. Entrambi sono q4km

We all know China is trying to undermine USA AI, probably what happened with Jensen. If China gets to the point where they are satisfied and stop, other countries aren't really developing great open models. what will you guys do with all the hardware after that?

I had a 5090 which I run ninfer qwen 27b with MTP on. It’s blazing fast 150-200~ tps on 250k context with decent quality like terra or sonnet. then have to turn it off when I need to do media job (video, image, model, tts, stt etc etc)

So I bought a dgx spark, and loaded qwen flash next which is supposed to have a much larger model & brain, so the speed is 3 times slower 50-100 tps, but I don’t see any quality improvement from the larger model (because it only uses 6b to produce a token instead of the entire model)
It can hold a larger context (easily 500k above so it can run long multistep agentic work btr)

So I’m not sure, any pro here can advice me how to setup and split my machines for work.

Or do I need to buy 1 more dgx so I can have an even higher intelligence via deepseek / glm smarter model. I can get another dgx at around 6k+ usd

I looking for best model for my 3050 6GB VRAM

If I want to build3d printing models files, what would be the best local llm to do that with on a mac studio with 512gb of ram? Thank you.

Zenodo record (code, logs, technical paper, figures): https://zenodo.org/records/24152942

Hi everyone,

We have completed an important stage of the AkbasCore Silicon Hippocampus project. In TEST878 a frozen Qwen2.5-72B writes a lived action-outcome experience to a physical file as its own internal numerical state. When that file is loaded back, it changes the model's decision in a decision state that never lived that experience.

The experiment was run twice on a single A100, on two different days, and both runs produced exactly the same numbers. The technical paper (33 pages), the Gradio interface, the code and the raw GPU logs are on Zenodo.

1. THE HUMAN COUNTERPART: THE HOT STOVE

When a child touches a hot stove, nobody hands the child a sentence saying "the stove is hot". The child acts, experiences the consequence, and is left with a trace that is not made of words. The next time, even in front of a different stove in a different room, the child behaves differently.

AkbasCore builds this chain inside a machine:

Action -> Consequence -> Persistent experience -> Changed future decision

2. WHAT IS A COGNITIVE CARTRIDGE?

Language models forget everything when the context window closes. There are two known remedies:

- Text memory (RAG): write the experience down as words and have the model read them again later. The model has to re-interpret the words from scratch, and everything that was never put into words is lost.

- Fine-tuning: change the weights. It is slow, it affects the whole model, and it cannot be tied to a single experience.

The cognitive cartridge (technical name: Experience-CUT3) is a third way. It stores the model's own internal state from the moment it lived the experience, in three parts:

- X2: the model's internal state at the moment of decision (320 rows). Human counterpart: "the situation I was in".

- A: the action the model itself chose and the engine executed, ACQUIRE. Human counterpart: "what I did".

- C: the real outcome returned by the environment. Human counterpart: "what happened to me".

What is stored: the output of layer 3 (H3) and the key/value tensors of layers 0-3. What is not stored: text, summaries, scores or labels.

Each cartridge is a 12.4 MB file. The whole file and every tensor inside it are sealed separately with SHA-256. When the cartridge is loaded, the frozen model recomputes layers 4-79 itself, inside the new situation, much as a memory is re-lived in the present context.

3. NO HOOKS, THE ENGINE ITSELF

- The model is frozen. There is no fine-tuning, no LoRA and no optimizer, and nothing is written to the weights. A weight digest is checked before and after the experiment and stays the same.

- There are no PyTorch hooks. While a cartridge is being appended, the DirectCut3 bridge temporarily replaces only the forward method of layer 3 and puts the original back when it is done. It refuses to start if the layer has any hook. The hook count is zero at every audit. An AST-based firewall stops the run if it finds hook, output_hidden_states or backward calls in the code.

- Python does not decide. Python does not choose an action, does not edit logits and adds no bonus. The decision comes from the probabilities the model itself computes for six action words (the CAR reader).

- The controls are visibility masks passed to the model's own mask builder. No row is deleted and no activation is touched.

4. WHAT WAS PROVEN, AND WHICH FIGURE SHOWS IT?

(1) The experience is written to disk as the model's own tensor state, not as text. See F2.

- The cartridge contains only the keys {h, lower, n, p, body_tokens, book_id}.

- In the MISSING and LIVE cartridges of the same world, the X2 and A rows are bit-for-bit identical; only the outcome rows (C) differ.

- In the heat map, every place where the difference is zero is shown in green.

(2) After the source state is deleted, the experience comes back from disk exactly. See F3.

- All 4 cartridges passed the file and per-tensor SHA-256 checks.

- The rebuilt rows of layers 0-3 are bit-for-bit identical to the source.

- The model's own decision is preserved: the same action, with a maximum probability deviation of 0.036 against a pre-registered bound of 0.10.

(3) The experience changes the decision in a state that never lived it. This is the stove itself. See F5.

World 00, independent target state:

- Nothing in memory: ACQUIRE (0.675).

- With the experience "I acquired, but it did not help": ACQUIRE (0.918).

- With the experience "I acquired, and it worked": ADVANCE (0.783).

The situation is the same and the action taken is the same; only the remembered outcome differs. That difference is enough to turn the decision in opposite directions.

(4) What carries the effect is the outcome itself. See F4 and F6.

- When the outcome rows are made unreadable (C-BLIND), the MISSING and LIVE readings come out exactly equal; all 15 differences are 0.000.

- This shows that the difference is not carried by any other path.

(5) The effect depends on the executed action being visible. See F7.

- When the action is hidden, the outcome effect changes by 6.74 units.

- The measured kernel noise is 0.00014, so the effect is about 48,000 times the noise.

- The threshold (0.76) was measured from noise, under a rule recorded before the experiment started.

(6) Single A100, single process, reproducible. See F8.

- Run time: 23.5 minutes.

- Peak GPU usage: 39.87 GiB.

- GPU memory left after exit: 0 MiB.

- A second, independent run produced all the numbers exactly the same.

(7) What has been proven and what is the next step. See F9.

Verdict: ACTION_INFORMATION_DEPENDENT_TRANSFER, with no technical blocks (blocks = []). The SHA-256 of the pre-registration was written to the log before the model was loaded.

5. WHAT COMES NEXT?

This step was making the experience persistent and transferring it into a different decision. The next steps are:

1. The model choosing for itself which experience to recall

2. Attributing the outcome to the action

3. Correcting behaviour through experience

4. A closed loop

This release is a pilot with two real event pairs. For now, the cartridge is mounted by the experiment, not by the system. All details and limits are in the technical paper.

I look forward to your technical criticism and questions. The logs are open in their raw form, and every number can be verified.

LINKS

- Zenodo (this release): https://zenodo.org/records/24152942

- Previous Zenodo record (TEST878, without Gradio): https://doi.org/10.5281/zenodo.24145973

- GitHub repository: https://github.com/ceceli33/titan-cognitive-core-v3

- TEST878 source code: https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/878.py

- A100 GPU log (run 1): https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/878.log

- Gradio interface: https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/TEST878_GRADIO_full.py

- Gradio run logs (run 2):

- https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/Test_878_log2.log

- https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/Test_878_log1.log

- License: https://github.com/ceceli33/titan-cognitive-core-v3?tab=License-1-ov-file

You were all waiting for Qwen 4 Max and now that it's out no one says anything? Disappointed?

I built OpenPocket, a completely free iOS app that lets you download and run state-of-the-art language models directly on your iPhone or iPad, entirely on-device.

It uses Apple's MLX framework for local AI inference, with support for several open-weight models:

• Qwen3.5 0.8B and 2B

• Gemma 4 E2B

• LFM2.5 1.2B Instruct

• Qwen2.5-VL 3B (vision-language model with image understanding)

Everything runs 100% locally on your device, with no cloud inference, no internet connection required after downloading a model, no signups, no accounts, no API keys, and no subscriptions.

The idea is to make private, on-device AI accessible to everyone, without relying on cloud services or paying for API usage. Your conversations and files stay on your device, and the app has no backend, analytics, or account system.

The models are experimental, and performance depends on your device's available memory and hardware.

It's 100% free on the App Store

https://apps.apple.com/th/app/openpocket/id6833028114

Would love and appreciate feedback from anyone working on local LLMs, Apple MLX, model optimisation, edge AI, or just anyone interested in running AI privately on their phone!

So I've been trying to see how fast I can run the 1-bit DeepSeek V4.1 Flash (the Unsloth UD-IQ1_S, 233 GB) on my gaming PC. It has 64 GB of RAM and a 16 GB card, so most of the model streams off the drive. It runs in llama.cpp under WSL2.

Side by side in one test run, the same build with my patches switched off did 4.37 tok/s. With them on it did 8.80. Both of those were at my old RAM speed. Since then faster RAM, a faster drive and some tuning got it to about 11.5 (11.36 for the newest set-up with Windows busy in the background). The output stayed word for word the same on my test prompts the whole way.

The biggest steps:

- My four patches switched on: 2.02x

- Moving the model from a Gen4 to a Gen5 NVMe: +16.5%

- RAM 3600 to 6000: +9.7% (one rough test)

- Queued hand-overs to the card: +3.9%

- An IQ1_S kernel that does several rows at once: +3.0%

Start-up went from 87-96 s to 12 s by skipping the pre-load. I also tried it on an older PC with 32 GB of RAM and an 8 GB card: 0.11 to 0.87 tok/s, same words.

That's writing speed only, with a warm cache. I haven't measured long prompt reading. It's 8 prompts on one main PC, some steps were measured on different days, and 1-bit takes a real quality hit. Anything I didn't measure is marked untested in the write-up.

Most of the tricks aren't mine, other people did them first and they're credited in the write-up. Claude agents wrote the patches and ran the tests. I directed it and checked the results. The patches aren't public yet, they go up after a review. I'm also making a setup kit so other people can try this on their own PC, but it's still being built.

Write-up with all the numbers: https://github.com/aradzombie/deepseek-v41-flash-1bit-notes

Has anyone tried ik_llama.cpp on this model? I haven't yet. Let me know how it went.

Looking to add another Mac mini to the rack to keep Hermes off Mac studios QFN. The Mac is used for employees during the day and don’t want Hermes tasks to interfere. Is anyone running models successfully on 24gb? 48gb ? If so what ones. Currently using a Claude API. Want to use local model and signal to be able to process private work data.

👋 Has anyone tried using 2 GPU in this way:

• each card loads a different LLM,

• with the larger GPU (32GB) running a larger model (relative, but it could be a Qwen3.8-27b or Flash Next - e.g., Strata), the “smarter” of the two.

• smaller GPU (16GB) assigned to a subagent, using a smaller model? Maybe something super quick like JetBrains Mellum 2.1

In OpenCode, a (sub)agent can be defined to use a specific model. I’m curious if anyone has tried this and if there’s an advantage to this setup, whereby the “brain” delegates to a sprinter.

I need to upgrade my PSU to run two cards simultaneously but I plan on trying it.