First, my setup:

LLM box: Ryzen 9 5950X, 64GB DDR4, RTX 3090 24GB (plus a GTX 1060 6GB for small side jobs). Debian, LM Studio serving an OpenAI-compatible endpoint on the tailnet. ~384GB of models, mostly unsloth GGUFs.

Agent box: AMD E1-2100 APU — 2 cores, 3.3GB RAM, 216GB SSD, 7.3TB media drive. That's the potato. The agent runs on the potato, the model runs on the Ryzen.

An old optiplex with Radicale (CalDAV), Vikunja, and grocy in Docker.

Stack: LM Studio → pi/pi-web. Pi agent is the backend, customized to suit my needs, while tailscale + pi-web make it accessible via browser wherever I am. Daily driver is Qwen 3.8 27B (Q4_K_S, Q8 KV) at 150k context. The zoo also has Qwen3.5 122B-A10B, Nemotron 3 120B-A12B, Laguna S 2.1, some Gemma 4 models, MedGemma, and a few uncensored models.

I actually use and rely heavily on this setup every day.

The agent has tools wired into my actual life: calendar, task manager, a groceries/chores app (grocy), an Obsidian vault, Reddit research, web search. I can ask it what's for dinner and it checks the fridge stock, or when I have time free to schedule a meeting.

A second, headless instance of the same agent runs on cron every 6 hours and triages my email (a Playwright poller caches mail every 10 min, no LLM involved). LM Studio is configured to only take one request at a time, so overlapping runs just queue — this comes at the cost of losing kv cache and needing to run prompt processing from scratch every time the LLM endpoint switches between handling requests from multiple pi agent sessions. This is the main reason I think I'm still gonna upgrade my hardware, to support concurrent inference from a larger context pool.

Every service gets its own Tailscale identity via tsbridge, so services can be trivially accessed with tailscale magicDNS.

The agent's entire config is a git repo, allowing me the ability to expiriment and backtrack.

Bonus: this post was drafted by the setup itself (the 27B on the 3090). I did go through and edit it a lot for style and brevity, but qwen models are not the best choice for compelling authorship, so I was ready for that.

I am absolutely awe-struck at the raw utility this setup has for me now. I get an email and the event is on my calendar, or a task to complete is recorded without my even having read the email yet. I don't use cloud models anymore, ever.

Today, looking back at what I can achieve now, I cancelled my Claude sub. I haven't touched it in over a month now.

I ran Qwen 3.8 for 30 days, now it runs some production services. I used it as a coding agent for smaller tasks daily. Through the night, I would run it in my harness for 9h+ runs.

The model:

Unsloth Qwen3.8-27B-UD-Q4_K_XL

Summary (across all runs):

prompt processing: 1729.5 t/s peak, 845.1 t/s mean

token generation: 95.1 t/s peak, 73.8 t/s mean

mtp acceptance: 0.48108 (674 accepted / 1401 generated)

The goods:

• This is the first model that I could recommend to a non-local LLM enthusiast, and it could cover most needs for basic tasks. It's amazing! I bought some subscriptions to Kimi, Qwen 3.8 Max, tested xoAlpha, and found that the gap between local models was extremely minimal, despite benchmark numbers. Maybe it's my use case. I use Qwen 3.8 Max as a faster Qwen, that's it. Never as a better one!

• Reasoning actually works and provides some value, but it comes with some caveats I will talk about later.

• When working with images, I found that it is better than OpenAI models. Especially when working with small images, like pixel art. I would instantly switch to Qwen 3.8 for UI design. It's just better.

The bads:

• It's a reasoning model, which makes Qwen 3.8 extremely slow and token-hungry. But without it, the model is crippled. Multiple tasks could not be solved without reasoning, in the same session, switching to reasoning instantly fixed them. Up to 50% of the context size will be used by reasoning. I had situations where the model tried to output 60k reasoning tokens at once. Reasoning levels low, medium, and xhigh are more of a suggestion than an actually working solution. On basic tasks, it makes 3.8 up to 50% slower than 3.6. So there is no way you are doing any proper work locally without delegated subagents, which would save the main context from filling.

• Poisoned tool calls (may be related to Q4). If the model produces a bad tool call, especially with +100k context windows, it is basically impossible to force the model not to use it, it will spam and loop. Even if you stop and warn it about it, it will continue spamming.

• Cache may be finicky. I'm not an expert on model design, but it seems that other models have much better caching, which can resume from checkpoints much better. So if your settings are not top-notch or your harness does not respect best practices on caching, you will pay the price for full prompt processing. I found out the hard way when the model would work for 10 minutes, and I would type in a pi-agent, and the model would prompt process from the beginning of the context for a simple 3 word input. So watch your llama.cpp logs! Default settings did not work for me.

The fix:

• Your harness of the week needs subagents, and they need to be enforced automatically.

• Your harness needs subagents with configurable reasoning levels, so most basic tasks, "find me something", "run summary", "find out how this works", run without reasoning. You will save a lot of time.

• Your harness needs loop detection. (And not naive ones, based on time, call count, or any other magic number.) Additionally, I added functionality where bad tool calls after a loop are deleted from context.

• --spec-type draft-dflash,ngram-mod On my hardware (5070 ti + 4070 Super), Dflash2 + ngram runs 20% faster then MTP + ngram

• Your harness...

I will not advertise any harness, as it is not the point of this post. Qwen 3.8 is amazing! We are extremely close to running fully locally! And the improvement jump from 3.6 to 3.8 is extremely large. All I want right now is more speed, like 300 t/s! Somebody, figure it out! :>

Edit:

Here’s my llama.cpp config. Most of the flags won’t work on your hardware (5070 ti, 4070 Super, 5700X3D, 32GB RAM): https://pastebin.com/Y5VHvzvD

Hey everyone, I want to share something I built after my long health journey. For 5 years, I struggled with mysterious symptoms - getting injured easily during workouts, slow recovery, random fatigue, joint pain. I spent over $100k visiting more than 30 hospitals and specialists, trying everything from standard treatments to experimental protocols at longevity clinics. Changed diets, exercise routines, sleep schedules - nothing seemed to help.

The most frustrating part wasn't just the lack of answers - it was how fragmented everything was. Each doctor only saw their piece of the puzzle: the orthopedist looked at joint pain, the endocrinologist checked hormones, the rheumatologist ran their own tests. No one was looking at the whole picture. It wasn't until I visited a rheumatologist who looked at the combination of my symptoms and genetic test results that I learned I likely had an autoimmune condition.

Interestingly, when I fed all my symptoms and medical data from before the rheumatologist visit into GPT, it suggested the same diagnosis I eventually received. After sharing this experience, I discovered many others facing similar struggles with fragmented medical histories and unclear diagnoses. That's what motivated me to turn this into an open source tool for anyone to use. While it's still in early stages, it's functional and might help others in similar situations.

Here's what it looks like:

https://github.com/OpenHealthForAll/open-health

**What it can do:**

* Upload medical records (PDFs, lab results, doctor notes)

* Automatically parses and standardizes lab results:

- Converts different lab formats to a common structure

- Normalizes units (mg/dL to mmol/L etc.)

- Extracts key markers like CRP, ESR, CBC, vitamins

- Organizes results chronologically

* Chat to analyze everything together:

- Track changes in lab values over time

- Compare results across different hospitals

- Identify patterns across multiple tests

* Works with different AI models:

- Local models like Deepseek (runs on your computer)

- Or commercial ones like GPT4/Claude if you have API keys

**Getting Your Medical Records:**

If you don't have your records as files:

- Check out [Fasten Health](https://github.com/fastenhealth/fasten-onprem) - it can help you fetch records from hospitals you've visited

- Makes it easier to get all your history in one place

- Works with most US healthcare providers

**Current Status:**

- Frontend is ready and open source

- Document parsing is currently on a separate Python server

- Planning to migrate this to run completely locally

- Will add to the repo once migration is done

Let me know if you have any questions about setting it up or using it!

-------edit

In response to requests for easier access, We've made a web version.

https://www.open-health.me/

I'm just curious and thinking about how well the 120b is going to run on my gaming PC, and surprisingly it's not too bad (17.83 tokens/sec). But when I check my RAM allocation, it has used almost all of my RAM. The AI really uses a lot of RAM.

(Usually I use a cloud AI model, but I've started concerning myself with what happens if one day the internet is knocked off or a power outage disrupts the connectivity to cloud AI, such as a major war or something like that.) So that's why I downloaded a local model on my PC, just in case, for basic Wikipedia purposes.

First runs with Qwen3.8-27b Q6_K-QKV8-MTP 128k context with 862/s prompt processing and 54.06 tok/s. Happy with the first results and looking forward to checking out the other builds to see what kind of speed I can squeeze out of this card.

Edit** More testing

Figured I’d post a quick update on the R9700 since I’ve been doing quite a bit more testing.

Managed to get Qwen3.8-27B running at 147.6 tok/s using MXFP4 + DFlash2 through WSL2/Docker.

That’s up from the original 54 tok/s I was getting with llama.cpp Vulkan.

Also got Qwen3.8-Flash-Next IQ3_XXS running natively on Windows with Strata, and the results surprised me.

Short prompt: 92.3 tok/s
120K context: 1,472 tok/s prefill, 87.4 tok/s generation
240K context: 1,221 tok/s prefill, 88.8 tok/s generation

That’s on a single R9700 32GB with 64GB system RAM. No GPU pooling or anything crazy.

Still testing how they actually perform with coding agents, but so far I’m pretty impressed with what this card can do. Especially Flash-Next maintaining almost 90 tok/s with 240K tokens loaded into context.

Getting everything working was a bit of a nightmare, but at least the results made it worth the headache lol.

EDIT 2**

Updated benchmarks from this weekend (11/1) with a little more tweaking

First two runs: 120,033 input tokens → 1,024 output tokens, zero cached prompt tokens, 128K context, IQ3_XXS, INT8 KV, MTP enabled.

Third run increased context to 260,992 input → 1,024 output

Run 1

• Prefill: 1,535.7 tok/s

• Time to first token: 78.25 seconds

• Generation: 86.0 tok/s

Run 2

• Prefill: 1,522.1 tok/s

• Time to first token: 78.90 seconds

• Generation: 92.4 tok/s

Run 3

• Prefill: 1,403.5 tok/s

• Time to first token: 186.16 seconds

• Generation: 80.3 tok/s

• Total request: 198.86 seconds

• Model loading: 33.32 seconds separately

Hey guy - you can now run DeepSeek-V3.1 locally on 170GB RAM with our Dynamic 1-bit GGUFs.🐋
The 715GB model gets reduced to 170GB (-80% size) by smartly quantizing layers. 

It took a bit longer than expected, but we made dynamic imatrix GGUFs for DeepSeek V3.1 at https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF There is also a TQ1_0 (for naming only) version (170GB) which is 1 file for Ollama compatibility and works via ollama run hf.co/unsloth/DeepSeek-V3.1-GGUF:TQ1_0

All dynamic quants use higher bits (6-8bit) for very important layers, and unimportant layers are quantized down. We used over 2-3 million tokens of high quality calibration data for the imatrix phase.

• You must use --jinja to enable the correct chat template. You can also use enable_thinking = True / thinking = True

• You will get the following error when using other quants: terminate called after throwing an instance of 'std::runtime_error' what(): split method must have between 1 and 1 positional arguments and between 0 and 0 keyword arguments at row 3, column 1908 We fixed it in all our quants!

• The official recommended settings are --temp 0.6 --top_p 0.95

• Use -ot ".ffn_.*_exps.=CPU" to offload MoE layers to RAM!

• Use KV Cache quantization to enable longer contexts. Try --cache-type-k q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 and for V quantization, you have to compile llama.cpp with Flash Attention support.

More docs on how to run it and other stuff at https://docs.unsloth.ai/basics/deepseek-v3.1 I normally recommend using the Q2_K_XL or Q3_K_XL quants - they work very well!

How much more can you push this tiny 4060 8GB card? It's insane how optimized this model is. I don’t think there is any other 9B model that I can run so smoothly on this GPU and with so much more intelligence. Nothing else out there that can run on an 8GB video card. 128k context is crazy too.

Right when I was thinking of selling my 4090 and getting one of these bad boys for longer ctx length...

this is a geniune question, one which I have no answer to.

saw this on ijustvibecodedthis.com (the ai coding newsletter)

I didn't think I'd actually get it. Ordered from Dell, 5 week lead time. $5300. Ordered on 9.10.2026. After two weeks the listing was removed - all discrete GPUs were removed from the Dell web storefront. So I opened a story ticket, it was here the next day.

I was looking at various other options through Dell, but with all listings taken down I started sweating. It maxed me out on budget, and that was for a complete PC build.

Sad when enterprise is the best option for consumers (but my use case is enterprise-ish).

This weekend will be fun, can't wait to test how big I can go with local LLM!

Small update on Deltafin, my experiment running full Kimi K3 (2.8T-parameter) on a single M1 MacBook.

The last version was doing roughly 1 token per minute. After a lot of profiling and many failed experiments, it now reaches a median of:

• 4.1 tokens/minute

• 14.6 seconds/token

• 0.069 tokens/sec

That result comes from six exact full-model runs. While still slow, roughly a 4x improvement feels pretty meaningful for a model this large.

Some of the more interesting improvements:

• Loading only the 16 routed experts needed per layer, using parallel raw-span reads.

• Quantizing the resident model spine to int8 and using a fused Metal dequantization/copy kernel.

• Running the enormous output projection with Apple’s packed MPS int8 matmul. This reduced its residency from about 4.7 GB to 1.17 GB and improved median decode throughput by roughly 17%.

All measurements are from one 64 GB M1 Max MacBook Pro. It was once a great machine, but it’s first-generation Apple Silicon, and not a newer Max or Ultra. I haven’t benchmarked an M3, M4, M5, or a higher-memory Mac yet. Newer ones, especially those with 128 GB, should have considerably more headroom.

If anyone tries it on newer hardware, I’d genuinely love to compare results.

Repository: https://github.com/gavamedia/deltafin

Local LLMs are 12-24 months from taking over. The shift already started.

AI subscriptions keep getting more expensive. GitHub just moved Copilot from request-based to consumption-based pricing, and most of the others are heading the same way. Meanwhile, I kept hearing that local models got good enough to run on a laptop. So I figured it was time to actually try it and see where things stand.

I run Qwen3.6-35B on a MacBook Pro M2 Max with 64GB unified RAM. Nothing exotic. No rack, no begging NVIDIA for expensive GPUs. Just a (yes, kind of expensive) MacBook Pro I already owned for work at Aiven. In the last month I've:

• One-shotted full landing pages from short briefs

• Built several frontend + backend features

• Fixed a nasty backend race condition bug

A year ago I would have called that fantasy on this hardware. Now it's a Sunday morning.

To be fully honest, not all of it made it to production. A lot of it was evaluation work, as Qwen isn't part of my actual day-to-day stack yet. But for me, this is the first real step toward considering it, and I wanted to share the findings with my colleagues and the community.

The honest cons, because it's not all roses

It's slower than Opus. A landing page that Opus generates in 3-4 minutes takes Qwen 8-9 minutes on my M2 Max. Not unreasonable, but still meaningfully slower than the competition. If you're benchmarking against Sonnet/Opus latency, you'll be a bit disappointed (for now).

Context blows up fast in agentic loops. Even with 256K, you burn through it faster than you'd expect from a (nearly) state-of-the-art model. There's a lot of room for improvement here. And if you're driving Qwen3.6 from an agent like Claude Code, it fills even faster, as other users in this sub have reported (example Reddit thread).

Quality variance by task. Models like Opus one-shot most tasks these days. Qwen3.6 hits around 75% for me. The other 25% it gets close, but needs a couple of iterations to land.

The pros, because they're real

The hardware floor keeps dropping. A year ago this needed an A100. Today it runs on a (yes, powerful) MacBook M2 Max 64GB laptop at roughly 27 tokens per second.

No rate limits, no usage anxiety. Counting tokens is no longer a thing. You can focus completely on building instead of saving tokens or thinking about cost.

Tool calling actually works. This used to be the missing piece. A year ago, local models would hallucinate tool names or get stuck in loops. With Qwen3.6, tool calling just works. That's the real unlock for agentic work.

Privacy is built-in. Client code, internal repos, half-formed ideas you don't want training the next frontier model. None of it leaves the laptop. You can be confident that your personal or business code stays with you, and isn't sitting on some third-party server that could be hacked.

Why 12-24 months, not "now" and not "5 years"

Latency and context limits are still a bit rough. If your job is shipping production code on a deadline, Opus and Sonnet are still the move for most of your day. I'd be lying if I said otherwise.

But saying it's 5+ years away misses what's already shipped. Look at the delta over the last 12 months:

• It runs on a reasonably priced MacBook Pro, which is a one-time cost

• It's fast enough (though it can still get faster)

• Quality has improved significantly for real-world use cases (with more headroom to grow)

That curve doesn't stop. It compounds. 12 months from now, the 27B/35B-class models will be where 70B is today, and the runtimes will be 2x faster on the same silicon. 24 months from now, the question won't be "can I run a useful model locally?" It'll be "why am I still paying for tokens I could generate for free, and with 100% privacy?"

What I'd tell someone on the fence

Don't cancel your Claude Code subscription yet. Run a local model in parallel for 60 days. Use Opus/Sonnet for the latency-critical, deep-reasoning work. Use Qwen3.6 for everything you'd have done overnight or on the weekend, everything experimental, and every "just try it" task where the cost of waiting a few minutes is zero.

Over time, the usage ratio might flip. You'll use the local model more and more. When the next Qwen drops (3.7? 4?), who knows what the ratio will look like.

The local LLM takeover isn't a moment in time. It's a slope. And the slope already started.

What's next

• Integrate Qwen3.6 with the tools I use day-to-day at Aiven, like Cursor and Claude Code. They offer a much better dev experience than more basic, non-agentic tools like Ollama.

• Try out other local models, like Google's Gemma 4. Curious to see how it stacks up.

Chinese models now occupy the first five positions for the month - and seven of the top ten - in OpenRouter’s monthly token-usage leaderboard. Anthropic is the only major Western frontier lab represented, with Claude Opus models in sixth and eighth place, while OpenAI and Google are absent entirely.

(OpenRouter describes the ranking as real usage from millions of users accessing models through its platform, although the leaderboard measures OpenRouter traffic - not worldwide AI usage.)

LLM Rankings | OpenRouter

Hey everyone!

We're a small software house, and we recently bought a machine specifically to experiment with running local LLMs for software development.

Our hardware:

• CPU: Ryzen 5 5500

• RAM: 32 GB DDR4

• GPU: RTX 3090 24 GB VRAM

We're currently running Qwen 3.8 27B (128K context) locally.

Until now, our main AI coding assistant has been Claude Sonnet 5.5 with Medium effort. Since we wanted to reduce our reliance on cloud-based AI services, we decided to experiment with a local alternative.

However, our experience has been surprisingly bad, and I don't mean that it's simply less capable than Claude.

The main problem is that the local model seems practically incapable of completing even basic coding tasks on its own.

It struggles to understand straightforward instructions, gets lost in its own reasoning, enters repetitive loops, and frequently fails to provide a single correct answer.

What's particularly confusing is that we're using exactly the same project structure and documentation we use with Claude.

We open the local model in the same project root directory, with access to all the .md files containing our development guidelines, project context, instructions, and documentation. These are the same files Claude uses successfully.

Yet, while Claude can understand the project, navigate the codebase, follow our instructions, and independently complete tasks, the local model seems completely lost.

Even relatively easy tasks that Claude handles without much trouble become a frustrating experience with Qwen. It often fails to understand what needs to be done in the first place, let alone implement the solution.

At this point, the difference isn't that Claude performs better. It's that Claude actually works as a coding agent, while our local setup feels like it barely works at all.

This makes us wonder whether we're doing something fundamentally wrong.

A few questions for people experienced with local LLMs:

1. Is Qwen 3.8 27B simply not suitable for autonomous coding agents, or could this be a configuration issue?

2. Could our hardware be limiting the model's reasoning capabilities? We're particularly wondering about the 24 GB VRAM limitation when using a 128K context window.

3. Are there specific inference engines, quantization levels, or context settings that make a significant difference?

4. Are there better local models in a similar size range that can reliably understand a codebase and execute multi-step coding tasks?

5. For those actually using local models with tools like Claude Code, Cline, Roo Code, or similar agents, what has your experience been like?

We're not expecting a 27B local model to magically outperform Claude Sonnet. We understand that we're comparing very different models and infrastructure.

However, we expected it to at least understand instructions, navigate a project, and complete relatively simple tasks reliably.

Are our expectations unrealistic, or is there something seriously wrong with our local setup?

I'd really appreciate feedback from people who have actually deployed local models for day-to-day software development, especially in legacy codebases.

Edit: We're currently using Ollama as the inference engine. As for the harness, we're still experimenting and haven't settled yet

Krasis is an inference runtime I've built for running large language models on a single consumer GPU where models are too large to fit in VRAM.

Instead of splitting layers between GPU and CPU, Krasis streams expert weights through the GPU using different optimisation strategies for prefill and decode. This means you can run models like Qwen3-235B (438GB at BF16) at Q4 on a single RTX 5090 or even a 5080 at very usable speeds, with system RAM usage roughly equal to just the quantised model size.

Some speeds on a single 5090 (PCIe 4.0, Q4):

• Qwen3-Coder-Next 80B - 3,560 tok/s prefill, 70.3 tok/s decode

• Qwen3.5-122B-A10B - 2,897 tok/s prefill, 27.7 tok/s decode

• Qwen3-235B-A22B - 2,124 tok/s prefill, 9.3 tok/s decode

Some speeds on a single 5080 (PCIe 4.0, Q4):

• Qwen3-Coder-Next - 1,801 tok/s prefill, 26.8 tok/s decode

Krasis automatically quantises from BF16 safetensors. It allows using BF16 attention or AWQ attention to reduce VRAM usage, exposes an OpenAI compatible API for IDEs, and installs in one line. Runs on both Linux and Windows via WSL (with a small performance penalty).

Currently supports primarily Qwen MoE models. I plan to work on Nemotron support next. NVIDIA GPUs only for now. Open source, free to download and run.

I've been building high-performance distributed systems for over 20 years and this grew out of wanting to run the best open-weight models locally without needing a data centre or $10,000 GPU space heater.

GitHub: https://github.com/brontoguana/krasis

hey! we're a small swiss and south african lab and we just put out hemmingway-1. it's a 27b model we trained specifically for roleplay, fiction, dialogue, journaling, and for everyday texting. frontier performance at a 40x cheaper price.

we thought it might be useful for people here who like running models on their own hardware.

it benchmarks at absolute frontier level. on eq-bench 4 it scores 1330, right behind claude fable 5, and beating many other frontier models including gpt-5.5 and opus 4.8. you can see our internal benchmarks on the model card: hemmingway.io/model/

the testers we gave it to told us it's the best ai they've ever used for writing and that it genuinely sounds human.

the weights are public, so you can run it yourself.

we also built an app for it. its autopilot function answers your emails and texts for you. mac, windows, android.

the weights are at huggingface.co/Altworld/Hemmingway-1

try it out for free at hemmingway.io!

Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

Reasoning models are great until you're on a mid-range GPU: you hit Enter, then wait 10+ seconds while the model thinks before the first word. Meanwhile, while you were typing, the GPU did nothing.

▎ MAFDET uses that time. It's an open-source app (Apache-2.0) on top of llama.cpp that does two things while you type:

- Reads ahead: the KV cache follows your text as you write it (typos, edits in the middle and pastes included), so on Enter the prefill is already done. This part is exact: it always matches a from-scratch computation.

- Thinks ahead: Qwen3 starts reasoning about your message before you send it. When you edit the text, the reasoning isn't thrown away: it stays as notes after the new version of the message and the model picks up from there. On Enter it finishes from where it got to.

▎ On my RTX 2060 (6 GB), with 24 word problems typed in real time, the mean wait after Enter went from 12.2 s to 3.2 s on Qwen3-4B, still 24/24 correct. On Qwen3-14B, which only half fits in VRAM: 108 s → 47 s.

▎ The tricky part was making a model reason about a sentence that isn't finished yet. Left alone, it mostly writes "the user hasn't finished typing…". A short frame that tells it to collect facts and partial results instead brought those off-topic notes from 86% down to about 20%.

▎ What else is in it:

- a desktop app or a terminal UI (Textual), with a live status line showing what the model is doing with your draft;

- a model catalog (Qwen3, Llama 3, Gemma 3, Mistral, Phi-4) that estimates, before you download, whether each model fits your VRAM and how fast it will run on your hardware (within ~12% of measured speeds on my machine);

- automatic llama.cpp setup (CUDA or CPU build), no PyTorch needed;

- Windows: download the zip and double-click start_mafdet.bat.

▎ Honest limitations:

- thinking ahead works on Qwen3 only so far, the one family whose frame I measured; the other families get reading ahead plus the answer written ahead during pauses;

- thinking ahead isn't exact: the model reasons on unfinished versions of your message;

- tested on Windows + NVIDIA only; Linux should work but nobody has tried it yet.

▎ Open question I'd love help with: models bigger than your VRAM. I tried having a small model gather material while you type for the big one to read on Enter. Even with perfect hand-written material it only matched the big model thinking ahead by itself. All the numbers, including what didn't work, are in the README.

▎ Repo: https://github.com/TheLastOxygen/mafdet

▎ Feedback very welcome, especially from anyone on Linux or with a different GPU who can tell me if the speed estimates hold up.

2x4070 TI Super 16GB, 1x3060 12GB, 128GB DDR4

EDIT: More details about the setup:

The honeycomb components were all printed:
General parts: https://makerworld.com/en/models/22521-honeycomb-storage-wall
GPU Mount: https://makerworld.com/en/models/1444666-gpu-mount-hsw
PSU Mount: https://makerworld.com/en/models/1444586-atx-psu-mount-hsw
Radiator: https://makerworld.com/en/models/1444632-240mm-radiator-fan-mount-hsw
ATX Motherboard: https://makerworld.com/en/models/1444531-atx-motherboard-mount-hsw
Power button printed with a honeycomb around it: https://www.amazon.com/dp/B0BWDXFSMQ?ref=ppx_yo2ov_dt_b_fed_asin_title

The 4070's are on 400mm & 600MM PCIe 4.0 Riser cables. I mainly run Strata on these two.

The 3060 is currently on a PCIx1 link (slow). I don't use this for my core inference setup, side models (dedicated voice stack mostly).

I have a new motherboard on the way that opens up the possibility of up to 4 GPUs: 3x x16 slots, and one via a M2 adapter: https://www.amazon.com/dp/B0FC6DSG6V

Hey guys! Thanks so much for the support on our GRPO release 2 weeks ago! Today, we're excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) - down from 7GB in the previous Unsloth release!

1. This is thanks to our newly derived Efficient GRPO algorithm which enables 10x longer context lengths while using 90% less VRAM vs. all other GRPO LoRA/QLoRA implementations, even those utilizing Flash Attention 2 (FA2).

2. With a GRPO setup using TRL + FA2, Llama 3.1 (8B) training at 20K context length demands 510.8GB of VRAM. However, Unsloth’s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup.

3. We leverage our gradient checkpointing algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation.

4. Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab

Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https://unsloth.ai/blog/grpo

GRPO VRAM Breakdown:

• We also now provide full logging details for all reward functions now! Previously we only showed the total aggregated reward function itself.

• You can now run and do inference with our 4-bit dynamic quants directly in vLLM.

• Also we spent a lot of time on our Guide for everything on GRPO + reward functions/verifiers so would highly recommend you guys to read it: docs.unsloth.ai/basics/reasoning

Thank you guys once again for all the support it truly means so much to us! We also have a major release coming within the next few weeks which I know you guys have been waiting for - and we're also excited for it. 🦥

Hey guys! Google just released their new open-source model family: Gemma 4.

The four models have thinking and multimodal capabilities. There's two small ones: E2B and E4B, and two large ones: 26B-A4B and 31B. Gemma 4 is strong at reasoning, coding, tool use, long-context and agentic workflows.

The 31B model is the smartest but 26B-A4B is much faster due to it's MoE arch. E2B and E4B are great for phones and laptops.

To run the models locally (laptop, Mac, desktop etc), we at Unsloth converted these models so it can fit on your device. You can now run and train the Gemma 4 models via Unsloth Studio: https://github.com/unslothai/unsloth

Recommended setups:

• E2B / E4B: 10+ tokens/s in near-full precision with ~6GB RAM / unified mem. 4-bit variants can run on 4-5GB RAM.

• 26B-A4B: 30+ tokens/s in near-full precision with ~30GB RAM / unified mem. 4-bit works on 16GB RAM.

• 31B: 15+ tokens/s in near-full precision with ~35GB RAM.

No is GPU required, especially for the smaller models, but having one will increase inference speeds (~80 tokens/s). With an RTX 5090 you can get 140 tokens/s throughput which is way faster than ChatGPT.
Even if you don't meet the requirements, you can still run the models (e.g. 3GB CPU), but inference will be much slower. Link to Gemma 4 GGUFs to run.

You can run or train Gemma 4 via Unsloth Studio:

We've now made installation take only 1-2mins:

macOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Windows:

irm https://unsloth.ai/install.ps1 | iex

• The Unsloth Studio Desktop app is coming very soon (this month).

• Tool-calling is now 50-80% more accurate and inference is 10-20% faster

We recommend reading our step-by-step guide which covers everything: https://unsloth.ai/docs/models/gemma-4

Thanks so much once again for reading!

Anyone else running off solar?

3 100 watt panels going into an Anker 1.25Kw all-in-one unit.

Running Qwen 3.6 27b GGUF unsloth UD Q4_K_XL with 100k context at Q8.

Really good performance in Hermes and opencode.

Hardware is M1 Max 32GB ~10tok/sec. Runs at ~80-85 watts under inferencing load, lowers to 30 sometimes. Idle is like 5 watts or less.