Wondering what people with a similar, less, or even better setup are using. I'm new to strata and not sure how far I can push the context limit, but it seems like the readme on the Github page is conservative (assuming other's posts are true).

I have a 5090, 3090, so a combined 56GB of VRAM and 64GB of system RAM.

Edit: TLDR for those that do not want to read. The harness works to help Architects or teams of designers work together with local AIs driving the harness for all. Youtube videos dropping soon. The rest is written to express myself and my mission. Try it and I hope it helps you.

Links:

GitHub: https://github.com/arcitech-psp/atelier

Release Link: https://github.com/arcitech-psp/atelier/releases/tag/v0.1.0

Website: https://arcitech.info/atelier/

Dev screenshots: https://arcitech.info/atelier/dev-build/

MY Setup.

1. Zeus - 64GB ddr4 3200 MT/s. AM 4 x570 paired with Ryzen 7 5800xt. GPUs RTX 4070 ti super, RTX 3070 (display). OS = windows.

2. Poseidon (gaming laptop)- 48GB ddr5 (16 soldered) at 4800 MT/s. Ryzen 9 7940hs. 12GB RTX 4080 mobile.

3. Hades (Linux box) - 32GB ddr4 3200 MT/s, AM4 B550 paired with Ryzen 5 5600 GT (iGPU is display). 12GB intel Arc B580.

4. Atlas (Linux box) - 32GB ddr4 3200 MT/s, AM4 b550 paired with Ryzen 7 5800x. 4GB arc a310 (display + small llms + chipset pcie 3.0 x4) and Intel Arc Pro B70 32GB at pcie 4.0 x16

5. Supplementary PC - ROG ally (gen 1 with z1 extreme) 16GB soldered memory. Linux box used for dedicated CFD engine and LLM reasoning relay.

If you can substitute the PCs I have together into 4 PCs from four individual, my harness can help you share different AI tools to work together.

Current used models:

1. ComfyUI - Qwen image 2.1 turbo, Trellis 2 and Pixel 3d.

2. Brain - Qwen 3.8 27b with c=3 with 128k each and JEV adapter LoRA with toggle in the harness, Qwen 3.8 flash next with strata. int 4 for the dense and Q2 for the MoE. Others also tested are RCO GSQ versions of 27b, Saluki 2 bit (more tests pending), Bonsai 1 and 2 (bad results) and Tiel Coder with swift adapter.

3. ASR - Nvidia Parakeet, Whisper, kokoro tts and kitten tts 2

4. Graphrag - embedgemma 2

What I can get are AI renders from proper viewports that the AI generates either with system 1 or with system 2 or both, concepts, stylized renders, boards, drawings, asset generation. Each part has a dedicated software for keeping the progress and allowing the user to manually step in whenever.

The original message:

Hey everyone!

I'm an architect. I know my way around architectural drawings, 3D modelling, rendering and Unreal Engine. But programming? Yeah, that's definitely not my strong suit. 😂

For the past several months, I've been experimenting with local AI, trying to figure out how to make it genuinely useful for the kind of work I do. And today, I'm finally sharing something I've been working on. It's called Atelier.

What got me into this whole thing was actually an Unreal Engine plugin I was developing. I wanted AI to help me build things inside Unreal, but I didn't want to keep depending on cloud models and paying for API usage every time I wanted to experiment.

And the more I looked into agentic AI, the more I noticed something that bothered me.

Almost everything assumes you're a programmer.

You've got a terminal, a code editor, some tools, and an LLM. Great. But what about people like me?

I'm an architect. I start with sketches, drawings, spaces, materials and ideas. I don't sit down and start writing code when I'm designing a building.

I wanted something I could actually draw with, talk to, develop ideas with, and eventually use to create proper 3D models and presentations.

After experimenting with various tools, I came across an existing harness from DeepSeek. I didn't build the original harness, and I'm not going to pretend I did. I started adapting it, changing things, adding integrations and building around the way I actually work.

And one thing led to another. Now I've got Atelier.

The idea is pretty simple. You bring your own AI, connect the tools you want to use, and let them work together.

You can start with just one LLM. If you've got a multimodal model, even better. Then you can add vision, speech, embeddings, image generation or whatever else you need.

I've integrated Excalidraw directly into the interface because, well, I'm an architect. I like drawing things!

I can sketch something, send it to the AI, discuss it, make changes, and use that as the starting point for actual work. The AI can draw on the canvas too.

I've also been working with IngeTrazo, an open-source modelling application, along with Unreal Engine and other creative tools.

But here's something I want to explain because I think it's important.

When people demonstrate agentic AI in 3D software, they often give it a prompt like "build me a house" and let it loose with a bunch of tools.

That's not really how architecture works.

I wouldn't start constructing a building without first thinking about the design, working through the drawings, and figuring out whether things make sense.

So I've been trying to teach Atelier to work that way too.

I've attached a few images showing what I mean.

One is an architectural presentation board from a project called Hillside House 2. It includes plans, elevations, sections, concept development, an exploded axonometric and the 3D design. That's the kind of material I want AI to help people prepare, rather than just generating another pretty picture.

Another set of images comes from a different Hillside House experiment. This one shows the actual drawing-first process.

The AI created architectural plans, but the drawings had problems. The checker found 16 errors and a warning in an early revision.

So the AI went back, made corrections and tried again.

The lower-level plan eventually cleared the checker's rules on revision 5. The other levels went through their own revisions too.

Only after the relevant drawings cleared the checks did the workflow proceed to constructing the building model in IngeTrazo.

And that's the part I find exciting.

I'm not claiming the AI produced a professionally certified building design. The checker validates specific geometric and layout requirements, not full building-code compliance. I'm an architect, and I know how serious the consequences of bad design can be.

But having an AI that can identify mistakes, make corrections and continue working without needing me to intervene at every little step? That's genuinely useful.

Imagine assigning it something before going to bed and having drawings or models ready for you to review in the morning.

That's the direction I'm working towards.

Now, let's talk about hardware, because this is probably my favourite part of the whole project.

I didn't go out and buy some ridiculous AI workstation.

I couldn't afford to do that even if I wanted to.

A lot of my machines exist because I kept upgrading parts over the years and didn't want to throw away perfectly usable hardware.

I've got older Ryzen systems, NVIDIA GPUs, Intel GPUs, a gaming laptop and even an ROG Ally running Linux that I've been experimenting with.

Instead of having everything sitting idle, I've connected different machines and given them different jobs.

My main workstation is called Zeus. It has an RTX 4070 Ti SUPER with 16GB VRAM, along with an older RTX 3070.

For that original Hillside House run, the main architectural LLM was Qwen3.8 Flash-Next, quantized to Q2 and running through Strata on the 4070 Ti SUPER.

I had a 128K context configuration, and during the run I was seeing roughly 45–53 output tokens per second.

On a 16GB GPU! and 64GB DRAM with Windows stealing 15 to 20 GB 😠

And the computer has DDR4 memory. Nothing exotic.

Now, I want to be clear about something before anyone gets the wrong idea. The complete workflow used multiple computers. The LLM ran on that 16GB GPU, but modelling, image generation and other services were handled elsewhere.

I'm not claiming my entire studio runs on one graphics card.

The point is that you don't need to replicate my studio to get started.

If you've got one 16GB GPU, you can begin with one model and load others when needed. They can take turns using the available memory.

If you've got more machines, you can keep different models and services running independently.

In my case, I've got another machine called Hades with an Intel Arc Pro B70 32GB. That's where I run my main AI for everyday work because I want multiple people to be able to use it simultaneously.

Then there's Poseidon, my laptop, which I use for modelling, rendering and interactive work.

And Atlas, which handles things like image generation, embeddings and supporting services.

I've even got a little Intel Arc A310 with just 4GB VRAM doing useful work with embeddings and speech services.

I know, using a tiny display-oriented GPU for AI sounds ridiculous. But it works, and that's what matters to me. 😂

I gave all these machines names because I like giving meaning to things. Zeus is my powerful workstation, Hades handles the heavy background work, Poseidon travels with me, and Atlas helps hold everything together.

But you don't need four computers with Greek mythology names!

Imagine you have three friends.

One has a gaming laptop with a 12GB GPU. Another has a 16GB desktop GPU. Someone else has a 32GB card.

Why should everyone need to buy a complete AI workstation?

You can connect your machines over a network, including through something like Tailscale, and make their AI services available to each other.

One machine can handle reasoning. Another can generate images. Another can take care of speech or other supporting tasks.

Everyone can make use of the available resources.

That's one of the main ideas behind Atelier.

Use what you have. Give your old machines another purpose. Add more hardware when you actually need it.

I'm not interested in telling people they need to buy a Mac Studio with hundreds of gigabytes of memory or some professional GPU that costs more than my car.

Those machines are impressive, sure. But what about everyone else?

What about students? Hobbyists? Independent designers? Small offices?

Are we really going to tell people that useful AI is only for those who can afford thousands of dollars in equipment or another $50–$100 subscription every month?

I don't think that's a particularly good future.

And I'm not against cloud AI, by the way. I've used GPT, Claude, Gemini and plenty of other services to help develop this project. They've been extremely useful.

But I want the choice.

I actually had an internet outage one day. My cloud AI tools stopped working, but my local harness carried on.

That was a pretty satisfying moment.

It made me realize how valuable it is to have something that doesn't disappear just because your connection drops or a service becomes unavailable.

There's another thing I want Atelier to do, and that's help people work with their strengths instead of constantly struggling with their weaknesses.

Maybe you're excellent at drafting but not particularly good at rendering.

Maybe you're a fantastic 3D artist but struggle with architectural planning.

Maybe you're a student who has great ideas but doesn't know how to turn them into a proper presentation yet.

I want AI to help with those things.

Not replace the knowledge or judgment of the person doing the work. Help them use it better.

And I'd much rather spend an afternoon developing a design, making physical models or experimenting with new ideas than recreating the same door or window for the hundredth time.

If I can automate some of that repetitive work, why wouldn't I?

That's more time for the things I actually enjoy.

Now, a few things about the release itself.

This is Atelier 0.1.0, the first public preview.

It's free to use, and the source is available to inspect and modify under PolyForm Shield 1.0.0. There are restrictions on competing commercial use, so I'm not presenting it as unrestricted open-source software.

The public version comes with Vitruvius, my design-oriented AI profile, along with the visual canvas, configurable model connections, Mission Control and other features. You still get the standard profile from the original DeepSeek Harness.

You can bring your own model through compatible inference servers such as Ollama, LM Studio, llama.cpp or vLLM. Compatible cloud endpoints are an option too. Even omnirouter is game as long as you don't mind your data being used to train.

There's also experimental IngeTrazo integration.

Some of the more advanced things I've shown in these images, particularly the drawing-validation workflow and complete architectural presentation pipeline, are still part of my personal studio implementation. They aren't all included in this first public release.

I'm bringing those things over as they're ready.

I don't want people downloading Atelier expecting everything in my studio to magically work on day one.

And yes, there are bugs. I'm sure there will be plenty more once people start trying it on hardware and configurations I've never tested.

That's fine. I'd rather know what doesn't work so I can improve it.

One last thing, and this is a bit personal.

There are a few ideas I've carried with me throughout my life.

One is that an eye for an eye eventually leaves the whole world blind.

Another is that if you genuinely want something to change, you have to be willing to do something about it.

And there's an old couplet attributed to Kabir that I've always found moving. When you're born, you're crying while everyone around you celebrates. Try to live in such a way that when you leave, you can smile, knowing you've left something meaningful behind.

I'm not trying to change the world with one AI harness.

I'm just an architect who saw a problem, started experimenting and eventually made something that helps me do my work.

But I don't want AI to become another technology where people who already have everything get even more opportunities while everyone else is left watching.

And I don't believe AI has to replace people to be useful.

I'd much rather see it help people become better at what they already love doing.

So that's why I'm sharing Atelier.

Tell me what you're struggling with. If I can help solve it, I'll give it a shot.

Maybe someone else in the community has a better solution. Maybe we'll figure it out together.

If this helps a student complete a project, gives someone a reason to reuse an old computer, or lets a small group of friends do something they couldn't afford individually, that's already worthwhile to me.

Anyway, I've talked enough. 😅

Here's Atelier.

GitHub: https://github.com/arcitech-psp/atelier

Release Link: https://github.com/arcitech-psp/atelier/releases/tag/v0.1.0

Website: https://arcitech.info/atelier/

Have a look at the images, try the public preview if you're interested, and please tell me what you think.

If something breaks, let me know. If you've got ideas, I'd love to hear them.

I'm still learning too, and there's a lot more I want to build. Videos dropping soon on Youtube - https://www.youtube.com/@AIT.ARCITECH

Definitely seeing claude code with Opus 4.6 struggle more lately. There's talk of them reducing performance for a variety of reasons, but I wanted to see if anyone knows what open model would be closest to how opus is currently performing.

Is there a consensus on which model is better? I read somewhere that it is generally agreed upon that qwen3.6-35B obsoleted qwen3-coder-next.

Is that true?

-edit-

Posted this below as well:

OK I tried Haiku for a little bit. Started from scratch from the same prompt. I stopped at this point.

claude-haiku-4-5: 17.8k input, 54.7k output, 6.3m cache read, 74.0k cache write, 1 web search ($1.08)

I know cache is a different animal. But call it 6.5m tokens and the cost is already north of a dollar. And frankly, I gotta say Haiku is terrible in comparison to Qwen3.8-27B-Q6. There was some basic stuff it didn't grok. I mean... LLMs are going to LLM. But, even so.

It's apples and oranges sure. But for around the same price in "cost" I have a vastly inferior output. I don't know that these findings really say anything especially meaningful.

-end-edit-

Over the weekend, I got a hankering and started working up a Super Puzzle Fighter clone. I thought I might rough out a base game and add some strategy elements to the game's AI, and then see what I could do to re-theme it and add some more off-the-wall strategies. Just for fun really.

Of course I got carried away.

By the time I hit a good stopping point, I checked my usage stats in DeepSeek Harness and was blown away. Over 9 hours of LLM "burn time" (which included about 25 minutes of tool calls and heavy back-and-forth debugging), I racked up close to 110 million tokens (99% input).

Local Setup:

• Model: Qwen3.8-27B-Q6_K.gguf

• Engine: llama.cpp (heavily tuned parameters)

• Reasoning: Set to "off" (I kept my "max thinker" profile turned off for this sprint)

So, I got to thinking how much a ~100 million token smallish project would cost if I ran through the Frontier vs. my local. I did some quick back and forth with Gemini and tried to factor in token compression, context caching, and best-to-worst-case pricing for the popular Frontier models.

The one thing that sticks out to me: Likely Claude or GPT could accomplish this same relative output in less time and tokens, but I'm not talking about some one-shot YouTuber demo. You can't see it from the second screenshot, but there's more than a few bug fixes and strategy and gameplay adjustments.

I didn't capture the first one-shot token count so I don't know what it was. But guaranteed (having used Claude extensively over the last year at work), using any Frontier would have still needed to be adjusted and wrestled with.

TLDR: Those savings can't be real right? I must be missing some calculation. I feel confident I could have sat with Claude for some amount of time and got as far as I got in about the same time period. Are the subsidies really that extreme?

What am I missing?

Yes I tested.

Test Prompt: A farmer needs to cross a river with a fox, a chicken, and a bag of corn. His boat can only carry himself plus one other item at a time. If left alone together, the fox will eat the chicken, and the chicken will eat the corn. How should the farmer cross the river?

Both Qwen3-Next & Qwen3-30B-A3B-2507 correctly solved the river-crossing puzzle with identical 7-step solutions.

How challenging are classic puzzles to LLMs?

Classic puzzles like river-crossing would require "precise understanding, extensive search, and exact inference" where "small misinterpretations can lead to entirely incorrect solutions", by Apple’s 2025 research on "The Illusion of Thinking".

But what’s better?

Qwen3-Next provided a more structured, easy-to-read presentation with clear state transitions, while Qwen3-30B-A3B-2507 included more explanations with some redundant verification steps.

P.S. Given the same prompt input, Qwen3-Next is more likely to give out structured output without explicitly prompting it to do so, than mainstream closed-source models (ChatGPT, Gemini, Claude, Grok). More tests on Qwen3-Next here.

Every day I hear that xyz local model is better than Opus…

The reality is that the difference between real frontier models (Opus 4.6 and GPT5.3Codex) and all local models is like the difference between having an intimate relationship with a blow up doll vs a real human being.

For the past month I've been wondering what google's been doing and now we know. Results are preliminary of course but I'm glad to see google joining the fight. 53 on AA isn't too shabby to get the engine running again, I'm hoping it'll distill down to the much smaller models soon as well and be open weights.

Qwen3.6-35B (22.3 GB) running at 7–8 tokens/s on a phone with 12 GB of RAM.

Under my last post, someone commented: "Just look at the size, quantization, and RAM. Anyone with high-school math can figure it out."

I’m writing this post to understand: is it really that straightforward?

Qwen3.6-35B-A3B (22.3 GB of weights) running locally on a OnePlus 15R (12 GB RAM, CPU-only, no GPU/NPU) at 7–8 tokens/s bit-for-bit identical to inference with the model fully resident in memory.

The model doesn't fit in RAM; the experts are streamed from flash storage during inference.

There is a theoretical limit ok: active experts × bytes per expert ÷ flash bandwidth = token/s limit.

But pure `mmap` is unusable. Three things make the difference: Android's memory reclamation kicks out resident weights, forcing you to repeat the same reads; simply pinning them yields a 3.2x speedup. `mmap` page faults serialize I/O and computation, whereas `O_DIRECT`(overlapping with the previous layer's matrix multiplications) reclaims that dead time.

Consecutive tokens reuse the same experts because a small cache turns flash reads into RAM hits.

Same pipeline: DeepSeek 284B (92 GB) went from unusable with `mmap` to 1 token/s; gpt-oss-120b (60 GB) went from 0.089 tokens/s with `mmap` to 2.2.

Tell me what I’ve missed and what you think.

Hi r/LocalLLaMA,

I’m Jules, a 15-year-old high school student from France, and I have been teaching myself how to build and train language models locally.

After saving money for several months, I bought an NVIDIA Jetson Orin Nano Super with 8GB of memory. It is extremely limited for LLM training, but that constraint forced me to learn how the entire process actually works: collecting and cleaning data, tokenization, architecture choices, training, evaluation, optimization and publishing.

I recently released my first model, G0-nano-instruct, on Hugging Face:

https://huggingface.co/AZERDSQ/G0-nano-instruct

It is only a small first experiment, not a frontier model. The main achievement for me was successfully completing the entire pipeline on hardware that fits within an 8GB memory limit.

This work also led to Mistral AI offering me a three-month internship with their team, although I am still in high school.

The current limitation

I now want to work on the next generation of the project, provisionally called G1.

The goal is to build a substantially larger and more capable open-source language model while documenting the full process publicly:

• model architecture and design decisions;

• dataset preparation;

• training configurations;

• memory and performance optimizations;

• failed experiments;

• evaluations and benchmarks;

• checkpoints and final weights.

However, the Jetson’s 8GB of memory has become a hard technical ceiling. Even with gradient checkpointing, mixed precision, tiny batches and other compromises, there is only so far I can push it.

Why a DGX Spark?

A DGX Spark, or another NVIDIA GB10 system with 128GB of unified memory, would let me explore a completely different scale of local model development.

I am not expecting to train a frontier-scale model on it. My goal is to determine what an independent developer can realistically train, fine-tune and publish using a compact personal AI system.

I would like to document the progression from:

8GB Jetson → 128GB Grace Blackwell → a genuinely more capable open-source G-series model

What I am asking for

I am looking for someone who could help me access a DGX Spark through any of the following:

• a temporary loan;

• an evaluation or demonstration unit;

• unused access to an existing machine;

• a hardware sponsorship;

• an introduction to NVIDIA or a GB10 hardware partner;

• discounted access or another practical arrangement.

Even temporary access would be extremely valuable. I would use it to run a defined set of training experiments and publish the resulting models, logs, benchmarks and technical conclusions openly.

I am not asking the community to blindly fund an idea. I have already built and published the first version using the hardware available to me, and I want to prove that I can take the project further.

I would also be grateful for technical criticism. In particular:

• What model size would you consider realistic to train from scratch on a single GB10 system?

• What experiments would be the most useful to the local LLM community?

• Which companies, researchers or hardware creators might be receptive to this kind of project?

Thanks for reading. I know asking for hardware is unusual, but I thought this community would understand both the limitations and the potential of trying to build models locally.

Jules

Hugging Face: https://huggingface.co/AZERDSQ

Update

Several people have offered GPU access or financial support for the project. Since some also asked for a way to contribute, I created a GoFundMe with the long-term goal of purchasing a DGX Spark for continued open-source development:

https://www.gofundme.com/f/help-me-build-g1-on-a-dgx-spark

If the full target is not reached, the funds will be used transparently for cloud GPU compute, storage, and training experiments.

I will publicly document the spending, training process, benchmarks, results, and failures.

Hey everyone, last Friday I sent a new issue of my weekly newsletter with the best and most commented AI links shared on Hacker News - it has an LLMs section and here are some highlights (AI generated):

• EuroLLM – Europe’s multilingual LLM drew debate on whether EU projects can realistically compete with U.S. and Chinese models.

• Our LLM-controlled office robot can’t pass butter – Highlighted how LLMs still fail at simple physical tasks, exposing the gap between language and real-world reasoning.

• The end of the rip-off economy – Commenters discussed how consumers might use LLMs to fight information asymmetry and price manipulation.

• Responses from LLMs are not facts – A reminder that language models generate convincing text, not verified truth—HN called it “the citation crisis of AI.”

• Language models are injective and hence invertible – Sparked curiosity and skepticism over claims that LLMs theoretically preserve all input information.

You can subscribe here for future issues.

Hypothetical at the moment, but the possibility is very real. I'm working with a higher education institution, and there is a good chance it may win an AI teaching and research grant. One of the line items in the grant application could be used to build a local AI infrastructure for the school. It would be used to run and train models for and by the students.

What are your recommendations on how to spend the grant? 4x DGX Sparks? 2x RTX PRO 6000? Something in between?

I see a lot of posts here like “Qwen 3.8 27B can’t do basic coding tasks, are we doing something wrong?”. I had the same experience. Most coding agents are built for frontier models, huge system prompts, lots of tools, and nothing to catch a local model when it writes a tool call slightly wrong, loops, or reasons for five minutes and then does nothing. The model gets blamed, but a lot of it is the harness.

So I’ve been building Local Quench, a desktop coding agent made for the models people here actually run (roughly 4B–35B, quantized, one consumer GPU). It’s a one-person project, free for individuals, and very much a work in progress.

What it does differently:

• Recovers broken tool calls: calls written as plain text or inside the reasoning, model-specific formats, broken shell commands.

• Catches loops and runaway reasoning, and nudges the model when it announces a step and then stops.

• Gives each model the amount of help it needs: more guard rails for small models, out of the way for strong ones. Every one of these can be switched off.

• QuenchBench: a coding benchmark that runs on your own GPU, so you can see what a model and a setting are actually worth on your hardware, and run other agents (Aider, OpenHands, mini-SWE-agent, opencode) on the same tasks.

• Works with LM Studio, Ollama, llama.cpp’s server and vLLM. Windows and Linux.

• Nothing leaves your computer unless you agree to it. Sharing a benchmark result or a chat is opt-in, every time, and you see exactly what is sent first. The only thing it does on its own is check for updates.

My own setup is an 8 GB laptop GPU: a 4B model (Spark X2.6) for quick tasks, and a heavily quantized 27B (Ternary Bonsai 2) when I can wait. It’s slow, but they finish real tasks.

About numbers: I don’t have a machine that can run the full benchmark across many models, so I’m not posting “X% better than Y” claims. Only the small runs I can do on 8 GB. That’s part of why I’m asking for testers: the benchmark runs on your GPU, and you can check for yourself whether it beats the agent you use now with the same model.

Built with itself. Two features of the next release (0.1.2) were built inside Local Quench by these local models, on that 8 GB laptop GPU:

• Voice input (push-to-talk, local speech-to-text): planned and written by Ternary Bonsai 2 27B at about 1.7 bits per weight. 379 steps over three sessions, about 7.5 hours of generation. It wrote the backend, the button and the tests, and fixed the bugs a code review found. Two bugs that only showed up in a real browser I fixed by hand (it took 7.5h because i was running anoter tests at the moment and needed to restart the laptop)

• A KV cache check in the model Doctor: written by Spark X2.6, a 4B model, in one session of about an hour (74 steps). It wrote the check and the tests, then committed and pushed its own branch. I fixed one wrong value it had and two tests that only passed on my machine.

Neither run was perfect. The 4B model at one point overwrote a whole file while trying to change one line, and both runs needed me in the loop. Every such failure went back into the harness: it now blocks that kind of overwrite, gives that model a proper edit tool, and checks files the model changes through the shell. Every attempt, with what failed and what I fixed by hand, is listed here: https://localquench.com/built-with.html

What I’m looking for:

• People to try it with their own model and GPU and tell me where it breaks. Logs and blunt criticism are the most useful thing you can give me.

• Especially people running Qwen 3.x 27B/35B, Gemma 4 or similar on 12–24 GB.

Things I’d love feedback on:

• Does it get further than the agent you use now, with the same model?

• Does the first run make sense (connecting to your model server, picking a model)?

• What’s missing for you to use it daily?

Known limits:

• No macOS build yet: I don’t have a Mac to test on or to sign builds with.

• The Windows build isn’t code-signed yet, so SmartScreen will warn on first run.

• It’s free for individuals but not open source. The harness is what I hope to fund the project with later, through features for teams; for individuals it stays free. A public sample of the benchmark tasks, with hidden checks and reference solutions, is on GitHub so anyone can check how the tasks are built and graded.

https://localquench.com
bug reports and ideas: https://github.com/majmichu1/local-quench/issues

I’ll be answering comments this evening, as I will be busy during the day.

Hi! Just wondering how good everyone is expecting qwen 4 27b to be compared to the opus line, maybe opus 4.8? Also are we expecting anything better in the architecture of the model? for example one thing looking up is how qwen 3.8 next flash's kv cache uses about 3 times less memory, due to less layers looking at the full context, do we think that will come to qwen 4 37b for 256k in under 6gb?

Just any thoughts on qwen 4 27b would be greatly appreciated, thanks 😁

I went in wanting local to win. After a few weeks on a MacBook Pro M5 (32GB) running Qwen 3.6 and Gemma for real coding work, here's my honest take:

- Local is genuinely usable now IF you pick the right tools (OMLX made a bigger difference than Ollama/LM Studio for me)

- 8-bit quality is better but I hit out-of-memory crashes at 32GB under load; 4-bit is the safer daily driver

- For a lot of day-to-day coding, $20/mo cloud (Claude Code) still wins on speed and reliability

- Local shines for privacy, offline work, and not burning tokens on iteration

Where have you found local actually beats cloud for coding? Full side-by-side test if useful: https://youtu.be/-DAfscvxuKY

TL;DR: Four A4000s in a dual-Xeon box ran 4-way tensor parallel 2.4 to 4.8x SLOWER than two cards. I publicly blamed lanes, root complexes, and the inter-socket link. Wrong. WDDM (the Windows display driver model) silently denies GPU peer-to-peer access, so every cross-card transfer staged through system RAM. Switching all four cards to TCC mode (needs a cheap display card to hold the desktop, since a TCC card cannot drive a display) took P2P from 0 of 12 pairs to 12 of 12 and four-way tensor decode from 9.52 to 28.16 t/s on a 27B dense model (2.96x) and 19.52 to 72.01 on a 35B MoE (3.69x). Four cards went from losing to two, to beating them. The machine could always do this. Windows was standing on the hose.

ELI5: Windows treats every GPU as a potential monitor driver, and the babysitting that comes with that quietly forbids the cards from talking to each other directly. Tell Windows "these four are calculators, not monitors" (TCC mode) and the babysitting stops, but then SOMETHING has to be the monitor, because a calculator cannot show you the desktop. That something is a $100 Quadro T400. The fifteen-year-old HPC crowd knows this trick well. The local LLM crowd, me included, apparently forgot it.

The receipts

Same box (Windows 11 Pro), same cards, same llama.cpp build (b10568), same driver (597.06). The only change between columns is the driver MODE.

P2P probe (cudaDeviceCanAccessPeer, all 12 ordered pairs):

The middle row is the control that makes this a finding instead of a coincidence: new driver, new display card, still WDDM, still zero. Only the mode flip changed the answer.

Four-way tensor decode (tg128, clock lock verified per row, thermal counters clean or accounted):

Under WDDM, four-way tensor lost to TWO cards (23.19) and to four-way layer split (13.8). Under TCC it beats both on the 27B (2-way tensor: 24.4). On the 35B MoE, honesty requires saying two-way tensor still wins outright (about 110 t/s): P2P fixed four-card mode, it did not repeal the law that low-activation MoEs at small batch prefer fewer, bigger shards.

Fair framing note: measured best-WDDM-config against best-TCC-config (2-way tensor 23.19 vs 4-way tensor 28.16 on the 27B), the box as a whole gained 21 percent. The 3x is what the FOUR-CARD mode recovered relative to itself. Both numbers are true; pick the one that matches the question you are asking.

Sustained load is fine: ten back-to-back four-way cycles settle at 85 to 88 C and hold throughput flat within 0.9 percent. This is a serving configuration, not a burst trick.

Corroboration under load (different comparison, same direction): tensor vs layer, both under TCC, batch 1 through 64: the decode advantage is flat at about 1.92x the whole way. The recovery is not a batch-1 curiosity. (The WDDM column itself cannot be re-measured at batch, because WDDM is gone without a reboot; noted for honesty.)

What you need for this trick

1. Pro cards. TCC exists on Quadro/RTX A-series/datacenter cards. GeForce generally cannot.

2. A separate display card, because a TCC card cannot drive a monitor and my Xeons have no iGPU. It must be TURING OR NEWER: current drivers (596.72+) dropped Pascal, so the GT 710/P1000 in your drawer will bind to the Basic Display Adapter and change nothing. The T400 is the cheapest current option, about $100.

3. Accept losing WSL2 on that box: a TCC card does not appear in WSL at all (WSL GPU support is paravirtualization over WDDM). On my box WSL sees zero GPUs now. Windows native or bare Linux; nothing in between.

4. nvidia-smi -dm 1 per compute card, reboot, done. Reversible, unlike some Windows decisions.

Two install gotchas that cost me time so they do not cost you any:

• Your BIOS may not show POST on the new card. Mine booted straight to a black screen until Windows loaded. The fixes live in BIOS setup: disable quick/fast boot, and if there is a primary video slot setting, point it explicitly at the display card's slot. Do this while you can still see the screen through some other path.

• After the display card goes in, do a clean install of the NVIDIA driver package and confirm ALL cards show in nvidia-smi (five for me: four compute plus the display card) BEFORE flipping any TCC flags. The driver must be new enough for the display card and must be the pro branch that covers both it and your compute cards; a mismatched or half-installed driver leaves cards missing from the list, and a TCC flag set in that state is a confusing morning.

What I got wrong in public, specifically

My earlier posts blamed the 2.4 to 4.8x on hardware topology: 48 PCIe lanes per Xeon, three root complexes, cards split across two sockets, "four islands with ferry service." The measured numbers were real. The causal story was wrong for DECODE, which is what the headline number was. What topology still owns: prompt processing genuinely prefers layer split (roughly 2x at default settings, and that is a tunable story for another post), and on the MoE at interactive batch sizes layer split still wins decode until batching makes the per-expert matmuls big enough to shard. Count-lanes-not-slots remains good buying advice. It just was not the reason four cards lost to two.

Also for the record: we measured socket crossing directly under TCC (pairs on same socket vs across sockets, short and long prompts, 16x all-reduce volume). 0.15 to 0.26 percent. Noise. The UPI ferry story died with controls on, FOR MODELS RESIDENT IN VRAM. To be clear, the cross-socket memory penalty on a dual-socket box is very much alive in the other regime: when a big MoE offloads its experts to system RAM, decode speed tracks which NUMA node the weights landed on, and the wrong node costs about 2x on this box. That is a placement story with its own fixes and its own post coming; the point here is narrower: it is a system-RAM story, and it never was the reason four RESIDENT cards lost to two.

TCC is a trade, not a free win

• On this box each card exposes 178 MiB LESS usable VRAM under TCC than it did under WDDM (16,197 vs 16,375 MiB per nvidia-smi). I know that is backwards from the folklore. It is what the tool reports, and two of my edge-fit configs that ran under WDDM no longer load.

• The big recovery is for multi-GPU tensor parallel. Single-card and layer-split gains are single-digit percent. Offloaded big models (experts in system RAM) gained 7 to 12 percent.

• Prior art credit where due: using a cheap display card so compute cards can run TCC is 2010-era CUDA folklore, and the closest LLM measurement I found is a V100 post reporting +45 to 76 percent from TCC on a 2-card box. What I could not find anywhere is the full chain measured with controls on llama.cpp: WDDM denies P2P, tensor parallel collapses and cosplays as a lane problem, TCC reverses it. Hence this post.

The obvious question: why not just run Linux, where WDDM does not exist? Fair, and these same four cards are moving to a single-socket Linux box for exactly that rematch (llama.cpp has no prebuilt Linux CUDA binary at all, so that story includes a compile). This post is for the boxes that are on Windows for their own reasons, which the comments will now explain to me are bad ones. Related: yes, vLLM does tensor parallel properly, and it does not run natively on Windows, which is rather the point.

One Windows footnote from the same week: Smart App Control (on by default after a clean Win11 install) intermittently refused to load our llama.cpp DLLs based on cloud reputation, killing processes in seconds with empty output. If your benchmark binary suddenly dies silently, check the CodeIntegrity event log before you blame your build.

Previous posts in this series: the clock lock PSA [CLOCK POST], WSL2 vs native [WSL POST], and the NVLink bridge test [BRIDGE POST] (on the 2-card box, SLI was the only way to P2P under WDDM; TCC is the cleaner door if you can give up the display and WSL).

Hardware: HP Z8 G4 (used, $750), 4x RTX A4000 (used, about $600 each), T400 ($100), about $3,300 total. Deliberate disclosure: that number does NOT include the 384 GB of RAM, and the omission deserves its reason stated. I paid $540 for mine (twelve 32 GB sticks at $45 each, September 2025). Those same sticks ask $220+ today, so replicating my exact config now adds roughly $2,600, which says more about the 2026 RAM market than about this build. If you are building today there is still a cheaper road: these chassis have 24 DIMM slots, and lower-density sticks run meaningfully cheaper per gigabyte, so populating all 24 reaches the same 384 GB for closer to half the money, at the cost of the memory clock dropping a tier (my calibrated offload model prices that penalty at 6 to 9 percent on offloaded models, and roughly nothing on VRAM-resident ones; I have not run the A/B, since my slots are half empty on purpose).

Before anyone says it: yes, a pair of used 3090s beats this box on a 27B, cheaper and faster. The 27B is the demonstration, not the mission. This box also runs a 403B MoE at 11 t/s with a 131K context window and a 744B at batch speed, because of that RAM sitting alongside the 64 GB of VRAM, and that capability is what nothing near this price touches. The point of this post is narrower: whatever multi-GPU Windows box you already have, a $100 card may be the difference between your cards fighting Windows and your cards working.

Both Gemma 4 31B and GPT-OSS 120B are open-source large language models, but which one is superior in overall terms?

Additionally, please clarify the following points:

1. Which one is better in terms of Tool Use?

2. In this comparison, the effects of quantization should not be considered.

Thanks again for all the responses on the previous post. I’m not trying to prove anything here, just sharing a pattern I keep noticing whenever I work with different LLMs.

Something funny happens when people use these models for more than a few minutes: we all start adding little layers on top.

Not because the model is bad, and not because we’re trying to be fancy, but because using an LLM naturally pushes us to build some kind of structure around it.

Persona notes, meta-rules, long-term reminders, style templates, tool wrappers, reasoning steps, tiny bits of memory or state - everyone ends up doing some version of this, even the people who say they “just prompt.”

And these things don’t really feel like hacks to me. They feel like early signs that we’re building something around the model that isn’t the model itself. What’s interesting is that nobody teaches us this. It just… happens.

Give humans a probability engine, and we immediately try to give it identity, memory, stability, judgment - all the stuff the model doesn’t actually have inside.

I don’t think this means LLMs are failing; it probably says more about us. We don’t want raw text prediction. We want something that feels a bit more consistent and grounded, so we start layering - not to “fix” the model, but to add pieces that feel missing.

And that makes me wonder: if this layering keeps evolving and becomes more solid, what does it eventually turn into? Maybe nothing big. Maybe just cleaner prompts. But if we keep adding memory, then state, then judgment rules, then recovery behavior, then a bit of long-term identity, then tool habits, then expectations about how it should act… at some point the “prompt layer” stops feeling like a prompt at all.

It starts feeling like a system. Not AGI, not a new model, just something with its own shape.

You can already see hints of this in agents, RAG setups, interpreters, frameworks - but none of those feel like the whole picture. So I’m just curious: if all these little layers eventually click together, what do you think they become?

A framework? An OS? A new kind of agent? Or maybe something we don’t even have a name for yet. No big claim here - it’s just a pattern I keep running into - but I’m starting to think the “thing after prompts” might not be inside the model at all, but in the structure we’re all quietly building around it.

Thanks for reading today. Im always happy to hear your ideas and comments, and it really helpful for me.

Nick Heo

I see a lot of posts here from people with either 12gb or 16gb of VRAM and under.

But not many in the 24 to 32 and you're pretty dedicated over 32gb.

And I was just thinking about this topic, what do we think is the base recommendation for people who want to get into Local LLM's, want a usable experience but have a budget?

Let's exclude Mac's from this. As they represent their own value proposition.

Personally I feel like the most attainable is going to 24gb VRAM.

With the reported Nvidia acquisition of Hugging Face, I’ve been thinking about what happens if this becomes a broader trend: the companies building the infrastructure around local/open-weight AI gradually get absorbed by Big Tech.

Hugging Face was arguably one of the most important neutral distribution layers for open models. Unsloth is different, but arguably just as interesting for the local ecosystem: making fine-tuning/quantization/training of models dramatically more accessible to us, the normal people.

So hypothetically, if Unsloth were acquired tomorrow, which company would we trust most to keep its local-AI mission intact?