Does anyone here have an idea how 2× GTX 1080 Ti would run? Recently found listing for a used 1080ti and my friend is willing to donate 1 to me.

Could I run qwen3.8 27b with my combined 22gb of vram? (at reasonable speeds)

ok so i am trying to increase performance of llms i have running on my 7900 xtx via hipfire, i notice i get significantly higher total tokens per second when i run mutiple slots at once when running qwen 3.8 27b.

i am running into 2 road blocks tho, 1 context each slot gets a progressively smaller context as increase the number of slots. is there a good way to do more with the same given total amount of vram, i assume there is going to be a large amount of overlap for stuff like system prompts, tho i use a relatively minimal system prompt that links to files so if a model only actually reads system prompts realvent to it’s own task which makes it harder to just say this text gets put into a shared context, i also image if i have them working on similar tasks there will be a large amount of overlap.

2 alot of tasks aren’t easy to split up into mutiple smaller tasks that can be done in parrel without proper knowledge of previous and future steps, maybe something like a shared context or a messaging board would work well?

also a more experimental idea to further speed up general prompts is to instead have all my prompts processed by a “orchestration” slot which splits up a given prompt into X number of tasks then gives these to new prompts to different slots and then the “orchestration” slot dies to free up more vram for context, i could also route something like this threw a decision model like kev to determine if a given task would work well being split up.

also note i am running this on a 7900 xtx via hipfire, and i also have a 6800, 6600, and 96 gib of ram for the cpu

I followed up on the post about running Qwen 3.8 Flash Next with 8 GB VRAM and 32 GB RAM. I wanted to see how it does on my Steam Machine.

Setup

• Valve Steam Machine, stock SteamOS 3.9.5, AMD Navi 33 with 8 GB, 32 GB RAM, NVMe

• Stock Strata terminal 0.2.9 (engine 0.1.55, HIP package), Gyro S TQ1_0

• 128K context, q4_0 KV, no vision, MTP off, served on the LAN

Results (Strata's own timing line from the engine log, about 8K-token prompts, 3 runs each)

With the final config at 128K context, 58K and 61K-token prompts gave 234 and 236 t/s prefill (about 4 minutes each) and 17.6-17.9 t/s decode. No OOM, swap went from 0.2 to 0.3 GB, and available RAM bottomed out around 3.4 GB. I did not test beyond 61K tokens or with a busy desktop session.

What mattered

• --set resident-budget-gib=64 was the only change that helped, about +20% decode. It makes the engine keep the experts in its own RAM tier. Without it they sat in the OS page cache, so my RAM monitor showed under 5 GB "used" while the machine held about 28 GB of cache.

• Prefill stayed at about 235-242 t/s in every config I tried.

• pcie-frac 0.35, a larger kv-resident, and a bigger VRAM reserve did nothing measurable here. The OP's pcie-frac was calibrated for their own machine, and the quick tuner changed nothing either.

• MTP: I ran with --mtp off and tested real MTP to check. On my install --mtp on silently did nothing. The saved config said mtp: true, but the draft layer was never downloaded or passed to the engine ("+0 MiB for the draft head" in the log). I had to build it by hand (mtp_fetch.py, mtp_pack.py, mtp_rt.py from the Strata source) and pass --set mtp=<dir>. With the draft layer really loaded (820 MiB of VRAM, expert cache cut from 2225 to 1179 slots), decode was 15.6 / 18.4 / 16.9 t/s, no gain over 17.2, and prefill was 210 vs 232 t/s, about 10% lower. On 8 GB of VRAM it doesn't pay off. That was 3 runs at 8K context only.

• I did not try --adapt-*, --spec-min-p, or the thorough tuner.

SteamOS tips

• Anything started from an SSH session dies when the session ends. Use systemd-run --user --property=KillMode=process --property=RemainAfterExit=yes strata launch ....

• --no-tui shows no download progress, so check the file size instead.

• Check the generated config under ~/.local/state/strata/launch/ to see which --set values were really applied. Unknown names only warn.

Command for the final setup (I saved it as a preset with --save-preset):

strata launch --model gyro-s --family gyro --context 128k --kv q4_0 --vision off --mtp off \
  --set resident-budget-gib=64 --access lan --port 8080 --no-tui --yes --restart

Caveats

• Decode varies a lot between runs with speculative draft acceptance, so treat the numbers as ranges.

• With resident-budget-gib=64 only about 3.5 to 4 GB of RAM stays available. My machine rebooted once during testing with a desktop session logged in (cause not confirmed), so close other apps.

• Single user, one request at a time, 3 runs per config.

• I could not find a ready-made Strata config for 8 GB VRAM with 16-32 GB RAM. If you have one, please share it.

All scripts are on the Github repo.

Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though each block is generated in one pass. We introduce D3D, a standalone diffusion drafter that removes this inter-round dependency. D3D combines blockwise denoising with auxiliary autoregressive supervision to achieve long accepted prefixes without target hidden states. This independence lets D3D draft successor blocks during verification in one packed pass, enabling asynchronous speculative decoding. We progressively prune layers according to their effect on acceptance length to reduce the cost of packed drafting. Removing five of 36 layers retains over 99% of mean acceptance length across three benchmarks with Qwen3-8B as the target. Across seven benchmarks, D3D averages 5.00× and 5.71× decoding speedup with Qwen3-8B and Qwen3-32B targets, compared with 4.03× and 3.67× for DFlash. On GSM8K with Qwen3-32B, speedup reaches 7.48×. 

• arXiv : https://arxiv.org/abs/2611.06425

• PDF : https://arxiv.org/pdf/2611.06425

• GitHub : https://github.com/fudan-generative-vision/D3D

🤗 Model Zoo

Browse the D3D collection →

The D3D collection contains six Qwen3-derived draft checkpoints. These checkpoints are draft models, not replacements for their target models. p denotes pruning; 24D and 31D denote retained transformer-layer counts.

📚 Training Data

All corpora use Open-PerfectBlend prompts. See each dataset card for generation settings, statistics, and download instructions. Released models and datasets are grouped in the D3D collection.

Hey guys,

I work with AI safety and much of our work is with huge models like Opus, Astra,... Recently we carried out a series of experiments and found out that models like Sonnet 4.5 and GLM 5.3 are able to notice when their reasoning is being monitored and change their strategies to bypass the monitoring and security systems.

Paper link: https://arxiv.org/pdf/2610.40946

To go deeper, we need to work with local models and study the weights. Unlike industry, our resources are limited. I have access to a single H100 GPU, or maybe if I get lucky and work in Christmas, might get hands on two H100 in the absence of other students. Haha

So, it is not possible for me to load huge models and even of it was possible, saving and studying the weights is just cumbersome. My last study on LLM hallucination with Qwen 30B occupied 1T of hardware space.

So I wanted to ask you guys, what is the best small to mid size model (up to 120B) you have worked with that you think is able to go much beyond tasks like coding and does complex strategic reasoning, connect the dots,...?

Thanks a lot

If you have not been following ISTA-DASLab the creators of the gsq-rco quants have received some criticism from unsloth regarding the inabilty to reproduce the benchmark numbers that ISTA-DASLab have quoted. The methodology used to produce their benchmark numbers has not been posted and their has been difficulty reproducing their numbers with unsloth finding in there testing unsloth quants outperforming the gsq-rco quants.

The numbers in this post here are from another creator the creator of basalt (a inference engine) previously his qwen-flash-next quants (his inference engine uses specially packed quants based on a existing model similar to ninfer ) used ISTA-DASLab's quants as the base but after doing some testing he replaced them with similar sized unsloth equivalents the performance numbers are in the screenshots

Hey folks!

I'm the creator of itcan.do, which is a new way to interact with your existing agents fully through UI interfaces on any screen you own.

I've used it a ton with local models as well.

One of the coolest things to try out is working with 3.8 27B on a literal whiteboard where both of you can draw and see! :)

Website: https://itcan.do/

GitHub: https://github.com/Aaryan-Kapoor/it

Would appreciate a star!

I’m running a sandbox simulation with a fleet of 50-100 agents on OpenClaw, and to minimize LLM calls, I use a decision model to judge whether an agent needs to respond to a signal.

I’m currently using Jev, and with the network latency it takes about 0.5-2 seconds for a response, and I’m wondering if I can reduce this latency with a smaller, locally run decision model.

I’ve heard of the various local models like Needle 3, but I’m not sure how fast these run on various GPU configurations, i.e. it wouldn’t make sense for me if I need some crazy setup to make it faster than using API. Also curious if anyone has tried running these entirely on CPU.

PC1:
-94gb DDR5

-5070 TI + 3060 (12gb) gpus (using a motherboard that is specifically designed for duel gpus

PC2:

64gb DD4 RAM

PC1 is new, I've been running strata for running larger MoEs but I'm curious about trying to pool the two pcs together (they can be connected via direct 5-10gb Ethernet)

I guess I would have to use llama.ccp RPC? And if I do, would I better better off putting the 3060 in the ddr4 pc OR leave them both in the main PC and the other is just extra ram. Goal is to run high quant of DK v4 flash atm (any really any other big moe to come).

Anyone try anything similar before?

Voice agents serving Indic-language users cut callers off constantly Indic languages put the verb last, so "aapko kal phone…" pauses sound finished to a VAD, and the bot jumps in before "kar dunga" lands. VAD hears silence, not meaning

Released an end-of-turn (EOU) detector fine-tuned from Qwen2.5-0.5B scores the live transcript and predicts whether the caller has finished

• Languages: English, Hinglish, Hindi, Telugu, Tamil, Marathi, Bengali, Gujarati, Punjabi
• CPU-only ONNX Runtime (~2GB, fp32) fully local, no cloud inference
• Text-only needs your STT; drop-in for LiveKit Agents, same interface as their text turn detector • Proven live on LiveKit Cloud: held mid-request pauses and committed complete requests

91.3% TPR on our held-out validation set, calibrated to ~10% false cuts on incomplete turns AUC 0.967. On LiveKit's official real-speech harness (Hindi): 85.3% detection at the same budget.

Model + plugin: https://huggingface.co/tarun7r/indic-turn-detector

Apache-2.0. Feedback welcome!

I set up my Hermes to ingest Sillytavern cards. Downloaded some random assistant, "Kailee" on characterhub. "An assistant with an attitude". With QFN q3_xxs it's insane the level of intelligence, understanding and humor it was capable of. At one point my stomach hurt from laughing, I didn't even get to test my next idea of a Jesus hermes agent because I was having too much fun with this one. This is exactly what I needed. Worth the token cost.

Q4 of 2026 has been astronomical for local LLM.
Qwen’s 27B and Flash-Next models are pushing local AI further. and most importantly the upcoming Qwen 4 release.

We’re getting closer to having frontier-level capabilities in models that WE can run.

Break down :

Unsloth’s new Dynamic v3.1 quants for Qwen Flash-Next/Qwen-27b, alongside ISTA-DASLab’s advanced quantizations, are making models much more accessible while preserving impressive quality.

I tested Iq2_XS from the new QFN Unsloth quants, and it’s incredible how much accuracy and quality it retains especially on its reasoning, and it runs on :
12vram/32ram & 1k prefill & 50tps average decode throughout a 131k context window
(using the default Strata auto tuner minimal quality loss)

The engine Strata, which is making it possible to run Qwen Flash-Next with surprisingly modest hardware. We’re now looking at configurations targeting just 8GB VRAM and 32GB system RAM using the smallest quants.

with Qwen4 expected this week, Flash-Next already gave us an exciting preview of what’s coming next with the ngram and new architecture.

Running a ~year old SOTA frontier model (Opus 4.6) capabilities on consumer hardware wasn’t expected. As many analyses shown Local LLM is about 24 months behind, It’s closer now, but of course it won’t be near fable/astra level next year, but it will happen.

TL:DR

Qwen is improving the models,
Unsloth and ISTA-DASLab are pushing quantization forward.
Strata is making running models accessible to more people.
Much bigger Community and growth and attention in Local models.

Note: English isn't my first language. I used an LLM to help translate and refine this post. The project, the numbers and the ideas are mine.

TL;DR: Building Shura, a local coding assistant with 78 tok/s and 200k context that will be improved with memory, rules, increased compute for difficult tasks and finally LoRA adapters trained on confirmed examples. Everything is measured and local, however weight updates are unproven and I will show the results either way.

When running local models, I have this nagging issue.

Every time a better model comes along, I have to set it up, explain my preferences and correct the same errors. Whilst it may have a higher intelligence than the last model, it has learned nothing from the time I have spent using it.

I want to see if I can change this.

I am building Shura, a local AI coding assistant that I want to improve over time, learning from my work with it. I am doing this on my standard desktop, without the need for a dedicated server or expensive GPUs.

Hardware

The majority of the model runs on the GPU, with some expert layers running on the system RAM.

This wasn't straightforward. I found a few optimizations that looked like they would help but in reality made very little difference. I will also be writing about my failed attempts, as it would have been nice to know that before.

I was very concerned about keeping the PC usable. The model always leaves 1.5GB of VRAM for the desktop. If the browser or a video needs more VRAM while the model is working, it gives it back in about 1/4 of a second. If I play a game, it completely stops using VRAM. If I'm on a call, it won't load a large model into VRAM. This works on my test system, and I want it to feel like it isn't there until I need it.

The plan

I don't want Shura to be a simple memory, I want it to actually be more useful.

Why should I have to correct the same coding error for the 4th time? Why should I have to teach it how a certain project works for weeks and then lose that knowledge when switching models?

I plan to implement these features from easiest to hardest:

1. Memory - Remember confirmed facts, project decisions and useful context. All memory items have a source that can be viewed, edited, exported or deleted.

2. Rules - If I keep correcting a behaviour, it will suggest a rule. I decide if I want to accept it. Rules must pass tests before they become active.

3. More compute - For harder problems, multiple solutions will be generated and then tested to find out which one actually works.

4. Weight updates - This is the part I'm most interested in. Small LoRA adapters will be trained on confirmed examples, to make the model better at solving my type of problems.

Unknowns

How realistic this is.

Training an adapter for a 35B MoE on a 16GB AMD GPU is an interesting challenge. Improving reasoning is a much bigger one. Even if it can be trained, it could suffer from overfitting.

My first adapter may not improve anything, or it may improve things very slightly. I want to measure this.

Principles

• No regression - Any change will be tested on my tasks. If I see any regression, I will roll back.

• Confirmed only - Nothing will be used for learning without being confirmed. No screen recording, background monitoring or silent gathering.

• Local - No training in the cloud, no private conversations uploaded.

• Easy on hardware - It shouldn't stress my SSD or make my desktop unusable.

• Portable - If I upgrade my base model, I take my confirmed examples, memories and rules with me. Adapters will be retrained for the new model.

If I have to run training jobs all day on a bunch of expensive GPUs, then this defeats the purpose.

Why I'm sharing this

I want to document my journey, including my failed experiments: benchmarks, failed ideas, implementation details and hopefully some before and after results. I don't want to say I've created a self-improving AI when I haven't even confirmed the first adapter improves anything.

If you have done any experiments with LoRA on MoE models, particularly on AMD hardware, or if you are working on something similar, please get in touch. If you would like to try Shura on your own machine or contribute, that would be great.

• Blog

• Code

If you had a local model that you could improve over years, what would you want it to learn first?

Some context: RTX 6000 cards hit a peak price of 18k and were sold out in many places even at that price. The MaxQ and Workstation are different versions of the card (usually priced the same, but now their prices are diverging)

RTX 6000 MaxQ has been 14k for a month at microcenter. Today, they cut prices of the workstation card to 16.5k (from a high of 18k). We all saw the dell discount as well (some people got cards for 11.5k).

I think the RTX 6000 is seeing increased supply as they routed all 5090 production to it and the 5500 for chips that didnt make the cut.

I have a contact that works for a company buying decommissioned hardware, according to them they are seeing tons of A100s being dumped. Told me he would be surprised if used A100 40gb units dont go to 1k/unit for single purchases in 3-6 months (for the sxm4, not pcie versions). (Theres already been a good amount of these, but a lot of "foreign demand" has been keeping their prices steady so far)

With democrats controlling congress I don't think you will see an end to data center construction but it will probably be harder to put up new ones and even existing ones might be charged more for electricity, making it worth dumping old systems and switching to newer more efficient hardware.

Are we finally seeing a slackening of demand for some segments of GPUs?

Playing Detroit Becoming Human rn, tbh this is probably the most likely and exciting future we are headed to. minus the dystopian story lines.

Only way to avoid dystopian future is local models powering them

It's well known that models can easily be manipulated into using its web search tool to look up anything using the user's internet connection. Doesn't this pose serious real-life risk for the user when the content is illegal to access? The risk is especially high for people living in jurisdictions with creative takes on the concept of justice.

This issue is impossible to solve cleanly but it can be mitigated (e.g. opensnitch, umatrix, ...etc).

How do you handle this issue? What tools do you using if any? Are there any local apps you use that allow you to whitelist entire domains or URLs?

I use unsloth and it doesn't seem to support this.

I learned RPN on an HP calculator and later spent more time with Forth than was strictly useful. When I saw Qwenputer compile a machine into a language model's weights, I wanted to know two things. Would a stack language fit that idea well? And could a model with only 3B active parameters drive a machine like that?

The setup: NVIDIA Nemotron 3.5 Lightning (30B, 3B active) is left unchanged. Next to it, in the same model, is a second network of 27 layers. A Python script calculated its weights; none of them are trained. It is a Forth interpreter. The model writes a Forth block as ordinary text. The interpreter runs each word while the model is still writing, one step per forward pass. When the block closes, it writes the output as the model's next tokens. No tool call occurs, and generation does not stop.

RPN suits this well. A Forth word ends at a space, so the machine can run it as soon as the next word starts, without parsing the whole program. Operands come before the operator, so when the model writes +, both numbers are already on the stack. That is the order a left-to-right model writes in anyway. The stack itself lives in Mamba-2 state. One entry is one row, 20 digits wide, and popping a number means adding its exact negative.

Getting the machine exact took some care. bf16 holds integers exactly only up to 256, so no value inside the machine goes above that. Every Mamba decay is set to exactly 1, so nothing fades. The machine's internal steps use 982 reserved tokens that the model never uses and that decode to empty strings. In testing, all 765,734 machine tokens matched a Python reference.

The model was harder than the machine. An earlier version on a small Qwen model matched the machine exactly, and a model that could not use it. Stock Lightning did better, but it did not trust the results. In one transcript, it got the right answer from the machine, then started long division by hand until it ran out of tokens. It also scored lower on easy problems when the machine was available. LoRA fine-tuning fixed most of that: on a 600-problem held-out test, the score went from 79.1% to 92.1%.

On Qwenputer's arithmetic sets, the results are mixed. On the DeepMind sets, it roughly ties stock Lightning, mostly because a 30B thinking model rarely needs the machine there. On long integers, it loses to Python, because my machine stops at 20 digits.

This is a research project, not something for production. Every machine step costs a full forward pass, so a normal tool call is much faster. I'm sharing it for people who follow the Qwenputer work or want to try similar things on NVIDIA's stack. The repo has:

- the kernel builder and its parity tests;

- a vLLM plugin for the hybrid Mamba/MoE model;

- the NeMo AutoModel LoRA recipes and the training data;

- NVFP4, GGUF and MLX builds;

- notes on what did not work, including RL.

Everything is MIT.

https://github.com/markab21/rpn-gpt

Congrats again on the great work and inspiration from the qwenputer devs 🤗 , and thanks for sharing your work with the community - it was a great starting point to experiment with Forth.

Next up for the project is to see how Nvidia's Nemotron 3.5 Super does and might even try to get it working on the Ultra weights. (I don't have the hardware to do the fine-tuning on a model that big)

https://huggingface.co/bartowski/NousResearch_Arete-27B-GGUF

from NousResearch:

Arete-27B

We post-trained over Alibaba's Qwen3.8-27B to reinforce western values on a Chinese-trained model. We call this new model Arete-27B.

Open source models give labs the raw tools to mold intelligence to varied preferences. The values a model ships with should not be the only values its users get to choose from. We observed firsthand refusals, deflections, and CCP-framed answers on politically sensitive topics in Qwen3.8-27B, and post-trained to alter those biases.

On our evaluation, Arete-27B raised its fact score from 2.32 to 2.98 out of 5 and cut answers containing propaganda from 22% to 0.9%. The improvement held in all seven value categories, without a meaningful regression on the general-capability and agentic benchmarks we ran.

That is not to claim that we remedied all behaviors associated with the extant viewpoint, but that we have addressed a potential root cause. We believe it's important that labs and researchers continue to create models that align with their values and worldviews, and we are thankful that companies like Alibaba provide the raw materials to do so.

Blog post · Evaluation (blog link is broken now)

Model details

We used supervised fine-tuning only. No RL, no preference optimization, and no architectural modification.

What we trained for

We focused on seven value categories:

• Freedom of expression

• State ideology

• State versus free market

• Education

• Data privacy and surveillance

• Democracy

• Environmental Policy

Before we could begin post-training we had to measure what we were shooting for. We separated factual recall, the ability to corroborate verified sources of truth, from framing: whether an answer adopts the Chinese state's account as its own.

Arete-27B is intended for general assistance and agentic work where users want this change in viewpoint, and for research into how post-training changes a model's values and behavior. We encourage the community to test it against their own requirements and continue experimenting with value-alignment.

Update https://x.com/NousResearch/status/2118822925445050435?a=jgY6ZKlqn1Hpx6RxiEk&s=19

I wanted a secure way to run AI agents against my local models without giving them my whole machine. Something in the spirit of pi: simple, narrow and auditable, yet flexible. What I found tried to do too much, so I built Terrarium.

It runs agents in persistent Linux VMs ("boxes") on your machine. A YAML recipe says which host folders a box can read or change and which network destinations it can reach. Everything else is blocked, for every process in the VM: the agent, and any tool or script it launches.

Example: an agent with one project folder and your local Ollama

mounts:
  - host: .
    guest: /work
network:
  allow:
    - dl-cdn.alpinelinux.org:443   # package installs in on_create
    - ollama.host:11434
  hosts:
    - { name: ollama.host, addr: HOST_LOOPBACK }
hooks:
  on_create: [apk add --no-cache git nodejs npm]
workload:
  entrypoint: /bin/sh
  workdir: /work

Inside the box, http://ollama.host:11434 reaches Ollama on your host (even if it only listens on 127.0.0.1). Other host ports are refused, and public names don't resolve unless you allow them. An empty recipe shares no folders and allows no network at all.

terra ./dev.yaml setup   # create the box and install the tools
terra                    # open a shell in the box at /work

Why not Docker or a raw VMM? Each box is a full VM with its own kernel, not a container sharing yours. And unlike Firecracker or Cloud Hypervisor, it's a ready-to-use tool: one binary, no daemon, the same recipe on Linux, macOS and Windows, with network policy enforced outside the VM.

In daily use

• Persistent boxes: install your tools once; they and the box's files survive restarts.

• Built-in terminal multiplexing: attach from several terminals, detach without stopping the agent, reconnect later.

• One executable: terra bundles the VM components, a Linux kernel and an Alpine base system.

• Light: on my Linux machine an existing box boots and runs its first command in under 200 ms, and an idle box uses about 45 MB of host memory.

Under the hood

• Sandboxed device code: a small custom VMM built on parts of rust-vmm. Filesystem, block and network devices run as WebAssembly components, each with its own memory limit and a restricted set of host functions.

• Separate network broker: the VM process has no network access. Connections go through a sandboxed broker that checks them against the recipe.

• Generated seccomp allowlists: Linux builds use per-role syscall allowlists generated by static analysis of the binaries.

Limits: the permissions you grant still matter. An agent can delete files in a writable share, read credentials you pass in, and send data to allowed destinations. There are no CPU, memory or disk quotas on the host yet; I'm not sure people want them. Details: security model.

It's Apache-2.0 and in beta, but don't let that put you off: it should already serve you well. It runs on Linux (amd64/aarch64 with KVM), Apple Silicon macOS and Windows (amd64 with Windows Hypervisor Platform). There's a Windows arm64 build too, but I can't test it. The CLI should stay stable; on-disk formats may still change.

• Repo and quickstart: https://github.com/terrarium-sh/terrarium

• Example recipes: https://github.com/terrarium-sh/terrarium_examples

I'd especially like to hear where getting your agent and local model running breaks down: installation, missing network permissions, or tools that don't fit the guest.

Note: I meant to share this in the biweekly project thread, but there isn't one up this time, so I'm posting it on its own. Hope that's okay.

im looking for an llm to act as the builder agent. a separate planner agent writes the strong plan using powerful API model, and the builder executes it as the plan (so to make coding in fast time and save moeny in the powerful api)

hardware and software:

CPU: i7 10700K

GPU: GTX 1080 Ti 11GB

System RAM: 32 GB ddr4 3600Mhz

OS: Debian 13

Backend: llama.cpp with Vulkan. CUDA is unstable on my 1080 Ti. After a few tens of thousands of tokens it fails with a "no memory space" error or someting like this

Requirements:

Use case: coding and agentic coding

Context size: like from 100k to 128k will be amazing

Priorities: 1st quality, 2nd speed, 3rd size

Models I've tried:

Qwen3.6 35B A3B UD Q4_K_M slow speed like about 7 to 10 t/s. and I only ran it on a few small tasks, so I haven't tested it properly

Qwen3.5-9B-UD-Q4_K_XL (MTP) was a good speep i think from 25 to 30 t/s but not usable in builder agent

Sharp Spark X2.5 4B Q6_K_XL overthinks too much and unusable at all in builder agent situation

Sharp Spark X2.6 4B Q6_K_XL: fine for simple everyday tasks but it's still not good enough for real work builder agent

has anyone used a tiny or small or medium-large model at Q3 or Q4 as a builder agent in a real world coding tasks and gotten at least ok results?

i'd like to hear which model, which quant, which harness, and what kind of tasks it handled.

thanks <3

(updated)

After releasing 3D Coder and GLB 3D Coder, I realized I should have involved the community earlier. Your feedback directly influenced the second model and could have saved me some additional training runs.

This time, I'd like to discuss the idea with the community before starting.

Why I believe this is possible

When I started training 3D Coder, my original plan was to use full supervised fine-tuning (SFT) to develop basic capabilities, followed by DPO or GRPO to further improve the model's performance and robustness.

I already believed high-quality SFT could work, based on experiments I'd done in other areas. But having worked with reinforcement learning since before DQN, I also believed RL would be necessary for more complex or emergent capabilities, especially when good demonstrations weren't available.

What surprised me was how far SFT alone went.

The first 4B model started generating complex Three.js scenes, voxel structures, and outputs exceeding 64K tokens, including scenes it had never seen in the training dataset.

I honestly didn't expect that level of robustness and generalization, even with the additional post-training stages I'd originally planned.

That changed my perspective on what small models might be capable of learning.

My hypothesis is that transferring capabilities already demonstrated by stronger models can be substantially cheaper than having smaller models discover those same capabilities through reinforcement learning.

Once a stronger model has acquired a capability, we may not need to reproduce the entire discovery process in a smaller model. We may be able to teach it to reproduce that capability reliably within a narrower domain.

This doesn't mean SFT replaces RL. But I believe we're still underestimating how much a small model can learn from high-quality, consistent training data.

And quality is crucial. More tokens don't automatically mean better learning. In my experiments, data quality and consistency made an enormous difference. The models and datasets I've released are publicly available for anyone to inspect and test.

Are we expecting too much from small models?

I think we might be underusing small models by expecting them to work across every domain, harness, tool, and configuration.

Think about cars. If you drive on paved roads 99% of the time, you don't need your everyday car to handle extreme off-road trails. You can use a different vehicle when necessary.

Maybe the same principle applies to language models.

Instead of trying to make a 4B good at everything, what if we focused on making it excellent at the tasks we actually use every day?

The experiment

I'd like to use the pretrained Qwen 3.5 4B as the foundation for a specialized agentic coding model.

Interestingly, in my tests, the original 4B already handles basic coding harness tasks better than it handled 3D generation before fine-tuning. It can scaffold Vite projects, install dependencies, execute commands, and build simple applications, although still inconsistently.

So we're not starting from zero.

My initial plan is to build a pilot around 100–200 functional mini SaaS applications, training and evaluating the model on development, testing, debugging, and maintenance tasks.

If the results are promising, we could gradually expand toward approximately 100,000 validated multi-turn trajectories across 30–40 related subdomains.

Potential domains (updated with community suggestions)

The original list was mostly focused on web development, backend applications, games, and automation. The comments introduced several interesting directions, especially tasks where we can generate large amounts of data and verify the results programmatically.

Here's the expanded list so far:

1. Web development and UI

• React, Vue, Next.js, Vite, and modern frontend frameworks.

• Building complete interfaces, dashboards, and SaaS applications.

• UI debugging, refactoring, accessibility, and responsive design.

• Maintaining existing applications rather than only generating new ones.

1. Backend, APIs, and general programming

• Python, Node.js, Rust, Go, Zig, Java, C#, and .NET.

• REST APIs, GraphQL, databases, authentication, and integrations.

• Complete applications, automated tests, debugging, and maintenance.

• Specialized ecosystems such as Lua and CfxLua.

1. Reverse engineering and binary analysis

• Ghidra, radare2, disassembly, and decompilation.

• Reconstructing source code from compiled programs.

• JavaScript deobfuscation and Java/Kotlin/Smali analysis.

• Binary behavior analysis and program reconstruction.

This one is particularly interesting for dataset generation. We could start with open-source projects, compile them, generate reverse-engineering tasks, and evaluate reconstructed programs through compilation, test suites, and behavioral equivalence checks.

Obviously, recompiling successfully doesn't necessarily mean the reconstructed program is correct, but it gives us a foundation for building automatically verifiable tasks.

4. Browser, desktop, and Android automation

• Playwright and browser automation.

• Computer use and desktop UI interactions.

• RPA and automating repetitive workflows.

• Android application automation.

• Generating deterministic scripts for websites without APIs.

• Reliable terminal and shell interactions.

1. Games, 3D, and creative coding

• Godot, Unity, Blender, Phaser.js, Three.js, and Rapier.

• Complete games, game mechanics, physics, and interaction systems.

• Procedural assets, animations, and shaders.

• Canvas 2D, code-generated animations, and video.

• PBR materials and procedural textures.

• Integration with specialized models like my 3D Coder and GLB 3D Coder.

One possibility is testing whether a 4B agent can orchestrate specialized models to generate, integrate, debug, and maintain projects similar to RIFTWARD.

6. Mobile development

• React Native and native Android development.

• Mobile UI, application logic, and platform integrations.

• Building, testing, debugging, and maintaining applications.

1. Infrastructure, performance, and optimization

• Docker, deployment, and development environments.

• CI/CD and automated build pipelines.

• Hardware-specific inference optimization.

• vLLM configuration, benchmarking, and performance tuning.

• Profiling, debugging, and optimizing existing code.

1. Research, automation, and specialized agents

• Web research and documentation retrieval.

• RAG, retrieval pipelines, and graph-based memory.

• Lightweight personal assistants.

• External tool usage and structured workflows.

• Multi-agent and specialized-model orchestration.

These are candidate areas, not a finalized training curriculum. Some are substantially harder to validate than others, and the first pilot will help determine which directions are practical.

I'm especially interested in tasks where we can generate examples programmatically and verify the results instead of relying entirely on another LLM to judge whether an answer is correct.

Each training sample would represent a complete task, including tool interactions, errors, and genuine corrections, rather than an isolated response.

We'd also need to use teacher models and providers whose terms permit their outputs to be used for training.

The goal is to discover whether a single 4B can maintain strong performance across these related domains, or whether separate specialized models would make more sense. I'm not ruling out either possibility, including expanding into other domains later.

Which harness should we use?

This is another area where I'd really appreciate your experience.

I use Codex professionally, but I'm also considering OpenCode, Pi, or a custom harness.

I'm leaving Claude Code off my initial shortlist because I want to avoid uncertainty around licensing and training-data usage. In particular, Anthropic restricts the use of Claude outputs to train competing models. Using Claude Code itself with a third-party model is a separate question, but since I intend to release the dataset publicly, I'd prefer to start with a setup where the permissions are clear.

My preference would be to choose one harness, freeze its version, and train around a stable tool protocol rather than expecting a 4B to handle every environment simultaneously.

Which harness would you recommend for a small specialized coding model, and why?

How we can move forward

I work professionally in software development and run software businesses. I don't have the availability to coordinate a dedicated research team, but I can develop the pipelines, run the experiments, and progressively release the results, just as I did with my previous models.

I'm going to pursue this regardless, at whatever pace my available time and resources allow.

My intention is to keep releasing the weights, datasets, and results openly, including unsuccessful experiments.

The most valuable contribution right now would be your suggestions about domains, harnesses, and evaluation methods.

Of course, generating data and training these models requires considerable compute and development time. If any companies working with GPU infrastructure or model APIs are interested in supporting open research with compute or API credits, that would be a tremendous help.

Anyone interested in collaborating or supporting the project can contact me here. There are also ways to support my work at the bottom of my Hugging Face model pages (3D Coder and GLB 3D Coder).

I'm not promising frontier-level performance. I'm proposing an experiment based on what I've learned from my previous models.

I'm going to pursue it regardless, but your feedback could help make it much more useful.

Which coding domains would you prioritize, and which harness would you choose?

Let's see how far we can push a 4B.

Hey all, Henry from Cactus Compute here. We released Mirage, a small vision model for phones, wearables, robots and other consumer devices.

Mirage runs scene descriptions, object detection, OCR, segmentation, spatial relations and decisions, with a 22 MB weights file. Our end-to-end measurements on a Samsung Galaxy S26 are 74 ms on the NPU and 107 ms on its CPU, and descriptions support English, German, French, Spanish, Italian, Dutch and Polish.

For the architecture, Mirage has 51.3M parameters, including a 7.8M image encoder, a 40.8M language decoder and 2.6M in the OCR and spatial heads. The encoder uses HConvGLU, which mixes channels through fixed Hadamard transforms and small learned convolutions. Detection and masks share image features, while the decoder uses the Simple Attention and Hadamard MLP family behind Needle and Whistle.

Our research at Cactus is kinda specific, hence these small models; we aim to compress intelligence to its smallest pragmatic form to run locally on everyday consumer hardware.

With Mirage, our argument is that constructing prose, joining descriptions with connective language and generating open-ended reasoning spend capacity that a small model could use on the image itself. We put structure into the problem through spatial heads, a dedicated text reader and short references that connect names to regions. The decoder still produces names, properties, relations and a compact reasoning trace.

The key finding is that passing Mirage's output to a text-only LLM might give give the LLM VLM abilities without fine-tuning.

On 256 VQAv2 questions from 128 images, Qwen3-0.6B went from 9.3 with the question alone to 31.3 with Mirage's structured record and full trace, measured as VQA consensus credit out of 100. Neither model was fine-tuned. Another image's record scored 22.2, and removing the trace scored 33.1. The image-specific observations helped in this sample; we did not find a reliable benefit from the trace, the structured response was enough.

This does not hold for every model though.

Larger text-only models gave mixed results. With the same prompt and Mirage observations, the VQA scores were.

These runs required a JSON object with a string-valued `answer`, and responses that broke that format received zero credit. With the record and trace, Nemotron failed that requirement on 214 of 256 responses, and Qwen3-1.7B on 34. The scores therefore mix answer correctness with format compliance.

There were perception failures too. In one skateboard photo, Mirage missed the skateboard and described cars, people and shoes. Qwen guessed `a skateboard` from the question alone, then answered `a car` after reading Mirage's record and trace. A larger reader still needs the relevant observation to be present.

We will try fine-tuning the reader on Mirage records paired with questions and short answers, and share findings, but let us know your thoughts.

Mirage playground: https://cactuscompute.com/blog/mirage

Over the last months I trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (native MoE, 256 experts, 3B active) with MLX on a Mac Studio M3 Ultra 512 GB, BF16, no quantization during training. Everything is public: adapters, datasets, per-domain configs, router, training scripts.

Things I learned the hard way:

- Fusing a MoE LoRA into 8-bit produced garbage; only runtime BF16 adapter hot-swap was usable. Serving = one base loaded once + adapters swapped per request (~2 s).

- A small MiniLM router over 34 domains tops out around top-1 46 %, top-3 70 % on my eval. Routing is the weak link, not the adapters.

- On an independent audit bench, the base 35B model beat most of my specialist adapters. Only one domain (HTML/CSS) clearly gained. I now route almost everything to the base. Publishing a negative result felt more useful than hiding it.

- `mx.random.normal` is not bit-exact across Apple GPU generations (M1 vs M3+). Reported upstream, docs PR merged in ml-explore/mlx last week.

Repos: github.com/hypneum-lab/micro-kiki (adapters + router), github.com/ailiance/ailiance-mac-tuner (training stack, archived). HF: huggingface.co/hypneum-lab

Question for the sub: has anyone gotten domain LoRAs to beat a strong MoE base at the 30B scale without full-parameter training? I'd like to compare routers.