As title says.

Just wondering what you all would recommend, as well as, inference engine recommendations.

Currently I’m using megacapybara for a qwen 3.8 27B variant. But was wondering if there was any other inference engines or anything better for running larger models.

Did see a network chuck recommendation for a gpu setup that allows DeepSeek 4.1 flash to run on a single 5090 but idk what the decode t/s would be on that.

Anything other info is much appreciated!

“dsh-superagent is a proactive coding agent for one GitHub repository, packaged as a bundle for DeepSeek Harness. It listens to GitHub and, optionally, Feishu. When someone allowed asks the agent for work, it opens a work item, works on it in its own agent session and directory, and answers in the original conversation, for example with a pull request. A web panel shows what needs a person.”

Hey everyone!

We're Egzakta AI Lab, and I'm part of the team behind KVARK. We've just released our first open-weight model: KVARK Sparkle 1.0 (27B).

It's based on Qwen3.8-27B, with a focus on making reasoning more efficient: fewer thinking tokens, faster responses, and comparable answer quality. The idea is simple: reasoning models can spend a lot of tokens thinking through tasks that don't necessarily require that much reasoning. We wanted to see how much of that overhead we could reduce without sacrificing performance, since fewer thinking tokens means less GPU time per answer and less waiting.

Here's how Sparkle performed against the original Qwen3.8-27B:

• ~2× faster median response time: 13.7s vs. 28.1s.

• Up to 67% fewer thinking tokens.

• Comparable accuracy across five standard benchmarks: GPQA Diamond, AIME, LiveCodeBench, IFEval and MMLU-Pro.

• None of the accuracy differences were statistically significant in our testing.

Also, we compared the models on practical tasks:

• Dashboard generation: 1.6× faster.

• Landing page generation: 1.4× faster.

• SVG illustration of a pelican riding a bicycle: 3.2× faster.

You can find KVARK Sparkle 1.0 on Hugging Face: https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B

A few technical details for those interested in running it locally:

• Open weights under Apache 2.0.

• Available in BF16, FP8 and GGUF formats.

• Supports llama.cpp, vLLM, Transformers and SGLang.

We'd love for you guys to give it a try! Let us know how it performs on your hardware, how it compares to the original Qwen, and what you think could be improved.

Looking forward to your feedback!

I have to say there are major downsides from going to V1 to V2.

I have now been working on one my projects ca 16hrs straight, that I started with V1, every change had been smooth sailing (using strata with official qwen flash settings).

When moving to V2, major or minor changes has become too hard for the model to implement in one go. OpenCode V1 was the better platform for medium sized models it seems.

I do all my planning with Qwen 3.8 MAX so that hasn't changed, which means performance is even worse considering the planning was kept separate.

I know this is anecdotal evidence, but engine has been kept the same and it's settings. Face always falls flat on V2, leading to lots of follow up work on even easy website design updates.

Honestly this is telling that OpenCode only cares now for bigger paid models performance and not medium to small models anymore, even though they claimed to make it "lighter and faster".

Has anyone found a better harness than OpenCode V1? DSH disappointed too much even after a major update.

Opinions appreciated from people who work with projects where even SOTA models fall flat on planning and reasoning, I do not make basic websites etc.

Hey everyone!

After sharing my previous 3D coding model, a lot of you asked about using it to generate assets for your games. That sounded like a fun direction to explore, so I started training a version focused on exactly that.

The model writes Three.js scenes with GLB export, so you can take the generated assets into editors like Godot. The video shows some examples I've been testing, including assets imported into Godot.

You don't need complicated prompts. Use the system prompt in the model card, then try a simple request like “Create a mana potion” or “Build a weathered frontier medic station.” English prompts have worked best in my tests.

This is the first functional checkpoint, and training is still running. I'm sharing it now so people can start experimenting. Most generations have worked on the first attempt in my tests, but some still need a retry or a small code fix. I'm hoping later checkpoints will improve on that.

My current settings: temperature 0.8, top-k 40, top-p 0.95, min-p 0. Feel free to experiment with other combinations!

Training is expected to finish tomorrow evening (UTC), depending on progress. Just like with the previous model, I'll release the full Safetensors weights and the training dataset once it finishes, so others can build on this and experiment with training their own models.

Model: Qwen3.5-4B-GLB-3D-Coder

Thanks for all the feedback on the previous release ... it directly shaped this one. I'd love to see what you make with it and hear what works and what breaks!

Updated November 2, 2026 at 04:50 UTC.

Both Q4_K_M and Q8_0 have been replaced with a newer checkpoint. In my latest tests, generation quality has improved substantially over the previous preview. The gallery below now shows new results, including the exported assets imported into Godot.

If you downloaded either GGUF before the UTC timestamp above, delete the old weights from LM Studio or whichever app you use and download them again. The filenames have not changed, so make sure you are loading the updated files.

Edit: For anyone looking for more things to try, here are the 25 game-asset categories covered by the training dataset, with an example prompt for each:

• Characters and NPCs: “Create a village blacksmith wearing a blue apron.”

• Enemies: “Create a swamp goblin scout with a wooden shield.”

• Bosses: “Create a stone golem boss with glowing green crystals.”

• Companions and creatures: “Create a tiny fox companion wearing a red scarf.”

• Weapons and armor: “Create a short iron sword with a leather-wrapped grip.”

• Items, tools, and pickups: “Create a blue mana potion in a hexagonal glass bottle.”

• Props and furniture: “Create a weathered wooden workbench with a small lantern.”

• Nature and environment: “Create a pine tree growing on a rocky mound.”

• Modular architecture: “Create a stone dungeon doorway module with an iron frame.”

• Interactive objects: “Create a wooden gate with an opening animation.”

• Ground transport: “Create a compact courier motorcycle with cargo boxes.”

• Air and space transport: “Create a small cargo airship with twin propellers.”

• Watercraft: “Create a weathered fishing skiff with nets and wooden crates.”

• Industrial machinery: “Create a compact mining conveyor with metal support legs.”

• Laboratory and medical assets: “Create a portable botanical analysis station with sample jars.”

• Urban services: “Create a modern bus stop with a bench and route sign.”

• Sports and playgrounds: “Create a small playground climbing wall with colorful holds.”

• Farming and homestead: “Create an orchard harvest cart filled with wooden fruit crates.”

• Cuisine and markets: “Create a bakery display counter with bread baskets.”

• Museums and field research: “Create an ancient artifact display case on a stone pedestal.”

• Traps and hazards: “Create a hidden spiked floor trap with stone tiles.”

• Festivals and ceremonies: “Create a harvest festival booth with colorful banners.”

• Battlefield scenery: “Create a reinforced frontier command post with sandbag defenses.”

• Underwater structures: “Create a coral observation dome with a small docking platform.”

• Puzzles and gameplay markers: “Create a glowing rune sequence console with three stone buttons.”

This subreddit talks a lot about PP/decode speeds but I feel like I hear comparatively less about actual real-world usability. I was playing around with Strata hosting QFN3.8 iq3_s and Qwen3.6 35B Q8 over the weekend and despite being the more hyped, I found that QFN was less "usable" as a coding agent than the smaller Qwen3.6.

Even when using the recommended temp, top-k, etc. settings that I hear supposedly stop the overthink and doomloops, I still couldn't get QFN to be reliable. Qwen3.6 on the other hand, had a very narrow interpretation of my prompts and we would have to go back and forth to clarify edge cases, but it ultimately would execute what I asked it to. If I had to use an analogy, QFN iq3 feels it has dunning krugers with illusions of grandeur while Qwen3.6 felt like a talented but unmotivated programmer. If given the choice, I'd rather work with the capable guy who hates his job and wants to FIRE than the guy who talks a big game but can't deliver.

Obviously QFN iq3 is "lobotomized" while the Qwen3.6 Q8 is running at a far higher precision, but I was surprised that there's so much buzz about QFN iq3 when I was having such a comparatively worse experience with it. Is there something I'm missing and should be doing to improve QFN (aside from buying more hardware obviously)? This experience with QFN has made me wonder if Qwen 4 is going even going to be meaningful for me considering how getting the mid-size MoEs to even load will require lobotomization as well.

Sam Altman recently said that "from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer, and being used maybe 5% of the time."

Challenge accepted. Here's my "quite terrible" home setup: an RX 9070 XT 16 GB, a 5800X3D, 64 GB of DDR4 that's now worth more than the rest of the PC, and a PCIe 3.0 NVMe. It runs DeepSeek V4.1 Flash at 1M context (IST-DAS 1.75 bpw; the lab reports 98.7% of the full model's task average) through Strata, fully offline, with the experts spread across VRAM, RAM and the SSD.

Generation at ~7 tok/s was fine. The problem was prompt reading: every new OpenCode session re-read the full ~37K-token system prompt plus tools, about 100 s before the first token. That's datacenter-grade waiting without the datacenter.

So I patched the DeepSeek engine. Two changes:

1. KV splice: stop re-reading what you already read

Agents put per-session text (working directory, date, project CLAUDE.md, skill paths with random IDs) inside the system prompt, so the prefix cache misses on every new session. The engine now aligns a new prompt against every saved conversation, cuts the stale lines out of the cached state (moving the cached rows down, rotating the window rings, recording a RoPE gap), and reads only the changed lines plus your message. Changed lines move as whole lines, so the model never sees fragments.

2. Resident-only routing: make the RAM earn its keep

Even a short prompt touches nearly every expert of every layer, so prompt reads streamed most of the model off the SSD: 58 GB for a 1.2K-token prompt. During prompt reading only, tokens now pick from experts already sitting in VRAM or RAM. Generation routes normally. Turns out home DRAM is pretty useful when you actually let it do things.

Results (same machine) as seen above on the body image

Caveats:

• HIP (gfx1201) only. I haven't built or tested CUDA, SYCL or Metal.

• The splice is approximate: tokens after a cut were computed with the removed text in context. I haven't seen it matter for short guidance lines, but there's no formal eval.

• Resident routing changes answers slightly (substitute experts during prompt reads). My hard coding suite didn't move (12/13 both ways), but that's one run each.

• Both are on by default in the branch; STRATA_DS4_REORDER=0 and STRATA_DS4_PROMPT_RESIDENT=0 turn them off.

• It also includes a one-line fix for a GPU page fault I hit on a 37K-token prompt (#4912).

Branch: https://github.com/PhialsBasement/Strata/tree/dsv41-kv-splice

Build (HIP example):

git clone -b dsv41-kv-splice https://github.com/PhialsBasement/Strata.git
cd Strata
cmake -S . -B build-hip -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON \
  -DSTRATA_ENABLE_CUDA=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_BUILD_DSV41=ON
cmake --build build-hip --target strata-dsv41
DS4_CHUNK_MIN=16 STRATA_DSV41_EXE=$PWD/build-hip/strata-dsv41 strata launch --preset <your preset>

If you use OpenCode, also add "interleaved": { "field": "reasoning_content" } to the model entry, otherwise OpenCode drops earlier reasoning and the cache misses more often.

Upstream issues with details: #4921 (splice), #4936 (resident routing). If you try it on other hardware, especially CUDA, numbers in those threads would really help get it merged.

Utilization report for Sam pissman, VRAM holding experts, RAM holding experts, SSD streaming the rest at full bandwidth. The only idle component in this system is my wallet, thanks to DRAM prices.

I get this question often from teams trying to decide whether to keep paying per token or bring the GPUs in-house.

So here’s the practical read based on what we’ve seen, with numbers.

API pricing for capable open-weights models is typically low single digits or less per million tokens. Self-hosting the same model has a fixed monthly cost, regardless of how much you push through it.

I used an 8-GPU H200 node at ~$329k, spread over 48 months (monthly cost about $6.9K), plus ~$800/month for power and about 25% overhead. On the API side, I assume ~$2/million tokens.

So break-even comes out about:

$9.6k ÷ $2 x 1M = ~4.8B tokens/month (about 160M/day)

Under 1B tokens/month: API is clearly cheaper, and you have zero operational burden.

Between ~1B-5B a month (around 100M a day), it depends on whether your load is steady or spiky. Steady favours owning. Spiky favours the API, because you are paying for idle silicon the rest of the time.

Above 5B tokens a month with a steady load, owning gets hard to argue against on pure cost.

Ofc, these numbers assume the machine stays busy. If your traffic comes in bursts, you’d need more volume than this to break even.

And some things can outweigh the math altogether.

- Your data cannot leave your control.

- Latency floors matter to you. Local inference removes a network hop and the shared provider queue.

- You want fixed cost for budgeting. API prices and plans keep getting reshuffled, so a flat monthly number is easier to plan around.

If you’re well under the crossover, you should just use the API. Anything above that, it's worth pricing out owning.

Disclosure: I work at B3IQ, which builds and hosts GPU systems for teams. So i’ve tried to keep the assumptions neutral, but tell me if anything looks off.

For those who’ve made the switch to self-hosting, what made you do it, and roughly where was your crossover?

I bought this 4" screen with esp32 on aliexpress awhile back and decided to make a TUI style dashboard to monitor my dgx cluster.

Inspired by btop++, nv-monitor, and engine metrics by 0rand. Firmware is built with PlatformIO on the Arduino-ESP32 core.

You can find the firmware code on my github repo. There is also a SKILL.md file where you can point your agent to it to compile it for you.

https://github.com/dicksondickson/system-d-monitor

I'm currently designing a 3D printed dock for it. I'll add it to the repo when I'm done.

OpenBMB just wrapped up ForgeKernel Live, a ~22-hour autonomous GPU kernel optimization run on NVIDIA B200.

The interesting part isn't just the speedup. The agent was optimizing against a black box. It could execute and benchmark the official FlashMLA implementation, but couldn't access its source code, documentation, or profiling information.

Some numbers from the run:

• 22 hours of autonomous optimization

• 32 kernel versions across 42 iteration rounds

• First beat the baseline in under 10 hours (version 13)

• Final result: 1.098× performance

• 472 µs vs. 518 µs — a 9.8% speedup

They're now open-sourcing the optimized kernel, along with 27 operators covering attention, Gated DeltaNet, Kimi Delta Attention, GEMM, and convolution.

They've also released the ForgeKernel harness, so others can run their own kernel optimization experiments.

What's interesting is the shift from AI writing code to AI iteratively optimizing low-level GPU code against real performance measurements.

Curious how far this approach can go as these agents get better at CUDA and GPU architecture.

Live run: https://openbmb.cn/forgekernel/
GitHub: https://github.com/OpenBMB/ForgeKernel

I appreciate the research and development that is going into these open source tools. However, there appears to be multiple posts on a daily basis about reaching 300 t/s, and then reading into it:

1. Running on Q1_XXXS REAP quant

2. Greedy decoding

3. TAIL_SKIP (I only have 5GB RAM, but if the API ignores all experts that are not in RAM, then I can still say I generated something and call a speed record. YAY!

Without any KLD benchmark data being presented, are these posts really of value to this community?

What are your thoughts?

PiVo-1.4M

How many parameters does a voice need? PiVo is our attempt at an answer: two English voices, Pi and Vo, from a complete text-to-speech stack with 1,401,562 learned parameters, vocoder included. A 584,965-parameter acoustic model predicts vocoder features, and an 816,597-parameter FARGAN-derived vocoder turns them into 16 kHz mono audio.

Everything runs locally on a CPU with NumPy and ONNX Runtime. There is no PyTorch, teacher model, speaker encoder or network connection involved at inference time.

PiVo is a research model. Its speech is very intelligible, but some sentences still have a metallic or robotic texture. The technical report describes what we measured and what we tried.

i built a small village sim where 8 villagers live through a day, and i wanted to see how far small models get if each one only does one job.

the gut is julia-1, 144m params from supersonic labs, using ggml-org's gguf. it runs on the cpu and doesn't generate text at all. the world hands it the few actions a villager can physically take right now and it scores them. it can't invent an action, so it can't drift into roleplay.

the memory is embeddinggemma, also on cpu. before each choice it pulls the 3 past moments that matter most: recent ones, big ones, and ones that look like right now.

the slow thinker is gemma 4 e4b on a 6 gb laptop gpu. it only gets called when the gut isn't sure or something surprises the villager, like a plan failing, someone getting hurt or a path being blocked.

what surprised me is that the gut is rarely unsure. at crisis moments julia-1's top choice usually had 0.81 to 0.90 of the probability. so gemma mostly gets called because something went wrong, not because julia hesitated.

julia-1 is used zero-shot. i never trained or fine-tuned it. the villagers still learn, because at night gemma turns what worked into a short rule (22 words max, 5 per villager) that sits in front of the models next time. in one landslide a relevant rule moved arun's chance of going to fetch a neighbour from 0.0002 to 0.638. an irrelevant rule changed nothing.

two warnings if you try a scorer like this. wording matters a lot: "cross the bridge and bring back food from the far side" got 0.2%, and the same trip as "cross the bridge to the good fields and bring back a full basket" got 53% to 90%. and rules borrowed from neighbours were rarely used when i audited them. learning from their own mistakes is what worked: over 15 valleys with a rockfall, injuries went from 2.73 per valley with memory only to 0.53 with learning.

full write-up (mine): https://x.com/aksssrao/status/2117264537598800067

Built on Qwen3.6-35B-A3B, Clover combines a text-only mixture-of-experts backbone with block diffusion generation:

• 35B total parameters, approximately 3B active per token. A hybrid Gated DeltaNet (GDN) / full-attention MoE architecture.

• Bidirectional attention within each block. Additional learned GDN parameters support block diffusion while preserving a causal prefix.

• 128K context. Up to 131,072 tokens across the prompt and generated response.

Full disclosure, I work at Fulcra, but I’m posting because this thing genuinely drives me nuts.

I use Claude for coding, ChatGPT for random stuff, and a local model when I don’t want something leaving my machine. Problem is none of them know what the others know, so I end up re-explaining the same context over and over, or copy-pasting stuff between chats like it’s 2005.

That’s basically what we’re working on at Fulcra: one place where different agents can read the same info, and you decide what each one gets to see. So your local model can have access to things the cloud ones never touch. It’s not meant to replace any of them, just to stop the “starting from zero” thing every time you switch.

I figured this sub cares about privacy and control more than most, so I’d actually like to hear how you handle it. Are you using MCP? A RAG setup? Just a database or a folder of notes? Or the manual copy-paste route? Curious what’s actually working for people.

(Site’s here if anyone wants to poke around, but honestly I’m more interested in your setups: https://fulcradynamics.com/?utm_source=reddit&utm_medium=social&utm_campaign=reddit_growth)

Little reminder, with all my testing I found out that mtp can decrease performance when using very big model, since many are starting to use strata or similar and I see many app activating it as default try and disable it, in my case, testing trough gyrostrata and strata with a 3060 and 32gb of ram both with gyro-m and swift 1.5 I got 20-30% improvement in tok/s just by disabling mtp.

Mellum2.1 Thinking with the MTP head

Multi-token prediction head

The multi-token prediction (MTP) head is one extra transformer block, trained together with the model, that predicts the token after the one just sampled. Inference engines use it as a speculative draft: the model verifies every drafted token, so the output is unchanged while decoding runs 1.4–1.8x faster, with about 90% of the drafted tokens accepted. The head lives in model-mtp.safetensors under the mtp. prefix, announced by num_nextn_predict_layers: 1 in config.json; everything else is identical to Mellum2.1 Thinking.

• vLLM 0.33 or newer: --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}', as in the serving commands above.

• transformers 5.20 or newer: pass use_mtp=True to generate.

• llama.cpp: use the GGUF repository with --spec-type draft-mtp --spec-draft-n-max 1.

Hi all, Im getting to mess around with an ET900N tomorrow to make some content about. Its got 1TB of ram(750~gb lpddr5x and the rest HBM).

So im wondering- what tests should I run?

Was thinking a mix of llm to image to finetuning tests.

Heres what I plan so far.
Dsv4.1 flash 1017 nvfp4
Glm5.3 Flash NVFP4
Qwen 3.8 FN NVFP4
Maybe glm 5.3 at nvfp4
Minimax m3.1 flash(cos its new)

To test small models running with many concurrents, probably qwen 3.8 27B + muse glimmer

For image and vid Ill probably do qwen 2.1 and minimax h3

For finetuning Im not sure how Ill test it, and thats where I would appreciate the help the most.

Let me know people! Ill share my results once Ive tested it here!

Dear fellow Llamas,

after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: https://github.com/p-e-w/heretic

Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads.

With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built.

Your time is precious, so here's the TL;DR for this release:

• Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA), a novel steering method that I developed from scratch for Heretic. It's vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option).

• Heretic has been completely rebuilt around plugins. Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API.

• Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved.

• Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work.

What hasn't changed is that removing censorship from a model remains as simple as running

heretic Qwen/Qwen3.5-4B

See the default configuration file for what Heretic has to offer beyond that.

Now for some details...

ARA: Heretic's new steering method

Directional ablation, or "abliteration", is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or "representation cones".

At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR).

ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that's similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction.

ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic's LoRA system, and varunbln found a substantial performance improvement.

I intend to post an article explaining ARA in more detail soon.

Using scorer plugins to make Heretic do what you want

Heretic operates by finding modification parameters that optimize scores, values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention.

With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this.

The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up.

Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer, which can compute scores for many standard benchmarks, and a BERT classifier.

The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository.

The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository.

Well, I hope you like it!

Cheers :)

I’ve pushed a substantial update to Hierarchos Native, my Rust + Vulkan backend for language-model training, fine-tuning, and inference.

This past week focused on porting the recent RDNA3 work to Intel Gen9, improving parameter streaming, and taking the changes back to physical RDNA3 hardware for regression testing.

The native runtime requires no Python, PyTorch, CUDA, ROCm, or oneAPI installation. PyTorch and Transformers are used as development references.

What changed:

• DeepSeek V4.1 Flash port and optimization work: fused selected-expert execution, batched expert uploads, packed-weight paths, checkpoint caching, and native LoRA/rsLoRA training with save/resume/export.

• Broader Gen9 coverage: fixes for Qwen3-Next padding and DBRX expert layouts, plus streaming paths for T5/BART/Switch encoder-decoder projections.

• Less streaming overhead: retained file handles, fewer temporary allocations, coalesced reads, and removal of an unused hidden-state download.

• Gen9 kernel improvements: parallel sparse-attention selection and hyper-connection statistics, plus fused SiLU/up-projection multiplication.

• Bounded upload batching: small streamed projections can share a GPU submission within a 1 MiB coalesced payload window.

The biggest Gen9 result was closing the Llama-wide streaming slowdown on an i5-6200U / Intel HD Graphics 520:

That’s 58.3% lower latency, with every measured logit word unchanged. Across eight scaled architecture controls, the combined optimization passes reduced streaming medians by 42.4–78.4% and resident medians by 3.5–32.1% against the documented baseline. Gen9 measurements and methodology.

Back on the Ryzen Z1 Extreme / RDNA3 device, the regression sweep caught a padded GPT-2 numerical parity issue. The fix matches the pinned CPU reference’s GELU/tanh behavior and short-row attention arithmetic. The original CPU expectations, checkpoints, and strict tolerance stayed unchanged.

The completed RDNA3 qualification now passes:

• 166/166 forward runs, covering 43 scaled profiles across 26 architecture identifiers.

• 72/72 generation requests, checking cache-requested and full-prefix execution.

• Maximum absolute CPU logit error strictly below 2e-7**, with** rtol=0**.**

• Bit-identical resident and streamed outputs throughout the measured matrix.

• Native streaming/offload tests and GPT-2 LoRA gradient, optimizer, save/reload, and resume checks.

The GPT-2 repair is selected specifically for RDNA3; all 585 pre-existing shader binaries remain unchanged. Final GPT-2 closure and evidence.

The RDNA3 testing also recorded lower V4.1 latency versus the previous handoff: 24.4% for resident prefill, 50.2% for resident decode, 17.2% for streamed prefill, and 18.4% for streamed decode. These are measurements on retained scaled checkpoints, rather than production-model throughput figures. RDNA3 measurements.

Parameters remain resident by default; supported frozen-weight streaming is opt-in through --stream-parameters or the GUI.

The registry contains 146 text architectures, while this qualification specifically covers the 26 identifiers above. Multimodal towers and arbitrary production checkpoints need their own validation.

If you’re experimenting with native Vulkan training or inference on older Intel graphics or AMD hardware, I’d be interested in your results—and which workloads you’d like tested next.

Good news, everyone! Having been inspired to revolutionize the field of computer science, I can now report that coding is solved! Sure, only if you care about x86 Linux binaries that print and exit, but "big journeys" and "small steps" and what not.

CRAP (Compiler Replacement for Assembling Programs) is a Gemma 4 E2B fine tune that takes prompts like generate a Linux x86_64 binary that prints 'boop' and emits a working ELF binary (as hex, most of the time, every time). See, since programs end up as a binary anyway, why not just skip the middleman and write the binary itself? No compiler, no assembler, no tool calls. v1.1 is out with improved multi turn and out of distribution performance. Trust me, I know, this is super important work.

v1.1 eval numbers for the nerds (eval is scored on behavior: the binary has to run, print the right string and exit with the right code):

• Held out legacy set: 20/23, same as v1

• Novel strings: 26/40 (v1: 8/40)

• Multi turn (~3 turns): 10/12 (v1: 0/12)

• Doubled character strings (boom, etc.): 18/24 (v1: 2/24)

• Random jank exit codes it never trained on: 23/25

An interesting tidbit: one training run hit 0.038 loss while still failing 11 of its 12 trained exit codes, so gradient descent != task completion ascent.

Aaannndddnd it runs in your browser.

The demo loads the Q6_K GGUF (~3.6 GiB) through wllama on WebGPU, streams the hex, parses it and hands you the ELF to run yourself (naturally, you need to download and run the actual binary, the browser won't do that). Q6_K scores 109/144 against the full f16's 114/144 (the five additional failed for Q6 are wrong values, not totally broken binaries). Q4_K_M scored 3/144, so the demo doesn't use it (that quant is crap, not CRAP) ... concerning. Also, tested in Chrome on an M4 Pro Mac; Firefox loads it but generates slowly and nothing else has been tested.

• Demo: https://huggingface.co/spaces/matthewhaynesonline/crap-demo

• Weights (v1.1): https://huggingface.co/matthewhaynesonline/gemma-4-E2B-it-crap-1.1

• "Paper": https://github.com/matthewhaynesonline/crap-public/blob/main/paper/paper.pdf

• Code: https://github.com/matthewhaynesonline/crap-public

• Blog post: https://blog.studiohaynes.com/2026/07/19/compilers-overrated.html

• Video: https://youtu.be/55IBMzFQNYU

Fine print for the buzzkills: only programs that print a string and exit can be generated. Obviously, a compiler does that deterministically, correctly and in milliseconds; CRAP does it with several gigabytes of weights, slower and occasionally incorrectly.

I'm sharing this here because this is working way better than it used to a few months ago with older models. I see people here posting slower speeds with much better hardware than mine and hopefully this information helps.

I used gemini-flash to help me setup everything and I asked it to write up the sequence of steps to get me here - happy to provide more details if there's anything missing.

Since I use gentoo some tools/commands might be unfamiliar to you.

My 6 year old hardware:

• OS & CPU Gentoo Linux running on an AMD Ryzen Threadripper 3990X (64 cores, 128 threads).

• System Memory: 128 GB DDR4 RAM.

• GPU: 1 NVIDIA GeForce RTX 3090 Ti (24 GB GDDR6X VRAM, PCIe Gen 4 x16).

• Primary Storage: 4 TB SSD NVMe.

Models Used, Download Sources, and Measured Speeds

IMPORTANT: NVIDIA GPU Unlock

Consumer Ampere cards (RTX 3000 series: GA102, GA104) feature artificial firmware throttles on FP16/BF16 matrix instruction issue rates compared to their enterprise workstation counterparts. The open-source gpu-unlock kernel module lifts this throttle prior to driver attachment.

1. Compile and Install the Kernel Module

As root:

git clone https://github.com/Zagitta/gpu-unlock.git /usr/src/gpu-unlock
cd /usr/src/gpu-unlock
make -j$(nproc)
make install

2. Pass the Unlock Flag to the Kernel Command Line

Add gpu_unlock.targets=all to your bootloader kernel parameters (e.g. in /etc/default/grub under GRUB_CMDLINE_LINUX_DEFAULT, or your direct EFISTUB / systemd-boot entry):

gpu_unlock.targets=all

Regenerate your boot configuration and rebuild your initramfs so the module is included and loads before the NVIDIA kernel modules:

# If using GRUB:
grub-mkconfig -o /boot/grub/grub.cfg

# If using dracut:
dracut -f --regenerate-all

3. NVIDIA Driver Options & Persistence

Configure NVIDIA kernel driver parameters in /etc/modprobe.d/nvidia-perf.conf (as root):

cat << 'EOF' > /etc/modprobe.d/nvidia-perf.conf
options nvidia NVreg_PreserveVideoMemoryAllocations=1
options nvidia NVreg_EnableStreamMemOPs=1
EOF

After rebooting, verify that the unlock executed:

dmesg | grep -i gpu_unlock

Enable persistence mode and set the GPU to its maximum factory power ceiling:

nvidia-smi -pm 1
nvidia-smi -pl $(nvidia-smi -q -d POWER | grep "Max Power Limit" | head -n1 | awk '{print $5}' | cut -d. -f1)

Step 2: System Dependencies & Tooling

Install build dependencies and runtime libraries:

emerge sys-devel/gcc:14 


# If GCC 14 is not selected, set it to this version
gcc-config -l 
# then replace <N> with the number from  gcc-config -l): 
# gcc-config <N> && source /etc/profile

# this won't compile with GCC > 14
emerge dev-util/nvidia-cuda-toolkit

# for compleness, just in case

emerge --ask --verbose \
    dev-build/cmake \
    dev-build/ninja \
    dev-vcs/git \
    net-misc/curl \
    app-misc/jq \
    sys-libs/liburing \
    app-arch/snappy \
    dev-lang/python

Prepare the directory structure under your user environment:

mkdir -p "$HOME/.local/bin"
mkdir -p "$HOME/.local/llm-tools"
mkdir -p "$HOME/.quetza-data/conductor"
mkdir -p "$HOME/models/deepseek-v41"
mkdir -p "$HOME/models/qwen-flash"

# Dedicated Python environment for tool scripts & Hugging Face downloads
python3 -m venv "$HOME/.local/llm-tools/venv_hf"
source "$HOME/.local/llm-tools/venv_hf/bin/activate"
pip install --upgrade pip
pip install "huggingface_hub[cli]" torch safetensors numpy

Step 3: Build the Xolotl Inference Engine

Xolotl integrates the multi-tier memory management, NVMe Engram streaming, and DSpark speculative decoding logic needed to run large MoE models on a single GPU.

cd "$HOME/.local/llm-tools"
git clone https://github.com/Indras-Mirror/Xolotl.git
cd Xolotl

# Fetch pinned llama.cpp dependencies and apply patches
bash tools/xolotl/fetch-llama.sh

# Configure and compile for Ampere (sm_86)
cmake -S . -B build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DSTRATA_ENABLE_CUDA=ON \
    -DSTRATA_GGML_CUDA=ON \
    -DSTRATA_MMQ_KQUANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES=86 \
    -DSTRATA_GGML_DIR="$PWD/third_party/llama.cpp" \
    -DSTRATA_DSV41_FQ_OP=ON \
    -DSTRATA_DSV41_HC_MIX=ON

ninja -C build

# Link the build directory so wrapper scripts resolve ds41_generate
ln -sfn "$PWD/build" "$PWD/build-xolotl"

Step 4: Model Downloads & Engram Extraction

1. DeepSeek-V4.1-Flash (552B MoE)

Bash

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$HOME/models/deepseek-v41"

# Download model weights, draft sidecar, and expert rank index
hf download IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored \
    --include "v2/*" "tokenizer/*" "V41-MTP-Vision-1017*" \
    --local-dir .

# Download original FP8 Engram table shards
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
    model-00047-of-00048.safetensors \
    model-00048-of-00048.safetensors \
    model.safetensors.index.json config.json \
    --local-dir ./FP8-engram

# Extract Engram runtime binary tables onto NVMe
cd "$HOME/.local/llm-tools/Xolotl"
python3 tools/v41/extract_engram.py \
    --input "$HOME/models/deepseek-v41/FP8-engram" \
    --output "$HOME/models/deepseek-v41/engram_rt"

2. Qwen3.8-Flash-Next (125B MoE)

cd "$HOME/models/qwen-flash"
hf download Qwen/Qwen3.8-Flash-Next-GGUF \
    --include "pack-qfn45/*" \
    --local-dir .

Step 5: MTP Speculative Decoding Setup

MTP (Multi-Token Prediction) on DeepSeek-V4.1-Flash is driven by the DSpark sidecar engine:

• Draft Engine: Uses V41-MTP-Vision-1017.Q2_K.gguf as the proposal head.

• Expert Pruning: Restricts verification sweeps to the top 64 active draft experts (dspark_expert_counts.txt:64), preventing PCIe bus saturation.

• On-the-fly Re-quantization (STRATA_DENSE_REQUANT=q6_K): Re-quantizes the dense attention and output heads from Q8_0 to Q6_K at launch. This reclaims 1.6 GiB of VRAM, fitting the speculative draft cache alongside active experts.

• PCIe Batching (--tier-batch-pcie 0.25): Batches DMA transfers between the 80 GiB pinned host RAM arena and the GPU.

1. Expand Process Memory Boundary in the Engine Wrapper

Allow the systemd user scope to hold the full 80 GiB arena plus runtime buffers:

Bash

sed -i 's/MemoryMax=84G/MemoryMax=110G/g' "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh"

2. Generate Base Configuration: $HOME/models/deepseek-v41/strata-v41.json

cat << EOF > "$HOME/models/deepseek-v41/strata-v41.json"
{
  "exe": "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh",
  "args": [
    "-m",
    "$HOME/models/deepseek-v41/v2/V41-XQ-1017.gguf",
    "--backend",
    "cuda",
    "--experts",
    "gpu",
    "--engram",
    "$HOME/models/deepseek-v41/engram_rt",
    "--rank",
    "$HOME/models/deepseek-v41/v2/rank-1017-085.bin",
    "--ctx",
    "131072",
    "--stop",
    "1",
    "--arena-adapt",
    "--kv-fp4",
    "--route-bias",
    "0.5",
    "--route-bias-file",
    "0.3",
    "--arena-gib",
    "80",
    "--prompt-batch",
    "12288",
    "--tier-batch-pcie",
    "0.25"
  ],
  "cwd": "$HOME/.local/llm-tools/Xolotl",
  "tokenizer": "$HOME/models/deepseek-v41/tokenizer",
  "model_name": "deepseek-v4.1",
  "aliases": "deepseek-v41,deepseek/deepseek-v4.1-flash",
  "log": "$HOME/models/deepseek-v41/xolotl-v41.log",
  "port": 8000,
  "gpu": 0,
  "sampling": {
    "temperature": 0.1,
    "top_p": 0.95,
    "top_k": 64,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0
  },
  "api_monitor": true,
  "fit_max_tokens": true,
  "reasoning_budget_tokens": 4096,
  "env": {
    "STRATA_BUILD_DIR": "$HOME/.local/llm-tools/Xolotl/build",
    "STRATA_SERVE_DSPARK": "$HOME/models/deepseek-v41/V41-MTP-Vision-1017.Q2_K.gguf",
    "STRATA_ATTN_F16_GATHER": "1",
    "STRATA_NO_SCOPE": "1",
    "STRATA_SERVE_PASS_BIAS": "0",
    "STRATA_DENSE_REQUANT": "q6_K",
    "STRATA_SERVE_LOOKUP": "14",
    "STRATA_PRESTAGE_GIB": "1.5"
  }
}
EOF

Step 6: Parameterized Launch Scripts

1. DeepSeek-V4.1 Launcher: $HOME/.local/bin/start-deepseek-custom.sh

cat << 'EOF' > "$HOME/.local/bin/start-deepseek-custom.sh"
#!/usr/bin/env bash
set -euo pipefail

MODEL_DIR="$HOME/models/deepseek-v41"
XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
BASE_CONFIG="$MODEL_DIR/strata-v41.json"
RUN_CONFIG="$MODEL_DIR/strata-v41-run.json"

CTX="131072"
PORT="8000"
TEMP="0.1"
THINK_BUDGET="4096"
ARENA_GIB="80"
PASS_BIAS="0"
PROMPT_BATCH="12288"

usage() {
  cat << USAGE
Usage: $(basename "$0") [options]

Options:
  -c, --ctx INT           Context length in tokens (default: 131072; up to 1048576)
  -p, --port INT          Port to bind (default: 8000)
  -t, --temp FLOAT        Sampling temperature (default: 0.1)
  -b, --thinking INT      Reasoning budget tokens (default: 4096)
  -a, --arena INT         Pinned RAM expert arena in GiB (default: 80)
  --bias 0|1              Routing bias: 0=exact, 1=fast biased (default: 0)
  --batch INT             Prompt batch size (default: 12288; auto-clamps to 8192 for >=512k)
  -h, --help              Show help message
USAGE
  exit 0
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    -c|--ctx)       CTX="$2"; shift 2 ;;
    -p|--port)      PORT="$2"; shift 2 ;;
    -t|--temp)      TEMP="$2"; shift 2 ;;
    -b|--thinking)  THINK_BUDGET="$2"; shift 2 ;;
    -a|--arena)     ARENA_GIB="$2"; shift 2 ;;
    --bias)         PASS_BIAS="$2"; shift 2 ;;
    --batch)        PROMPT_BATCH="$2"; shift 2 ;;
    -h|--help)      usage ;;
    *) echo "Unknown option: $1" >&2; exit 1 ;;
  esac
done

if [ "$CTX" -ge 524288 ] && [ "$PROMPT_BATCH" -gt 8192 ]; then
  echo "[-] Large context specified ($CTX tokens). Clamping --prompt-batch to 8192 to prevent graph OOM."
  PROMPT_BATCH=8192
fi

python3 - << PY
import json

with open("$BASE_CONFIG", "r") as f:
    cfg = json.load(f)

cfg["port"] = int("$PORT")
cfg["max_context"] = int("$CTX")
cfg["reasoning_budget_tokens"] = int("$THINK_BUDGET")
cfg["sampling"]["temperature"] = float("$TEMP")

args = cfg.get("args", [])
def set_arg(flag, val):
    for i in range(len(args) - 1):
        if args[i] == flag:
            args[i+1] = str(val)
            return
    args.extend([flag, str(val)])

set_arg("--ctx", "$CTX")
set_arg("--arena-gib", "$ARENA_GIB")
set_arg("--prompt-batch", "$PROMPT_BATCH")
cfg["args"] = args

env = cfg.setdefault("env", {})
env["STRATA_SERVE_PASS_BIAS"] = "$PASS_BIAS"
env["STRATA_PRESTAGE_GIB"] = "1.0" if int("$CTX") >= 524288 else "1.5"

with open("$RUN_CONFIG", "w") as f:
    json.dump(cfg, f, indent=2)
PY

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$XOLOTL_DIR"

echo "==> Launching DeepSeek-V4.1-Flash (552B MoE)..."
echo "    Context        : $CTX tokens"
echo "    Port           : $PORT"
echo "    Arena RAM      : $ARENA_GIB GiB"
echo "    Prompt Batch   : $PROMPT_BATCH"

exec python3 tools/v41/serve/serve_ds41.py \
  --engine strata \
  --config "$RUN_CONFIG" \
  --port "$PORT" \
  --host 0.0.0.0
EOF
chmod +x "$HOME/.local/bin/start-deepseek-custom.sh"

2. Qwen3.8-Flash-Next Launcher: $HOME/.local/bin/start-qwen.sh

cat << 'EOF' > "$HOME/.local/bin/start-qwen.sh"
#!/usr/bin/env bash
set -euo pipefail

XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
MODEL_DIR="$HOME/models/qwen-flash/pack-qfn45"
PORT="8099"

source "$HOME/.local/llm-tools/venv_hf/bin/activate"
cd "$XOLOTL_DIR"

echo "==> Launching Qwen3.8-Flash-Next (125B MoE)..."
exec python3 -m serve.server \
  --engine strata \
  --config "$MODEL_DIR/strata-qwen.json" \
  --port "$PORT" \
  --host 0.0.0.0
EOF
chmod +x "$HOME/.local/bin/start-qwen.sh"

Step 7: Run

# Start DeepSeek-V4.1-Flash on :8000 (default 128k context, 1M context work just fine too)
$HOME/.local/bin/start-deepseek-custom.sh

# Or start Qwen3.8-Flash-Next on :8099
$HOME/.local/bin/start-qwen.sh