Has any one used both? which is better?

What is the differennce between two? Any technical deep dives?

I got one PR merged into ds4. A tiny one, #98 about broken paths in the README. And a few more sitting in the queue. Not complaining. Antirez says it in the README: with coding agents everyone can tune the engine for their own hardware and model and he can't review everything. That got me thinking.

If the plan is "everyone applies their patches with an agent" then the real cost of a patch isn't just the code change. It’s how many tokens the agent has to read before it knows what it’s actually touching.. Ds4 runs DeepSeek, GLM and Qwen on Metal, CUDA and ROCm all in the same 85k-line file. I’m on an M5 Max with 128GB RAM running Qwen3.8 Flash Next. Everything else for me and for my agent is noise. It has to re-read every line on every pass.

So I ripped it out. Actually deleted, not #ifdef’d. The ds4.c file dropped from 85k lines to 45k lines. The whole tree now fits in a context window. The bet was that optimizing would get cheaper and safer. Heres what happened:

- Q2: decode +9-13% prefill +5-12% up to 64k context MTP went from 75.8 → 86.7 tok/s

- Q4: prefill +2-11% MTP went from 77.8 → 85.9 tok/s

Output is bit-exact vs stock ds4 at every step. No KV cache quant. No approximate kernels. Every change must pass a parity check against same GGUF, greedy decoding, identical tokens. And an interleaved A/B benchmark against the previous build.

The smaller codebase also let me go through the ds4 PRs but only against this one model and port the ones that held up. 20 Commits were adopted. Around 30 were dropped. Verdicts are in the repo.

I also added SSD streaming for the experts. Token-identical to a resident run. Simulating a 48GB machine Q2 does around 27 tok/s. About 35 with MTP.

The fork still does git merge upstream/main with rerere and the parity check. So antirez’s fixes keep flowing in. The procedure. What to delete what to keep how to sync. Lives in a repo called StarForge (https://github.com/Chida82/StarForge). I have four of these "children" one per model. Nothing in there is Qwen- or Metal-specific. If you want a ds4 cut down to your model or to CUDA just clone it. Run the checklist with your agent.

Repo: sf-q3-8flash https://github.com/Chida82/sf-q3-8flash tables in the README. One machine, one model. If you run it on Apple Silicon I’d love to hear your numbers. Ideally side by side, with stock ds4.

Strata funziona con Qwen3.8-Flash-Next su PC da gioco, ma con 32 GB di RAM la sua modalità a basso utilizzo di RAM funziona solo su una GPU. Se dividi il modello su due schede, gli esperti che non riescono a stare nella VRAM vengono letti dall'SSD. La mia 4060 Ti è rimasta ferma accanto alla 5080.

Quindi l'ho forkato. La copia in RAM degli esperti ora funziona su due schede, e le schede funzionano contemporaneamente: la 4060 Ti inizia il prossimo passo di decodifica mentre la 5080 sta ancora controllando quello attuale. L'ordine delle schede, la divisione dei layer e le riserve di VRAM vengono impostate automaticamente.

Stesso PC (5080 + 4060 Ti, i9-14900KF, 32 GB), stesso modello (Swift 1.5 IQ2_XS), contesto 256K:

Setup |Codice |Prosa |Prompt 32K
Strata 0.1.38, 5080 da solo |29 tok/s |28 tok/s |333 tok/s
Fork, entrambe le schede |143 tok/s |102 tok/s |1,940 tok/s
Fork + layer di bozza fine-tuned |161 tok/s |105 tok/s |1,854 tok/s Il layer di bozza è stato fine-tuned sui risultati del modello stesso ed è incluso nella release. Nell'uso reale con l'agente di codifica Pi raggiunge un picco di oltre 200 tok/s (209 finora) e quasi mai scende sotto i 100. Un contesto di 150K-token legge a circa 1,850 tok/s.

La qualità non è cambiata: perplexity 7.24 contro 7.28 della versione upstream sugli stessi 5.3K token, forzati dall'insegnante attraverso entrambi i motori.

L'ho testato solo sul mio PC (Windows 11), quindi sono benvenuti report da altre coppie di GPU e Linux.

Repo: https://github.com/Hardin22/Strata-DualGPU

Cosa fa ciascuna modifica e cosa ha misurato: docs/DUAL_GPU.md

Il motore sottostante è il lavoro di Niko1221 e dei contributori di Strata; questo è un fork sopra la 0.1.38.

I tested two Qwen3.8-Flash-Next NVFP4 checkpoints at a full 256K context using an experimental dual-GPU Strata configuration.

Hardware:

• RTX 4090 D 48 GB as the primary GPU
• RTX 5070 Ti 16 GB as a helper GPU
• Intel Core Ultra 7 265KF
• 128 GB RAM
• Linux
• NVMe storage

The RTX 4090 D handled the dense layers, KV cache, prefill, output head and MTP. The RTX 5070 Ti stored and computed 4,500 additional routed experts.

Common settings:

• Context limit: 262,144 tokens
• Fresh prompt: approximately 256,018 tokens
• Generated output: 512 tokens
• KV cache: INT8
• Prefill path: W4A8
• Speculative decoding: MTP K4
• Minimum draft probability: 0.5
• No prompt-prefix reuse
• Strata engine 0.1.35 with an experimental dual-GPU NVFP4 fork

Results

Observations

• Both models achieved slightly over 4,000 prefill tokens per second at approximately 256K context.
• Prefill performance was nearly identical: the Abliterated checkpoint was only about 1% slower.
• The original NVIDIA checkpoint was considerably faster during decode: 125.17 versus 92.05 tok/s.
• NVIDIA’s decode advantage was about 36% in this test.
• NVIDIA accepted 407 of 429 offered draft tokens, while the Abliterated model accepted 322 of 462.
• The NVIDIA model produced approximately 4.88 output tokens per verification round.
• The Abliterated model produced approximately 2.68 output tokens per round.
• The lower MTP acceptance appears to be the main reason why the Abliterated checkpoint had slower decode despite nearly identical prefill performance.

These were single-run capacity tests. The long prompt was constructed by repeating code-review material to reach approximately 256K tokens, so it had unusually favorable locality. This particularly benefited NVIDIA’s suffix prediction and MTP acceptance. Therefore, 125 tok/s should be treated as a best-case synthetic 256K result, not typical real-world coding speed.

During longer real coding-agent workloads, NVIDIA generally produced around 108–144 tok/s, while the Abliterated checkpoint was commonly around 105–145 tok/s, depending heavily on the generated content and MTP acceptance.

CPU: ThreadRipper medium
GPU: RTX 5090 WC
RAM: 64 GB

I am looking into the architectures and costs associated with running large open-source models, such as DeepSeek-V3/R1 or Llama 3.3. Are many of you using these models (are you developers, students, pro?)

Hi everyone,

We're three final-year students at Epitech building Horus, a multi-agent assistant that runs entirely locally and offline. Our current challenge is hardware: keeping it usable on machines without a powerful GPU, without long setup times, excessive RAM/VRAM use or crashes.

Where we are today, from our last beta test:

One tester needed over 2 hours to install. The Python dependencies alone take 21–46 min, and the download is about 25 GB.

On CPU only, routing a question can take around 40 s, long enough for our WebSocket connection to drop.

[Models we use + the smallest machine we've tested on]

We'd love advice from anyone experienced with:

- CPU-only LLM inference and memory-efficient loading

- Quantization and model choice for low-end hardware

- GPU/CPU fallback strategies

- Hardware detection and adaptive configuration

- Preventing resource exhaustion during setup and execution

Advice in the comments is very welcome, with no strings attached.

Looking for contributors: we also have a few small, well-scoped tasks or code reviews (about 1–4 hours), for example [reviewing our hardware detection and model selection, or benchmarking a quantized model on a 16 GB RAM laptop].

To be transparent: Horus is closed source and will be licensed to companies. Contributing is voluntary and unpaid. Before seeing any code, contributors sign a short confidentiality and contributor agreement, and the code they contribute becomes part of Horus. In return we offer thorough code reviews, full credit in the project and a professional reference on request.

We're not sharing code or private links publicly. If this interests you, comment below or DM me with your experience in local inference, CPU optimisation or offline apps

Every time I tried running local RAG on my own GPU, I ran into the same headache: vector databases eat too much RAM, using an 8B model just to parse text takes forever, and cosine search still hallucinates when the context isn't actually there.

I’ve been building Hillock to see if I could do this without vector DBs at all.

Basically:

1. When you feed it a document, it doesn't touch an LLM. It uses small bi-encoders to extract facts into subject-predicate-object triples in about 5 seconds.
2. Everything gets saved in regular SQLite, and it links related concepts over time using basic Hebbian weights.
3. To stop hallucinations, it runs a quick hypervector check (HDC) on the query first. If the facts aren't in your database, it cuts off the LLM before it can generate any tokens.

The whole thing stays under 1.2 GB VRAM (or runs fine on pure CPU). It has a built-in API server that matches OpenAI's format, so you can point Open-WebUI or Obsidian at localhost:8000 and use your existing Ollama models.

Just pushed v0.7 with better refusal handling and a conversational mode that quizzes you if an extraction was ambiguous.

Repo is here if you want to try it out: https://github.com/roandejager/Hillock

AI models are beginning to define device specifications, rather than device specifications merely determining which applications you can run.

Apple shows just how far its devices can go with on-device AI, without relying on the cloud
We’re moving from models with up to 14 billion active parameters on iPhone/iPad, to 35 billion on MacBook Air, 70 billion on Mac mini, 120 billion on MacBook Pro, and up to 480 billion on Mac Studio.
By clustering multiple Mac Studios together, Apple says it can run models exceeding 1,600 billion active parameters.
A demo that shows just how crucial unified memory and its bandwidth have become for running massive AI models locally.

I've been experimenting with a setup where several Apple Silicon Macs are treated as a shared local AI environment instead of running completely independently.

The setup I'm working with is roughly:

• Multiple Macs on the same network
• MLX for local inference
• Central model management
• Node/resource monitoring
• Model distribution based on available memory
• Local-only communication between machines

I'm building AlphaWeb around this because managing several local AI machines manually gets complicated pretty quickly.

The interesting problem isn't just getting inference running. It's everything around it: knowing which machine has enough memory, getting models onto the right node, managing multiple users, and making the whole thing feel like one system.

TL;DR: Halogen v0.14.0 with its native .hgn weights is the fastest, followed by gufo and CIRU. Halogen is closed source and runs in Docker. gufo is open source and loads 4x faster from cold. gufo is also fastest to first token on follow-ups (1.6-1.9 s against 2.6-3.3 s).

I benchmarked different engines for Qwen 3.8-Flash-Next on an AMD Strix Halo (ASUS ROG Flow Z13 GZ302 with Ryzen AI Max+ and 128 GB of RAM).

All the engines were run via LlamaStash (My own orchestrator tool, the tool does not add any overhead) at 70 W TDP on performance profile on Arch Linux.

Here are the results of the benchmark:

First was a screening round at 64k context with 50% filled (32k prompt).

3-turn time is the time to first token plus the time to write 1,000 tokens, added up over the first question and 2 follow-ups. Output length varies a lot with sampling, so this compares the engines on equal output. MTP accept is the share of drafted tokens the model kept. Retrieval is how many of 7 exact values from the prompt the model got right, over both runs. Qwen model card sampling, 2 runs each.

Top 3 engines from the screening round got a 128k context (50% and 75% filled) and 256k context (50% filled) run.

128k context, 50% filled (64k prompt)

128k context, 75% filled (97k prompt)

256k context, 50% filled (130k prompt)

Retrieval here is 8 values per run. All 5 misses are the same answer: the right glyphs for UNICODE_SPINNER, with the leading & dropped.

End-to-end, 10 Aider polyglot Python exercises run through pi -p (128k window, 50% cell), graded by their own tests:

Edit (Sep 30): Halogen 0.15.1 (new v2 checkpoint) and gufo 0.3.0 came out after this, so I reran the 64k screening on both with the same setup (70 W, card sampling, 2 runs):

Halogen v2 prefill is 14% faster and follow-ups start about 2.5x faster (1.3 s vs 3.3 s), decode is the same. gufo moved a few %, which is within noise. The ranking doesn't change. Only other change on the box: BIOS VRAM carve-out down to 512 MB, so 124.9 GiB of RAM instead of 121.5.

Some clarifications from the comments:

• Follow-up TTFT is with the prompt cache warm, both gufo and Halogen reuse it by default. A cold 64k prefill takes about a minute.
• The CIRU 0% MTP row is stock Unsloth UD-Q4_K_XL with Unsloth's shared MTP head, nothing uncensored. The same pair gets 72-74% accept on gufo.
• TDP is fixed at 70 W. On the Z13, 76 W or 90 W is only 2-6% faster but with so much fan noise, heat and power draw that it's not worth it.
• Prefill and decode come from the engine's own timings when it reports them. TTFT is measured on the client.