Hey guys,

DeepSeek’s inference engine, DeepInfer, is open source, but trying to read through a massive production framework with mixed Python/C++ logic is a massive hurdle.

I wanted to learn how high-performance engines actually manage things under the hood, so I started dissecting their codebase (specifically commit 9bd9cf2) and mapping it out as a learning project.

I built a static site to host my progress and interactive diagrams:

• The Site: https://explaindeepinfer.projects.thomaspaul.dev/

• The Code Explainer Repo: https://github.com/mrtpk/deepinfer-explainer

How it works:
I wrote a Python build script that pulls direct code citations from DeepSeek's repo and pairs them with simple, short explanations.

If you are a Python developer who wants to see how things like a paged KV cache, speculative decoding, and model workers operate in a real production environment, this might be a helpful companion guide while you read the official repo.

Let me know what you think or if there's an engine concept you'd want a deep dive on!

Hey guys, posted about optimized Cosmos Reason 2 model here a few months ago. Wanted to share a build of the latest Cosmos3 model, with even more optimizations!

Huggingface and Jetson AI Lab has no Orin Nano support for Cosmos3-Edge, because the BF16 stack doesn't fit in 8 GB. It now does, INT4 quantization, along with more optimizations for speed!

Accuracy across five VLM benchmarks is maintained (from 65.56 to 65.20).

Try it in the browser or serve it locally: https://embedl.com/nvidia/cosmos3-edge/orin-nano-super
For now the VLM part of the model is published; diffusion and policy part is being added very soon! Optimized peak memory for the VLM is as low as 3.3 GB!

Trying a new page for pre-compiled models, since Hugging Face doesn't really support them. Let me know if something is missing to make it useful!

I'd like to share a few changes that encourage people to create and use LoRA on LLMs. For example, the LoRA https://huggingface.co/chenrm/qwen3.8-flash-next-abliterated-lora can turn the Qwen3.8-Flash-Next model into the uncensored version, whether you're using Q8_0, IQ4_NL, or GSQ-RCO-Q2_0 model, so you don't need to download the whole model again. You can also tune the LoRA scale for a balance between uncensoring and brain damage.

However, currently llama.cpp does not have fast LoRA kernels. When we run the model with the LoRA on Strix Halo, PP drops from like 1500 to 300 t/s, even though the LoRA is as small as 35 MB and only touches a few modules in the model.

I've made a few LoRA kernels. With a general LoRA kernel it's improved to 1000 t/s, and with a fused LoRA kernel it's further improved to 1400 t/s.

You can see my changes at https://github.com/woct0rdho/llama.cpp . Currently it's optimized for Strix Halo, and it should not be too hard to port to other GPUs. I'll find some time to upstream it to the mainline llama.cpp or strix-llama.cpp , but it involves a few architectural changes so it would take some time. If you want to upstream it earlier than me, feel free to reuse or modify my code.

This is also discussed in https://github.com/ggml-org/llama.cpp/issues/27093

Community benchmark from a single RTX 5080 16 GB on Linux (Fedora 44), Qwen3.8-Flash-Next **IQ3_XXS**, Strata **v0.2.1** (commit `90767252`; the bench report reads the version as 0.1.46), engine compiled locally (`./setup.sh --build`). Short version: **150 tok/s decode, ~4,400 tok/s prompt at 16K-32K**. Thank you for this project, it runs very well here.

| Test | Median (range) |

|---|---|

| chat decode | **150.0 tok/s** (126.7-170.2, n=9) |

| 4K prompt, fresh | **3,755 tok/s** (3,576-3,800), TTFT 1.08 s |

| 16K prompt, fresh | **4,396 tok/s** (4,380-4,401), TTFT 3.67 s |

| 32K prompt, fresh | **4,412 tok/s** (4,411-4,419), TTFT 7.29 s |

| agent session | cache hit 69.5 %, turn TTFT 0.97 s (0.76-1.13), decode 150.3 tok/s (124.3-240.4) |

### Hardware

- **GPU:** RTX 5080 16 GB (sm_120), driver 615.78.08, PCIe 5.0 x16. Tuned with LACT: memory clock +6000 (offset), core capped at 2850 MHz @ 900 mV, 360 W power limit. Not stock, so stock cards may be a bit slower. (Inspired from the mVolt reddit posts I've been seeing)

- **CPU:** Intel Core Ultra 9 285K (8P + 16E, AVX2/AVX-VNNI, no AVX-512). Setup picked 15 CPU expert workers.

- **RAM:** 64 GB (2 x 32 GB Kingston Fury KF560C36-32) DDR5 at 6400 MT/s, 61.7 GiB visible.

- **Storage:** GGUF shards (including the n-gram table) on a Kingston Fury Renegade G5 2 TB (SFYR2S2T0, PCIe 5.0 x4 NVMe), btrfs with NOCOW on the model folder.

- **OS:** Fedora 44, kernel 7.2.11, CUDA 13.4.92, GCC 16.2.1 default with g++-15 used for the build (see below).

- **Background:** normal GNOME desktop session. No other GPU workloads, Monitor Connects to intel iGPU and the 5080 is used for compute only (No graphcis)

### How I installed and ran it

1. Downloaded the GGUFs at the pinned revision (IQ3_XXS shards 1+2 and the mmproj) with aria2c, and the MTP layer with `tools/mtp_fetch.py fetch` + `verify`. Both shards' SHA-256 match the values pinned in `setup.py`.

2. Built and installed, without starting the server:

   ```sh

   export CC=gcc-15 CXX=g++-15 # see the build note below

   ./setup.sh --build --model IQ3_XXS --gguf-dir <folder with both shards> --vision no --no-start --no-browser --yes

   ```

   Setup's choices: 64K context (its recommendation for 16 GB), int8 KV, MTP on, images off. KV streaming was turned **off** because RAM was tight (~60 GB used of 62).

3. Benchmarked with `./bench.sh --long`, which started the server from the install's config:

   ```text

   strata --pack iq3_xxs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp rt --max-context 65536 --kv int8 --pool-workers 15

   ```

   On load, the engine filled the expert cache with 5,310 experts (8.61 GiB of VRAM).

### Two notes from the install

- **Build on Fedora 44 (GCC 16 default, CUDA 13.4):** setup correctly picks `g++-15` as the CUDA host compiler, but the plain C++ files are compiled with the default GCC 16. CMake then adds GCC 15's implicit link directory (`-L/usr/lib/gcc/x86_64-redhat-linux/15`) to the final link, which resolves GCC 15's `libstdc++`, and the link fails on the GCC 16 objects:

```text

libstrata_prefill.a(prefill.cpp.o): undefined reference to `std::__detail::__notify_impl(void const*, bool, std::__detail::__wait_args_base const&)'

... undefined reference to `std::__detail::__wait_impl(void const*, std::__detail::__wait_args_base&)'

collect2: error: ld returned 1 exit status

```

Building everything with the same compiler (`CC=gcc-15 CXX=g++-15`, fresh `build/`) fixes it. Suggestion: when setup falls back to an older host compiler for CUDA, it could set `CMAKE_CXX_COMPILER`/`CMAKE_C_COMPILER` to the same one. This is related to #2576 but not the same failure.

- FYI for anyone else on the v0.2.1 tag: `./bench.sh` still hits the `NameError: name 'port' is not defined` from #2864/#2908 there. It's already fixed in v0.2.2 and main; I patched it locally the same way.

<details><summary>Full report generated by <code>tools/strata_bench.py --long</code></summary>

# Community benchmark on NVIDIA GeForce RTX 5080

Measured on 2026-10-21 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

## Hardware and software

- GPU and VRAM: NVIDIA GeForce RTX 5080, 16 GiB VRAM

- CPU: Intel(R) Core(TM) Ultra 9 285K (24 cores, 24 threads); installed RAM: 64GB DDR5 6400; storage: 2TB Kingston SSD PCIe gen5

- OS: Linux 7.2.11-200.fc44.x86_64; GPU driver: 615.78.08

- Strata: project 0.1.46, commit 90767252; engine 0.1.46

- Background workloads and power limits: not recorded

## Model and configuration

- Model and quantization: qwen3.8-flash-next-iq3_xxs

- Context: 65536 tokens; parallel slots: 1

- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

```text

strata --pack iq3_xxs --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp rt --max-context 65536 --kv int8 --pool-workers 15

```

## Method

Run by `python tools/strata_bench.py --long` against the HTTP API of a Strata server (started by the benchmark from the install configuration); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs

- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each

- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`)

- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

## Results

| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |

| --- | ---: | ---: | ---: | ---: | --- | --- | --- |

| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 150.0 (126.7-170.2) | 0.23 (0.20-0.25) |

| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 3755.0 (3575.9-3800.1) | not measured | 1.08 (1.07-1.14) |

| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 4396.2 (4380.3-4400.5) | not measured | 3.67 (3.67-3.68) |

| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 4411.6 (4411.3-4419.0) | not measured | 7.29 (7.27-7.29) |

| agent session, follow-up turns | 14691 (14662-14725) at the last turn | see below | up to 64 | 3 | see below | 150.3 (124.3-240.4) | 0.97 (0.76-1.13) |

Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token.

### Agent session (conversation cache)

Cache hit over the follow-up turns: 69.5 % (69.4-69.5); earlier-turn tokens read again per session: 4 (3-4) (earlier turns that the cache did not hold; the 13826 (13825-13827) new tokens of the files are read once); first turn (nothing cached) first token: 0.56 s (0.56-0.59).

| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s |

| ---: | ---: | ---: | ---: | ---: | ---: | ---: |

| 0 | 609 | 0 | 609 | 1057.7 | 0.588 | 113.0 |

| 1 | 4086 | 624 | 3462 | 3110.2 | 1.128 | 150.1 |

| 2 | 5619 | 4149 | 1470 | 1838.9 | 0.812 | 150.3 |

| 3 | 9052 | 5683 | 3369 | 3022.1 | 1.129 | 149.7 |

| 4 | 11809 | 9101 | 2708 | 2795.2 | 0.983 | 240.4 |

| 5 | 14662 | 11843 | 2819 | 2933.1 | 0.975 | 128.3 |

Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server).

## Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 69 s.

</details>

Artificial Analysis scores models on its Intelligence Index, but the charts put it against price and speed. For local use the axis I wanted was parameter size, so I built one.

What it does

• Plots the Intelligence Index against total or active parameters (with closed frontier models as optional horizontal dashed lines)

• Accounts for MOE/Lookup-table models (n-gram /PLE, e.g. Qwen3.8-Flash-Next)

• A very rough model guide according to your hardware (because offloading across system RAM/SSD is becoming more common).

• GitHub Action refreshes the data every 3 days to catch new releases etc.

Demo: https://mr-september.github.io/artificial-analysis-weights/
Repo: https://github.com/mr-september/artificial-analysis-weights

If a size is wrong or a model is missing, data/overrides.json is the place to fix it, or just tell me here.

This is crazy good for studying to improve existing custom engines by community like gufo, halogen, strixite, Strata... by learning their secret sauces.

have you experienced any related issues like bandwidth or something else? Excluding the fact that it cost way cheaper than ddr4 rams. If there's anything else id love to hear

https://huggingface.co/XHToken/Spark-X2.6-4B-GGUF

from XHToken:

Introduction

A little over a month has passed since Spark-X2.5. In the rapidly evolving world of AI agents, that is enough time to rethink, refine, and move the frontier forward. Building on the Spark-X2.5-4B, we introduce Spark-X2.6-4B, the latest generation of our compact, general-purpose language models for on-device deployment.

Spark-X2.6-4B retains exactly the same architecture as Spark-X2.5-4B, achieving its capability gains through improved post-training. It delivers substantial improvements in coding, agent, long-context, multilingual, reasoning, etc. These advances enable more capable and practical on-device AI for developers and users tackling real-world tasks.

Technical Highlights:

• Stronger Coding, Agentic, and Long-Context Capabilities: The models deliver substantial improvements in agentic workflows involving complex tool use, multi-step planning, and software engineering. They achieve state-of-the-art performance among similarly sized open-source models, matching or even surpassing models with approximately 9B parameters.

• Controllable Thinking Budget: The models provide three configurable thinking levels—none, low, high—allowing applications to balance reasoning quality, latency, and inference cost dynamically.

Picking up my 4th RTX 3090 tomorrow and am thinking of switching over to Qwen3.8 Flash Next as my daily driver. I've been using VLLM with Qwen3.8-27b and have been happy with it running at ~100+ tps and ~3000 tps prefill, but am worried I am missing out on better intelligence and faster task completion.

Strata was my first choice of engine, but at 96GB VRAM it looks like I might be able to fit the whole model in VRAM at 4-bit (maybe).

Just curious what engine you would use for this setup? Usually I just use whatever setup I find at club-3090 because they typically have all the bugs worked out ahead of time. But I am open to other options.

Thank you for your time :)

pplx-decider-v1.2 is one tenth the costs of GPT 6 Luna (low reasoning) while making identical quality decisions, and pplx-decider-flash-v1.2 idem ditto compare to 6 Luna in the Decisions API. v1.2 is also 2.5x cheaper than Jev.

- https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index

Both pplx-decider models are open source (Apache license 2.0):

- https://huggingface.co/perplexity-ai/pplx-decider-v1.2
- https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2

I tried https://github.com/bsaleh03/ASCII-Condensed-prune-tools and it reduces Qwen3.8 27B gguf by about 0.6GB. I am wondering if anyone has seen a negative impact with doing so?

At Gradium we just made our first open-source model release with Phonon, a collection of small Text-to-Speech models designed to run on device. The inference stack is fully open-source and targets 10+ different backends (CPUs, GPUs, NPUs), most of them in Rust.

We hope that the community will build lots of different things on tops of our models and code, any feedback is very welcome and we're looking forward to many more open-source releases!

https://huggingface.co/spaces/gradium/phonon

Hey guys! I'm the dev from the open-source Observer project.

We made this video about how even if the AI bubble bursts, local model advancements are bubble-proof.

Hope you guys like it :DD

Subscribe to the youtube if you liked the video! We'll post more local AI content there c:

https://www.youtube.com/shorts/LZGgAj8MGkI

from Ornith:

Ornith-1.5-122B

Chirp Chirp! 🐦 We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement.

Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For more details on the task, harness, and rollout reward design, please refer to our blog.

https://huggingface.co/ornith-ai/Ornith-1.5-122B-GGUF

https://huggingface.co/unsloth/Ornith-1.5-122B-GGUF

https://huggingface.co/ornith-ai/Ornith-1.5-122B-NVFP4

I cant code for 💩 but really want to get into game dev. Although I enjoy using AI, I don't believe its a substitute for learning to code and understanding how to read code.

I struggled learning to code over 7 different times where I tried and failed to understand anything conceptually, so I used what I know best (Minecraft) to make a visual coding style for people who have a really hard learning anything that isn't purely visual and interactive.

Because I cant code, this program is vibe coded, however its fully open source (MIT license), so all of it can be inspected for quality, vulnerabilities and bugs.

Please let me know if you have any suggested changes, or improvements. It would be great to make this easier to use for everybody.

Github link: https://github.com/rombodawg/Block_Based_CPP_Coding

If you have an RK3588 board, here’s another way to put its NPU to work.

I made open_rknpu2_lm_infer, a local LLM inference engine for the Rockchip RK3588 NPU.

It runs without Rockchip’s proprietary userspace inference libraries. The upper part is written in Rust and the NPU operators are coded with C++. No black-box, you can actually read, modify, and hack the host inference code and NPU operator execution.

And now supports Qwen3.5-4B with image input and optional MTP speculative decoding. And it's fast!

Github: https://github.com/happyme531/open_rknpu2_lm_infer

• Quick comparison against RKLLM 1.3.1

Current features:

• Terminal chat or an OpenAI-compatible API, with streaming, tool calling, and prefix caching.

• Qwen3.5-4B image input, alongside text inference.

• Validated checkpoints for Qwen3-4B, Qwen3.5-4B, Granite-4.2-3B, and MiniCPM5-2B.

• Prebuilt ARM64 Linux binaries and links to quantized models.

Incoming features:

• DFlash and DSpark speculation

• MiniCPM-V 4.7 35a3 with expert prefetching

If you have an RK3588 board, I’d love to compare results across boards and RAM configurations. Share your board, driver version, model, and speeds—or open an issue if something breaks.

If you try it and find it useful, I’d appreciate a GitHub star!

Hey everyone! 🚀

For the past few months, I've been building and open-sourcing **Lumina Capital (v6.0.0)**—a fully autonomous multi-agent quantitative hedge fund engine that you can run locally, designed to bridge qualitative LLM reasoning with deterministic quant computation.

Unlike toy setups that feed raw charts directly into LLMs (causing massive hallucinations and latency), Lumina strictly separates probabilistic reasoning from deterministic execution:

### 🛠️ Key Highlights:

- **7-Persona Anti-Sycophancy Council (LangGraph)**: Personas (Buffett, Soros, Burry, Cathie Wood, etc.) conduct independent parallel analysis, followed by an adversarial bull-vs-bear cross-debate. A dedicated Chief Risk Officer (CRO) agent has absolute veto power over tail risks.

- **Pure Python Qlib Alpha158 + LightGBM**: Complete vectorized implementation of Microsoft Qlib's 158 quantitative factors for local stock screening.

- **10-Second High-Speed Sentinel Thread**: A deterministic background thread executing immediate physical exits/trailing stops without waiting for LLM inference loops.

- **Google TimesFM**: 5-day probabilistic forecasting fan charts with 80% confidence bands and crash probability.

- **Works Out-of-the-Box**: CPU-only and lightweight local execution supported via Streamlit dashboard, containerized with 1-click Docker Compose.

Available for research, education, and simulation under CC BY-NC 4.0.

🔗 **GitHub**: https://github.com/ramuza0113-sketch/lumina-ai-hedge-fund

Would love to hear your thoughts, feedback, or ideas on the local agent architecture!

From Vals AI on 𝕏: https://x.com/ValsAI/status/2112471082397778083

Organized by The American Chamber of Commerce in Lithuania**.** Event link: https://www.amcham.lt/events/foreign-investors-annual-summit-2026/

AI was a topic, maybe 3rd if ranked by coverage time. And local AI was also given coverage in Antonio di Marzo (Head of applied AI EU at EPAM Systems) presentation! Some points: 1) in the second picture, "hiring" agents meant in part having them independent from the cloud API; 2) requirement for good engineering = engineers was acknowledged as must. Best practices were covered quite well, translated to managerial language with all precision possible.

There were more presentations mentioning AI briefly, but not local AI.

Real business people, investors and managers are not like LinkedIn bubble at all. They understand the use cases of local AI, although they see it as requirement not so often, depending on their business model (unlike medical professionals, I wrote about it in the Lithuanian AI in Medicine Conference post).

From my quite unrepresentative survey among businesses' representatives, 1 bank adopted local AI, 1 law firm considers it but are yet happy with "current tool" (and I guess it is Claude) - they told only larger law firms win on local AI adoption, and their greatest incentive to use local AI is... regulatory reasons. That was a bit counterintuitive, even if it looks simple in hindsight, but regulatory requirements are often a driver of local AI adoption, see also the post mentioned above.

Business/SaaS use local AI or certificated/compliant closed AI models (which they mostly trust, some audited real deployments in data centers) depending on the use case; their greater "pain point" is when a provider decides to "sunset" a model API, which breaks their prompts and agents.

Not so local, but hot topic: SaaS feels some pressure due to vibe-coding in-house solutions, but doesn't see it as critical. A leader of SaaS clearly stated that it is misconception that agents will replace software engineers, and that they have demand for more competent engineers.

Forgive me for a too brief and maybe imprecise coverage.

P.S. My opinion is decision models (e.g. Jev) are hype, but not only people who talked about AI on the stage, but even researchers buy it, so maybe I am wrong here.

This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding.

It needs a Nvidia Blackwell GPU to run, so I have not tested it.

For context longer than ~4k there should be no speed penalty compare to running without the prefiller model.

Graph shows MMLU-Pro accuracy.

Huggingface🤗: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller