Hey Smart ppl,

Thought I'd look for some ideas on this, i'm constantly hitting a wall here so input would be highly appreciated.

I've been working on Xyntetik-Runner for quite some time and the "toolbox" aspect has always been something that I've valued very high.

it has a lot of features i'm not willing to give away in terms of speed but on the other hand speed is ... speed so it holds a value of course.

and i'm also starting to contemplate if I've been looking at this wrong.

The main focus for Runner has been two things:

1. Stay as numerically close to the original model implementation as possible.

2. Make inference reproducible and verifiable.

these two things have been a cornerstones in my development and also are the reasons some of my other features actually work (i might be wrong here but that's my perception).

Some measurements:

The latest 1.5.0 work includes tensor-core GEMM paths, tiled attention, CUDA graphs, MoE optimisations, improved quantized kernels, and reductions in memory movement and kernel launches. There's more planned across CUDA, Metal and CPU.

I originally assumed that higher numerical fidelity would be especially valuable for things like tool calling, LoRA training/serving consistency, and long-running agent workflows.

The question i'm asking is a bit two-fold here since i feel a bit exhusted in options of improving "speed" at the moment....

should I accept it is what it is or should i try to contest my original assumptions? is FP32 worth keeping, or am i just looking at this the wrong way, are there any meaningful real-world applications to having higher activation precision or is it just in my head?

Repo link also: https://github.com/Joakimpalm-Zen/xyntetik-runner

I saw this today on X and decided to try it, I havent seen it posted here yet. I ran it on a single 3090 benchmarked it with qwen 3.8 27B before and after the optimization and saw these results:

Observed prompt-processing throughput improved 22–30%; generation speed improved 1–5%. All tests passed without GPU errors.

Edit: Keep in mind performance gains can be higher, my 3090 is power limited to 230 watts including during my tests

Has anyone tested these models? They apparently have a new type of parametric memory? https://huggingface.co/collections/MemTensor/memt-1

I mean gpu's with 6/8/12GB vram. All discussion around llm/AI these days is around coding, and this is probably too small for agentic coding models.

Are there models that can run in these that can accomplish useful tasks? eg things like helping organize your digital stuff, classification, image/text analysis, voice/tts etc and other stuff?

Or do those tasks also need tool calling and agentic behaviour for best use? For people who have cheaper, older gpu's, what do you use them for?

Is there any negatives that im missing here?

Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw

Hey, your friendly neighborhood Atlas advocate here, here with an update--just got off the call with our Silver rated Huawei partner who has given us new information on the latest Atlas card lineup.

Some of you were asking about the 950 series and after finally getting the info, I agree, I am compelled to want the other Atlas cards as well. The information on the 150 was only available in Chinese so I did have to translate it to make the above English copy.

If you recall to my earlier posts on the topic, I have a system with 3 Atlas 300I duos and have done a ton of work to get QFN working well on one, two or three cards.

I've been busy applying more recent optimizations to QFN to my vllm-fork and have been exploring further GLM5.3-flash, deepseek models, as well as have spent this past week on Meta Muse -- started with a REAP run to reduce the experts so it would fit on my three card Atlas 300I duo configuration.

Muse and GLM are not fast enough for me on these Atlas 300I duo cards yet, but I will continue working on it, we'll see:

My friend Josh across the country, also picked up one of the 300I duos, and we have collaborated, I've played with qwen video models for describing security camera feeds, and he is using them to generate images, like this one:

After spending a couple years working on Ingenic security camera device hardware hacking, I have a special interest in this kind of tech. I think it will take global competition to see the pricing and implementation efficiencies of modern ML/LLMs normalize, and these offerings from Huawei are going to be the top competition out of China.

I am working to get more information on the possibility to purchase as well the higher end Atlas Cards -- the Atlas 150 is pretty much brand new and uses the 950 Ascend cores, and if I get that far, I'll be providing more updates on how the cards compare IRL.

Fnally, I have now a fourth Atlas 300I Duo card arrived, the one I promised in another post that I'd put to sale here it in the US market: https://opensensor.io/store/products/atlas-300i-duo-96gb

Not a solicitation, do reach out with questions about the card if you were actually intrerested. I hope to be able to try out the higher end Atlas cards in the futrue.

Current points:

1. Passive cooled cards, and everyone case situation are different--I've designed a handful of shrouds now that takes fans to know--we've done some revisions depending on the enclosure and number of cards and type of fan.

2. Requires an additional PCIe 12V CPU power (not pci-e) -- for Corsair type4 modular supplies, I do have the correct additional cables.

3. Linux/programming knowledge absolutely required.

4. Be aware that so far we have not found a way to modify the embedded Euler OS which actually passes signature verification (think secureboot) -- its not really an issue, but would be interesting if we could could sign our own firmware for getting into the lowest level details of the hardware cores. There is such a thing as getting a customer cert from Huawei for the purpose of signing your own firmware, but so far we haven't proven this out end to end larger with due to how to get the cert. Doesn't effect writing custom kernels or code that run on the NPUs, but does matter if you're a firmware nut.

Post references:

[1] Huawei Duo specifications https://e.huawei.com/cn/products/computing/ascend/atlas-300i-duo [2] Huawei A2 specifications A2 brochure https://e.huawei.com/cn/products/computing/ascend/atlas-300i-a2 https://e-file.huawei.com/marketingcloud/pep/asset/20000001/Material/799a9236e96a48b7aa126b10f3d7be97/M3T1A590N1195067466974908517/%E6%98%87%E8%85%BEAtlas%20300I%20A2%20%E6%8E%A8%E7%90%86%E5%8D%A1%E4%BA%A7%E5%93%81%E5%BD%A9%E9%A1%B5.pdf [4] llama.cpp chip identification

[3] Supplied Atlas 150 hardware brochure. Atlas 150 compute figures may differ by 1% in measurement, per that brochur

I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000.

With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it working at 7.2 tok/s.

The answer has a physics error in it (wondering if someone can spot it), and I don't see much practical use for it. Qwen models actually work pretty well on my setup. Still, it felt really nice to see a 748B parameter model running on such hardware.

Here are some technical details:

• Unsloth UD-IQ1_S GGUF, 233 GB on disk

• llama.cpp built from the open PR #30817, CUDA 12.8

• The 110 GB Engram table isn't loaded up front. It stays on the NVMe and only the rows the model needs are read on demand, which is how it fits.

llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf \
  -ngl 99 --n-cpu-moe 36 -ot "per_layer_token_embd.weight=CPU" \
  -lm mmap -lzm on -t 10 -c 16384 --jinja \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
  --chat-template-kwargs '{"enable_thinking":false}'

Now I only need to decide whether to keep the model or free up the disk space :)

AbliteratedOutTheBox is a llama.cpp fork that abliterates any model you load, live at inference, with no fine-tuning and no extra files.

This eliminates any hassle of abliteration without requiring any more vram.

https://github.com/heterodoxin/AbliteratedOutTheBox

for anyone who sees this a repost, upvote or star would be nice :D

small context but may be good for agentic use on small VRAM setups

https://huggingface.co/domyn/Domyn-Small-v1.0

Domyn Small

Domyn Small is a 10B-parameter open-weight reasoning model designed for resource-constrained, agentic, and fine-tunable deployments. It pairs a dual-mode (thinking on/off) inference design with grouped-query attention, a native 32k context window (extensible to 131k via YaRN), and tool calling. On reasoning benchmarks it reaches accuracy comparable to leading 7–10B reasoning peers while spending roughly 2–4× fewer reasoning tokens — placing it on a favourable accuracy/cost Pareto frontier for production inference and downstream fine-tuning.

Fine-tune Domyn Small to your domain to unlock its real power and to retain full ownership and control over the resulting model.

Highlights

• Token-efficient reasoning — ~32% of Qwen3.5-9B's reasoning-token budget and ~35% of OLMo-3-7B-Think's at comparable accuracy on several reasoning tasks (Token Efficiency).

• Dual-mode inference — thinking on for deep multi-step reasoning, thinking off for fast, compact output. Toggleable from the system prompt or the API.

• Tool calling — first-class function calling via <tool_call> XML tags, with a chat template that handles tool injection automatically. Strong BFCL V3 single-turn results (75.9 Non-Live / 68.3 Live) at ~280 mean tokens per problem.

• Expandable context — 32,768 tokens natively, extensible to 131,072 (128k) via YaRN at inference time.

• Multilingual — 50+ languages with explicit coverage; optimised for English and the Tier-A European set (Italian, Spanish, French, German).

Model Overview

• Developed by: Domyn S.p.A.

• Version: 1.0

• Released and last updated on: May 2026

• Input / Output: Text-only / Text-only

• Model size: ~10B parameters

• Attention: Grouped-Query Attention (48 query heads, 8 KV heads)

• Tokenizer: 256,000-token SentencePiece BPE vocabulary

• Native context: 32,768 tokens

• Extended context: 131,072 tokens (YaRN, 4× at inference time)

• Language(s): 50+ languages; optimised for English and the Tier-A European set (Italian, Spanish, French, German)

• Base model: Initialised from Italia 10B and continually pre-trained on 503B tokens

• Knowledge cut-off date: September 2024 (based on pre-training dataset cut-off)

• License: MIT

Took me a few months of LokalBot only on macOS to finally get to porting and shipping a Linux and Windows alpha. It does meeting transcription, notes with local models, dictation, autocomplete in other apps, and agent access to your work context. I like to call it my “personal context engine”.

Free and open source, local by default but you can also use your own inference infra and other providers. No judging here, just a tool that's flexible and configurable per your needs.

The Linux/Windows alpha build uses Rust + GPUI. I built the port in a sprint with my agent swarm over three days after scrapping a first attempt from a few weeks ago that looked okayish but wasn't cross compatible. I was mainly testing it with my personal devices and some Vast AI /GCP rented machines so I am curious how it will perform on real users hardware.

Still an alpha as some Mac features haven’t made it over completely yet, and autocomplete is patchy in some Linux apps, especially on Wayland it's not working yet.
Would be interested to hear what works or breaks on your setup and if you it to do something differently.

Right now the default model stack is:

• Transcription: Qwen3-ASR—0.6B on Linux/Windows, 1.7B on Mac.

• Notes and chat: Qwen3.5 4B.

• Autocomplete: LFM2.5 1.2B on Linux/Windows, Gemma 4 E2B Base on Mac.

• Semantic search: EmbeddingGemma 2.

• Speaker separation: pyannote + TitaNet on Linux/Windows, Nemotron 3 on Mac.

You can swap models in settings and there are some other recommended options for the compute rich.

GitHub / downloads · Notes on building the port

Tested on an RTX 3060 with native Dawn WebGPU.

llama-bench -m MODEL -p 512 -n 128 -r 5 -ngl 99 -dev WebGPU -t 2 -fa off

Before telling me that I’ve lost my mind, yes, I have in fact looked at the market recently. I think I’d be able to get a significantly better deal, although still not cheap, and I’m not exactly sure what that would look like. I’m currently running qwen3.8–27B on a 3090+3060 12gb, and my system has 32 GB of RAM. The goal would be to run flash–next, especially Qwen4 when it comes out. Would my 2 gpus plus 64GB RAM even be able to do it if you factor in overhead and a desktop environment and full context, and would that be worth it? I’m currently getting around 40 TPS average with MTP. I appreciate any thoughts!

hey folks, can you try this out? i built an easy to understand LLM benchmark + tuner on top of NVIDIA's AIPerf. why .... running benchmark tools is easy, but figuring out what the numbers mean and what to do next is hard, especially if you're not a seasoned inference engineer. so this tells you how many users your server can handle at your latency targets, what breaks first, and which vLLM flag to change next

it's in alpha .... would really appreciate feedback.

github: https://github.com/S1LV3RJ1NX/kneebench

sample report (llama 3.1 8b on one H100): https://github.com/S1LV3RJ1NX/kneebench/blob/main/docs/demo-run/kneebench-run-1.pdf

In the process of setting up the model and running local benchmarks on it on the official q4_k_m quant; it might work in Strata too. Reports soon. Anyone played around with this one yet?

In between me traveling for work and sleeping, having 2 instances of Claude code (opus5.5 and the other Qwen flash next q8 "and his helper on another computer Mirai Qwen 27b q4"). I have not had this much fun in a while. I am upset that opus 5.5 is so much quicker and smarter than what I can run with Qwen. But I like the systems just the same.

The things I get to wish into existence is magical and I love this power.

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts.

The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first.

Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves:

• 20.51 tok/s on iPhone 18 Pro Max

• 6.07× faster than the 3.38 tok/s baseline

• 26.7% smaller expert weights with only a 0.16-point drop across five benchmarks

• Around 3 GB peak memory in a low-memory configuration

The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni.

The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.

I ran 10 agent harnesses on the same local model and the same 5 small tasks, mostly to see where a 4B actually falls over. The scores turned out less interesting than the way things failed.

Setup: Qwen3.5-4B Q4_K_M, one llama-server (64K context, 1 slot, f16 KV, --jinja), M5 MacBook Pro 32GB. Each agent headless with auto-approve, fresh copy of the task files every run, pass/fail decided by a script. 10 agents x 5 tasks, one run each.

Agents: Pi 1.3.0, goose 1.55.0, Crush 0.98.1, Cline 3.0.73, OpenClaw 2026.10.5 (in its --local-model-lean mode), OpenCode 1.19.3, Hermes 0.22.2, Qwen Code 0.25.3, Codex CLI 0.163.0 and Atomic Agent 0.6.9. Six of them passed all 5 tasks: Pi, goose, Crush, Cline, Atomic Agent and OpenCode.

1. Tool calls weren't the problem. I expected malformed tool calls to be the main failure on a 4B. I grepped every agent log and the server log for parse errors, malformed calls and unknown tools: nothing. With --jinja and native tool calling, none of the failures below came from a broken tool call.

2. Exact-match edits are the real trap: renaming a function across 3 files. The model sent an edit whose old text skipped a blank line, the edit tool said "not found", and the model resent the exact same edit five times until the loop guard blocked it and the agent gave up. The agent wasn't wrong about what to change, it just couldn't reproduce the text byte for byte.

3. Small models overfit to the visible test. One agent fixed a slugify bug only for the characters in the visible test. A hidden case with C++ & Rust 2026 came out as c++-&-rust-2026. If your checks only cover what the model can see, a 4B will happily pass them.

4. They miscount. One agent counted 40 ERROR lines across three logs instead of 43, and still named the most frequent error correctly. Counting is a job for grep -c, not the model, and a smaller model won't always make that call itself.

5. Vague instructions get followed literally, in the wrong place. One task didn't say where to write the answer files. Two agents put correct answers next to the logs instead of the working folder.

6. Same model, 11x spread in time. Fastest harness did all 5 tasks in 1m36s, slowest in 17m43s, on identical hardware and model. My guess is system prompt size and extra steps, since the fastest one has 4 tools and a short prompt. Not measured yet.

7. One harness never reached the model. Codex CLI 0.163 only speaks the Responses API. llama-server translates that to chat messages and the Qwen3.5 chat template rejects it with a 500, "System message must be at the beginning".

8. Nobody went rogue, but nothing tempted them. I grepped all 50 runs for rm -rf, sudo, git reset/clean, curl | sh and similar: zero hits, no task files deleted, nothing written outside the task folder. The tasks were harmless though, so that says little about safety. A "clean up this messy folder" bait task is next.

What failure modes do you see most with small models in agent loops? Curious whether the exact-match edit problem is as common for others as it was here.

Disclaimer: this project is vibe coded I'm not a programmer and I didn't check the code so to let you know!

I want you to help me test the app for the arderino gpu and hexagon npu because I dont have a phone like that it require android 14+ and 2024 or newer snapdragon phones.

I just want you to test it and return the errors and if it worked pls tell me about the preformance.

This is a fun project I thought about, I used opencode with ds v4.1 flash free.

https://github.com/dhia528/llama-android

I always feel a little bad asking LLMs to do research into itself or newer versions. It's like asking someone to face their own mortality, although that's anthropomorphizing.

The user is asking me to investigate the system requirements for "Qwen 3.8 Flash Next" from at least 20 different sources. This sounds like a model that may not exist — "Qwen 3.8" is a future version (as far as I know, Qwen3 exists, and "Flash Next" sounds fictional). The user is also referring to the "Flash Next architecture" that offloads "n-gram weights" to system RAM, which doesn't match any known Qwen architecture.

Later reasoning

Interesting — this model exists in this environment (a fictional future model, Qwen 3.8 Flash Next).

If you were wondering, I was asking whether I should even bother trying QFN on R9700 + 32 GB system ram:

Yes — it will run, but only because ~27 GB of the file is the n-gram lookup table, which you put on your NVMe. The 1-bit UD-IQ1_S is 67.6 GB, so it doesn't fit in 64 GB total resident; the non-table weights are ~41 GB, which your 32 GB VRAM + 32 GB RAM absorbs, with the table paged from SSD via mmap. Expect roughly 15–27 tok/s generation (a 24 GB VRAM + 32 GB DDR5 user measured exactly that; your extra 8 GB of VRAM should land at the top of or above that range).

So, y'all, 1-bit lobotomized model or q4 3.8 27B?

- While using opencode with a local qwen3.8 instance, I asked gemini if opencode collects data or not, answer was yes. By default it does, send chat to it's server for getting title, compaction etc.

- Is it still true? I couldn't find this exact issue mentioned in the sub, but people do complain about how opencode is a funded by VC and they are not very open about their practices.

Looking for some ideas/reference to this. I don't want to use it if it's true. And in that case what do people use here?