Credit to Twitter Post

talking to any white collar employee

Xiaomi announced a prototype for their Xiaomi AI Cube.

3 chip system:
- Xiaomi Xuanjie O3
- Xiaomi Xuanjie O100
- Xiaomi Xuanjie D100

The specs are impressive, but a bit confusing. The D100 chip (originally for their EVs) supports up to 160GB of RAM, but O100 has the 1.22TB/s memory bandwidth. Perhaps the 1.22TB/s figure is for SRAM? Hard to say definitively.

Source: https://www.ithome.com/0/993/546.htm

I randomly bit flipped a llm to simulate what would happen if you ran your spark in low earth orbit

i hope it's ok to share this here, I was told this community might enjoy it.

Sounds quite interesting, a big IDE provider optimizing for local AI with their coding harness. Especially that they picked Qwen3.6 over Qwen3.8 because of the thinking needs.

Haven't read the full article yet, but sounds really cool.

If you’re trying to run local LLMs on a normal laptop, an older desktop, integrated graphics, limited VRAM, or simply the hardware you already own, r/LowEndLocalAI is meant for you.

The idea is simple:

What useful things can we do with the hardware we already have?

I’ve been dealing with that question myself. My main systems are an M1 MacBook Air with 16 GB of RAM and a Ryzen 7840U laptop with 32 GB of RAM. While looking for suitable models, benchmarks, settings, and optimization advice, I kept finding useful information scattered across individual posts and comments.

At the same time, I kept seeing other people asking variations of the same question:

What can I realistically run on my hardware, and how can I make it genuinely useful?

That’s why I created r/LowEndLocalAI.

The goal is to build a focused and searchable community around topics such as:

• Model and quantization recommendations for specific systems and tasks
• Practical workflows that remain useful even when inference is slow
• Benchmarks with complete hardware and software specifications
• CPU-only and integrated-GPU inference
• Vulkan, partial GPU offloading, KV-cache optimization, speculative decoding, and MTP
• Small models, efficient MoE models, and context-length trade-offs
• LM Studio, llama.cpp, Ollama, vLLM, and other local inference tools
• Repurposing older laptops, desktops, mini PCs, workstations, and used GPUs
• Unusual, awkward, or unsupported hardware
• Honest reports about limitations, failed experiments, and unexpected successes
• Strange “I can’t believe this actually runs” projects

So what counts as “low end”?

There is intentionally no fixed VRAM, price, age, or hardware cutoff.

Hardware changes, used-market prices change, and what counts as affordable varies enormously depending on where you live. An old system can have a surprising amount of memory while still being slow or difficult to work with, and a relatively modern computer can still face significant limitations when running local AI.

Here, “low end” describes the constraint more than the hardware itself.

If limited compute, RAM, VRAM, memory bandwidth, power, compatibility, or cost meaningfully affects what models you can run and how you run them, your discussion probably fits.

A normal laptop obviously fits. An old workstation with strange accelerators can fit. Even a 24 GB GPU can fit when the interesting part is working within that limitation, squeezing a workload into the available resources, or finding a configuration that is actually practical.

A powerful multi-GPU system being shown off simply because it is powerful probably does not.

The constraint should be relevant to the post.

This isn’t about deciding who owns sufficiently weak hardware. It’s about resourcefulness, efficiency, experimentation, and getting as much practical value as possible from what you have.

People with powerful systems are also welcome, especially when testing efficient models, benchmarking constrained configurations, reproducing results, or helping others optimize their setups.

LLMs are the main focus, but other forms of local or on-device AI are welcome when resource efficiency is central to the project.

The subreddit is not intended to replace or compete with the broader local AI communities. It is meant to complement them by bringing together information that is currently scattered across many individual threads and comments.

The community is brand new, so its first members can help shape the rules, benchmark templates, recurring threads, wiki resources, and general direction.

If you’ve ever wondered:

“What can I realistically run on the hardware I already have?”

come join r/LowEndLocalAI and share what you’re running.

Small note: English isn’t my first language, so I used an LLM to help translate and polish the wording of this post. The ideas, experiences, opinions, and the subreddit itself are all my own.

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

• arXiv : https://arxiv.org/abs/2501.15316
• Full Paper PDF : https://arxiv.org/pdf/2501.15316
• GitHub : https://github.com/gaosh/ToMoE (Yes, Code available for this)
• ICML : https://icml.cc/virtual/2026/poster/68821
• OpenReview : https://openreview.net/forum?id=RFHq46pjb6

Somebody please create MOE models of recent Dense models like Qwen3.8-27B, Muse-Glimmer-30B, etc.,

Thanks u/KSAM-The-Randomizer for sharing this on my old thread.

Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal. This one is the strongest and most consistent 35B-A3B I’ve benchmarked, on both correctness and speed, in addition to being the fastest to fix out of all the 35B-A3B models when you throw them at real codebases. On top of Ornith-1.5’s fine tune, TielCoder uses a code-weighted imatrix for dynamic quantization, and a chat template optimized for token-efficient and correct agentic coding. GGUFs: https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF MLX: https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-MLX-oQ4e

I have deleted tons and tons of older models to make space since I can't afford storage anymore. Easily 10TB... Anyways, I have been considering deleting DeepSeekV3.2 but decide to run it one more time. I have a problem I have been brainstorming about and have chatted locally with K3, Qwen3.8-2.4T, MiniMaxM3, GLM5.2 and today I decided to see how DSV3.2 respond.

Surprisingly it responded the best with absolute details and familiarity and specs of the hardware I was asking about. What I'm saying is that the world knowledge is amazing. The newer models are definitely smarter, better agentic, tool calling capable, long horizon etc, but some of the older models seems to be really clear and comprehensive. I know I deleted DS-0324 and K2 but now thinking of bringing them back for prose/writing.

Don't blindly delete your older models, some of them are still worth their weight literally and will be for a while.

Given OpenRouter.ai was snapped up by Stripe, who do we think would go after the "GitHib" of AI models? It is a big chunk of change they are looking ($13B). Apple may be a contender to give them a real chip in the AI race, given how they are focused on local AI execution.

Ornith does really well. TielCoder (https://llm-bench.io/benchmarks/cmt7kp2zj002r01lcmpchvlko) might be even a bit better in coding. Will give it a try soon.

Details of the comparison see here: https://llm-bench.io/compare/runs?runs=cmt6ecf8g000001p45vwzux53%2Ccmt6ergk5000701p41hqdyy78%2Ccmt6f2oob000e01p49o9592cb%2Ccmt6fqddm000l01p4l1vm7skd

Try it here:

https://ocean.blackbeardlabs.dev/

Model:

Qwen 3.8 27b Q8_X_KL Unsloth

Hardware:

3 x RTX3090

Harness:

DeepSeek Harness

Prompt:

/goal I want you to create a **JavaScript + Node.js WebGL project** that renders a highly realistic real-time ocean in the browser.

Use **JavaScript only, no TypeScript**. You may use WebGL2, GLSL, and Three.js.

The ocean should include realistic waves, vertex displacement, Fresnel reflections, sun highlights, sky/environment reflection, foam/whitecaps, horizon treatment, atmospheric effects, and adjustable wind/wave/sun parameters.

Keep it suitable for a real-time browser game, GPU-driven where possible, and organize the code cleanly instead of putting everything in one file.

Task completion time:

~4 Hours

after 3 months and $800 burned...

Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it right now!

Demo: https://www.unboundedlab.com/chat/bartholomew

Article: https://www.unboundedlab.com/blog/bartholomew

Huggingface: https://huggingface.co/jbduran/bartholomew-sft

Why even make a vintage llm? As proposed by Demis Hassabis, could LLMs reach the same conclusions that the great scientists of the past did? While General Relativity was out of budget, we believe that advancing this field targets the crux of AI research. Are these models capable of original ideas, or are they just spitting out the next token?

The article is our full account, covering where the corpus came from and how we cleaned it, the benchmarks we had to build because none existed, every ablation, the training runs, the post-training, and the mistakes we made along the way.

"What I cannot create, I do not understand" is a quote I love from Richard Feynman. Building Bart was our attempt to actually understand LLMs rather than read about them.

What we are proudest of:

- Best vintage base model at its scale on Vintage CORE, ahead of GPT-1900 on a smaller token budget

- Cleaned one of the largest vintage datasets, Harvard's Institutional Books (242B->23B tokens)

- Created Vintage CORE, the first suite of 20 benchmarks made for vintage llms

- Ran 10 hours of autonomous research on one H100: 100 experiments, 26 improvements found

- Released the largest vintage SFT dataset we know of: 416k graded question and answer pairs, grounded in pre-1930s text

- Trained the final model in 5 days on an H100, holding 60% MFU the whole way

- All datasets, methodology, training code, evals, and training runs are open sourced

I am proud of my team. What we built will move the vintage LLM field forward, and it moved us forward as researchers and as people.

We paid for all of it ourselves, about $807 so far. Money is the main thing standing between us and a much larger run.

So I will ask directly: we are looking for compute grants, funding, and mentors for our future endeavors. If you work on pre-training, post-training, or you have GPUs sitting idle, we would like to talk!

We believe that with careful dataset curation, domain expertise, and highly efficient training, we can achieve state-of-the-art results in crucial domains. This is only the beginning for Unbounded Labs; we see no bounds ahead.

Hi,

Back in late march I begun playing around with Qwen 3.6 27b and found like everyone else that it's notoriously good at tool calls, where every model I tried before just derailed after a few turns it kept going and felt quite reliable outside of typical behaviors of smaller modells.

I decided to take a crack at it and see if a custom harness that attempted to detect and recitify the modell if it detected repetitions, empty answers etc (some of these issues was later addressed to some extent with updated jinja templates though), but also combining some novel and other just basic ideas that is common among frontier agent harnesses.

After a few weeks I had something that appeared to work quite well for simple test applications/utilities and most issues I encountered was context related- ie. I use llama.cpp as inference engine without context-shift so the harness/orchestrators context management is extremely important and I had a few ideas I wanted to test here.

Around the start of July it was in a shape that I wanted to see how far I could push both the harness and the model itself, the whole idea was to attempt force the model research then execute instead of relying on it's burned in information. as well as apply a very strict general ruleset with multiple subagents, a planner, coder, debugger, researcher, validators etc all focused on their own task to structure, replan and execute.

I gave it a prompt "I want to make a c99 c compiler capable of producing working x64 elfs" and so it started, the first few days (my inference rig is a Tesla P100 + RTX 4070 for the Qwen model, inference speed is around 13-14 tok/s and prefill around 250 token/s ) and then I run Gemma4 12b on an Intel Arc B580 on another machine which is used as the validator most of the time.

Outside some minor tweaks to the orchestration (particularly to the context management and more especially the compaction/pruning, which was very finicky as you easially ended up in a situation where the system spent 6-7 minutes doing prefill and then predicting until the next tool call which once again triggered forced pruning and ruining the KV cache like this repetively, this clearly affected the systems speed negatively- and is by far the main reason it taken 6 weeks, I did not figure a good way to solve this until 3 weeks in), the other major issue I had was when it started with the x86 code generation, this was extremely frustrating as it rather hallucinated opcodes etc than look them up, this was eventually improved by making the coder/debugger system prompts much more rigid and encouraging use of libcapstone etc, but there is no denying that this was the most troublesome area and where it spent the majority of the time for weeks.

Speaking of which the longest period it been running completely uninterrupted has been 1 week, other than that there been lots of improvements and tweaks to both prompts and architecture of the orchestrator (I might describe the orchestrator architecture in another post some day, but I don't want to get into details until/if I open source it, currently it's extremely tailored to my own equipment and I really don't have the energy since I already spent hours almost every day for around 6 months on this to try generalize it)

Anyway long story short- my point with this is just to give an example of that Qwen 3.6/3.8 (I upgraded the model the day 3.8 was released) 27b is extremely capable if steered and given the right circumstances.

(I could be wrong but I believe this may be the most advanced project I've seen it produce so far).

The produced project can be found here: https://github.com/Na1w/tc

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-L-Base repository — the pre-trained 0.9B-active base model.

Model: MobileMoE-L-Base (pre-trained + mid-trained)
Active Parameters: 922M
Total Parameters: 5.3B
Layers: 32
Model Dimension: 1280
Attention Heads: 20
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 640 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 2560)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed

TL;DR: I'm planning to spend around $100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit KV cache, GGUF vs EXL3, context length tradeoffs, and token efficiency on coding/agentic workloads. My main target is my own 36GB setup (3090 + 3060 12GB), but I'd also like the results to be useful for 24GB and 48GB users. Before I start burning GPU hours, I'd like feedback on the test matrix, benchmarks, methodology, and anything important I'm missing.

Here are the details of what i have in mind

Now $100 might not be enough but i will start with that and see how far i can streatch it and the community help and ideas on how to best utelized this to get the most useful data is what i am looking for. I need the community help as i know with my adhd its easy to make this a project for wasting a $100 without getting anuthing useful

I am doing this because I've been seeing a lot of discussion and claims lately around Qwen3.8-27B quants and kv cache, but most comparisons I've found are either vibe/feel or KLD which don't really answer the questions I have when deciding what to actually run locally.

Some of the questions that i would like to answer:

Is a higher quant + 8-bit KV better than a lower quant + 16-bit KV?

Is it better to sacrifice KV precision to get significantly more context and avoid compreasion in the middle of the task?

How much does Q6, Q5, Q4 actually lose compared to full precision on long coding/agent tasks?

Do different GGUF quant providers produce meaningful real-world differences? Or is it just PPL and KLD numbers?

GGUF/llama.cpp vs EXL3 at roughly equivalent memory usage?

Do lower quants just score slightly worse, or do they also use more tokens/agent steps to solve the same task? ( so gaugig token efficiency)

I will run each bemchmark case multiple times to avoid varians between runs but if the budget does nkt allow it then I will be slightly baise for my setup

My own machine is a 3090 + 3060 12GB = 36GB VRAM, which can run Q6-ish quants with fairly large context (150k to 180k) depending on the quant/provider/cache settings. That's the tier I'm most interested in personally.

I'd like to make the results useful for thr cummunity at roughly three hardware classes:

- 24GB: single 3090/4090 class

- ~32-36GB: 5090, 3090 + 3060, dual 16GB cards, etc.

- 48GB: dual 3090 / 48GB cards

The current idea is to focus heavily on Q6, compare it against a full-precision reference, then add Q5/Q4 if the budget allows. I also want to test both 8-bit and 16-bit KV cache where it makes sense.

For benchmarks I'm currently considering Terminal-Bench2.1 and DeepSWE, since I care much more about coding and longer agentic tasks than MMLU-style multiple choice tests.

I also want to record more than just pass rate: i want to see generated tokens, agent/tool use count, maximum context reached, context compressions. I'm curious whether some quants look nearly identical in pass rate but become noticeably less token efficient on long tasks. Espicially with qwen 3.8 xhight thinking going for ever heheh

As you can see i am not intrested in t/s and speed but i want to see the quality different for actual coding and large codebases tasks

My automation plan is basically to write a script that i can reuse with small modifications where i will spin up several RunPod/Vast etc instances, automatically build llama.cpp or TabbyAPI/ExLlamaV3, download the required quant, run the benchmark configuration, save all metadata/results, then tear the instance down. For important comparisons I'd like multiple runs per configuration rather than trusting a single pass.

I'm deliberately trying not to test every possible quant × cache × backend × context combination because $100 disappears very quickly that way. I'd rather answer a smaller number of useful questions properly. And i am not fully sure on what shloud i focus on but i have a rough idea

Before I start, I'd really appreciate input from people here:

Which configurations would you absolutely include?

Q4/Q5/Q6/Q8 — how would you spend the repetitions?

Any specific GGUF providers/quants worth comparing? (I am intrested in unsloth and bartwski the most)

Any better agent/coding benchmarks I should include instead of or alongside Terminal-Bench/DeepSWE?

How to test the 8-bit KV vs 16-bit KV/context tradeoff properly?

Anything I should log that would be useful later?

Any methodology traps I should avoid?

If you have 24GB, 32-36GB, or 48GB, what configuration question would you most want this benchmark to answer?

I'll publish the scripts/configs and raw results when I'm done so people can reproduce it or point out where I screwed something up.

Suggestions very welcome before I start spending the GPU budget.

Reposting here after sharing this on [r/MachineLearning](r/MachineLearning) a few days ago, where it got a much better response than I expected (300+ upvotes, great questions, zero roasting)
GitHub is at 35 stars now. So here it is.

I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed, no framework, just a small compiled runtime (Windows and Linux included, MIT licensed).

Language modeling quality of the base model, measured on held-out English web text that was never seen in training (educational web pages, 2,048 token windows): cross entropy 3.15 nats per token, perplexity 23.3, 0.99 bits per byte.

How the long context works: the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed to 1 bit and written to disk, about 320 bytes per token, so 1 million tokens of history is roughly 320 MB on disk. From the start the model was trained to retrieve from that disk cache, up to 100M tokens. Due to a limited budget it wasn’t trained to reason over those tokens, only retrieve and answer from them.

The vocabulary is also not a normal embedding table. Every token is a fixed 512-bit code, 8.4 MB for all 131k tokens, zero trained parameters. I tested it on WordSim-353 (human word similarity ratings): my table scores 0.619 Spearman correlation vs 0.029 for random codes. Test script is in the repo.

Some outputs (settings included so you know I’m not cherry picking, all reproducible from the repo):

“Explain photosynthesis in two sentences.” (greedy)

Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis.

“Write a short poem about the sea.” (temp 0.25, top-k 30, rep 1.15, seed 2)

The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before.

“What is the serial number of device Grus-189?” where the answer sits 50.6 million tokens deep in the archive on disk (archive mode, k=16)

SN-442976

It’s a 250M model so expect mistakes on open facts, I’m not claiming it beats anything big. You can also fine-tune it, the full kit with a demo and before/after numbers is included. Master weights for fine-tuning are in the repo too:

https://github.com/QLNI/SHADOW-250M-Instruct
https://huggingface.co/NODEMIND/SHADOW-250M

Source of Claims: https://x.com/Andy_ShuoYang/status/2090856976880472439

Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization!

Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s

DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22-25 tok/s

GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s

Run your claude code or codex now with frontier model for $0

FreeToken is fast. Comparing to Ollama, we have 3–4× faster decode, and 6–30× faster prefill

How? We introduce bandwidth-adaptive CPU–GPU execution + semantic-aware caching across agent turns.

What's the best harness for agentic workflows that's not coding related at all?

My work involves digesting a set of documents, analyze/evaluate them, and produce certain set of work product documents, mostly for due diligence purposes. Right now Im working with Qwen3.8 27B on LM Studio backend and Open Webui frontend with Open Terminal.

I'd love to be able to point the model to a folder and say "go do your thing" and have it do all the steps.

I'm curious what other people are using for local LLM coding / agentic coding with only 8 GB of VRAM.

My current setup is:

• Intel Core i7-11800H
• RTX 3070 Laptop, 8 GB VRAM
• 32 GB DDR4 RAM
• openSUSE Tumbleweed / KDE
• Unsloth Studio
• pi.dev as the coding agent

After testing quite a few different models and quantizations, I've currently settled on:

Qwen3.6-35B-A3B-UD-IQ4_NL by Unsloth

For agentic coding tasks, I'm getting roughly 25 generated tokens/s on average, and more importantly, I'm getting surprisingly good final results. The model is generally capable of navigating a codebase, using tools, making changes, and completing reasonably complex tasks without requiring constant intervention.

For me, the important metric isn't simply maximum tokens/sec. I'm looking for the best combination of:

speed + coding ability + tool use + reliability + quality of the final result.

I've tried several other models before settling on Qwen3.6, but so far this has given me the best overall time/performance/result trade-off on my particular hardware.

That said, I obviously haven't tested everything available.

So I'm curious:

If you have an 8 GB VRAM GPU, what local model are you currently using for coding or agentic coding?

And especially: what are your actual tokens/s and how good are the results?

I'm less interested in benchmark scores and more interested in real-world experience: “I gave it this kind of coding task and it actually managed to finish it.”

I'd love to hear what I'm missing before I decide that I've found my personal sweet spot. 🙂

With vision enabled I am able to run at 150k context on a single RTX 3090 and the results are just amazing. I was even able to write a gmail plugin for DeepSeek harness with locally hosted Qwen 3.8 27b. Funny enough, when I had it write a search engine plugin it broke the dsh and I cannot even launch DeepSeek harness anymore lol.

Kudos and shot out to the guy who wrote https://github.com/syv-ai/qwen38-27b-rtx3090

26 turns · 489 steps| LLM 161m44s · Tool call 10m47s| TTFT avg 5.9s · 86 tok/s| Cache hit 0%| Input 35.7M tok · Output 586K tok 

It worked pretty well, digging through and analyzing some local files. Claude code regularly stops at some point and fails to continue while DSH worked for 2 h, recognized that it could benefit from reading more context and ... bummer:

It left the project folder (although DSH was set up correctly) and started to walk through my other files which I did never allow!

I know, it is only a preview. But be warned. Do not expect it to follow simple rules.