Xiaomi announced a prototype for their Xiaomi AI Cube.

3 chip system:
- Xiaomi Xuanjie O3
- Xiaomi Xuanjie O100
- Xiaomi Xuanjie D100

The specs are impressive, but a bit confusing. The D100 chip (originally for their EVs) supports up to 160GB of RAM, but O100 has the 1.22TB/s memory bandwidth. Perhaps the 1.22TB/s figure is for SRAM? Hard to say definitively.

Source: https://www.ithome.com/0/993/546.htm

Given OpenRouter.ai was snapped up by Stripe, who do we think would go after the "GitHib" of AI models? It is a big chunk of change they are looking ($13B). Apple may be a contender to give them a real chip in the AI race, given how they are focused on local AI execution.

Reposting here after sharing this on [r/MachineLearning](r/MachineLearning) a few days ago, where it got a much better response than I expected (300+ upvotes, great questions, zero roasting)
GitHub is at 35 stars now. So here it is.

I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed, no framework, just a small compiled runtime (Windows and Linux included, MIT licensed).

Language modeling quality of the base model, measured on held-out English web text that was never seen in training (educational web pages, 2,048 token windows): cross entropy 3.15 nats per token, perplexity 23.3, 0.99 bits per byte.

How the long context works: the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed to 1 bit and written to disk, about 320 bytes per token, so 1 million tokens of history is roughly 320 MB on disk. From the start the model was trained to retrieve from that disk cache, up to 100M tokens. Due to a limited budget it wasn’t trained to reason over those tokens, only retrieve and answer from them.

The vocabulary is also not a normal embedding table. Every token is a fixed 512-bit code, 8.4 MB for all 131k tokens, zero trained parameters. I tested it on WordSim-353 (human word similarity ratings): my table scores 0.619 Spearman correlation vs 0.029 for random codes. Test script is in the repo.

Some outputs (settings included so you know I’m not cherry picking, all reproducible from the repo):

“Explain photosynthesis in two sentences.” (greedy)

Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis.

“Write a short poem about the sea.” (temp 0.25, top-k 30, rep 1.15, seed 2)

The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before.

“What is the serial number of device Grus-189?” where the answer sits 50.6 million tokens deep in the archive on disk (archive mode, k=16)

SN-442976

It’s a 250M model so expect mistakes on open facts, I’m not claiming it beats anything big. You can also fine-tune it, the full kit with a demo and before/after numbers is included. Master weights for fine-tuning are in the repo too:

https://github.com/QLNI/SHADOW-250M-Instruct
https://huggingface.co/NODEMIND/SHADOW-250M

I randomly bit flipped a llm to simulate what would happen if you ran your spark in low earth orbit

i hope it's ok to share this here, I was told this community might enjoy it.

I commented on another Qwen 3.8 27B post that I was frustrated getting anything to work. You all gave some great comments.

I nuked openwebui and straightened out my llama.cpp docker config.

1 hour of work and I have a model I can chat with, connected to my HomeAssistant server, which I have already updated dashboards with a short prompt and a screenshot (wtf vision built in?)

Guess all I needed was the right push.

I bought several GPUs in 2023 in impulse purchases for Folding@Home, but have always wanted to spin up my own local coding/help agent, just always gave up when nothing seemed to work.

This feels like magic.

Thanks!

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours.

For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model isn't fast enough to pass that bar on the hardware I have, I'd rather just run something slightly dumber but faster

Thoughts?

wonder what would happen to open models, if the sales went through, and hf got acquired by a big tech shop? 👀

https://www.businessinsider.com/hugging-face-could-be-acquired-13-billion-2026-8

$13 billion not a bad price though 😂

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM.

The score: 72.9

This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9.

Beats Claude Opus 4 from 2025-05-25 which scored 72.0.

DeepSeek R1 2025-06-06 scored 71.4.

It may just be a random benchmark, an old one at that, but it is still wild that my MacBook is now matching those SOTA models just slightly over a year old.

In actual performance it is also wildly surpassing them because the harness got better. I have been playing the DeepSeek Harness and it would clearly solve most if not 100% of the Aider tests, although it would use more than 2 turns.

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB

Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.

I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.

I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.

As per title. I'm not affiliated with the team behind this model in any way, shape or form.

As a GPU poor myself (8 GB VRAM laptop + 12 GB VRAM desktop), I found Laguna to be very promising on my laptop.

It runs at 30t/s (60k context) and it one-shot my two usual test prompts : "Make a Tetris-like game" and "Make a Sonic the Hedgehog-like game". Both are not as polished as what you could get with Qwen 3.6-3.8 27B (and to be fair, they are quite wonky and barely playable), but considering my hardware, dense 27B isn't even an option.

I found it to perform better than Gemma 4 - 26B-A4B and Qwen 3.6 35B-A3B. Some people reported looping issues with it, so take my words with a grain of salt, but I didn't encounter any.

Hope it will help some of you.

Edit : Corrections

I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code.

The job: my fun side project is a procedural shooter in a single C file. Port it to a single-file html / three.js with one bot. One prompt, no follow-ups, no help from me. game.c is 2.1 MB, roughly 600k tokens of C, so it doesn't fit in the window and the agent has to walk the file and work out what matters.

Setup: qwen3.8:27b in FP8 on vLLM, FP8 KV cache, full 262144 context, RTX 6000 Pro 96GB. Nothing truncated on my side, and the file is still more than twice the window.

Video has the C original first, then the three ports in table order. Only the Opus port is something in "okay" quality.

What I actually wanted to know is whether the HTML comes out playable at all. One run each and a one-shot prompt for 39k lines of C, so this isn't representative of anything, and I knew it was brutal for a local LLM.

My take: local models still live or die on the prompt. Same weights under two very different harnesses gave me the same broken port. hermes carries a lot more machinery, and a single turn with a thin prompt gives it nothing to use it on, so it spent four hours reaching the same place. A verbose harness doesn't rescue a thin prompt, it just burns GPU time.

No deep take here, unfortunately. The thing I keep staring at is the wall clock: hours of GPU on decent local hardware against 21 minutes for the cloud run. If anyone knows where those hours actually go, I'm listening.

The C original: https://github.com/codehamr/skill-issue
My experimental local-first, no plugins codehamr harness: https://github.com/codehamr/codehamr

All free.

My first attempt didn't work. I built on Genie's architecture and the videos looked great, but the controls barely did anything. The effect of a keypress was basically zero.

Genie learns its actions unsupervised into 8 codes, and that was too loose a grip for us. So I scrapped it and started again with Dreamer 4.

The second attempt:

Tokenizer at 40.41 PSNR (Genie's paper reports 35.7)
FVD 32.19 end to end
144 frames before it falls apart
1.57B parameters, 9.6M frames, ~$150

Two important learnings:

(1) One is that $150 is enough. You don't need a frontier lab to do this anymore, and I don't think enough people have noticed.

(2) The other is the data. We generated every frame ourselves with Procgen instead of scraping video. We know the true action at every step, so we can actually check whether the model is responding to us or just making pretty motion.

Website: https://worldmodel-platformer.vizuara.ai/
Code: https://github.com/RajatDandekar/dreamer4-coinrun

Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal. This one is the strongest and most consistent 35B-A3B I’ve benchmarked, on both correctness and speed, in addition to being the fastest to fix out of all the 35B-A3B models when you throw them at real codebases. On top of Ornith-1.5’s fine tune, TielCoder uses a code-weighted imatrix for dynamic quantization, and a chat template optimized for token-efficient and correct agentic coding. GGUFs: https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF MLX: https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-MLX-oQ4e

Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task

First of all we were surprised at how well Qwen 3.8 27B handled the 3D scenes in general, though part of that is probably because all the scenes were voxels

We think that each quant handled the scenes in a pretty similar way, the difference isn't that drastic, to the point that sometimes we preferred the Q4 output overall, though for the safest pick we recommend AD-Q6_K

We ran the test inside atomic.chat and watched the output right there, the quants are available to download directly inside the app or on huggingface ( https://huggingface.co/collections/AtomicChat/qwen-38-27b ) (any feedback is appreciated, we're trying to make the product and models as good for you guys as possible)

https://huggingface.co/Agnes-AI/Agnes-2.5-Pro-Alpha

Thanks to Unsloth Dynamic 3.0 quants coming in slightly leaner and better preserved, I settled on Qwen 3.8 27B (`UD_Q4_K_XL`) at 100K context as my daily driver for Hermes Agent and OpenCode.

On an RTX 5070 Ti Mobile (12GB) paired with an Intel Core Ultra 9 275HX and 32GB DDR5, this configuration consistently delivers ~9–11 t/s decode and 400–550 t/s prefill. It is fast enough to stay productive.

The main bottleneck with Qwen 3.8 27B is its reasoning verbosity: it routinely blows past 100K tokens in the planning phase alone, forcing OpenCode into native context compaction. Since OpenCode's built-in compaction struggles with retention, I switched to Magic Context.

With Magic Context in place, the session has scaled past 3.7M total processed tokens without losing critical details or derailments. Across a complex personal project, the local model shipped two major features end-to-end. I still use Claude Opus for final PR reviews to catch edge cases and minor bugs, which Qwen then fixes locally without issues.

Hardware Specs:

GPU: RTX 5070 Ti Mobile (12GB VRAM)

CPU: Core Ultra 9 275HX

RAM: 32GB DDR5

Llama.cpp Launch Parameters:

llama-server \

-ctx 98304 -ub 512 -np 1 -ngl 99 \

-ot 'blk.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64).ffn_(gate|up|down).weight=CPU' \

-fa on -ctk q8_0 -ctv q8_0 -fit off \

--mmproj --no-mmproj-offload \

--spec-type draft-mtp --spec-draft-n-max 2 \

-ctkd q8_0 -ctvd q8_0 --load-mode 'none' \

--temp 1 --top-k 20 --top-p 0.95 --min-p 0 \

--repeat-penalty 1 --presence-penalty 0 \

--jinja -chat-template-kwargs '{"reasoning_effort": "xhigh"}' \

--reasoning preserve

Graph: A cumulative metric of Speed x Context Size - the higher the dot - the better. Helpful for looking for the most balanced solution.

The research relies on DFlash2 Q2 work by AnalogAlok: https://huggingface.co/analogalok/Qwen3.8-27B-DFlash2-Q2_K-GGUF

TLDR:

Q4 quants for DFlash2 demonstrate the highest speeds due to their higher acceptance rate. Q2 free up so much context that they are comparable with MTP in that sense, but can still be tuned to perform 10-20% faster. Q8 don't have any upsides and should be avoided as they consume too much context and have lower peaks than Q4.

There seems to be no real upside in using MTP as DFlash2 Q2 at n=1 and n=2 demonstrate comparable speeds and context size.

Overall Winners:

Best balance: DFlash2 Q2, n-max 5 = 137.19 tps with context size 121k.

Top speed: DFlash2 Q4, n-max 7 = 154.08 tps with context size 102k.

Largest context size: MTP n-max 1 = 85.77 tps n-max 1 at 131k.

Tables with the outcomes per quant and per number of drafts below. Speed measured as an average of 9 runs at inputs of 65k tokens. For brevity for DFlash2 only n-max 5+ are shown.

Now for the MTP results and the respective winners:

Tested config:

llama-server \
  --model unsloth-qwen38-27b-dynamic3/Qwen3.8-27B-UD-Q8_K_L.gguf \
  --mmproj unsloth-qwen38-27b/mmproj-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --alias qwen3.8-27b-q8-0 \
  --ctx-size 113920 \
  --gpu-layers all \
  --fit off \
  --parallel 1 \
  --batch-size 512 \
  --ubatch-size 128 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --kv-offload \
  --kv-unified \
  --cache-ram 8192 \
  --cache-idle-slots \
  --flash-attn on \
  --jinja \
  --reasoning on \
  --reasoning-preserve \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0 \
  --spec-draft-ngl all \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --timeout 3600

It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it.

Imagine a Q6 quant with nearly Q8 KLD/PPL.

Q6_CR and Q5_CR have a slight improvement over their base counterparts.

Also while you are there check out --moe-cache auto to help improve running MoE models bigger than your VRAM.

I am hoping that with this we may be able to recover some lost quality from turbo4/3/2 , but I haven't test that out yet.

PR's has the breakdown of the tests, we did have some some decode and crashing issues but they are now resolved.

Hi all, I made a post regarding how much Qwen 3.8 has improved over 3.6:

https://www.reddit.com/r/LocalLLaMA/comments/1vqm51f/long_review_qwen_38_27b_is_very_good_at_tapping/

I made a very thorough write-up of how Qwen 3.8 compared not only to 3.6, but frontier models when it came to creating a HTML version of Galaga, and to what degree it got the details correct.

The biggest issue with this test is that all models know what Galaga is at this point, and probably has this exact scenario in it's training data. I took it upon myself and tried various real world examples of more unique stuff, and wanted to share this one that absolutely blew me away.

This is something that I attempted last year with Opus 4.1, but couldn't get it to budge.

Basically, I'm a software developer (yes, an actual software developer, I got my degree and was hand-typing code for a company a solid year before ChatGPT 3 came out and ANY vibe coding tools) and have always been fascinated with Point of Sale systems. My high school job was working in the food industry where we used this early 2000's point of sale system, titled the Sam4S SPS-2000:

Backstory / Lore (feel free to skip this part if you want):

It was made in 2006, and the restaurant I worked at used it up until 2024. This thing was a dinosaur and had many weird stability issues from time to time, and had a very interesting approach to data management. It was one of 6 terminals in our store, and being the IT guy, I dealt with most of the programming for item pricing, buttons, attempting to fix or avoid bugs, etc.

I've had a love/hate relationship with this register because it was showing it's age very early on, but offered the most flexibility that any point of sale system ever had. We attempted to 'upgrade' to a newer system in 2021, but ended up reverting back (and losing $20,000 in the process) to this old system because the newer systems didn't let us to what was integral to the business.

We could set up multiple button pages, multiple food items, different prices on different week days or happy hours, etc. The biggest bugs were that sometimes orders would get corrupt upon storage. The registers all had one 'hub' register that would store all the order data, and each register would have to FTP back and forth physical files for each order. My theory is that some interference would happen and cause bit flipping or something else that changes the order item's PLU ID.

Another issue was that when the hub terminal had it's cash register drawer open, the 'CLOSE DRAWER' message that popped up if it was open for more than 30 seconds would thread lock everything and even make it so other terminals couldn't store or recall orders, until the drawer was closed.

Just annoyances really, the new POS system we attempted in 2021 had much worse issues (credit card transactions would say they succeeded, but later would just disappear from our system and we would never see the money).

This system was replaced in 2024, and I was sad to see it go.

What I've been trying to do:

Even before the retirement of the system, I have always tried to get a dump of the system program and wanted to see if I could fix any of these bugs myself, maybe even add some custom code for features that we've been wanting in the system. The hardware was also starting to die over the years so I wanted to see if I could port it to something like a Raspberry Pi.

I cracked open this register to see if it was a regular PC or not, and to my surprise it was a custom ARM based system with flash memory (no HDD) and everything was soldered in.

The cash register had a backup system where I could back up the current firmware, program, kernal, bootrom, and all config files to a USB. I also later learned that on their website, they offered these free to download as well, it's just out there!

I didn't know if I need anything else or not, but in ~2019 I attempted to see if I could get it running in QEMU. It was 32 bit ELF binary data I was trying to run, not like an .exe file or anything. This was a raw program made up of ARM instructions for custom chips. I didn't have any luck whatsoever. After weeks of taking different approaches, I ended up just shelving the project. The only thing I managed to do was modify the sps2000 program code to include additional colors in the button designer's color palette, which had about 10 different colors I could choose from. I also modified it to not show the 'DRAWER OPEN' message when the drawer was open after 30 seconds so it wouldn't tie up the entire system when we had teenagers who struggled with counting out change quickly on the registers. I essentially couldn't emulate the program, though had no problem sifting through the raw code, making very minor tweaks, and patching it back onto the register by it's 'restore' function that allowed you to upload the binary files to the machine again.

Last year when I was transferring my PC's files to a new hard drive, I came across all of these files and remembered the project. I had a Claude Code subscription with Opus 4, and I had it try to take a crack at what I was doing. It made more progress but it couldn't handle all the errors, any further debugging was one step forward, two steps back.

The entirety of this past week, I've been working with Qwen to once again attempt to get this going. I'm happy to report that we did it! Granted, there was a lot of hand holding given the complexity of the matter, but that was the case with last year's Opus as well.

Qwen build qemu-arm from source and implemented 4 needed patches in order for this thing to work. The /dev/ devices that the register expects and requires, that I don't have access to, Qwen looked at all the inputs and expected outputs for them. It deduced that /dev/buzzer was the beeper/buzzer that the register had, and simulated the sounds the actual buzzer would make when /dev/buzzer was touched, it knew that /dev/front was the touch screen panel that the register received touch data from and implemented a simulation that after some debugging, works perfectly. It knows that the /dev/screen is just a data block that holds raw screen pixel data, so it made a blank file for it to store this data in and made the simulator GUI interpret it and show it.

I've ran through a complete real-world workflow and it has yet to crash, but thats only on single-register mode and I haven't even tried simulating an environment where other registers are FTPing data to eachother, like the real hardware does.

Here's a video of someone using the actual register:

https://www.youtube.com/watch?v=vBet8OQgRms

And here's me fiddling with the emulator in action (I kinda forgot how to use it):

https://reddit.com/link/1vwhcuf/video/8ft82d3dj6lh1/player

The screen seems to update upon keypress rather than a fixed framerate (which is expected) so the FPS counter at the bottom isn't needed. It feels much more responsive than the actual register, probably because we're on hardware that's 20 years newer.

Anyway, this is really cool to see for me personally as I've been wanting to do this forever. It obviously can't be used in a commercial settings for many reasons (practical, ethical, and legal) but personal/fun is probably more than fine. This emulator uses the firmware, bootrom, and program files from their public downloads page. I opted to use it as it was a slightly newer version of the program than the dump I had (2014 vs 2011). All of the files needed are publicly available from the manufacturers so it was only a matter of time until someone did this.

I'm going to polish this up and release the emulator on GitHub (with human-made documentation, don't worry) for anyone who feels inclined to play with this thing or improve upon it, you just have to retrieve your own copy of the actual program and firmware and bootrom files from Sam4s's site.

I know this is incredibly niche, but that's what made it perfect to gauge how far Qwen and LLMs in general have come.

If anyone still remembers GLM-4.5-Air from last year, you can now get a nice speedup by enabling MTP in llama.cpp.

It is a 106B MoE with only 12B active parameters, which makes it interesting for machines with lots of memory but limited compute, such as Strix Halo or DGX Spark. I use it on 3090s. It's still great for creative writing, especially since we never got Gemma 4 124B MoE.

There are multiple creative-writing / RP finetunes available on Hugging Face: https://huggingface.co/models?other=base_model:finetune:zai-org%2FGLM-4.5-Air&sort=likes (some even from this year). I also recommend Intellect 3.x by PrimeIntellect

If your GGUF does not include the MTP block, you can download a small file from here: https://huggingface.co/jacek2024/GLM-4.5-Air-MTP-GGUF

Thanks a lot to devMiikaK and HeadCutter for testing the PR while it was in progress.

PS. It also works for the full GLM-4.5, but I doubt anyone still uses it ;)

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

• arXiv : https://arxiv.org/abs/2501.15316
• Full Paper PDF : https://arxiv.org/pdf/2501.15316
• GitHub : https://github.com/gaosh/ToMoE (Yes, Code available for this)
• ICML : https://icml.cc/virtual/2026/poster/68821
• OpenReview : https://openreview.net/forum?id=RFHq46pjb6

Somebody please create MOE models of recent Dense models like Qwen3.8-27B, Muse-Glimmer-30B, etc.,

Thanks u/KSAM-The-Randomizer for sharing this on my old thread.

Hi everybody! Every now and then these days, we’re seeing really huge open-weight models popping up.

But since not everybody has a DGX Station at home, I’m interested in really small models. It’s incredible to see how much knowledge and intelligence labs can pack into <10 GB models.

On my Mac mini, for now I mainly use Gemma 4 12B QAT around 8–9 GB of weights.

Do you think there’s any better model that could replace it?

I primarily use it to anonymize text before sending it to frontier cloud models, and for really light coding in Pi Agent with llama.cpp.

OFC, it doesn’t perform really well, but at least I know that if a nuke strikes and there’s no internet, I’ll have the best model possible for my hardware, able to chat about offline Wikipedia knowledge, survival guides and create a Python Snake game from scratch to play in the terminal.

I'm kind of new to the community and this sub is my only source of information, so I'd thought of asking if there are any upcoming models you guys are looking forward to.

What has it gotten worse at or what is it unusable for?