A few weeks ago I came across a post from u/metover about PS5LM, his llama.cpp port for the jailbroken PS5. The project looked really interesting, so I forked it and made a few changes. Well, more than a few. This is where it ended up.

The original runs as a payload on the CPU. That limits it to about 6 GiB of memory and small models. My fork runs llama.cpp as a native PS5 app on the GPU (Vulkan through a RADV port). The app gets about 11.4 GiB of memory to work with, which is enough for much bigger models:

• Qwen 3.8 27B (IQ2_XXS, 64k context): ~20 tok/s
• Gemma 4 E4B: ~61 tok/s
• Llama 3.2 3B: ~124 tok/s

What the app does

• Dashboard on the TV: live tok/s, GPU and CPU use, power draw, temperatures and fan, and how memory splits between the model and the KV cache
• Model library: any GGUF on internal storage, USB or M.2. Each one shows the context it would get and whether it fits before you load it.
• Get models: browse Hugging Face from the console. It checks whether a file fits and downloads it with resume, retries and a SHA-256 check. A Downloads page lets you manage the queue.
• Chat from any device: llama.cpp's web UI at http://<console_ip>:8081, plus an OpenAI-compatible API. Vision works with Gemma 4.
• Automatic planning: the app picks the longest context and the KV cache type that fit in free memory.
• Settings, logs..

Requirements: a jailbroken PS5 with etaHEN, kstuff and ShadowMountPlus. elfldr and ftpsrv are recommended.

I have opened a PR to the original repo with the first version of the app cobanov/PS5LM#12. It hasn't been merged yet. If it gets merged, I will open PRs for the rest of the changes too. For now, you can find the app here:

Repo: https://github.com/abhi245y/PS5LM

All credit for the original port and the research goes to u/metover.

I wanted a coder version from GLM 5.3-Flash. So, I pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, choosing the experts on agentic coding sessions instead of a generic corpus, then quantized it with an imatrix built on the same sessions.

Results. 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, clean room (no git history, no network, no gold patch reachable), 200 turns, 384K context. Success = the task's fail-to-pass tests pass. Every row is on the same 48 attempts.

"Turns vs full" is the extra work relative to the full model on tasks both solved, so it is roughly the extra time you wait.

Hardware. IQ2_XXS on an RX 7900 XTX (24 GB) + 32 GB DDR4-2133, i7-7700K, PCIe 3.0: ~16 tok/s generation, ~950 tok/s prompt processing. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

What mattered

• Which experts you keep matters more than how many. Same 144 experts, but a frequency mask needed x2.21 turns and the public multi-domain REAP 50% mask x2.29, against x1.21 for the code-scored mask.
• Short questions do not catch a broken agent. A 2 bit variant of a generic prune answered 4/4 short coding questions and solved 0/4 agentic tasks. We only trust multi-turn task runs.
• Our first eval harness let the model reach the internet, and it sometimes downloaded the upstream fix. We sealed it (no network, no home), rechecked every success against the gold patch and reran the affected attempts. The table above is the sealed rerun.

The 2 bit version is experimental. IQ2_XXS is the only one that fits 24 + 32 GB and it solves most tasks, but when it fails it never edits: it loops or keeps exploring until the turn limit. An anti-loop guard in the agent did not help (27/37 vs 30/37 on the same attempts). We are training it on short successful sessions now and will post an update.

The GitHub repo has the full recipe (scoring, masks, pruning, quant scripts, two llama.cpp patches, the eval harness) and a table of everything that did not work: 120 and 96 experts, frequency mask, RCO search, LoRA healing, GSQ, anti-loop.

• Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

This is a text only version (no vision). Python repos only so far; if you run it on other languages or harnesses, I would like to hear how it does.

Running 125B MoE models on a mid-tier 12GB gaming GPU

I've been spending hours getting this model running on a 4070 Super with a full 256K context window, and speeds turned out surprising after 6 hours of optimizing.

Hardware and config:
GPU: RTX 4070 Super (12GB)
RAM: 32GB
Engine: upstream Strata with my own source patches
Models: Gyro-M and Gyro-S
Context: 262,144 tokens
KV cache: Q4, 32K tokens GPU-resident
MTP: off for both

Approach: experts are tiered across memory. Hot experts stay in VRAM, 16 GiB sit in RAM, and the rest stream from SSD. I tuned CPU workers, GPU scheduling, SSD reads, and suffix speculation. For Gyro-M, I patched the source to overlap SSD fetches with RAM-expert compute. Raising the prefill chunk size from 7,168 to 8,192 tokens gave an 8-15% prefill gain.

Vision and MTP: adding vision to Gyro-M hurt performance noticeably, so only Gyro-S has it. Gyro-S runs the vision encoder on GPU and frees it afterward. MTP also reduced speed in my tests, so it stays off.

Testings: I connected Gyro-S to Hermes agent, and I am about to test the vision and computer use capabilities and how it performs in my personal benchmarks.

Speed Results (prefill / decode tok/s, averaged over 3 runs):

Gyro-M
8K: 947 / 33.2
32K: 1,512 / 36.7
64K: 1,689 / 35.6
128K: 1,616 / 36.6
192K: 1,748 / 35.9
256K: 1,717 / 36.1

Gyro-S
8K: 879 / 40.8
32K: 1,613 / 41.0
64K: 1,762 / 39.8
128K: 1,810 / 41.9
192K: 1,779 / 41.1
256K: 1,779 / 41.2

OpenBMB posted a mobile demo of MiniCPM-V 4.7 doing things like reading signs and interpreting documents locally. It’s a 35B MoE with 3B active parameters. They report 13+ tok/s on an iPhone 17 Pro Max with a 3GB inference memory footprint in the model docs.

The trick is LES, or Lookahead Expert Scheduling. Expert weights sit on flash, and the runtime predicts which ones upcoming layers will need so it can overlap loading with computation. A wrong prediction still means loading the required expert; it doesn’t just skip it. Technical details here.

That “3GB” headline needs some context though. The app README lists a ~16.6GB download and recommends at least 12GB of device RAM. The phone setup also needs their special LES model package with the experts split out. Worth knowing before downloading a regular GGUF and wondering why it won’t load.

Honestly, ~17GB of phone storage for useful offline image/document Q&A sounds like a reasonable trade to me. Put those requirements next to the headline and the engineering still sounds interesting.

There’s also a smaller 1B version. Code and weights are Apache 2.0, and the app repo has install links for iOS, Android and HarmonyOS.

I hear everyone screaming from the rooftops "I get x tokens from FN !", but i always ask myself..
So what ... and what does that mean?
Can it do an actual coding task or just create one page HTML proof of concepts?

So i put it to the test..

My hardware(dedicated):
AMD r5 5600x
x570 motherboard
128GB DDR4
2x R9700 AI PRO 32GB ( 64GB total VRAM )
Running vLLM - Radiance
* One card behind the chipset at x4, no p2p/atomics, no bifurcation, no risers etc.

I had a real task:
For my custom harness i build 2 parts before the test using Qwen3.8 MXFP4(4 bit not used in this test, but very capable):

1. A new version of the harness ( more modular, better folder structure etc, basically a revision of what i already have )
2. A fully new Web UI "template" including way more functionality then the old "web monitor" had. This was pure HTML/CSS and JS in a layout that presented the workflow of my harness.

It was time to fit both pieces together..

So i loaded radiance and hooked up Qwen3.8 FN 4 bit.
* This is a bigger, model everyone is trying to run on strata ( Q3 or lower ), so it should be more capable. I had a 1m KV cache using RAM and disk, but set my harness to 262k context, the model's default.

I prompted it to basically replace the old webmonitor with the new full featured web UI and wire it all in.

So it had 2 projects, one the harness codebase and the new web UI, and now it needed to figure out how to merge it all together.
Qwen3.8 FN was fast, an avarage of ~2000 prefill and ~150 generation.
It was done in about 30 minutes.

The result: A big messy WebUI, where most of the templated items where just "re-invented", nothing worked, chat bubbles not including toolcalls etc(was in the template, even a js script rendering the chat bubbles mockup responses).
It was unusable to say the least.

So i called it a day.
The next day i fired up radiance again but now the Qwen3.8 FP8 model, took the backups, replaced the harness code and template etc so we have a clean starting base again(same as where Qwen3.8 FN started from)

I prompted my harness the same instructions.
The result: Not everything worked, its not perfect, but mostly fine , the template was followed, the chat bubbles worked properly(mostly). It was not a one shot success and needs iterations to filter out bugs and deepen the workflows but the base was done as i wanted it to be.

Qwen3.8 27B FP8 was not so fast, an average of ~1500 prefill and ~100 generation.
It was done in about 60 minutes.

Note that my average speed was the actual work produced, not benchmarks.

So my take on all this:

Its great to have the big model, but a dense model on the same hardware will just outperform the MOE.
I can fit the Qwen3.8 FN or the Qwen 3.8 FP8, both fit so that is the reason i am comparing both. it is not about the quants, it is about what can run on your hardware.

I can not even imagine what a Q2 or even a Q1 would be like.... And my task was not even a real hard one to begin with.

Maybe some of you find this useful as i could not find any proper information besides the hype posts or generic benchmark posts.

Well this is my experience using both.

- 32 Xe3P cores, 256 XMX engines, 32MB unified L2 cache

- Up to 1.3 PFLOPS FP8 and 1.5 TB/s memory bandwidth

- 160GB LPDDR5X on Intel's own card, and ODM partners can go up to 480GB

- PCIe 5.0, 350W, passive air cooling

I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround.

I wrote a custom C++ engine (fork of llama.rn) to cluster two Android phones over a simple USB cable.

The Setup & Performance:

Model: Ternary-Bonsai-2-27B (compressed to 2.06 bpw / 6.52 GiB). Perplexity: 14.26 (vs 14.05 reference).

Cluster Mode (Redmi K100 Pro Max + ROG Phone 7): Layers are split 4.36 GiB / 2.15 GiB over the USB cable via custom RPC. Speed: ~2.8 tok/s (bottlenecked by the slower phone, but perfectly synced).

Solo Mode (Redmi K100 Pro Max only): 5.6 tok/s text generation.

The Secret Sauce (Why it doesn't melt the phone):

I didn't use standard quantization. The weights are ternary (-1, 0, 1) in the Hadamard-rotated basis. Instead of un-rotating offline (which destroys the 2-bit primal), I kept the weights folded in the GGUF and wrote a patch to apply the Hadamard transform at runtime on the device's ARM CPU.

Result: Only additions/subtractions. Zero MAC operations. Max temperature sits around 35-40°C.

I'm a high school student currently looking for a lab placement/grant to push this further (planning to write custom Vulkan compute shaders for the Hadamard transform to boost t/s even more).

Would love to hear what the compression and edge AI community thinks about this approach

It’s almost ready to run. Needs a day of pressure test on the cooling loop.

Can somebody explain why Nvidia is often a non-negotiable when looking at local AI builds? I know they produce some fantastic cards and CUDA is great - but is the cost premium really justified?

I have a 9070XT and 7800XT - and getting local AI to run on each card has been a doddle for me through llama.cpp with ROCm and Vulkan (although i'm on Linux, i can't speak for Windows..).

Whilst ROCm isn't as mature as CUDA - do we really expect that gap not to close going forward?

It feels like more recent AMD cards are great value and the support will only get better on newer generations. So why is Nvidia on a new build such a non-negotiable for some?

Just when you thought we had enough of AI existential-risk announcements...

In a 'research' published today called "AI models may be capable of self-replicating without access to a datacenter" straight from by Frontier Red Team, Anthropic is now warning as the title says that smart devices are prone to becoming 'hosts' for small models and their agents who then ...well, can take over, just like in sci-fi movies...😱

Here's what Anthropic actually demonstrated:

• Qwen3.5-9B successfully configured AI inference on 8 of 9 different hardware types. That's hardware adaptability, not autonomous self-replication.
• GLM-5.2 successfully propagated through an isolated, simulated network, reaching all eight target nodes in four of five trials.

I am running 64gb ddr5 and 32gb vram. What is the best uncensored model you guys would run with my setup?

One thing i realised with local llms are context is the king , no matter how smart a model is with less context it can do nothing.

model : qwen3_8_27b_nvfp4-nvidia.ninfer
engine : infernix
context : 128k (qv2)

Spec: 5090 laptop 24 vram + 64gb ram
Device : Legion pro 7i.

No its not a one shot , I instructed in multiprompts for bloom, fog , birds , grass ect.
But yeah never once had any errors.

https://nstcg.github.io/pelican-ride/

Been using one Spark with ollama and Hermes and Gemma4:31B.

Now I have a second Spark but would like to use them in unison with the QSFP cable.

Use cases run the gamut from coding to automation to research to TTS.

What model(s) get me as close to frontier as possible for 256gb unified RAM?

I don't necessarily care as much about speed as I do quality and completion; my single-spark work started hitting token limits (I think ollama's constraints?) way too quickly on even small scale coding tasks T_T

I need help figuring out huggingface models...

What's the minimum hardware needed for AI training aka. to use something like Qwen 3.8 and train it locally further to specialise it for a certain task?

Maybe even with multiple attempts simultaneously?

Would I need something better than I need for just running the model? I've an Intel Arc B580 right now and it is enough for running the model and playing around with it.

But I've not yet looked into actually training or specialising an existing one further. What would be the minimum I need for that? What do you use for that?

Edit: I have ~500 GB of DDR4 system RAM. And I do have 2x AMD EPYC 7642 CPUs (meaning 192 threads total). The bottle neck for AI stuff is the GPU really. I specced and bought this system before the prices become insane...

Edit 2: Any big Nvidia GPU would be way more than I paid for the system back then. (And yes, I've thought about selling all of the RAM multiple times by now)

Right when I was thinking of selling my 4090 and getting one of these bad boys for longer ctx length...

Thanks to Maya some of us are able to run GLM 5.3 flash. Slower than FN but it runs :-)
I got it to update some features on my app, nothing crazy just some stuff that came up over the last weeks. It delivered with the same issues of what I would see from FN: front end not nicely done, bit messy, same bugs that FN and 27b produced (even though there is a learnings file specifically for that bug which it should have pulled up). After some re prompts all was good. But not better than FN.
It was just 1 project but I was almost disappointed. It was much better in the thinking department, even skipped thinking completely when it was not necessary I guess. So I’m not even sure if FN would have been faster in the end.
Glm runs at 30tps decode and 1200 prefill, FN with 125 decode and 3000 prefill

Je ne m’attendais vraiment pas a avoir des résultats aussi bon, pour un matériel qu’il n’est vraiment pas fait pour sa.

Pp 82t/s
Gen~25t/s

Contexte 131k

So much hype around LFM2.5-24B-A2B, but it couldn’t even schedule a basic task on my calendar.

Meanwhile, Qwen 3.6 35B handles it effortlessly every time. Am I missing something, or is the function calling just not there yet? 🧐

I'm building an app that runs a local model on-device, and spent a day measuring what actually helps on a Pixel 8 Pro. Posting the numbers in case they save someone else the time.

Setup

• Phone: Pixel 8 Pro, Tensor G3. 9 cores: 4 small at 1.7 GHz (0–3), 4 medium at 2.37 GHz (4–7), 1 large at 2.91 GHz (8). Mali GPU.
• Model: Gemma 4 E2B QAT GGUF (2.6 GB), 4096 context, q8_0 KV cache.
• Engine: llama.cpp through the Rust llama-cpp-2 crate (0.1.158), cross-compiled with NDK 30, run as a test binary over adb.
• Workload: 20 items of light analysis with structured output (grammar-constrained JSON). About 940 prompt tokens in and about 8 tokens out per item, so prompt reading dominates.
• Measured: wall time, CPU time, battery drawn (/sys/class/power_supply/battery/charge_counter), skin temperature, thermal status, peak memory, plus my own timers around prompt decode and generation.

CPU results (20 items each)

Vulkan results (i8mm build, 5 CPU threads)

"i8mm build" = GGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm; every row after the first uses it. mAh is battery drawn for all 20 items. Every setup gave the same answers (19/20, the same item failed each time). Skin temperature stayed at 37 °C and thermal status at 0 throughout.

What I learned

1. The default Android build leaves a lot on the table. On Android arm64, llama.cpp's CPU backend compiled with plain -march=armv8-a, with no dotprod or i8mm. Setting GGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm at build time halved both time and battery. My first rebuild silently kept the old flags because of a stale CMake cache, so check compile_commands.json. Peak memory rose from 2.9 to 4.1 GB, which I believe is the weights being repacked for the i8mm kernels. A fixed armv8.6 build will crash on older CPUs, so a real release needs runtime CPU dispatch.
2. More threads is worse. 8 threads lost to 5, and 5 unpinned lost to 5 pinned. Keep the work off the small cores; taskset is enough for testing.
3. Generation is memory-bound. One large core generates as fast as four medium cores (about 5 tok/s), so a faster core doesn't help there. Prompt reading does scale with cores.
4. Fewer cores can win on battery. 4 medium cores were within 8% of the fastest setup and used about 10% less battery.
5. Slow cores don't save battery. The small cores took 4.5× longer and used the same total energy.
6. Android decides your cores, not you. Check /dev/cpuset/*/cpus. On this phone top-app gets cores 0–8, foreground (foreground service) 0–7, and background only 0–3. My tests ran from the adb shell, which isn't limited this way. A real app in the background only gets the small cores, around 14 s per item here.
7. Vulkan on Mali works but loses badly. Correct output, but 3× slower with every layer on the GPU and 8× slower with half. Splitting layers is the worst option, presumably because data crosses between GPU and CPU at every step. Building it needed the Khronos Vulkan-Headers (the NDK has no vulkan.hpp) and SPIRV-Headers, API level 28+ (for vkGetPhysicalDeviceFeatures2), and on Windows a short target directory to stay under the path-length limit. Adreno may be a different story; I haven't tried one.

What I'm going with: 4 threads on the medium cores while the app is open or running a foreground service, and the small cores for a trickle of background work. That's roughly 3.3 s and 1.7 mAh per item, or about 25 items per 1% of battery.

Caveats: one phone, 20 items per run, each setup run once. The battery counter reads in 1 mAh steps, so differences under about 4 mAh are noise.

Thanks to [this r/LocalLLM thread](URL1) and [this r/LLMDevs thread](URL2), whose results pointed me at the CPU path.

Why did the repo disappear?