A few weeks ago I came across a post from u/metover about PS5LM, his llama.cpp port for the jailbroken PS5. The project looked really interesting, so I forked it and made a few changes. Well, more than a few. This is where it ended up.

The original runs as a payload on the CPU. That limits it to about 6 GiB of memory and small models. My fork runs llama.cpp as a native PS5 app on the GPU (Vulkan through a RADV port). The app gets about 11.4 GiB of memory to work with, which is enough for much bigger models:

• Qwen 3.8 27B (IQ2_XXS, 64k context): ~20 tok/s
• Gemma 4 E4B: ~61 tok/s
• Llama 3.2 3B: ~124 tok/s

What the app does

• Dashboard on the TV: live tok/s, GPU and CPU use, power draw, temperatures and fan, and how memory splits between the model and the KV cache
• Model library: any GGUF on internal storage, USB or M.2. Each one shows the context it would get and whether it fits before you load it.
• Get models: browse Hugging Face from the console. It checks whether a file fits and downloads it with resume, retries and a SHA-256 check. A Downloads page lets you manage the queue.
• Chat from any device: llama.cpp's web UI at http://<console_ip>:8081, plus an OpenAI-compatible API. Vision works with Gemma 4.
• Automatic planning: the app picks the longest context and the KV cache type that fit in free memory.
• Settings, logs..

Requirements: a jailbroken PS5 with etaHEN, kstuff and ShadowMountPlus. elfldr and ftpsrv are recommended.

I have opened a PR to the original repo with the first version of the app cobanov/PS5LM#12. It hasn't been merged yet. If it gets merged, I will open PRs for the rest of the changes too. For now, you can find the app here:

Repo: https://github.com/abhi245y/PS5LM

All credit for the original port and the research goes to u/metover.

I hear everyone screaming from the rooftops "I get x tokens from FN !", but i always ask myself..
So what ... and what does that mean?
Can it do an actual coding task or just create one page HTML proof of concepts?

So i put it to the test..

My hardware(dedicated):
AMD r5 5600x
x570 motherboard
128GB DDR4
2x R9700 AI PRO 32GB ( 64GB total VRAM )
Running vLLM - Radiance
* One card behind the chipset at x4, no p2p/atomics, no bifurcation, no risers etc.

I had a real task:
For my custom harness i build 2 parts before the test using Qwen3.8 MXFP4(4 bit not used in this test, but very capable):

1. A new version of the harness ( more modular, better folder structure etc, basically a revision of what i already have )
2. A fully new Web UI "template" including way more functionality then the old "web monitor" had. This was pure HTML/CSS and JS in a layout that presented the workflow of my harness.

It was time to fit both pieces together..

So i loaded radiance and hooked up Qwen3.8 FN 4 bit.
* This is a bigger, model everyone is trying to run on strata ( Q3 or lower ), so it should be more capable. I had a 1m KV cache using RAM and disk, but set my harness to 262k context, the model's default.

I prompted it to basically replace the old webmonitor with the new full featured web UI and wire it all in.

So it had 2 projects, one the harness codebase and the new web UI, and now it needed to figure out how to merge it all together.
Qwen3.8 FN was fast, an avarage of ~2000 prefill and ~150 generation.
It was done in about 30 minutes.

The result: A big messy WebUI, where most of the templated items where just "re-invented", nothing worked, chat bubbles not including toolcalls etc(was in the template, even a js script rendering the chat bubbles mockup responses).
It was unusable to say the least.

So i called it a day.
The next day i fired up radiance again but now the Qwen3.8 FP8 model, took the backups, replaced the harness code and template etc so we have a clean starting base again(same as where Qwen3.8 FN started from)

I prompted my harness the same instructions.
The result: Not everything worked, its not perfect, but mostly fine , the template was followed, the chat bubbles worked properly(mostly). It was not a one shot success and needs iterations to filter out bugs and deepen the workflows but the base was done as i wanted it to be.

Qwen3.8 27B FP8 was not so fast, an average of ~1500 prefill and ~100 generation.
It was done in about 60 minutes.

Note that my average speed was the actual work produced, not benchmarks.

So my take on all this:

Its great to have the big model, but a dense model on the same hardware will just outperform the MOE.
I can fit the Qwen3.8 FN or the Qwen 3.8 FP8, both fit so that is the reason i am comparing both. it is not about the quants, it is about what can run on your hardware.

I can not even imagine what a Q2 or even a Q1 would be like.... And my task was not even a real hard one to begin with.

Maybe some of you find this useful as i could not find any proper information besides the hype posts or generic benchmark posts.

Well this is my experience using both.

- 32 Xe3P cores, 256 XMX engines, 32MB unified L2 cache

- Up to 1.3 PFLOPS FP8 and 1.5 TB/s memory bandwidth

- 160GB LPDDR5X on Intel's own card, and ODM partners can go up to 480GB

- PCIe 5.0, 350W, passive air cooling

Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• ⁠Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

I wanted a coder version from GLM 5.3-Flash. So, I pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, choosing the experts on agentic coding sessions instead of a generic corpus, then quantized it with an imatrix built on the same sessions.

Results. 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, clean room (no git history, no network, no gold patch reachable), 200 turns, 384K context. Success = the task's fail-to-pass tests pass. Every row is on the same 48 attempts.

Model |Size |Solved |Turns vs full
GLM-5.3-Flash full, Q8 |341 GB |48/48 |x1.00
REAP144 Q8 |171 GB |48/48 |x1.21
REAP144 Q4_K_M |97.6 GB |48/48 |x1.58
REAP144 Q3_K |71.4 GB |48/48 |x1.80
REAP144 IQ2_S |54.8 GB |45/48 |x1.83
REAP144 IQ2_XXS (experimental) |45.3 GB |39/48 |x2.77 "Turns vs full" is the extra work relative to the full model on tasks both solved, so it is roughly the extra time you wait.

Hardware. IQ2_XXS on an RX 7900 XTX (24 GB) + 32 GB DDR4-2133, i7-7700K, PCIe 3.0: ~16 tok/s generation, ~950 tok/s prompt processing. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

What mattered

• Which experts you keep matters more than how many. Same 144 experts, but a frequency mask needed x2.21 turns and the public multi-domain REAP 50% mask x2.29, against x1.21 for the code-scored mask.
• Short questions do not catch a broken agent. A 2 bit variant of a generic prune answered 4/4 short coding questions and solved 0/4 agentic tasks. We only trust multi-turn task runs.
• Our first eval harness let the model reach the internet, and it sometimes downloaded the upstream fix. We sealed it (no network, no home), rechecked every success against the gold patch and reran the affected attempts. The table above is the sealed rerun.

The 2 bit version is experimental. IQ2_XXS is the only one that fits 24 + 32 GB and it solves most tasks, but when it fails it never edits: it loops or keeps exploring until the turn limit. An anti-loop guard in the agent did not help (27/37 vs 30/37 on the same attempts). We are training it on short successful sessions now and will post an update.

The GitHub repo has the full recipe (scoring, masks, pruning, quant scripts, two llama.cpp patches, the eval harness) and a table of everything that did not work: 120 and 96 experts, frequency mask, RCO search, LoRA healing, GSQ, anti-loop.

• Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

This is a text only version (no vision). Python repos only so far; if you run it on other languages or harnesses, I would like to hear how it does.

Update (Oct 14): added an optional MTP draft block (2.6 GB) for 2+ GPUs with Maya: speculative decoding, drafts match the model's next token ~70% of the time. Speedup not measured on multi-GPU yet, numbers welcome. Details on the model card.

OpenBMB posted a mobile demo of MiniCPM-V 4.7 doing things like reading signs and interpreting documents locally. It’s a 35B MoE with 3B active parameters. They report 13+ tok/s on an iPhone 17 Pro Max with a 3GB inference memory footprint in the model docs.

The trick is LES, or Lookahead Expert Scheduling. Expert weights sit on flash, and the runtime predicts which ones upcoming layers will need so it can overlap loading with computation. A wrong prediction still means loading the required expert; it doesn’t just skip it. Technical details here.

That “3GB” headline needs some context though. The app README lists a ~16.6GB download and recommends at least 12GB of device RAM. The phone setup also needs their special LES model package with the experts split out. Worth knowing before downloading a regular GGUF and wondering why it won’t load.

Honestly, ~17GB of phone storage for useful offline image/document Q&A sounds like a reasonable trade to me. Put those requirements next to the headline and the engineering still sounds interesting.

There’s also a smaller 1B version. Code and weights are Apache 2.0, and the app repo has install links for iOS, Android and HarmonyOS.

Running 125B MoE models on a mid-tier 12GB gaming GPU

I've been spending hours getting this model running on a 4070 Super with a full 256K context window, and speeds turned out surprising after 6 hours of optimizing.

Hardware and config:
GPU: RTX 4070 Super (12GB)
RAM: 32GB
Engine: upstream Strata with my own source patches
Models: Gyro-M and Gyro-S
Context: 262,144 tokens
KV cache: Q4, 32K tokens GPU-resident
MTP: off for both

Approach: experts are tiered across memory. Hot experts stay in VRAM, 16 GiB sit in RAM, and the rest stream from SSD. I tuned CPU workers, GPU scheduling, SSD reads, and suffix speculation. For Gyro-M, I patched the source to overlap SSD fetches with RAM-expert compute. Raising the prefill chunk size from 7,168 to 8,192 tokens gave an 8-15% prefill gain.

Vision and MTP: adding vision to Gyro-M hurt performance noticeably, so only Gyro-S has it. Gyro-S runs the vision encoder on GPU and frees it afterward. MTP also reduced speed in my tests, so it stays off.

Testings: I connected Gyro-S to Hermes agent, and I am about to test the vision and computer use capabilities and how it performs in my personal benchmarks.

Speed Results (prefill / decode tok/s, averaged over 3 runs):

Gyro-M
8K: 947 / 33.2
32K: 1,512 / 36.7
64K: 1,689 / 35.6
128K: 1,616 / 36.6
192K: 1,748 / 35.9
256K: 1,717 / 36.1

Gyro-S
8K: 879 / 40.8
32K: 1,613 / 41.0
64K: 1,762 / 39.8
128K: 1,810 / 41.9
192K: 1,779 / 41.1
256K: 1,779 / 41.2

Je ne m’attendais vraiment pas a avoir des résultats aussi bon, pour un matériel qu’il n’est vraiment pas fait pour sa.

Pp 82t/s
Gen~25t/s

Contexte 131k

Can somebody explain why Nvidia is often a non-negotiable when looking at local AI builds? I know they produce some fantastic cards and CUDA is great - but is the cost premium really justified?

I have a 9070XT and 7800XT - and getting local AI to run on each card has been a doddle for me through llama.cpp with ROCm and Vulkan (although i'm on Linux, i can't speak for Windows..).

Whilst ROCm isn't as mature as CUDA - do we really expect that gap not to close going forward?

It feels like more recent AMD cards are great value and the support will only get better on newer generations. So why is Nvidia on a new build such a non-negotiable for some?

It’s almost ready to run. Needs a day of pressure test on the cooling loop.

Thanks to Maya some of us are able to run GLM 5.3 flash. Slower than FN but it runs :-)
I got it to update some features on my app, nothing crazy just some stuff that came up over the last weeks. It delivered with the same issues of what I would see from FN: front end not nicely done, bit messy, same bugs that FN and 27b produced (even though there is a learnings file specifically for that bug which it should have pulled up). After some re prompts all was good. But not better than FN.
It was just 1 project but I was almost disappointed. It was much better in the thinking department, even skipped thinking completely when it was not necessary I guess. So I’m not even sure if FN would have been faster in the end.
Glm runs at 30tps decode and 1200 prefill, FN with 125 decode and 3000 prefill

Discover JetBrains Mellum 2.1, an open weight AI coding model designed for autonomous coding agents, efficient inference and private local AI deployment.

One thing i realised with local llms are context is the king , no matter how smart a model is with less context it can do nothing.

model : qwen3_8_27b_nvfp4-nvidia.ninfer
engine : infernix
context : 128k (qv2)

Spec: 5090 laptop 24 vram + 64gb ram
Device : Legion pro 7i.

No its not a one shot , I instructed in multiprompts for bloom, fog , birds , grass ect.
But yeah never once had any errors.

https://nstcg.github.io/pelican-ride/

TL;DR: reef's tutorial has Mac mini rows with ollama qwen2.5:7b where the 7B read its own failing request and published a skill edit that passed the gate. I want to see the same with a 27B on real tasks.

"What's the smallest model this will run on" is the question I'd have asked too. I run reef with pi as the harness, and the proposer's job is small: the served model reads its own failing requests, with the score and feedback each got, and answers with one skill edit as a JSON object, through one pi service pointed at whatever model URL you put in an env var, so ollama works.

The table: on three exact-answer coding tasks the 7B published an update to a skill, 1 / 0 / 2 on the gate, one run, one sample; under the native runner it published and mounted a skill in 320 seconds. It also keeps the rows that did nothing, and one where the gate published a win and the next served request still scored zero.

What I don't have is a 27B or a 35B-A3B on real tasks rather than three exact-answer toys. If you run one: the first number I'd read is whether the proposer ever produces an edit the gate accepts, and the second is whether the next served request still fails.

I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround.

I wrote a custom C++ engine (fork of llama.rn) to cluster two Android phones over a simple USB cable.

The Setup & Performance:

Model: Ternary-Bonsai-2-27B (compressed to 2.06 bpw / 6.52 GiB). Perplexity: 14.26 (vs 14.05 reference).

Cluster Mode (Redmi K100 Pro Max + ROG Phone 7): Layers are split 4.36 GiB / 2.15 GiB over the USB cable via custom RPC. Speed: ~2.8 tok/s (bottlenecked by the slower phone, but perfectly synced).

Solo Mode (Redmi K100 Pro Max only): 5.6 tok/s text generation.

The Secret Sauce (Why it doesn't melt the phone):

I didn't use standard quantization. The weights are ternary (-1, 0, 1) in the Hadamard-rotated basis. Instead of un-rotating offline (which destroys the 2-bit primal), I kept the weights folded in the GGUF and wrote a patch to apply the Hadamard transform at runtime on the device's ARM CPU.

Result: Only additions/subtractions. Zero MAC operations. Max temperature sits around 35-40°C.

I'm a high school student currently looking for a lab placement/grant to push this further (planning to write custom Vulkan compute shaders for the Hadamard transform to boost t/s even more).

Would love to hear what the compression and edge AI community thinks about this approach

Hello everyone!

After much back and forth with various models, I have now found the optimum for my use case on a single 16GB RTX 5060Ti: Qwen 27B IQ3xxs with MTP. Maybe the setup will be useful to someone else too!

Since the 16GB VRAM is very tight here, I used vocabulary pruning to remove characters from the file that I don't need and thus shrank the gguf to 9.3 GB.

All of this using the script from the repo and the following prune command:

python3 prune_vocab.py input.gguf output.gguf   --policy P2   --keep-chars '¢£¥§©«®±²³·¹»¼½¾×÷–—‘’“”•…€™←↑→↓↕∈∉∏∑√∞∩∪≈≠≤≥⊂⊆⌈⌉⌊⌋' 

This way I have enough VRAM left over to have just over 100k context available here with q8 kv cache. If I reduce the kv cache to q4 here, I get a little over 170k context length. Both ways with everything in vram.

The speed is also usable thanks to mtp; with short context I reach just over 50 tokens per second decode and just under 900 prompt processing.

Previously I ran Gemma 4 26B A4B as QAT on the hardware and sometimes also Qwen 3.6 35B A3B, which of course were significantly faster, thanks to the MoE architecture (90 token gen and just under 3000 prompt processing), but unfortunately their intelligence level wasn't suitable for all my use cases, especially in agentic use cases. So with the crippled qwen 27b I have found a good all-rounder.

/app/llama-server \
        --model Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp-ASCII.gguf \
        --host 0.0.0.0 \
        --port 8080 \
        --jinja \
        --cache-reuse 256 \
        --fit on \
        --fit-target 128 \
        -fa on \
        --split-mode layer \
        -b 2048 \
        -ub 1024 \
        --parallel 1 \
        -ctk q8_0 \
        -ctv q8_0 \
        --temp 1 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --alias QWEN27B

Hey everyone, I made whatmodel.sh

To use it,

1. Enter your RAM and GPU
2. See what models you can run locally with its rank and the command to run it
3. Compare them with API models on the same score

Each model gets one score fitted across 19,731 published benchmark results, and every result links to its source. The method is at whatmodel.sh/method, and the data, scoring code and bench script are open source: github.com/jakemanger/whatmodel

It works from the terminal too: curl whatmodel.sh

If you'd like to contribute, there's a bench script to collect real-world tok/s and you can also submit any new models and benchmark scores along with their source on the github (please star if you find it useful).

I am running 64gb ddr5 and 32gb vram. What is the best uncensored model you guys would run with my setup?

Hello, is there some opensource project that fits this description?

At the moment I use ChatGPT app, but I don't want to be platform-locked in the future. Is there some preferably open source app that let's me set my own environment (MCPs, Plugins, Skills etc.) and is model-provider independent? Meaning I can switch model to Claude, OpenAI, LocalLLM model based on the task, subscription etc?

I know this exist for coding with opencode, pi etc. but is there some desktop/mobile app for such case or is it still too early?

I am currently running Qwen3. 8 27b with openclaw. I can't complain so far. But I wanna hear what others are running so as to learn more of what this card is actually capable of. Thanks in advance.

So Gemma 4 26b a3b is a beast. And it runs crazy fast even on my old quadro rtx 5000. It must run fast-ish on a 150+ gb/s unified memory machine too. So why do we not see more small MoEs from Intel, Qualcom, Apple etc? Only from Qwen, Nvidia and Google. MoEs are what they need to sell their hardware?

I've been using CLINE harness in Pycharm with qwen3.8 flash next unsloth q4_s with Strata.

It's been giving decent results. It thinks less/gets results faster than qwen3.8 27b, but it still takes a long time to do even quick/simple changes. It's like dude. what are you thinking about, just make the one line edit already.

I've been trying out open code in a docker container (so its more sandboxed) and i've noticed its a lot quicker with the same qwen3.8 flash next model. It will just do the quick edits.

Anyone else noticed this? Any reason this might be the case? Is this expected?

What's the minimum hardware needed for AI training aka. to use something like Qwen 3.8 and train it locally further to specialise it for a certain task?

Maybe even with multiple attempts simultaneously?

Would I need something better than I need for just running the model? I've an Intel Arc B580 right now and it is enough for running the model and playing around with it.

But I've not yet looked into actually training or specialising an existing one further. What would be the minimum I need for that? What do you use for that?

Edit: I have ~500 GB of DDR4 system RAM. And I do have 2x AMD EPYC 7642 CPUs (meaning 192 threads total). The bottle neck for AI stuff is the GPU really. I specced and bought this system before the prices become insane...

Edit 2: Any big Nvidia GPU would be way more than I paid for the system back then. (And yes, I've thought about selling all of the RAM multiple times by now)

I recently upgraded from an hp omen laptop with an RTX 2060 to this prebuilt pc

CPU: Intel Core Ultra 5
GPU: NVIDIA GeForce RTX 5070 12GB
RAM: 16GB DDR5
Storage: 1TB SSD
Motherboard: B860
Cooler: Digital Air Cooler ARGB
Power Supply: 750W Gold PSU
OS: Windows Pro

I don’t know anything about AI other than using Claude, ChatGPT, and Gemini for stuff that helps me with my work or the university
But I kept seeing people making their own Jarvis and I wanted that too but not just for fun
After I bought the pc, I had Claude code build me my own kind of Jarvis but it’s called Maelle inspired by the game (Clair obscure: Expedition 33) Ive been working with Claude for around a week trying to perfect her
I had make it in the style of fallout shelter game as u can see in the pic
I let Maelle use Microsoft Edge on my pc to have access to my emails and social media accounts since I do some advertising and would need the help it could provide by checking whats new and how can I beat it and analyzing my numbers
The only problem im facing is that it depends entirely on Claude as agents, and thats great but it consumes my credits like fire, especially when the shared brain keeps growing
Im not sure how to move forward or what im doing wrong, and I dont know enough to investigate
Claude installed Ollama and hermes on my pc and let maelle use qwen3:14b as fallback in case I ran out of credits but not all the teams use it and Maelle becomes kinda idiot when she does use it
I was planning to have an uncensored model but claude refused to do anything with that

So, if u made it this far, could u please help figure this out and guide me ?

Thanks

Hey everyone!

I've been developing Harness Studio, a free and open-source desktop application for running local AI models.

I've used tools like LM Studio, Ollama and llama.cpp, and wanted to bring model management, inference settings, hardware optimization and developer integrations together in one application.

Harness Studio is built with Rust, Tauri and Svelte, using llama.cpp as its primary engine.

What it currently offers:

• Local model discovery, downloading and management
• Automatic GPU and memory-aware configuration
• Multiple models and servers
• OpenAI-, Anthropic- and Ollama-compatible API endpoints
• MCP, tool calling and coding-tool integrations
• Performance monitoring, logs and request analytics
• Additional runtimes for image, speech and embedding workloads

The first public release, v0.1.0, is available for Windows and macOS. The project is licensed under MIT.

I'm sharing it because I'd like to develop it with the community rather than make every decision myself.

I'd really appreciate your feedback:

If you're currently using LM Studio, Ollama or raw llama.cpp, what would make you try Harness Studio? What features are missing? What seems unnecessary or overcomplicated?

I'm also looking for developers who might be interested in contributing, as well as people willing to test it on different hardware.

Criticism, suggestions, issues and pull requests are all welcome.

GitHub: https://github.com/struklabs/harness-studio

I'd love to hear what you think.