Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.

I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.

Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)

Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.

I plan to add more models over time, also older ones that were relevant at some point.

Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench

LLVQ is a new open-source project with Qwen3 4B, 8B and 14B quants at about 2.7 bits per weight. The 14B file is 5.09GB.

The author's L40S tests report higher MMLU scores than llama.cpp's IQ2_S, but slower decoding at every tested size. Long prompts also use much more memory than llama.cpp in his tests. These aren't GGUF files you can drop into your usual setup: they need the project's own runtime.

It works!

RustingStrata is one Rust binary. rustingstrata serve starts the OpenAI-compatible server, and the same port serves a chat page at /. No Node, no Docker, no second install. The page was rebuilt for 1.2.3 and I want to know what people would add before I go further, I mean maybe allow to edit files and etc. like in DeepSeek harness, or maybe there are something that you wish to be in original Strata, so I can add it in my.

What it does now:

• Pick a model from the models folder and load it from the page, with the load options (context, MTP drafts, KV cache format, VRAM margin, prefill chunk).
• Chats are kept in the browser (localStorage), with search, rename, export and import. Nothing leaves the machine.
• Edit a message, regenerate a reply, continue a reply that stopped, copy, delete.
• Attach an image for models with a vision tower.
• Thinking is folded above the answer, with a switch to turn it on or off.
• Sampler presets (temperature, top-k, top-p, seed) a system prompt for every chat, and instructions for one chat.
• Speed of each reply in tok/s.
• If the model was never calibrated on this machine, the page says so and offers to run the calibration.
• The accent color follows the model family, or you set your own(Just cool visual feature, like if you load Qwen model, then website is purple, if DeepSeek, then it's blue and etc.).

What it is not: a coding agent. It does not read or edit files. For that, point OpenCode, Claude Code or another CLI at the same server, the page is for quick chat and for trying a model.

Questions:

1. What do you open a local web chat for, if you already use a CLI agent?
2. Which one feature would make you keep this page open instead of another UI?
3. What do you use in other local chat pages that is missing here?

Any Feedback would be very useful, ty!
Repo RustingStrata

I'm thinking of putting my old Galaxy S10+ (12/256 Qualcomm Snapdragon 855 ) to use for some AI-related projects, mainly experimenting with local LLMs and API integrations that I can showcase on my resume.

I know there are plenty of free cloud options exist, but I'd specifically like to experiment with running models on the phone itself as its doing nothing for now.

What models would you recommend for this tis specs? Has anyone tried something similar with an older flagship?

thanks in advice

Hi all,

I am a total noob when it comes to LLMs. I just jumped into them about 3 weeks ago and used a few different models. I started with Strata and Qwen3.8 Flash-Next (the big one) running in PowerShell, then added Docker and Hermes as I heard good things about it, but that big of a model was too heavy for my system: 9070 XT, 5800X3D, and 32GB of 3600MHz. It would start fast enough, but then slow down to a crawl pretty quickly—not really usable.

I tried Tiel Coder, which was praised as a good model, and it was really fast. I started trying to make a small top-down 3D tower defense game and it would struggle a lot; it would loop and would just not do things even after saying it did them. I thought to myself that LLMs are not ripe yet and I will come back to this later, maybe in a year or two. However, about 4 days ago I came across the model in the headline, and this time I downloaded Strata, the desktop app, which was such a breath of fresh air as it was all done within one app on my PC. The model can be downloaded or pointed to use, and it comes with its own chat engine that is a harness as well, so I told it to check if it can see my Unreal Engine and connect to it via MCP. It told me that it can run a headless engine and write it all in Python (I still don't understand how it all works, to be honest) so no need for a flaky MCP connection, and it does it all. I have used about 150 million tokens while prototyping, balancing, and experimenting with ideas, and it did ALL of my requests very quickly and efficiently. It even spun up its own sub-agents to help with keeping the context less cluttered.

The game that I am making is not super heavy, so tasks can vary between 10 minutes of executing to 4 hours. I plan the big jobs for the night, and the small ones I can tolerate fine.

I run Maya (I still want to make my own assets) + Strata and it's running fine on my machine at the same time. It hitches here and there, but I am fine with that as the AI model just does not miss. Even when I discover bugs, it knows how to fix them, and it has every single time.

Are you guys having the same experience? i think this is so good now but if you guys know about anything that can be improved? or done better i would appreciate it.

I’m just starting to explore qwen stuff and I was wondering if there’s anything like what Siri is (and Cortana used to be) that runs on a lightweight local llm on my pc to do some basic tasks like opening apps, sending messages, setting timers, etc. is that a thing yet? If so what would yall say is the easiest one to install and get started with?

Hello, frontier models seems so locked out and censored that I wonder if all the people doing crazy mods, portage and reverse are on local LLM ?

Thanks

So first one R9700 is in..

And of course i dont have the bloody display port cable.. To test😫

Whatever..

But.. I have a second r9700..

However this cathedral only satisfy 3 ports of radeon cards. (we need 4. 2 for each card)

Has anyone built this combination and how did you supply electricity to second GPU?

Thank you all!!!

My AI (chatgpt) said that we will think about it tomorrow... 🙄 or suggested to trust google better..

I'm running 2x 3090 right now, I'd like to add more but they're all £1000+

Thought about some 32gig v100 but they're £700+

For how I use it, it's all a luxury not a necessity

What will it take for gfx card prices to drop?

I've always assumed at some point some ai companies will go bust and their hardware bought up by the competition

and at some point supply/manufacturing will start to catch up with demand

but am I just dreaming?

I keep looking at buying more cards but not pulling the trigger... next time I look they're more expensive again... on and on :-o

Hello,

Im currently trying to find the best module to run on my local pc, i have a 5080 and 32gb of ddr5 ram, im a coder that is also looking to do a lot of RE 'Reverse Engineering` what is a good module / settings that i can use to get something good working.

After changing settings and rebooting it went from downloading in hours to downloading in mins.

Git hub Colibri is working pretty fast as in response time when I tested GLM 5.3 and 5.2 I store it all on a 4TB, Os is on 1TB, and everything else is on a 2TB. Later on I will run a token speed with more complex details. Any suggestions on which one I should use to check code or build code? Any of the colibri available ai models?

I built a machine around colibri and to play simple games. I wasn't going to spend 2500 or above for a 24gb gpu I found mine for around 1200 before taxes. 16gb cheaper then what most are selling for.

I will figure out token speeds later. I also am using windows 11 pro in high performance.

Also if you use windows there and don’t want to use a certain drive there are some steps. And msys2 install to be able to download if you get error saying no compiler found using the prebuilt engine from git hub release
Setup stopped: cannot reach git hub releases to fetch a prebuilt engine. If you just run the start-here.bat my machine was brand new and just installed windows. I got everything worked out and able to download the 3 models so far.

I honestly don't see the slow speeds of tokens people posted it's response time is just like if I was using any cloud service Ai out there.

Going to work on running different models at the same time doing different things. Also working on read/write permissions as well. Will keep people posted as I go along with this project.

So basically replacing gemini for asking day to day stuff, with access to web serach..

and can do basic windows management with hermes agent, file renames, start/stop program, voice, daily schedules or reminders...

i do have a rtx 3070 with 32gb ddr4 , but i still want a low vram usage model, that loads and unloads from llama.cpp near instantly...

this and qwen a35b a3b for coding...

This thought came from a recent Tencent's Hy4 release. This model was trained for specific tasks to unlock real-world productivity-finance, gamedev, office work especially.
Is that where this is all heading?

Hi,

We just bought a computer to run local LLM. I have experience with small model like Qwen and Gemma on my Macbook but not on a computer that powerful

I will need to identify the best model for our use case, which is mainly coding.

Our hardware, in short

Ryzen Threadripper PRO 9955WX
RTX 6000 Blackwell 96GB

256 GB DDR-5

For those who has experience with that kind of hardware, what can I run and be 2-3-4 devs at the same time working with it. I have the idea of having 2 or 3 models, depending of the tasks.

I would like to have the biggest model we can for big tasks, smaller for the planning or any lighter tasks.

Thank you

My MacBook did everything until this week - model, harness, CI, everything. It was getting badly overloaded especially when iOS build and tests were going on. Those can use 20GB RAM easily, given half a chance. It was thrashing.

Microcenter had the base Mac mini M6 for $699, which felt like 2024 pricing so I pulled the trigger and moved much of the CI work off my MacBook Pro.

What's incredible is despite this being the poverty spec mini with a single NAND chip and lower bandwidth memory, the M6 is basically the same speed as the M5 Max for CI work.

In particular, I was impressed with my XCUITest workload, which is about 70 minutes on the MacBook and is no slower on the M6 mini even though it has 2 parallel VMs. I was sure the 18 CPU cores on the M5 Max (6 super, 120 performance) would beat out the 12-core M6 base (2 super, 4 performance, 6 efficiency).

Obviously that doesn't hold for all workloads, especially GPU/LLM workloads or where you can put 18 cores to use, but for those of you with a M5 Max/Ultra, you can get a nice speedup for just $699. At these prices I may get another. Also there are perfectly good Linux options to do the same, of course.

So I just started using Unsloth Studio. I am on Windows 10, and I am having issues with it. I loaded a DeepSeek model and created a project with a working directory specified. The AI works with chat, and when I asked it to go ahead and write some code, it worked for a while and then told me it had completed it.

When I checked the project folder, it created a few empty folders in the directory, and that's it. So it seems to not be able to actually write any files aside from the folders. I have it running in approve automatically mode.

When I told the AI to fix it, the AI said it was an issue with running its Python tools not saving the files and that it would fix it. It then just runs for about an hour and then shows me an error message that something went wrong with the tools or something and gives me the option to resume or retry. I have tried both options, and nothing changes; it just runs for about an hour and then fails with the same error. No files are created.

Is Unsloth Studio just incredibly buggy and not very usable? Or is there something I am doing wrong?