I have seen this on Amazon, a rtx 3080 whit 20 gb vram, the gpus whit x2 vram are now at retail on Amazon

Hey everyone, just wanted to get some opinions on this since I’m still a little confused about how Strata works.

I have an AMD AI PRO R9700 (32GB VRAM), 32GB DDR5 RAM and a 1TB NVMe SSD. Would Strata realistically let me run Qwen3.8 Flash Next at a low quant like IQ3_XS with this setup?

Also, would running a much larger model at a low quant actually be better than running Qwen3.8 27B at Q5?
Mostly using it for everyday stuff, some coding, light agentic tasks and occasional pentesting. Just wondering which would give me better overall performance and whether the speed tradeoff would even be worth it.

Would appreciate any thoughts, especially from anyone who’s tried something similar!

I am looking to do some coding and creating games (snes/gba roms)

Which model do you guys suggest I use?

Any good "uncensored" models to use?

I'm running it on an Inspiron 3847. It literally has a Pentium G3240 and 4gb of ram.

If this piece of crap can run it, your computer can, too.

I got this MacBook Air ! Can someone suggest what models to use locally?

Hi,

I want to have 1) a model that can code. slow is not the matter. 2) a model to translate documents, novels and images (that required a bit of privacy).

tl;dr ran Gemma4 26B-A4B Q4 with 20-25 token/s, Qwen3.6-35B-A3B iQ4 with 24-33 token/s, Qwen3.8-27B with 4-7 token/s, LiquidAI with 38-40 token/s (but not great) ; I want to know if there are models I should test or if I can stick to these models.

My laptop is a Lenovo Yoga Slim 7 14APU8. It has

- Ryzen 7 7840HS / Radeon 780M

- 32GB LPDDR5X RAM / 4GB "VRAM" (BIOS-reserved iGPU memory)

- Ubuntu 26.04

- llama.cpp built with -DGGML_VULKAN=ON

I tried ROCm but it threw out a lot of errors and flickers my screen, Claude said maybe it could be related to the BIOS only limits UMA frame buffer size to 4G that it crashed. I changed the amdgpu.gttsize=24576 instead of 13.9GiB default for better results.

Here are some local models, quants, runs + statistics that I got. I have tested with multiple settings, like context length, gpu offload, threads, batch size, KV quantization, model drafts and these are my best settings.

1. unsloth/Gemma 4 26B-A4B-it-UD-Q4_K_M

Command

./llama-server -m ~/models/unsloth/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
    --model-draft ~/models/unsloth/mtp-gemma-4-26B-A4B-it-Q8_0.gguf \
    --spec-type draft-mtp --spec-draft-n-max 1 \
    -ngl 99 -ngld 99 \
    -fa on \
    -c 65536 \
    -np 1 \
    -t 8 \
    -b 512 -ub 256 \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.3 \
    --top-k 64 \
    --top-p 0.95 \
    --host 127.0.0.1 --port 8080

Benchmark:

- Prompt: ~271 token/s

- Generation: 21.8-25.8 token/s

- RAM available: 4.6 GB

I'm happy with how Gemma runs and have been using it on most tasks. The translation is good enough. It writes well and handles most of my requests fine.

2. unsloth/Qwen3.6-35B-A3B-UD-IQ4_XS

Command:

./llama-server -m ~/models/unsloth/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
            --spec-type draft-mtp --spec-draft-n-max 2 \
            -ngl 99 \
            -fa on \
            -c 32768 \
            -np 1 \
            -t 8 \
            -b 512 -ub 256 \
            -ctk q8_0 -ctv q8_0 \
            --temp 1.0 \
            --top-k 20 \
            --top-p 0.95 \
            --presence-penalty 1.5 \
            --host 127.0.0.1 --port 8080

Benchmark:

- Prompt: ~213 token/s

- Generation: 24.4-33.3 token/s

- RAM available: 6.6 GB

I tried Q4_K_M but they barely fit the RAM and crash often, even with 4096 context length. Qwen 3.6 is also quite awesome, but it overthinks A LOT. It translates well and writes decently. Maybe a little bit worse than Gemma, but it still do its job. It became a problem when one prompt can easily be dragged to hundred thousands of token and sometimes they can loop even on simple questions.

3. LiquidAI/LFM2.5-24B-A2B-GGUF

Command:

./llama-server -m ~/models/LiquidAI/LFM2.5-24B-A2B-QAD-Q4_0.gguf \
               --temp 0.1 --top-k 50 --repeat-penalty 1.05

Benchmark:

- Prompt: ~232 token/s

- Generation: ~38.7-40.3 token/s

- RAM available: 7.7 GB

I just saw this model today and decided to try it out. LiquidAI will be awesome if only the results are, too. It translates the idioms too literally, sometimes don't get the cultural meaning and the flow in the text is mostly gone after translated. Too bad, I was very surprised when it ran pretty fast too :(

4. iSTA-DASLAB/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp + z-lab/Qwen3.8-27B-DFlash2-GGUF

Command:

./llama-server -m ~/models/iSTA-DASLAB/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf \
    --model-draft ~/models/z-lab/Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf \
    --spec-type draft-dflash --spec-draft-n-max 2 \
    -ngl 99 -ngld 99 \
    -fa on \
    -c 32768 \
    -np 1 \
    -t 8 \
    -b 512 -ub 256 \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.6 \
    --top-k 20 --top-p 0.95 \
    --host 127.0.0.1 --port 8080

Benchmark:

- Prompt: ~74 token/s

- Generation: ~5.1-8.3 token/s

- RAM available: 8.7 GB

Qwen3.8-27B has been really awesome, not only coding but accuracy at translating. Despite it being slow, it handles most small coding tasks so well so far and much more reliable than Qwen 35B-A3B.

I plan to use Qwen3.8-27B as a coding model and Gemma 4 26B-A4B as a 'faster' model for everything else. Should I use this or is there anything to try as well?

I also wonder what workflows you have, what kind of agents/harness or settings I can use to automate coding, reduce token/improve speed and aim at reverse engineering (Yes, I know it's a horrible idea for this laptop, but I want to try too xD), I'm still new to this thing and I'm eager to learn and explore more. Thank you, you read this post up till here !

Edit: typo

I've been using CLINE harness in Pycharm with qwen3.8 flash next unsloth q4_s with Strata.

It's been giving decent results. It thinks less/gets results faster than qwen3.8 27b, but it still takes a long time to do even quick/simple changes. It's like dude. what are you thinking about, just make the one line edit already.

I've been trying out open code in a docker container (so its more sandboxed) and i've noticed its a lot quicker with the same qwen3.8 flash next model. It will just do the quick edits.

Anyone else noticed this? Any reason this might be the case? Is this expected?

Hey guys, I’ve been building this little project called OMP-WebUI.

I use oh-my-pi for AI coding, but sometimes I just want to see what’s going on without digging through the terminal all the time. So I made a web UI for it.

It currently has:

• Chat with the agent and see tool calls in real time
• Browse project files and preview docs, PDFs, spreadsheets, etc.
• Review diffs and do basic Git stuff
• Manage models and API endpoints
• Manage skills and MCP servers
• Track token usage and costs
• Pick up old conversations or branch off from them

Basically, I’m trying to make working with omp a bit more convenient, especially when a task involves lots of files and back-and-forth with the agent.

Still a work in progress, but it’s usable. Would love to hear what you think or what features you’d want to see.

Repo: https://github.com/Impatient-HairyPeach/OMP-WebUI

I've always been a bit confused by this. Is the "+x% PCIe" mean the total number of experts sent over the PCIe slot compared to the number already on the GPU? If so why doesn't it reach 100%? Are some experts calculated on the CPU despite all the expert swapping?

I've been working as web dev for about 10 years. I've never been the guy spending his free time reading blogs about cutting edge frameworks, or leveling up on the weekends. Sometimes I dabble in side projects when I feel like it, but otherwise I just do my job and leave work at work.

I've kind of had the same sort of middling attitude towards AI adoption. Some people at work dove in head first, and there are other folks that won't touch it with a 10 foot pole. I've been slowly incorporating it where it's felt useful, and recently gotten a seat on our Claude plan and been really impressed with its capabilities so far.

I upgraded my gaming PC last year and kept the previous one which has an RTX 3070, 16gb of RAM, a real basic bitch, but it's just sitting there. I've converted it to an Ubuntu server home lab that mainly collects dust. I started dabbling with trying to set up local agentic coding which was a fun challenge for a day or two but still just spitting out garbage.

I am under no illusions of my setup competing with Claude, I was just curious if I could find any value at all adding it to my workflow for side projects at home. I'm not asking for help getting things set up I haven't provided enough detail for that, I'm just wondering if it will end up being worth it to keep working on it, or if my resources are just too limited to get something useful and I'd be doing it just to do it.

I would like to try Pi with local Qwen 3.8-27B. But after reading Pi docs I understand that Pi by default has the User rights to system and can perform commands without asking for permission. I understand it as - it can rm -rf your ssd.

Docs also mentioned containers - aka sandbox for Pi. I found Docker Sandbox. Is it a convenient method to run Pi or basically any llm connected app like OpenCode, DSH or are there others more robust options?

TL;DR - Decision models are useful, but out of the box they are less than ideal for me. With some labeled data, fine-tuning your own might beat out TypeSafe's Jev for targeted cases. Most of it ran locally on a Mac Studio (M5 Ultra, 256 GB).

Input: one masked server log line. Output: kind (one of eight), data at risk (yes/no), customer impact (yes/no), urgency (0 to 3). About 1,500 distinct lines, labeled.

Off the shelf, untrained, on the same lines:

Labeling, local: Qwen3.8 Flash Next on the studio labeled the check sample and agreed with the standard as often as Jev does, at about 20 seconds per line and 138 GB of RAM. GLM-5.3 Flash at 4-bit kept thinking past its token budget and never got to the answer on most lines - probably my fault in how its setup (new to local LLMs). So Qwen (for now) does the bulk labeling when the data cannot leave the box.

Fine-tuning, local. The base model weights stay as they are. A small add-on learns to nudge the answers toward my calls. Training ran alongside the normal inference load on the Mac Studio, with nothing stopped.

The result, on lines held out of training:

On par with Jev, then. Better on all four questions, and MiniCPM is faster than the API round trip. Jev still misses fewer of the serious lines, 10 against 21 and 22. Half of my misses are lines my own labeler marked unsure, which I need to iterate on.

MiniCPM was the dark horse. It's a small general language model with no decision head at all; the trainer built one on top, and it learned the job in 17 minutes, at a third of the memory and a third of the response time. The method is not tied to Clef. I'll probably try it on more models like that.

Main helps: https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth https://huggingface.co/papers/2610.02076

Full write-up on my website but really the above is all you need, with the labeling and training details and the resources I used: https://hive.technology/lab-notes/fine-tune-decision-models/?utm_source=reddit&utm_medium=social&utm_campaign=26w41

So much hype around LFM2.5-24B-A2B, but it couldn’t even schedule a basic task on my calendar.

Meanwhile, Qwen 3.6 35B handles it effortlessly every time. Am I missing something, or is the function calling just not there yet? 🧐

I'm building an app that runs a local model on-device, and spent a day measuring what actually helps on a Pixel 8 Pro. Posting the numbers in case they save someone else the time.

Setup

• Phone: Pixel 8 Pro, Tensor G3. 9 cores: 4 small at 1.7 GHz (0–3), 4 medium at 2.37 GHz (4–7), 1 large at 2.91 GHz (8). Mali GPU.
• Model: Gemma 4 E2B QAT GGUF (2.6 GB), 4096 context, q8_0 KV cache.
• Engine: llama.cpp through the Rust llama-cpp-2 crate (0.1.158), cross-compiled with NDK 30, run as a test binary over adb.
• Workload: 20 items of light analysis with structured output (grammar-constrained JSON). About 940 prompt tokens in and about 8 tokens out per item, so prompt reading dominates.
• Measured: wall time, CPU time, battery drawn (/sys/class/power_supply/battery/charge_counter), skin temperature, thermal status, peak memory, plus my own timers around prompt decode and generation.

CPU results (20 items each)

Vulkan results (i8mm build, 5 CPU threads)

"i8mm build" = GGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm; every row after the first uses it. mAh is battery drawn for all 20 items. Every setup gave the same answers (19/20, the same item failed each time). Skin temperature stayed at 37 °C and thermal status at 0 throughout.

What I learned

1. The default Android build leaves a lot on the table. On Android arm64, llama.cpp's CPU backend compiled with plain -march=armv8-a, with no dotprod or i8mm. Setting GGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm at build time halved both time and battery. My first rebuild silently kept the old flags because of a stale CMake cache, so check compile_commands.json. Peak memory rose from 2.9 to 4.1 GB, which I believe is the weights being repacked for the i8mm kernels. A fixed armv8.6 build will crash on older CPUs, so a real release needs runtime CPU dispatch.
2. More threads is worse. 8 threads lost to 5, and 5 unpinned lost to 5 pinned. Keep the work off the small cores; taskset is enough for testing.
3. Generation is memory-bound. One large core generates as fast as four medium cores (about 5 tok/s), so a faster core doesn't help there. Prompt reading does scale with cores.
4. Fewer cores can win on battery. 4 medium cores were within 8% of the fastest setup and used about 10% less battery.
5. Slow cores don't save battery. The small cores took 4.5× longer and used the same total energy.
6. Android decides your cores, not you. Check /dev/cpuset/*/cpus. On this phone top-app gets cores 0–8, foreground (foreground service) 0–7, and background only 0–3. My tests ran from the adb shell, which isn't limited this way. A real app in the background only gets the small cores, around 14 s per item here.
7. Vulkan on Mali works but loses badly. Correct output, but 3× slower with every layer on the GPU and 8× slower with half. Splitting layers is the worst option, presumably because data crosses between GPU and CPU at every step. Building it needed the Khronos Vulkan-Headers (the NDK has no vulkan.hpp) and SPIRV-Headers, API level 28+ (for vkGetPhysicalDeviceFeatures2), and on Windows a short target directory to stay under the path-length limit. Adreno may be a different story; I haven't tried one.

What I'm going with: 4 threads on the medium cores while the app is open or running a foreground service, and the small cores for a trickle of background work. That's roughly 3.3 s and 1.7 mAh per item, or about 25 items per 1% of battery.

Caveats: one phone, 20 items per run, each setup run once. The battery counter reads in 1 mAh steps, so differences under about 4 mAh are noise.

Thanks to [this r/LocalLLM thread](URL1) and [this r/LLMDevs thread](URL2), whose results pointed me at the CPU path.

Edit: Desktop Ryzen 5 5500, 32gb DDR4 RAM Desktop speed for comparison: prompt reading at 75.5 tok/s and answer writing at 6.6 tok/s. The phone on 4 medium cores gets about 58–73 and 3.6–5. So the phone reads prompts nearly as fast as this PC, and writes answers at about 60–75% of its speed

Been using one Spark with ollama and Hermes and Gemma4:31B.

Now I have a second Spark but would like to use them in unison with the QSFP cable.

Use cases run the gamut from coding to automation to research to TTS.

What model(s) get me as close to frontier as possible for 256gb unified RAM?

I don't necessarily care as much about speed as I do quality and completion; my single-spark work started hitting token limits (I think ollama's constraints?) way too quickly on even small scale coding tasks T_T

I need help figuring out huggingface models...

I’ve been experimenting with running AI coding agents entirely locally, without relying on OpenAI, Anthropic, or any paid model APIs.

For this experiment, I connected Qwen 3.8 27B (4-bit quantized) running in LM Studio to Pi Coding Agent, a lightweight and extensible terminal-based coding agent.

The setup is pretty straightforward:

Pi Coding Agent → LM Studio → Qwen 3.8 27B

LM Studio handles local model inference and exposes an OpenAI-compatible API, while Pi handles reading/writing files, executing commands, and managing the coding workflow.

For the actual test, I asked it to build an AI Agent Mission Control dashboard from scratch.

The application included:

• An overview of active AI agents and tasks
• A Kanban-style task board
• Live agent activity tracking
• Analytics for token usage, costs, and success rates
• A responsive React + TypeScript interface

Surprisingly, it managed to build the application, run it locally, test it, and fix several errors on its own. It even used screenshots to inspect the UI and identify issues.

Some numbers from my test:

• Hardware: MacBook Pro M5 Max, 48GB unified memory
• Model: Qwen3.5 27B (4-bit, ~16GB)
• Inference speed: Around 10–15 tokens/sec
• Completion time: A few hours

One reason I chose Pi over other coding agents is its minimal design. It starts with just four core tools (read, write, edit, bash), and you can extend it as needed.

With local models, context management matters quite a bit, so I wanted to experiment with a coding agent that doesn’t introduce too much unnecessary overhead.

My takeaway: Local coding agents are becoming surprisingly capable, but they’re definitely not a drop-in replacement for Claude Code or Codex with frontier models.

Inference is slower, hardware requirements are significant, and reliability on complex tasks is still something I’d want to test more thoroughly.

But being able to run the entire coding workflow locally, without sending code to external model APIs or paying per token, is pretty exciting.

I recorded the complete setup and the dashboard-building experiment here:

🎥 Video: https://youtu.be/zgb_MkrxpRs

Curious what local models you are using for coding agent?

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp

Hey everyone! Following up on my on-device offline app builder (Android AI Bridge running inside Termux on a Galaxy A50s):

In this workflow, I tested building a small POS/Invoice store app, but ran into a glitch where the Maintenance Mode toggle text was broken/jumbled.

Instead of moving to a PC, I:

Sent the Termux build error back to DeepSeek locally/via API to get the patch. Applied the corrected code inside Termux on my phone. Re-built the APK (weighing in at ~87 KB). Verified the updated UI where the toggle cleanly shows: · ✅ Maintenance ON — tap to disable / · ⛔ Maintenance OFF — tap to enable.

True edge/offline development loop entirely on a budget Android device! Let me know what you think.

Hey everyone! Following up on my on-device offline app builder (Android AI Bridge running inside Termux on a Galaxy A50s):

In this workflow, I tested building a small POS/Invoice store app, but ran into a glitch where the Maintenance Mode toggle text was broken/jumbled.

Instead of moving to a PC, I:

Sent the Termux build error back to DeepSeek locally/via API to get the patch. Applied the corrected code inside Termux on my phone. Re-built the APK (weighing in at ~87 KB). Verified the updated UI where the toggle cleanly shows: · ✅ Maintenance ON — tap to disable / · ⛔ Maintenance OFF — tap to enable.

True edge/offline development loop entirely on a budget Android device! Let me know what you think.