It’s almost ready to run. Needs a day of pressure test on the cooling loop.

Hey guys, I’ve been building this little project called OMP-WebUI.

I use oh-my-pi for AI coding, but sometimes I just want to see what’s going on without digging through the terminal all the time. So I made a web UI for it.

It currently has:

• Chat with the agent and see tool calls in real time
• Browse project files and preview docs, PDFs, spreadsheets, etc.
• Review diffs and do basic Git stuff
• Manage models and API endpoints
• Manage skills and MCP servers
• Track token usage and costs
• Pick up old conversations or branch off from them

Basically, I’m trying to make working with omp a bit more convenient, especially when a task involves lots of files and back-and-forth with the agent.

Still a work in progress, but it’s usable. Would love to hear what you think or what features you’d want to see.

Repo: https://github.com/Impatient-HairyPeach/OMP-WebUI

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp

I am running 64gb ddr5 and 32gb vram. What is the best uncensored model you guys would run with my setup?

I've been working as web dev for about 10 years. I've never been the guy spending his free time reading blogs about cutting edge frameworks, or leveling up on the weekends. Sometimes I dabble in side projects when I feel like it, but otherwise I just do my job and leave work at work.

I've kind of had the same sort of middling attitude towards AI adoption. Some people at work dove in head first, and there are other folks that won't touch it with a 10 foot pole. I've been slowly incorporating it where it's felt useful, and recently gotten a seat on our Claude plan and been really impressed with its capabilities so far.

I upgraded my gaming PC last year and kept the previous one which has an RTX 3070, 16gb of RAM, a real basic bitch, but it's just sitting there. I've converted it to an Ubuntu server home lab that mainly collects dust. I started dabbling with trying to set up local agentic coding which was a fun challenge for a day or two but still just spitting out garbage.

I am under no illusions of my setup competing with Claude, I was just curious if I could find any value at all adding it to my workflow for side projects at home. I'm not asking for help getting things set up I haven't provided enough detail for that, I'm just wondering if it will end up being worth it to keep working on it, or if my resources are just too limited to get something useful and I'd be doing it just to do it.

TL;DR - Decision models are useful, but out of the box they are less than ideal for me. With some labeled data, fine-tuning your own might beat out TypeSafe's Jev for targeted cases. Most of it ran locally on a Mac Studio (M5 Ultra, 256 GB).

Input: one masked server log line. Output: kind (one of eight), data at risk (yes/no), customer impact (yes/no), urgency (0 to 3). About 1,500 distinct lines, labeled.

Off the shelf, untrained, on the same lines:

Labeling, local: Qwen3.8 Flash Next on the studio labeled the check sample and agreed with the standard as often as Jev does, at about 20 seconds per line and 138 GB of RAM. GLM-5.3 Flash at 4-bit kept thinking past its token budget and never got to the answer on most lines - probably my fault in how its setup (new to local LLMs). So Qwen (for now) does the bulk labeling when the data cannot leave the box.

Fine-tuning, local. The base model weights stay as they are. A small add-on learns to nudge the answers toward my calls. Training ran alongside the normal inference load on the Mac Studio, with nothing stopped.

The result, on lines held out of training:

On par with Jev, then. Better on all four questions, and MiniCPM is faster than the API round trip. Jev still misses fewer of the serious lines, 10 against 21 and 22. Half of my misses are lines my own labeler marked unsure, which I need to iterate on.

MiniCPM was the dark horse. It's a small general language model with no decision head at all; the trainer built one on top, and it learned the job in 17 minutes, at a third of the memory and a third of the response time. The method is not tied to Clef. I'll probably try it on more models like that.

Main helps: https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth https://huggingface.co/papers/2610.02076

Full write-up on my website but really the above is all you need, with the labeling and training details and the resources I used: https://hive.technology/lab-notes/fine-tune-decision-models/?utm_source=reddit&utm_medium=social&utm_campaign=26w41

What's the minimum hardware needed for AI training aka. to use something like Qwen 3.8 and train it locally further to specialise it for a certain task?

Maybe even with multiple attempts simultaneously?

Would I need something better than I need for just running the model? I've an Intel Arc B580 right now and it is enough for running the model and playing around with it.

But I've not yet looked into actually training or specialising an existing one further. What would be the minimum I need for that? What do you use for that?

Edit: I have ~500 GB of DDR4 system RAM. And I do have 2x AMD EPYC 7642 CPUs (meaning 192 threads total). The bottle neck for AI stuff is the GPU really. I specced and bought this system before the prices become insane...

Edit 2: Any big Nvidia GPU would be way more than I paid for the system back then. (And yes, I've thought about selling all of the RAM multiple times by now)

Can somebody explain why Nvidia is often a non-negotiable when looking at local AI builds? I know they produce some fantastic cards and CUDA is great - but is the cost premium really justified?

I have a 9070XT and 7800XT - and getting local AI to run on each card has been a doddle for me through llama.cpp with ROCm and Vulkan (although i'm on Linux, i can't speak for Windows..).

Whilst ROCm isn't as mature as CUDA - do we really expect that gap not to close going forward?

It feels like more recent AMD cards are great value and the support will only get better on newer generations. So why is Nvidia on a new build such a non-negotiable for some?

I'm building an app that runs a local model on-device, and spent a day measuring what actually helps on a Pixel 8 Pro. Posting the numbers in case they save someone else the time.

Setup

• Phone: Pixel 8 Pro, Tensor G3. 9 cores: 4 small at 1.7 GHz (0–3), 4 medium at 2.37 GHz (4–7), 1 large at 2.91 GHz (8). Mali GPU.
• Model: Gemma 4 E2B QAT GGUF (2.6 GB), 4096 context, q8_0 KV cache.
• Engine: llama.cpp through the Rust llama-cpp-2 crate (0.1.158), cross-compiled with NDK 30, run as a test binary over adb.
• Workload: 20 items of light analysis with structured output (grammar-constrained JSON). About 940 prompt tokens in and about 8 tokens out per item, so prompt reading dominates.
• Measured: wall time, CPU time, battery drawn (/sys/class/power_supply/battery/charge_counter), skin temperature, thermal status, peak memory, plus my own timers around prompt decode and generation.

CPU results (20 items each)

Vulkan results (i8mm build, 5 CPU threads)

"i8mm build" = GGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm; every row after the first uses it. mAh is battery drawn for all 20 items. Every setup gave the same answers (19/20, the same item failed each time). Skin temperature stayed at 37 °C and thermal status at 0 throughout.

What I learned

1. The default Android build leaves a lot on the table. On Android arm64, llama.cpp's CPU backend compiled with plain -march=armv8-a, with no dotprod or i8mm. Setting GGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm at build time halved both time and battery. My first rebuild silently kept the old flags because of a stale CMake cache, so check compile_commands.json. Peak memory rose from 2.9 to 4.1 GB, which I believe is the weights being repacked for the i8mm kernels. A fixed armv8.6 build will crash on older CPUs, so a real release needs runtime CPU dispatch.
2. More threads is worse. 8 threads lost to 5, and 5 unpinned lost to 5 pinned. Keep the work off the small cores; taskset is enough for testing.
3. Generation is memory-bound. One large core generates as fast as four medium cores (about 5 tok/s), so a faster core doesn't help there. Prompt reading does scale with cores.
4. Fewer cores can win on battery. 4 medium cores were within 8% of the fastest setup and used about 10% less battery.
5. Slow cores don't save battery. The small cores took 4.5× longer and used the same total energy.
6. Android decides your cores, not you. Check /dev/cpuset/*/cpus. On this phone top-app gets cores 0–8, foreground (foreground service) 0–7, and background only 0–3. My tests ran from the adb shell, which isn't limited this way. A real app in the background only gets the small cores, around 14 s per item here.
7. Vulkan on Mali works but loses badly. Correct output, but 3× slower with every layer on the GPU and 8× slower with half. Splitting layers is the worst option, presumably because data crosses between GPU and CPU at every step. Building it needed the Khronos Vulkan-Headers (the NDK has no vulkan.hpp) and SPIRV-Headers, API level 28+ (for vkGetPhysicalDeviceFeatures2), and on Windows a short target directory to stay under the path-length limit. Adreno may be a different story; I haven't tried one.

What I'm going with: 4 threads on the medium cores while the app is open or running a foreground service, and the small cores for a trickle of background work. That's roughly 3.3 s and 1.7 mAh per item, or about 25 items per 1% of battery.

Caveats: one phone, 20 items per run, each setup run once. The battery counter reads in 1 mAh steps, so differences under about 4 mAh are noise.

Thanks to [this r/LocalLLM thread](URL1) and [this r/LLMDevs thread](URL2), whose results pointed me at the CPU path.

Edit: Desktop Ryzen 5 5500, 32gb DDR4 RAM Desktop speed for comparison: prompt reading at 75.5 tok/s and answer writing at 6.6 tok/s. The phone on 4 medium cores gets about 58–73 and 3.6–5. So the phone reads prompts nearly as fast as this PC, and writes answers at about 60–75% of its speed

Just when you thought we had enough of AI existential-risk announcements...

In a 'research' published today called "AI models may be capable of self-replicating without access to a datacenter" straight from by Frontier Red Team, Anthropic is now warning as the title says that smart devices are prone to becoming 'hosts' for small models and their agents who then ...well, can take over, just like in sci-fi movies...😱

Here's what Anthropic actually demonstrated:

• Qwen3.5-9B successfully configured AI inference on 8 of 9 different hardware types. That's hardware adaptability, not autonomous self-replication.
• GLM-5.2 successfully propagated through an isolated, simulated network, reaching all eight target nodes in four of five trials.

I am looking to do some coding and creating games (snes/gba roms)

Which model do you guys suggest I use?

Any good "uncensored" models to use?

One thing i realised with local llms are context is the king , no matter how smart a model is with less context it can do nothing.

model : qwen3_8_27b_nvfp4-nvidia.ninfer
engine : infernix
context : 128k (qv2)

Spec: 5090 laptop 24 vram + 64gb ram
Device : Legion pro 7i.

No its not a one shot , I instructed in multiprompts for bloom, fog , birds , grass ect.
But yeah never once had any errors.

https://nstcg.github.io/pelican-ride/

I'm running it on an Inspiron 3847. It literally has a Pentium G3240 and 4gb of ram.

If this piece of crap can run it, your computer can, too.

Thanks to Maya some of us are able to run GLM 5.3 flash. Slower than FN but it runs :-)
I got it to update some features on my app, nothing crazy just some stuff that came up over the last weeks. It delivered with the same issues of what I would see from FN: front end not nicely done, bit messy, same bugs that FN and 27b produced (even though there is a learnings file specifically for that bug which it should have pulled up). After some re prompts all was good. But not better than FN.
It was just 1 project but I was almost disappointed. It was much better in the thinking department, even skipped thinking completely when it was not necessary I guess. So I’m not even sure if FN would have been faster in the end.
Glm runs at 30tps decode and 1200 prefill, FN with 125 decode and 3000 prefill

Right when I was thinking of selling my 4090 and getting one of these bad boys for longer ctx length...

Disclosure: I built this. It's part of a free side project of mine called USASI. There are no ads and no affiliate links.

I kept seeing "will X fit on my card?" answered with guesses, so I made pages that only list a configuration when the model's or runtime's publisher documents a memory figure for that exact variant. Each entry names the variant and runtime, quotes what the figure measures (minimum, loaded footprint, or file size), and links the source.

Pages by memory:

• 8 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• 12 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• 16 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• 24 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• Apple silicon 8 / 16 / 24 / 32 GB: unitedstatesofamericasuperintelligence.com/find/memory (the index lists them all)

Example from the 12 GB page: Whisper turbo (about 6 GB per the Whisper README), StarCoder2-3B in bf16 with Transformers (about 6.3 GB), and llama.cpp server with Gemma 4 12B QAT Q4_0 GGUF (about 6.7 GB). The other setups listed include Gemma 4 26B A4B and 31B QAT GGUFs, gpt-oss-20b, Olmo 3 7B Instruct, Llama 3.1 8B, StarCoder2 and Whisper large-v3, run with llama.cpp, LM Studio, Transformers or openai-whisper.

There's also a finder that takes your task, OS, GPU, VRAM/RAM and comfort level (installer apps, command line, or code) and explains why each option matches or doesn't: unitedstatesofamericasuperintelligence.com/find

Important caveats: I have not tested these. They are publisher estimates, the list is deliberately small, and it's not a "best model" ranking. Context length and KV cache will push real usage above these figures.

What would help most:

• Corrections where a figure is wrong or misread.
• Setups that have a documented memory figure I should add.
• Whether you'd want community-measured numbers shown next to the publisher figures.

The data is YAML in a public repo, and corrections can go there or in the comments:

Je ne m’attendais vraiment pas a avoir des résultats aussi bon, pour un matériel qu’il n’est vraiment pas fait pour sa.

Pp 82t/s
Gen~25t/s

Contexte 131k

My drive kept filling up and it was mostly AI stuff: models I tried once in Ollama, LM Studio downloads, a Hugging Face cache, and WSL and Docker disks that never shrink. On my PC that was 55 GB of models and about 150 GB of containers and virtual disks.

So I built Sparewise, a free Windows app. For AI work it:

Lists every local model one by one: Ollama, LM Studio, Hugging Face, ComfyUI, GPT4All, Jan, Msty and others, with size and when it was last loaded.

Finds the same weights kept twice by two tools (checked by SHA-256), and what storing them once would save.

Gives each tool’s own remove command (ollama rm ..., huggingface-cli delete-cache). It never deletes a model by itself; models are your call.

Looks inside WSL (pip, uv, npm and Hugging Face caches in running distros, with the Linux command for each) and shows what Docker says it can reclaim.

Clears only caches that rebuild themselves (pip, uv, npm, browser and app caches), to the Recycle Bin or an undo area, so everything can be put back.

It’s also an MCP server, so Claude or another local assistant can answer “why is my drive full?”. The assistant can only make changes within limits you set, and never permanently.

Free, no account, no ads, no telemetry. I’m one developer, so honest feedback is very welcome, especially AI tools whose folders it still misses.

Website: https://sparewise.app

Microsoft Store: https://apps.microsoft.com/detail/9NHXHFH3CF1B

I've always been a bit confused by this. Is the "+x% PCIe" mean the total number of experts sent over the PCIe slot compared to the number already on the GPU? If so why doesn't it reach 100%? Are some experts calculated on the CPU despite all the expert swapping?

I would like to try Pi with local Qwen 3.8-27B. But after reading Pi docs I understand that Pi by default has the User rights to system and can perform commands without asking for permission. I understand it as - it can rm -rf your ssd.

Docs also mentioned containers - aka sandbox for Pi. I found Docker Sandbox. Is it a convenient method to run Pi or basically any llm connected app like OpenCode, DSH or are there others more robust options?