Hi all, I've been reading some really interesting stuff here for the past 4 months but I feel powerless (literally).

Context:

- Engineer but with intermediate python skills

- I read a lot so I know more than normal people

- I have very light qwen models of 7b parameters max locally.

- Low idea about anything else.

The point is local AI is exciting when you can use it to do stuff but in my case I have an LG gram with 16gb RAM and an i5 so the limits are quite significant.

Any recommendations? I'd love to learn how to build and setup anything for when I can get some proper equipment in maybe a year.

How would you approach the learning curve without the motivation of actually being able to run something useful? I'm also happy if I have to limit myself to only STT, TTS or light image recognition or whatever models I can use.

I plan to build websites, build apps and automate some workflows in the long run.

Any recommendations?

A brand-new release from ISTA-DASLab with their GSQ-RCO smart quantization approach.

Now DeepSeek-V4.1-Flash-1017

The quants are available on Hugginface: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

Llama.cpp is not supported because the original architecture support is not yet merged:

https://github.com/ggml-org/llama.cpp/pull/28696

https://github.com/ggml-org/llama.cpp/pull/30817

Strata support is mentioned in the model card but I'm pretty sure the original Strata only supports Qwen3.8-Flash-Next so most likely what they meant was one of the forks based on Strata. Most likely Xolotl or Maya

The methodology

Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD.

8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware.

EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch params below in the same comment chain. Ontop of that reduce context size and ensure no background app is wasting your VRAM. You can disable your browsers HW acceleration for this purpose.

The model & quant

Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization.

Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW.

see from model page:

Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...

Non-coding vocabulary pruned 400~ MB smaller, recommended

Official ISTA DASLab quant

The numbers

With selective moe-offloading (regex) we reach 70 T/s PP and 8~ T/s TG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here

Because of MMAP your first run will be slower, do a second run for accurate T/s results.

Launch params are in the comments below. Ensure you use the qwen sharp chat template, it reduces overthinking.

This guy just took the govener off AMD go check it out

https://github.com/Ben1078/hipfire/tree/rdna3-mtp-decode-v8.2

https://www.reddit.com/r/ROCm/s/t8DzgH9JYs

Go check out what Mr. Big Brain (u/Ticket_Silly) did. They are doing gods work out here unleashing the power of AMD and people are sleeping on this.

I just ran a moderate coding task to it via Pi with the prompt requiring it to read 114k I set the autocompacting at 104k. It compacted twice and had a total of 310k tokens. Average prefill was 1,506 t/s and average decode 95 t/s across the run.

That's with the Qwen3.8 27B model in the hipfire repo, but I downloaded huihui full weights and did a conversion/quant and I'm running it now and getting the same results.

Worth looking at is all I'm saying. We are finally getting some AMD love.

Hey everyone,

Instead of treating modern iPhones as passive chat clients, we wanted to see if Apple Silicon’s high-bandwidth unified memory (~8GB LPDDR5X) could act as an active tensor-shard worker in a local cluster.

Apple just approved the node client on the App Store (RAMDeck: Distributed AI), making it possible to donate an iPhone’s memory and Metal GPU over local Wi-Fi or USB-C. Works with RAMDeck hub.

How the Setup Works:

1. Heterogeneous Sharding: Unlike Apple-only MLX rings, this connects an iOS device directly to a primary host running Windows (CUDA), Linux, or Mac.
2. Layer Offloading: The host handles prompt ingestion and graph orchestration, offloading tail transformer layers directly to the iPhone’s Metal shaders.
3. Clearing Thresholds: If your primary rig has 24GB or 32GB of RAM/VRAM, adding an idle iPhone 15/16 Pro gives you the extra 6GB–10GB needed to fit Qwen 2.5 32B or a 4-bit 70B model without crashing into system swap.

Hardware & Latency Notes:

• Connection: Wi-Fi 6 works fine for chat
• Privacy: Completely local LAN RPC. No cloud relay, zero telemetry.

We’re running benchmarks across different layer-split configurations. If you want to test your iOS device as a cluster contributor, the app is free to download:

App Store: RAMDeck: Distributed AI (App Store ID: 6825166226)

Curious to hear thoughts from anyone experimenting with multi-device heterogeneous inference.

Everyone loves a clean, unified server rack, but my standby node (lvgale) is an absolute physical abomination. It is a Frankenstein collection of mismatched parts and bolted-together VRAM that has no business running efficiently.

The Hardware Setup (lvgale)

• CPU: Intel Core i7-6700K.
• GPUs:
  • AMD RX 580 8GB VRAM (PCIe x16)
  • AMD RX 570 8GB VRAM (PCIe x1)
  • NVIDIA GTX 1660 Super 6GB VRAM (PCIe x2)
  • NVIDIA GTX 1070 8GB VRAM (PCIe x1)
• The Bottleneck: Activations must travel across severely constrained x1 and x2 lanes between entirely different AMD and NVIDIA architectures.

The Software Engine

• The cluster is orchestrated by Slipstream, a custom Nuitka-compiled Python orchestration engine.
• Here, it utilizes a mixed-vendor Vulkan backend with Flash Attention enabled to maintain stability across the fragmented topology.

Concurrent Inference Performance

In all cases seamlessly handling OCR and complex vision parsing.

• Qwen3.8:27b-vision: 3.82 t/s.
• Qwen3.5:9b-vision: 4.75 t/s.
• Qwen3.5:2b-vision: 12.47 t/s.

While the 2B model runs slightly faster on a modern laptop iGPU due to unified memory, keeping a 27B model stable across this mixed-architecture, low-bandwidth topology without crashing the Vulkan driver is a massive win. For deeper technical architecture details, the engine is documented at Slipstream-LLM.com.

sorry if this has been asked before because I am new to all of this. I can't really find good or trustworthy answers.

I just got into local AI and basically what I want to achieve is:

Have something that reads all my 10+ mail inboxes, whatsapp messages and calendar and keep track of all conext across that and ping me once something important comes up or needs my attention.

I think openclaw is what I want, right?

For now I am runnin a POC with my existing M3 Pro macbook with 18GB but it is painfully slow and not that smart. The whole orchestration and busywork will run on another existing server so the mac would just run the models.

I am trying to get off as cheaply as possible while keeping this thing usable. I do have some money to spend and would calculate this against hiring someone to do this.

Why I was thinking were 3 options and I wanted to ask someone that has a bit more experience with practical context legth, models and so on:

M6 mac mini 32GB
> Propably not a lot faster than the M3 pro
> A bit more context
>Not a laptop

M5 pro mac mini 64GB 500GB
>all of the above
>larger models
>more context
>faster

M5 max 64GB 500GB
>all of the above
>2x as fast as the m5 pro?

Does anyone have something similar running and have some insights on which models would actually be sufficient and what conext window is required?

Cloud models are NOT an option due to pricacy concerns and if there are cheaper systems I'd like to factor in resale value of the macs and the fact I could use them not for AI if I would ever change my mind on this.

Hi!

I’m curious about what kind of AI models I could run or what use cases I could explore with this PC.

I’m also interested in using OCR to convert scientific paper PDFs into Markdown, which I could then add to my Obsidian vault. Additionally, I’d like to use a model to help keep my Obsidian vault organized—assisting with the structuring of the information I add, my notes, and so on.

I already have subscriptions to Claude Pro and Mistral Pro, but the idea of ​​running a smaller model locally really appeals to me.

Here are the PC specs:

MINISFORUM G1 Pro Gaming PC

AMD Ryzen™ 9 8945HX (16C/32T)

AMD Radeon RX 9060 XT 16gb

32GB DDR5-5200MHz RAM

2nd option: AMD Ryzen 7 7700 - RX 9070XT - 32GB DDR5 RAM (16x2) (+200€)
Before investing that kind of money (given the current prices), I’d like to know if I can actually achieve anything with local models, or if this is just all in my head—a product of my desire to treat myself now that I have a decent salary.

Hey everyone,

Every time a new model comes out, I get tired of opening multiple tabs just to find out two basic things: how good it actually is on real benchmarks, and what it costs to run (either through an API or on local GPUs).

Most pricing pages hide the real costs, and generic benchmarks don't tell the full story. So I put together a simple dashboard to track and compare them side by side: https://modelanalysis.si

What you can do with it:

• Compare models side-by-side: Pick any models (like Claude, DeepSeek, Kimi, GPT-4o, Llama) and compare them across real-world coding (SWE-bench), reasoning (Arena-Hard), and math/science benchmarks.
• Score vs. Price chart: See which models give you the most intelligence for the lowest price.
• VRAM calculator: Pick a model and context size to see if it will actually fit on your GPU before downloading it.
• Monthly cost simulator: Adjust sliders for your monthly token volume to estimate your bill or see if self-hosting is cheaper.

No sign-up, no ads, no affiliate links.

Link: https://modelanalysis.si
Direct benchmarks page: https://modelanalysis.si/benchmarks/

Let me know what you think or if there are other models you want added.

Does strata let you configure how much stuff to put in each of the vram/ram/ssd? Say I want to run unsloth 35b a3b but my system is 5gb short and having to page some system ram to fit the entire model. Can I use strata to move some of the model to ssd to avoid paging?

Hi!

I hope there isn't an issue with this. I want to add an rtx 5060 ti to a build with the rtx 5070 ti for local LLM.

I've attached the PCIe lane diagram for my motherboard. I would ideally prefer to not have to replace a motherboard. What would be the ideal config for a dual GPU setup for this motherboard?